TITLE: CAN WEB INDICATORS BE USED TO ESTIMATE THE CITATION IMPACT OF CONFERENCE PAPERS IN ENGINEERING? A thesis submitted in partial fulfilment of the requirements of the University of Wolverhampton for the degree of Doctor of Philosophy October 2018 This work or any part thereof has not previously been presented in any form to the University or to any other body whether for the purposes of assessment, publication or for any other purpose (unless otherwise indicated). Save for any express acknowledgments, references and/or bibliographies cited in the work, I confirm that the intellectual content of the work is the result of my own efforts and of no other person. The right of Kuku Joseph Aduku to be identified as author of this work is asserted in accord- ance with ss.77 and 78 of the Copyright, Designs and Patents Act 1988. At this date copy- right is owned by the author. Signature---------------------------------------------------- Date ---------------------------------------------------------- i Abstract: Although citation counts are widely used to support research evaluation, they can only reflect academic impacts, whereas research can also be useful outside academia. There is therefore a need for alternative indicators and empirical studies to evaluate them. Whilst many previous studies have investigated alternative indicators for journal articles and books, this thesis explores the importance and suitability of four web indicators for conference papers. These are readership counts from the online reference manager Mendeley and citation counts from Google Patents, Wikipedia and Google Books. To help evaluate these indicators for conference papers, correlations with Scopus citations were evaluated for each alternative indicator and compared with corresponding correlations between alternative indicators and citation counts for journal articles. Four subject areas that value conferences were chosen for the analysis: Computer Science Applications; Computer Software Engineering; Building & Construction Engineering; and Industrial & Manufacturing Engineering. There were moderate correlations between Mendeley readership counts and Scopus citation counts for both journal articles and conference papers in Computer Science Applications and Computer Software. For conference papers in Building & Construction Engineering and Industrial & Manufacturing Engineering, the correlations between Mendeley readers and citation counts are much lower than for journal articles. Thus, in fields where conferences are important, Mendeley readership counts are reasonable impact indicators for conference papers although they are better impact indicators for journal articles. Google Patent citations had low positive correlations with citation counts for both conference papers and journal articles in Software Engineering and Computer Science Applications. There were negative correlations for both conference papers and journal articles in Industrial and Manufacturing Engineering. However, conference papers in Building and Construction Engineering attracted no Google Patent citations. This suggests that there are disciplinary differences but little overall value for Google Patent citations as impact indicators in engineering fields valuing conferences. Wikipedia citations had correlations with Scopus citations that were statistically significantly positive only in Computer Science Applications, whereas the correlations were not statistically significantly different from zero in Building & Construction Engineering, Industrial & Manufacturing Engineering and Software Engineering. Conference papers were less likely to be cited in Wikipedia than journal articles were in all fields, although the difference was minor in Software Engineering. Thus, Wikipedia citations seem to have little value in engineering fields valuing conferences. Google Books citations had positive significant correlations with Scopus-indexed citations for conference papers in all fields except Building & Construction Engineering, where the correlations were not statistically significantly different from zero. Google Books citations seemed to be most valuable impact indicators in Computer Science Applications and Software Engineering, where the correlations were moderate, than in Industrial & Manufacturing Engineering, where the correlations were low. This means that Google Book citations are valuable indicators for conference papers in engineering fields valuing conferences. Although evidence from correlation tests alone is insufficient to judge the value of alternative indicators, the results suggest that Mendeley readers and Google Books citations may be use- ful for both journal articles and conference papers in engineering fields that value confer- ences, but not Wikipedia citations or Google Patent citations. ii Table of Contents List of Tables ..................................................................................................................................... vi List of Figures .................................................................................................................................. viii List of Appendices ............................................................................................................................. ix Publications from this Thesis ............................................................................................................. xi Acknowledgements ........................................................................................................................... xii Chapter 1: Introduction ........................................................................................................................... 1 1.1 Background ................................................................................................................................... 1 1.2 Research problem .......................................................................................................................... 1 1.3 Research Aim and Objectives ....................................................................................................... 2 1.4 Terms and acronyms used in this thesis ........................................................................................ 3 1.5 Thesis Structure ............................................................................................................................ 5 Chapter 2: Literature Review .................................................................................................................. 6 2.1 Citation analysis ............................................................................................................................ 6 2.1.1 Citation-based indicators .......................................................................................................... 6 2.1.2 Problems with using citation analysis for research evaluation ................................................. 7 2.2 Non-journal publication citations .................................................................................................. 9 2.2.1 Patent citations .......................................................................................................................... 9 2.3 Social media metrics: Overview ................................................................................................. 11 2.3.1 Scholarly use of the social web ............................................................................................... 12 2.3.2 Advantages and disadvantages of altmetrics .......................................................................... 13 2.3.3 Validity assessment methods .................................................................................................. 15 2.3.4 Validity assessment results ..................................................................................................... 16 2.5 Mendeley and other reference managers..................................................................................... 17 2.5.1 Introduction to Mendeley ........................................................................................................ 18 2.5.2 Correlations between Mendeley readers and citation counts .................................................. 20 2.5.3 Field differences in the use of Mendeley ................................................................................ 23 2.5.4 Mendeley users ....................................................................................................................... 26 2.5.5 Mendeley reader count outliers ............................................................................................... 26 2.5.6 Other important studies about Mendeley ................................................................................ 27 2.6 Google Books citations ............................................................................................................... 27 2.7 Google Patents citations .............................................................................................................. 28 2.8 Wikipedia citations ..................................................................................................................... 29 2.9 Impact assessment for conference papers ................................................................................... 32 2.10 Summary ..................................................................................................................................