Meaningful METRICS A 21st-Century Librarian’s Guide to Bibliometrics, Altmetrics, and Research Impact Robin Chin Roemer & Rachel Borchardt Association of College and Research Libraries A division of the American Library Association Chicago, Illinois 2015 The paper used in this publication meets the minimum requirements of American National Standard for Information Sciences–Permanence of Paper for Printed Library Materials, ANSI Z39.48-1992. ∞ Library of Congress Cataloging-in-Publication Data Meaningful metrics : a 21st century librarian’s guide to bibliometrics, altmetrics, and research impact / edited by Robin Chin Roemer and Rachel Borchardt. pages cm Includes bibliographical references and index. ISBN 978-0-8389-8755-1 (pbk. : alk. paper) -- ISBN 978-0-8389-8757-5 (epub) -- ISBN 978-0-8389-8756-8 (pdf ) -- ISBN 978-0-8389-8758-2 (kin- dle) 1. Bibliometrics. 2. Bibliographical citations--Evaluation. 3. Scholarly publishing--Evaluation. 4. Research--Evaluation--Statistical methods. 5. Communication in learning and scholarship--Technological innovations. I. Roemer, Robin Chin, editor. II. Borchardt, Rachel, editor. Z669.8.M43 2015 010.72’7--dc23 2015006338 Copyright ©2015 by The Association of College & Research Libraries, a division of the American Library Association. All rights reserved except those which may be granted by Sections 107 and 108 of the Copyright Revision Act of 1976. Printed in the United States of America. 19 18 17 16 15 5 4 3 2 1 This work is licensed under Creative Commons license CC BY-NC 4.0 (Attribution-NonCommercial Use) Table of Contents Foreword .................................................................................................v PART 1. IMPACT Chapter 1: Understanding Impact ..................................................3 Chapter 2: Impact in Practice ........................................................13 PART 2. BIBLIOMETRICS Chapter 3: Understanding Bibliometrics .................................. 27 Chapter 4: Bibliometrics in Practice ........................................... 71 PART 3. ALTMETRICS Chapter 5: Understanding Altmetrics .......................................99 Chapter 6: Altmetrics in Practice ...............................................155 PART 4. SPECIAL TOPICS Chapter 7: Disciplinary Impact.....................................................181 Chapter 8: Impact and the Role of Librarians .....................209 Glossary ..............................................................................................233 Acknowledgments ......................................................................... 239 About the Authors ...........................................................................241 iii Foreword few days ago, we were speaking with an ecologist from Simon Fraser University here in Vancouver about an unsolicited job A offer he’d recently received. The offer included an astonishing inducement: Anyone from his to-be-created lab who could wangle a first or corresponding authorship of a Nature paper would receive a bonus of $100,000. Are we seriously this obsessed with a single journal? Who does this benefit? (Not to mention, one imagines the unfortunate middle authors of such a paper, trudging to a rainy bus stop as their endian-authoring col- leagues roar by in jewel-encrusted Ferraris.) Although it’s an extreme case, it’s sadly not an isolated one. Across the world, a certain kind of adminis- trator is doubling down on 20th-century, journal-centric metrics like the impact factor. That’s particularly bad timing because our research communication system is just beginning a transition to 21st-century communication tools and norms. We’re increasingly moving beyond the homogeneous, jour- nal-based system that defined 20th-century scholarship. Today’s scholars increasingly disseminate web-native scholarship. For instance, Jason’s 2010 tweet coining the term “altmetrics” is now more cited than some of his peer-reviewed papers. Heather’s openly published datasets have gone on to fuel new articles written by other researchers. And like a growing number of other researchers, we’ve published research v vi Foreword code, slides, videos, blog posts, and figures that have been viewed, reused, and built upon by thousands all over the world. Where we do publish tra- ditional journal papers, we increasingly care about broader impacts, like citation in Wikipedia, bookmarking in reference managers, press coverage, blog mentions, and more. You know what’s not capturing any of this? The impact factor. Many researchers and tenure committees are hungry for alternatives, for broader, more diverse, more nuanced metrics. Altmetrics are in high demand; we see examples at Impactstory (our altmetrics-focused non- profit) all the time. Many faculty share how they are including downloads, views, and other alternative metrics in their tenure and promotion dossiers and how evaluators have enthused over these numbers. There’s tremendous drive from researchers to support us as a nonprofit, from faculty offering to pay hundreds of extra dollars for profiles to a Senegalese postdoc refusing to accept a fee waiver. Other altmetrics startups like Plum Analytics and Altmetric can tell you similar stories. At higher levels, forward-thinking policy makers and funders are also seeing the value of 21st-century impact metrics and are keen to realize their full potential. We’ve been asked to present on 21st-century metrics at the NIH, NSF, the White House, and more. It’s not these folks who are driv- ing the impact factor obsession; on the contrary, we find that many high- level policy-makers are deeply disappointed with 20th-century metrics as we’ve come to use them. They know there’s a better way. But many working scholars and university administrators are wary of the growing momentum behind next-generation metrics. Researchers and administrators off the cutting edge are ill-informed, uncertain, afraid. They worry new metrics represent Taylorism, a loss of rigor, a loss of mean- ing. This particularly true among the majority of faculty who are less com- fortable with online and web-native environments and products. But even researchers who are excited about the emerging future of altmetrics and web-native scholarship have a lot of questions. It’s a new world out there, and one that most researchers are not well trained to negotiate. We believe librarians are uniquely qualified to help. Academic librar- ians know the lay of the land, they keep up-to-date with research, and they are experienced providing leadership to scholars and decision-makers on Foreword vii campus. That’s why we’re excited that Robin and Rachel have put this book together. To be most effective, librarians need to be familiar with the met- rics research, which is currently advancing at breakneck speed. And they need to be familiar with the state of practice—not just now, but what’s com- ing down the pike over the next few years. This book, with its focus on integrating research with practical tips, gives librarians the tools they need. It’s an intoxicating time to be involved in scholarly communication. We’ve begun to see the profound effect of the web here, but we’re just at the beginning. Scholarship is on the brink of a Cambrian explosion, a break- neck flourishing of new scholarly products, norms, and audiences. In this new world, research metrics can be adaptive, subtle, multidimensional, responsible. We can leave the fatuous, ignorant use of impact factors and other misapplied metrics behind us. Forward-thinking librarians have an opportunity to help shape these changes, to take their place at the vanguard of the web-native scholarship revolution. We can make a better scholarship system, together. We think that’s even better than that free Ferrari. Jason Priem Heather Piwowar Cofounders of Impactstory Glossary Section 1 IMPACT Chapter One Understanding Impact nce upon a time, there was no such thing as bibliometrics. Like its conceptual predecessor, statistical bibliography, bibliometrics is a Oconcept predicated on the widespread existence of printed mate- rial and the acceptance of a specific printed format (the journal article) as a fundamental means of communication between scholars and experts within a field. Within library and information science (LIS), we have seen many excellent books and articles published over the last 20 years, each telling its own version of the history of bibliometrics and predicting what lies in store for scholars and practitioners of bibliometrics with new advancements in technology, research methods, and general higher ed. This is not one of those books. This is a book that tells stories—some of which are about bibliometrics, others of which are about altmetrics, but all of which are about impact and human beings’ never-ending quest to measure, track, and compare their value. At this point, let’s take a moment to reflect on what we mean when we say the word impact, particularly in the context of an academic book. Impact is a word that we in the LIS field hear and use every day, yet it can be a surprisingly tricky word to define, at least without lots of additional context. For example, researchers in public health would certainly be dis- appointed by an English department’s assessment about what it means for faculty to produce “impactful” scholarship. This is because impact 3 4 Understanding Impact is a word with a variety of subtle definitions, each changing over time and with different audiences, geographies, and local institutional philos- ophies. The result