UC Davis Books Title Gaming the Metrics: Misconduct and Manipulation in Academic Research Permalink https://escholarship.org/uc/item/6096m1sp ISBN 9780262356565 Publication Date 2020-01-28 DOI 10.7551/mitpress/11087.001.0001 License https://creativecommons.org/licenses/by-nc-nd/4.0/ 4.0 Peer reviewed eScholarship.org Powered by the California Digital Library University of California Gaming the Metrics Infrastructures Series Edited by Geoffrey C. Bowker and Paul N. Edwards Paul N. Edwards, A Vast Machine: Computer Models, Climate Data, and the Politics of Global Warming Lawrence M. Busch, Standards: Recipes for Reality Lisa Gitelman, ed., “Raw Data” Is an Oxymoron Finn Brunton, Spam: A Shadow History of the Internet Nil Disco and Eda Kranakis, eds., Cosmopolitan Commons: Sharing Resources and Risks across Borders Casper Bruun Jensen and Brit Ross Winthereik, Monitoring Movements in Development Aid: Recursive Partnerships and Infrastructures James Leach and Lee Wilson, eds., Subversion, Conversion, Development: Cross- Cultural Knowledge Exchange and the Politics of Design Olga Kuchinskaya, The Politics of Invisibility: Public Knowledge about Radiation Health Effects after Chernobyl Ashley Carse, Beyond the Big Ditch: Politics, Ecology, and Infrastructure at the Panama Canal Alexander Klose, translated by Charles Marcrum II, The Container Principle: How a Box Changes the Way We Think Eric T. Meyer and Ralph Schroeder, Knowledge Machines: Digital Transformations of the Sciences and Humanities Sebastián Ureta, Assembling Policy: Transantiago, Human Devices, and the Dream of a World-Class Society Geoffrey C. Bowker, Stefan Timmermans, Adele E. Clarke, and Ellen Balka, eds., Boundary Objects and Beyond: Working with Leigh Star Clifford Siskin, System: The Shaping of Modern Knowledge Lawrence Busch, Knowledge for Sale: The Neoliberal Takeover of Higher Education Bill Maurer and Lana Swartz, Paid: Tales of Dongles, Checks, and Other Money Stuff Dietmar Offenhuber, Waste Is Information: Infrastructure Legibility and Governance Katayoun Shafiee, Machineries of Oil: An Infrastructural History of BP in Iran Megan Finn, Documenting Aftermath: Information Infrastructures in the Wake of Disasters Laura Watts, Energy at the End of the World: An Orkney Islands Saga Ann M. Pendleton- Jullian and John Seely Brown, Design Unbound: Designing for Emer- gence in a White Water World, Volume 1: Designing for Emergence Ann M. Pendleton- Jullian and John Seely Brown, Design Unbound: Designing for Emer- gence in a White Water World, Volume 2: Ecologies of Change Jordan Frith, A Billion Little Pieces: RFID and Infrastructures of Identification Morgan G. Ames, The Charisma Machine: The Life, Death, and Legacy of One Laptop Per Child Mario Biagioli and Alexandra Lippman, Gaming the Metrics: Misconduct and Manipulation in Academic Research Malcolm McCullough, Downtime on the Microgrid: Architecture, Electricity, and Smart City Islands Emmanuel Didier, translated by Priya Vari Sen, America by the Numbers: Quantification, Democracy, and the Birth of National Statistics Ryan Ellis, Letters, Power Lines, and Other Dangerous Things: The Politics of Infrastructure Security Gaming the Metrics Misconduct and Manipulation in Academic Research Edited by Mario Biagioli and Alexandra Lippman The MIT Press Cambridge, Massachusetts London, England © 2020 Massachusetts Institute of Technology This work is subject to a Creative Commons CC BY-NC-ND license. Subject to such license, all rights are reserved. This title is freely available as an open access edition thanks to the TOME initiative and the generous support of the University of California, Davis. Learn more at openmonographs.org. This book was set in Sabon by Westchester Publishing Services. Library of Congress Cataloging- in- Publication Data Names: Biagioli, Mario, 1946- editor. | Lippman, Alexandra, editor. Title: Gaming the metrics : misconduct and manipulation in academic research / edited by Mario Biagioli and Alexandra Lippman. Description: Cambridge, MA : MIT Press, [2020] | Series: Infrastructures | Includes bibliographical references and index. Identifiers: LCCN 2019010150 | ISBN 9780262537933 (pbk. : alk. paper) Subjects: LCSH: Scholarly publishing—Corrupt practices. | Learning and scholarship—Corrupt practices. | Research—Corrupt practices. | Communication in learning and scholarship—Moral and ethical aspects. Classification: LCC Z286.S37 G36 2020 | DDC 070.5—dc23 LC record available at https://lccn.loc.gov/2019010150 10 9 8 7 6 5 4 3 2 1 Contents Introduction: Metrics and the New Ecologies of Academic Misconduct 1 Mario Biagioli and Alexandra Lippman I Beyond and Before Metrics 25 1 Gaming Metrics Before the Game: Citation and the Bureaucratic Virtuoso 31 Alex Csiszar 2 The Transformation of the Scientific Paper: From Knowledge to Accounting Unit 43 Yves Gingras 3 Playing and Being Played by the Research Impact Game 57 Michael Power 4 The Mismeasurement of Quality and Impact 67 Paul Wouters 5 Taking Goodhart’s Law Meta: Gaming, Meta- Gaming, and Hacking Academic Performance Metrics 77 James Griesemer II Collaborative Manipulations 89 6 Global University Rankings: Impacts and Applications 93 Barbara M. Kehm 7 Predatory Publishing and the Imperative of International Productivity: Feeding Off and Feeding Up the Dominant 101 Sarah de Rijcke and Tereza Stöckelová vi Contents 8 Pressures to Publish: What Effects Do We See? 111 Daniele Fanelli 9 Ghost- Managing and Gaming Pharmaceutical Knowledge 123 Sergio Sismondo III Interventions: Notes from the Field 135 10 Retraction Watch: What We’ve Learned and How Metrics Play a Role 141 Ivan Oransky 11 PubPeer: Scientific Assessment Without Metrics 149 Boris Barbour and Brandon M. Stell 12 The Voinnet Affair: Testing the Norms of Scientific Image Management 157 Catherine Guaspare and Emmanuel Didier 13 Crossing the Line: Pseudonyms and Snark in Post- Publication Peer Review 169 Paul S. Brookes 14 Ike Antkare, His Publications, and Those of His Disciples 177 Ike Antkare 15 Fake Scientists on Editorial Boards Can Significantly Enhance the Visibility of Junk Journals 201 Burkhard Morgenstern 16 Altmetrics Gaming: Beast Within or Without? 213 Jennifer Lin 17 Why We Could Stop Worrying About Gaming Metrics If We Stopped Using Journal Articles for Publishing Scientific Research 229 Elizabeth Wager IV Mimicry for Parody or Profit 237 18 Making People and Influencing Friends: Citation Networks and the Appearance of Significance 243 Finn Brunton 19 Crack Open the Make Believe: Counterfeit, Publication Ethics, and the Global South 251 Marie- Andrée Jacob Contents vii 20 Fake Archives: The Search for Openness in Scholarly Communication Platforms 261 Alessandro Delfanti 21 Humor, Hoaxes, and Software in the Search for Academic Misconduct 271 Alexandra Lippman Acknowledgments 283 Contributors 285 Index 287 Introduction: Metrics and the New Ecologies of Academic Misconduct Mario Biagioli and Alexandra Lippman Traditionally, the assumption has been that academic misconduct emerges primarily in response to “publish or perish” pressures. Robert Slutsky, a UC San Diego cardiologist famously caught in 1986 reporting imagi- nary experiments, was, at one point, putting out one article every ten days (Lock and Wells, 2001). “Publish or perish,” however, is no longer the sole incentive for misconduct. New practices are emerging that are not limited to the production of fraudulent publications but are aimed instead at enhancing, often in unethical or fraudulent ways, the evalua- tion of their importance or “impact” (Biagioli, 2016). “Publish or perish” is merging with “impact or perish.”1 This is related to but different from the predictable gaming of aca- demic performance indicators one would expect from Goodhart’s law: as soon as an indicator becomes a target, gaming ensues, which fore- closes its ability to function as a good indicator.2 That may take the form, for instance, of massaging the definition of what counts as a “success- ful student” in metrics about schools’ performance, or of what counts as a “peer- reviewed” paper in faculty evaluation protocols. It could also involve aligning one’s practices to metrics-relevant parameters, like cap- ping classes’ enrollment to nineteen students to have them fit theUS News and World Report’s definition of “small class,” which is rewarded in its ranking of universities. But we now find authors and editors who move beyond this kind of gaming to create (rather than tweak) metric- enhancing evidence, such as citations to one’s work or to the work pub- lished in a given journal so as to boost its impact factor. We argue that the growing reliance on institutional metrics of evaluation does not just provide incentives for these kinds of manipulations, but also creates their conditions of possibility. They would not have come into being were it not for the new metrics- based “audit culture” of academia (Power, 1997; Strathern, 2000; Burrows, 2012). 2 Mario Biagioli and Alexandra Lippman Beyond Truth and Falsehood: Innovation in Manipulation As shown by the US federal definition, misconduct is construed in epis- temic terms: fabrication, falsification, and plagiarism.3 Accordingly, mis- conduct is equated to producing false statements like making up data, fudging data, and faking authorship—false statements within a publica- tion. Traditional fraud and misconduct continue to exist, and these defini- tions may be suitable to describe them. What they fail to grasp, however, are the new forms of manipulation that do not affect the epistemic status