Assisted Authoring for Avoiding Inadequate Claims in Scientific Reporting Anna Koroleva
Total Page:16
File Type:pdf, Size:1020Kb
Assisted authoring for avoiding inadequate claims in scientific reporting Anna Koroleva To cite this version: Anna Koroleva. Assisted authoring for avoiding inadequate claims in scientific reporting. Bioin- formatics [q-bio.QM]. Université Paris-Saclay; Universiteit van Amsterdam, 2020. English. NNT : 2020UPASS021. tel-02938856 HAL Id: tel-02938856 https://tel.archives-ouvertes.fr/tel-02938856 Submitted on 15 Sep 2020 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Assisted authoring for avoiding inadequate claims in scientific reporting Thèse de doctorat de l'université Paris-Saclay École doctorale n° 580 Sciences et Technologies de l’Information et de la Communication (STIC) Spécialité de doctorat: Informatique Unité de recherche : Université Paris-Saclay, CNRS, LIMSI, 91400, Orsay, France Référent : Faculté des sciences d'Orsay Thèse présentée et soutenue à Amsterdam, le 22 janvier 2020, par Anna KOROLEVA Sophia Ananiadou Présidente, Rapporteur Professeur, University of Manchester Paolo Rosso Rapporteur Professeur, Universitat Politècnica de València Ameen Abu-Hanna Examinateur Professeur, Universiteit van Amsterdam Evangelos Kanoulas Examinateur Professeur, Universiteit van Amsterdam Olivier Ferret Examinateur Ingénieur de Recherche, Université Paris-Saclay Nicolas Sabouret Examinateur Professeur, Université Paris-Saclay Patrick Paroubek Directeur de thèse Ingénieur de Recherche, Université Paris-Saclay Patrick M.M. Bossuyt Co-Directeur de thèse Professeur, Universiteit van Amsterdam Assisted authoring for avoiding inadequate claims in scientific reporting Anna Koroleva ISBN: 978-90-830376-4-6 Cover design: James Temple-Smith & Anna Koroleva Layout: Anna Koroleva Printed by: Print Service Ede - www.printservice-ede.nl This project was funded by the European Union’s Horizon 2020 research and innovation pro- gramme under the Marie Sklodowska-Curie grant agreement No 676207. ○c 2020 Anna Koroleva. No part of this thesis may be reproduced, stored or transmitted in any form or by any means without permission of the author or publishers of the included scientific papers. Assisted authoring for avoiding inadequate claims in scientific reporting ACADEMISCH PROEFSCHRIFT ter verkrijging van de graad van doctor aan de Universiteit van Amsterdam op gezag van de Rector Magnificus prof. dr. ir. K.I.J. Maex ten overstaan van een door het College voor Promoties ingestelde commissie, in het openbaar te verdedigen in de Agnietenkapel op woensdag 22 januari 2020, te 16.00 uur door Anna Koroleva geboren te Moskou Promotiecommissie: Promotor: prof. dr. P.M.M. Bossuyt AMC-UvA Promotor: dr. P. Paroubek Université Paris-Saclay Overige leden: prof. dr. A. Abu-Hanna AMC-UvA prof. dr. E. Kanoulas Universiteit van Amsterdam prof. dr. S. Ananiadou University of Manchester prof. dr. P. Rosso Universitat Politècnica de València prof. dr. N. Sabouret Université Paris-Saclay dr. O. Ferret Université Paris-Saclay Faculteit der Geneeskunde Dit proefschrift is tot stand gekomen in het kader van het European project "MIROR" GA No. 676207, met als doel het behalen van een gezamenlijk doctoraat. Het proefschrift is voorbereid in de Faculteit der Geneeskunde van de Universiteit van Amster- dam en in de Centre National de la Recherche Scientific (CNRS) van de Université Paris-Saclay. This thesis has been written within the framework of the European project "MIROR" GA No. 676207, with the purpose of obtaining a joint doctorate degree. The thesis was prepared in the Faculty of Medicine at the University of Amsterdam and in the Centre National de la Recherche Scientific (CNRS) at the Université Paris-Saclay. Contents Introduction 1 I Algorithm scheme and annotation 9 1 Automatic detection of inadequate claims in biomedical articles: first steps. Anna Koroleva, Patrick Paroubek. Proceedings of Workshop on Curative Power of MEdical Data, Constanta, Romania, September 12-13, 2017. 11 2 Annotating Spin in Biomedical Scientific Publications: the case of Randomized Controlled Trials (RCTs). Anna Koroleva, Patrick Paroubek. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 7-12, 2018 25 II Algorithms 43 3 Extracting outcomes from articles reporting randomized controlled trials using pre-trained deep language representations. Anna Koroleva, Sanjay Kamath, Patrick Paroubek. Submitted 45 4 Measuring semantic similarity of clinical trial outcomes using deep pre- trained language representations. Anna Koroleva, Sanjay Kamath, Patrick Paroubek. Journal of Biomedical Informatics - X 69 5 Towards Automatic Detection of Primary Outcome Switching in Articles Reporting Clinical Trials. Anna Koroleva, Patrick Paroubek. Submitted 101 6 Extracting relations between outcomes and significance levels in Random- ized Controlled Trials (RCTs) publications. Anna Koroleva, Patrick Paroubek. Proceedings of ACL BioNLP Workshop 2019, Florence, Italy, August 2019 121 III Implementation and interface 143 7 DeSpin: a prototype system for detecting spin in biomedical publications. Anna Koroleva, Sanjay Kamath, Patrick MM Bossuyt, Patrick Paroubek. Submitted 145 IV Difficulties 163 8 What is a primary outcome? A corpus study. Anna Koroleva, Elizabeth Wager, Patrick MM Bossuyt. Submitted 165 Summary 189 Samenvatting 195 Résumé 201 PhD Portfolio 207 Contributing authors 211 Acknowledgements 213 About the author 215 Introduction Many, if not all, of us came across a situation of being prescribed a medication that did not work as it was supposed to and, maybe, even worse, had some unpleasant side effects, leaving us wondering how such a substance could get into clinical use, why our doctor chose to prescribe it, and why there was no warning of side effects. Absence of effects, or even negative effects, of drugs can occur simply because persons differ, and even established and approved drugs may not work, or do not help, everyone. But what if there are other mechanisms as well that can lead to the dissemination of less effective drugs, to doctors prescribing them, and may lead the public into believing in their effectiveness? I had no idea that such mechanisms can exist before I started my PhD and learned about spin in reporting research results. The notion of "spin" comes from the domain of politics, where it denotes a form of propa- ganda and consists in passing on a biased message to the public, to create a certain (positive or negative) perception of a person or event (Tye, 1998; Jackson and Jamieson, 2007; Boardman et al., 2017). Political spin is often related to deception and manipulation. From politics, the term "spin" came into the domain of scientific research, where it refers to presenting research results in a more positive (or, rarely, more negative) way that the obtained evidence justifies. In 2010, Boutron et al. (2010) introduced and defined the term "spin"for research domain, in particular, for randomized controlled trials (RCTs) - clinical trials studying a new intervention by comparing it to an established intervention or to a placebo. In RCTs with non-significant primary outcome, spin is defined as "the use of specific reporting strategies, from whatever motive, to highlight that the experimental treatment is beneficial, despite a statistically nonsignificant difference for the primary outcome, or to distract the reader from statistically nonsignificant results" (Boutron et al., 2010). A variety of terms was usedto denote this phenomenon, such as distorted / misleading presentation, mis-/overinterpretation of research results; "spin" is now the most commonly accepted term for this phenomenon. If one looks at the examples of spin provided in the relevant articles, it may seem that the difference between "spinned" and fair presentation of results is very slight; as aresult, people often wonder if spin is really a problem. To answer this question, Boutron et al. (2014) 1 compared how clinicians perceive the treatments presented in abstracts with spin and in the same abstract rewritten without spin. The authors showed that the clinicians overestimated the effects of the treatment after reading a "spinned" abstract. Given that the abstract isoften the only part of an article available to wide public free of charge, readers often cannot read the full text of an article to assess the correctness of conclusions in the abstract. Hence, clinical decisions can be made on the basis of the information in abstracts; and so spin in abstracts can have a highly negative impact on clinical decision-making. Interventions with unproved efficacy or safety can get into production and clinical use (and maybe this is one of thereasons we have that negative experience with taking medications?). Spin in research articles can have a wider impact: it was shown to be related to spin in press releases and health news (Haneef et al., 2015; Yavchitz et al., 2012), and thus can impact the beliefs and expectations of the public regarding new treatments. The phenomenon of spin has started to attract attention not only of the research community, but also