Publication and Registration Bias in Randomized Clinical Trials

Mustafa Al-Durra

A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Institute of Health Policy, Management, and Evaluation University of Toronto

Publication and Registration Bias in Randomized Clinical Trials Mustafa Al-Durra Doctor of Philosophy Institute of Health Policy, Management, and Evaluation University of Toronto 2019 Abstract The research literature has identified the impact of publication bias in randomized clinical trials

(RCTs). The prospective registration of RCTs could mitigate the publication bias by allowing researchers to explore a source of trial information, that may or may not be published. This thesis offers a critical appraisal of the compliance in prospective registration and results publication of

RCTs.

This thesis includes three studies. The first study evaluated the publication bias and prospective registration in digital health RCTs in ClinicalTrials.gov. To expand the scope of this study beyond a single trial registry, the second study examined the global prospective registration, the reporting of the trial registration number (TRN), and the publication rates of digital health RCTs registered in any of the World Health Organization (WHO) registries. To improve the external validity beyond the realm of digital health RCTs, the third study provided an empirical analysis of the prospective registration, the reporting of TRN, and the selective registration bias in all

RCTs registered in any of the WHO registries, and published in any PubMed-indexed journal in

2018, ten years after the adoption of the 7th revision of the Declaration of Helsinki (DoH).

The overall results showed that (1) nearly one third of all digital health RCTs remain unpublished (27% and 34.5% in the first and second study respectively), (2) the compliance in prospective registration of RCTs remains low (29.3%, 38% and 41.7% in the first, second and third study respectively), and (3) the compliance in reporting the TRN in published RCTs was low (49%, 52% and 71.2% in the first, second and third study respectively), and (4) a new form of bias was detected “Selective Registration Bias”, with an increased (biased) tendency of the

investigators to register a trial only when intending to publish the trial. The majority of RCTs

(95.7% and 85.1% in the second and third study respectively), that were registered retrospectively, were registered within one year before submitting the manuscript for journal publication.

This thesis proposed several recommendations to inform the future development of best practices to mitigate the publication and registration bias in evidence-based medical research.

iii

Dedication

This thesis is dedicated to the memory of my beloved father, Basil Mustafa Ibrahim Al-Durra. A professional engineer, a polyglot, and a true mentor whom I lost when I was 23 years of age. I know I’ve made you proud with this work - a contribution to your legacy. Thank you for raising me the way you did. You shan’t be forgotten.

& to the kindness of my devoted mother, Dr. Basima Ali Mahmoud Shafaamri. A selfless mother, a life-long friend, and a medical doctor with an exemplar career. You instilled confidence, integrity and resilience in me. Thanks for believing in me and the wonderful motherhood over the past four decades. Mama - I’m eternally grateful.

Acknowledgments

This research was a multi-year effort that only succeeded because of the invaluable support and guidance I received from numerous mentors and pioneers in the field. To whom I owe a dept of gratitude. In particular, to Dr. Joseph Cafazzo, who stepped in as my supervisor, ever so briefly, and guided my research journey to safe shores. It is a privilege for me to have your co-authorship on my first published paper and all the three research papers of this thesis. I am thankful for your encouragements and for setting the path to the successful completion of my PhD thesis. Your passion and commitment to digital health research is a gold trophy to the University of Toronto and the eHealth Innovation at the University Health Network. To my world-class committee members, thank you for your thoughtfulness, rigor and continuous feedback. To Dr. Emily Seto, you were there for me when I need you the most. Not only did you provide prompt and critical insight to my research, but you addressed a very delicate academic challenge with the highest degree of professionalism, even when the resolution implied more unexpected work on your side. Thank you for your counsel and advice. To Dr. Robert Nolan, I very much enjoyed our discussions at your office when you kindly prioritized meeting with me before seeing your next patient. You educated me about the Hawthorne effect and the differences between intrinsic and extrinsic motivation in influencing targeted behaviors. Thank you for your countless revisions and intellectual contributions to this research. To Dr. Gunther Eysenbach, who supervised my master’s dissertation and extended the opportunity for me to join the PhD program under his supervision. Thank you for your enthusiasm and thoroughness in the inception of this thesis, and for the honor to have your senior co-authorship on the publication of the first paper of this thesis. Thank you to Dr. Quynh Pham, Dr. Patrick Ware, and Melanie Yeung at the eHealth Innovation center for your support and peer-mentorship over the past years. I take pride in completing this thesis on an academic, professional and personal level. I will be the first Iraqi born student to graduate with a PhD degree from the Institute of Health Policy, Management and Evaluation, who was a part time student and employed full-time in the private

sector (information technology industry) during the eight years of this PhD program. This accomplishment is the culmination of several years of my own research work, but more so of the support I received from many others throughout my life journey from Iraq to Germany and now Canada. I would like to take the honor to recognize a few, so bear with me. To the loving memory of my grandmother, Eugene (Sitto), my aunt, Nahida, my uncle Salman, my uncle Basim, my two brothers (Saif and Salam), and the rest of my extended family, thank you for your enormous support. Perhaps, I can describe one day how much you all mean to me. To the teachers of Baghdad College, and the professors of the University of Baghdad, thank you for all the knowledge, education and direction I received from you. You maintained the highest standards in education with very limited means during the many years of war. To my cousin Rabab Makki, and my first manager, Osama Matalka, thanks for believing in me and helping me land my first job during the most difficult year of my life. To the memory of Jürgen Leyde, thanks for offering me the opportunity to work for your company (Leyco Chemische Leyde GmbH) in Germany, become a polyglot like my father, and take advantage of the limitless career and professional development opportunities in Germany. To Ralf-Jörg Staufenbiel, the inclusiveness in your hiring decision, and the persistence in your mentorship has fundamentally transformed my career during my tenure at Avantem GmbH in Cologne, Germany. Dafür möchte ich mich an dieser Stelle nochmals für die Zusammenarbeit und für das mir entgegengebrachte Vertrauen recht herzlich bedanken. To Frank Körner, and Dr. Heimo H. Adelsberger, thanks for offering me the opportunity to peruse my post-graduate degree at the University of Duisburg-Essen, Germany. Because of your trust in my academic, professional and language competencies, I was exceptionally admitted to the VAWi master’s program. You also provided me with the option to complete my master’s thesis out of Toronto, Canada in collaboration with the University of Toronto. Thank you to Dr. Whitney Berta for her outstanding help when I was an international visiting student at the IHPME, and to the memory of Dr. Kevin Leonard, who interviewed me when I applied for this PhD program in summer 2010. Thank you to my Canadian employers who valued my pursuit of multi-disciplinary education and allowed for flexible working hours to accommodate my class time schedule. Sincere thanks to David Judd (MedCurrent Inc.), Kaytek Przybylski (Avanade Canada Inc.), and Marc Gagne (Microsoft Canada Inc.).

Thank you to all my friends who directly or indirectly supported me. A special thank you to my dear friend Dr. Zaid Al-Kaissy, who always made himself available whenever I needed his attention, and whose own success story is a stream of hope and motivation for me and many others. Finally, I truly thank you Sanya for being such an amazing wife, for always having that smile on your face, for your patience, determination and sacrifices to the success of this work. You are my inspiration. I look forward to writing the next chapters of our lives together.

vii

Table of Contents

Acknowledgments...... v

List of Tables ...... xv

List of Figures ...... xvi

Abbreviations ...... xvii

Chapter 1 Introduction ...... 1

Introduction ...... 1

1.1 Problem Statement ...... 1

1.2 Research Objectives ...... 2

1.3 Thesis Overview ...... 3

Chapter 2 Background ...... 6

Background ...... 6

2.1 Publication Bias in Biomedical Research ...... 6

2.2 Contributing Factors to Publication Bias ...... 8

2.3 Trial Registration to Mitigate Publication Bias ...... 9

2.4 The Case for Digital Health Trials ...... 12

Chapter 3 Publication Bias in Digital Health Trials Registered in the ClinicalTrials.gov Trial Registry ...... 14

Publication Bias in Digital Health Trials Registered in the ClinicalTrials.gov Trial Registry ...... 15

3.1 Abstract ...... 15

3.2 Introduction ...... 17

3.2.1 Background ...... 17

3.2.2 Research Objectives ...... 18

3.3 Methods...... 18

3.3.1 Data Source ...... 18

3.3.2 Inclusion Criteria ...... 18

viii

3.3.3 Justification of the Completion Date ...... 19

3.3.4 Exclusion Criteria ...... 19

3.3.5 Search Terms ...... 20

3.3.6 Data Extraction ...... 20

3.4 Results ...... 22

3.4.1 Screening Process ...... 22

3.4.2 Publication Rates ...... 24

3.4.3 Analysis of Trial Characteristics ...... 24

3.5 Discussion ...... 33

3.6 Limitations ...... 37

3.7 Conclusion ...... 38

3.8 Conflicts of Interest...... 39

Chapter 4 Registration and Publication Bias in Digital Health Trials Registered in Global Trial Registries ...... 40

Registration and Publication Bias in Digital Health Trials Registered in Global Trial Registries ...... 41

4.1 Abstract ...... 41

4.2 Introduction ...... 41

4.3 Research Objectives ...... 43

4.4 Methods...... 43

4.4.1 Data Source ...... 43

4.4.2 Statistical Analysis ...... 43

4.4.3 Inclusion Criteria ...... 44

4.4.4 Enrollment Date Justification ...... 44

4.4.5 Exclusion Criteria ...... 44

4.4.6 Search Terms ...... 44

4.4.7 Data Extraction ...... 45

4.4.8 Identification of Randomized Trials ...... 46

4.4.9 Identification of Publication ...... 46

4.5 Results ...... 46

4.5.1 Prospective Trial Registration and Publication Rates ...... 47

4.5.2 Enrollment-To-Publication Duration and Trial Size...... 52

4.5.3 Registration-To-Publication Duration and Retrospective Registration ...... 52

4.5.4 Recruitment Compliance and Inclusion of the TRN in Published Trials ...... 53

4.6 Discussion ...... 57

4.6.1 Retrospective Registration of Digital Health Clinical Trials ...... 58

4.6.2 Selective Registration Bias of Digital Health Clinical Trials ...... 58

4.6.3 Challenges in Oncology Trials...... 59

4.6.4 Compliance with Prospective Trial Registration in The Australian New Zealand Clinical Trials Registry ...... 60

4.6.5 Recruitment Compliance in the Middle East ...... 60

4.6.6 Recruitment Compliance in Low and Middle-Income Countries ...... 61

4.6.7 Adherence to Best Practices in Clinical Trials from Europe ...... 61

4.6.8 Time to Publication ...... 61

4.7 Limitations ...... 62

4.8 Conclusion ...... 63

4.9 Data availability ...... 63

4.10 Code availability ...... 63

4.11 Preprint Statement ...... 63

4.12 Authors Contribution ...... 63

4.13 Acknowledgements ...... 64

4.14 Competing Interests ...... 64

4.15 Funding ...... 64

Chapter 5 Prospective Registration and Reporting of Trial Number in Randomized Clinical Trials: Adoption of the ICMJE and the Declaration of Helsinki Recommendations ...... 65

Prospective Registration and Reporting of Trial Number in Randomized Clinical Trials: Adoption of the ICMJE and the Declaration of Helsinki Recommendations ...... 66

5.1 Abstract ...... 66

5.2 Introduction ...... 68

5.3 Research Objectives ...... 69

5.4 Methods...... 69

5.4.1 Data Source ...... 69

5.4.2 Tools for Statistical Analysis and Data Preparation ...... 70

5.4.3 Inclusion Criteria ...... 70

5.4.4 Exclusion Criteria ...... 70

5.4.5 Search Terms and Methodology ...... 70

5.4.6 Data Extraction ...... 71

5.4.7 Patient and public involvement ...... 72

5.5 Results ...... 72

5.5.1 Reporting of TRN ...... 75

5.5.2 Prospective Registration of Clinical Trials ...... 76

5.5.3 Authors Explanation for Delayed Registration of Clinical Trials ...... 79

5.6 Discussion ...... 81

5.6.1 Reporting of TRN ...... 82

5.6.2 Prospective Registration of Clinical Trials ...... 82

5.6.3 Compliance within the ICMJE and High Impact Factor Journals ...... 84

5.6.4 Registration Bias (Selective Registration Bias) ...... 86

5.6.5 Authors’ Explanation for Delayed Registration of Clinical Trials ...... 86

5.7 Implications for Editors, Investigators and Policy Makers ...... 88

5.7.1 Recommendations to the ICMJE ...... 89

5.7.2 Recommendations to Journal Editors ...... 89

5.7.3 Recommendations to the WHO ...... 89

5.7.4 Recommendations to Ethics Committees ...... 90

5.7.5 Recommendations to the Investigators and Authors ...... 90

5.8 Limitations ...... 91

5.9 Conclusion ...... 92

5.10 Contributors ...... 92

5.11 Funding ...... 93

5.12 Competing Interests ...... 93

5.13 Ethical Approval ...... 93

5.14 Data Sharing...... 93

5.15 Transparency Statement ...... 93

Chapter 6 Discussion ...... 95

Discussion ...... 95

6.1 Summary of Research Finding...... 95

6.1.1 Prevalence of Publication Bias in Digital Health RCTs ...... 95

6.1.2 Compliance in Prospective Registration of RCTs ...... 97

6.1.3 Compliance in Reporting the TRN within Published RCTs ...... 99

6.2 Conceptual Contribution ...... 100

6.2.1 Articulating a New Form of Bias in RCTs (Registration Bias) ...... 100

6.2.2 Extending the Definition of Prospective Registration ...... 102

6.3 Methodological Contribution ...... 104

6.3.1 Challenges and Opportunities in Identifying Unpublished RCTs ...... 104

6.3.2 Automation Strategy for Identifying the TRN in Published RCTs...... 107

6.3.3 Challenges in Identifying the TRN from the Full-Text RCT Manuscripts ...... 111

6.4 Limitations ...... 112

xii

Chapter 7 Conclusion ...... 115

Conclusion...... 115

References ...... 118

Appendices ...... 135

8.1 Appendix 1 - Evaluation of Trial Latest Completion Date ...... 135

8.2 Appendix 2 - Determination of Search Terms and Phrases ...... 136

8.3 Appendix 3 – Classification of Trial Condition Groups ...... 138

8.4 Appendix 4 – Classification of Trial Discontinuation Reasons ...... 139

8.5 Appendix 5 – Classification of Trial Major Technology ...... 141

8.6 Appendix 6 – Identification of Publication ...... 143

8.7 Appendix 7 – Global Distribution of All Included Trials ...... 147

8.8 Appendix 8 – Determination of Search Terms and Phrases ...... 149

8.9 Appendix 9 – Classification of Trials Condition Groups ...... 151

8.10 Appendix 10 – Identification of Prospective Trial Registration ...... 153

8.11 Appendix 11 – Classifications of Trials’ Major Technology ...... 157

8.12 Appendix 12 – Identification of Publication ...... 159

8.13 Appendix 13 – Funding Sources by Trial Registry and Location ...... 163

8.14 Appendix 14 –Global Distribution of Registered Clinical Trials ...... 164

8.15 Appendix 15 – Relationship between Trial Enrollment-To-Publication Duration and Trial Size ...... 166

8.16 Appendix 16 – Summary of Trial Enrollment-To-Publication Duration, Number of Published Trials, and the Cumulative Percentage of Non-Publication Rates ...... 167

8.17 Appendix 17 – Sample XML Export File from PubMed Results ...... 168

8.18 Appendix 18 – Details Describing the Search Methodology to Identify the TRN in Published Papers ...... 169

8.19 Appendix 19 – Trial Registration Numbers Examples of the WHO Trial Registries .....172

8.20 Appendix 20 – Trial Protocols not Found in the ICTRP Search Portal ...... 173

xiii

8.21 Appendix 21 – Classification of Trial Condition ...... 174

8.22 Appendix 22 – Classification of Trial Funding Sources ...... 175

8.23 Appendix 23 – Interval between Enrollment and Registration Dates in Retrospectively Registered Clinical Trials...... 176

8.24 Appendix 24 – Screening of Journal Publication Dates ...... 178

8.25 Appendix 25 – Reasons for Delayed Registration ...... 180

8.26 Appendix 26 – Scoping Analysis of Studies Reporting on Compliance in Prospective Trial Registration and Inclusion of TRN in Published Trial Papers ...... 182

8.27 Appendix 27 –Trial Registration Numbers and Patterns ...... 186

8.28 Appendix 28 – Logistic Regression Test of Recruitment Compliance by Trial Size ...... 187

xiv

List of Tables

Table 1. Analysis of RCTs by their Lead Sponsor Information...... 21

Table 2. Relationship between the characteristics of RCTs and nonpublication rate...... 25

Table 3. Results of the Pearson Chi-square test between start date of trials and prospective trial registration...... 30

Table 4. Summary of reasons for discontinuation...... 31

Table 5. Analysis of trial publication cycles (duration)...... 32

Table 6. Relationship between trial characteristics and prospective registration and nonpublication rates of included trials...... 48

Table 7. Relationship between trial registration-to-publication duration and retrospective trial registration...... 53

Table 8. Recruitment compliance and inclusion of TRN in published trials...... 54

Table 9. Journal Characteristics Relationship to Reporting of TRN...... 75

Table 10. Prospective Trial Registration Relationship to Trial Characteristics, Journal Impact Factors and Journal ICMJE Membership...... 76

Table 11. Relationship of Retrospective Trial Registration within the First Three Weeks after Enrollment between US based Trials and International Trials...... 80

Table 12. Relationship between Enrollment Interval, Journal Types, and Journals Impact Factors of Trial Registered Retrospectively in the ISRCTN Registry...... 81

Table 13. Results of the Pearson Chi-square test between Non-Publication Rates and Location of Digital Health RCTs ...... 96

Table 14. Differences in Formats and Patterns of Reported TRN in Published RCTs ...... 109

List of Figures

Figure 1. Trials included from the search results...... 23

Figure 2. Results of the publication-identification process...... 24

Figure 3. Time to publication of registered clinical trials in digital health...... 32

Figure 4. Included trials from the search results...... 47

Figure 5. Identification of trials publication...... 47

Figure 6. Relationship between Trial Enrollment-To-Publication Duration and Trial Size, Number of Published Trials and Non-Publication Rate...... 52

Figure 7. Search Results in PubMed...... 72

Figure 8. Screening and Identification of TRN...... 73

Figure 9. Delayed Trial Registration after Enrollment (Measured in Weeks)...... 80

Figure 10. Identifying Unpublished Trials...... 105

Figure 11. Identifying TRN in Published RCTs ...... 108

xvi

Abbreviations

ANZCTR Australian New Zealand Clinical Trials Registry API Application Programmable Interface BMJ British Medical Journal CTRI Clinical Trials Registry - India DoH Declaration of Helsinki EU-CTR European Clinical Trials Register FDA Food and Drug Administration ICD International Classification of Diseases ICMJE International Committee of Medical Journal Editors ICTRP International Clinical Trials Registry Platform IRCT Iranian Registry of Clinical Trials JMIR Journal of Medical Internet Research MeSH Medical Subject Headings NHS National Health Services NIH National Institutes of Health NLM National Library of Medicine NTR The Netherlands National Trial Register PDF Portable Document Format PRISMA The Preferred Reporting Items for Systematic Reviews and Meta-Analyses RCT Randomized Clinical Trial RegEx Regular Expressions SMS Short Messages Service TRN Trial Registration Number UHN University Health Network UK United Kingdom US United States UTN Universal Trial Number WHO World Health Organization XML Extensible Markup Language

xvii

Chapter 1 Introduction

Introduction

1.1 Problem Statement

Within the constructs of ethics and integrity of biomedical research involving human subjects, study results should be made available and accessible to the broader scholar community. Researchers, clinicians, and patients will then be able to assess the evidence around the efficacy and adversity of healthcare interventions [1]. Empirical studies indicate that not all results of biomedical research are successfully published in peer-reviewed journals. Perhaps, one of the first assessments of publication bias in scientific research was reported by Sterling in 1959 which found a high prevalence of studies with statistically significant results published in four psychology journals between 1955 and 1956 [2]. In 1979, Robert Rosenthal introduced the term “file drawer problem,” acknowledging the existence of selective publication bias in studies that reported positive and significant results [3]. Clinical trials are key to running interventional and observational biomedical studies on human participants and to advance evidence-based medical research [4]. Investigators have ethical and scholarly obligations to disseminate the knowledge generated by their trial results, in particular with respect to the study efficacy, safety, potential risk and adverse events [1,5,6]. Several studies have reported that half of all clinical trials remain unpublished in any peer-reviewed journal [7-10]. Failing to publish trial results can be considered a form of scientific misconduct when considering the use of scarce research resources invested in the execution of clinical trials without disseminating any knowledge back to society [10,11]. The phenomenon of publication bias in clinical trials was attributed to the tendency of investigators and journal editors to submit and publish findings that are positive and statistically significant [11-13]. Investigators of clinical trials reported lack of time as the key factor for not publishing their results in a peer-review journal along with other factors such as the lack of relevance of their results and disagreement with other coauthors [14,15]. The registration of clinical trials, first proposed by Simes in 1986, provides a means to mitigate publication bias by allowing researchers, scholars and healthcare professionals to explore a

source of trial information, that may or may not be published [11-13]. Trial registries could help mitigate selective reporting of positive outcomes by comparing outcomes reported in trial publication versus outcome measures indicated in the trial registration [14,16-20]. During the past two decades, this proposal triggered numerous calls demanding mandatory registration of clinical trials [21-26]. In response to these calls, a few initiatives were introduced to enable the implementation and adoption of clinical trials registries, such as the inception of the world's largest clinical trial registry, ClinicalTrials.gov, and the development of global standards for clinical trial registration dataset by the WHO [27,28]. Fulfilling the potential objectives of trial registries necessitates the adequate and prompt registration of clinical trials. Early trial registration may mitigate the bias associated with modification of pre-defined trial outcome measures due to preliminary analysis of the trial results [29,30]. Thus, the prospective registration of clinical trials at or before the enrollment of study participants was mandated and embraced by a number of international clinical research organizations, such as the International Committee of Medical Journal Editors (ICMJE) and the 7th revision of the DoH [31,32]. When a clinical trial is registered in a trial registry, a unique TRN is assigned for that trial in that trial registry. Investigators should consider including the TRN in all future dissemination of their trials results. The ICMJE and WHO recommends the inclusion of the TRN in the manuscript of published trials to enable linking the trial reports with the clinical trial registry information [33,34]. The implications of not including the TRN in the published trials would hinder establishing the linkage between registered and published trials and thus, impeding the evaluation of the quality and bias in reporting trial outcomes. 1.2 Research Objectives

The aim of this thesis is to evaluate the compliance with the guidelines with respect to trial registration and results publication of RCTs. When examining the publication bias of RCTs, this thesis focused on digital health RCTs. Investigators in digital health research may be overwhelmed with challenges that are unique to digital health research, such as the usability of the intervention under test, participant recruitment, and retention challenges that may contribute to non-publication rate and prospective trial registration [21,35-40].

When examining the registration bias of RCTs, this thesis focused on providing a critical analysis of prospective registration of RCTs as per the recommendations of the ICMJE, WHO and 7th revision of the DoH. This thesis addressed the following overarching research questions: Research Question 1: What is the prevalence of publication bias, and compliance in prospective trial registration, in digital health RCTs registered in ClinicalTrials.gov trial registry? Research Question 2: What is the global prevalence of publication bias, and compliance in prospective trial registration, in digital health RCTs registered in any of the seventeen WHO registries? Research Question 3: What is the global prevalence of registration bias, and the current empirical compliance in prospective registration of RCTs registered in any of the seventeen WHO registries? Research Question 4: What is the adoption level of ICMJE member journals to the recommendations of the ICMJE, WHO and the 7th revision of the DoH with: a. Enforcing the prospective registration of RCTs as a prerequisite for accepting trial publication? b. Publishing the TRN in the published trials? c. Disclosure justifying the publication of retrospectively registered trials? 1.3 Thesis Overview

In this thesis, the research questions were addressed in three interconnected and successive studies. The findings and knowledge obtained from each study have informed the design and methodology of the subsequent study. Each study was submitted for publication independently. The content of the following thesis chapters was based on the published or submitted manuscripts of the three research studies as per the following summary: Chapter 2 – Provides a review of existing literature with a summary of reported publication bias and compliance with registration of clinical trials. Chapter 3 - Addresses Research Question 1 above. This study examined the prevalence and characteristics of publication bias among digital health RCTs that were completed between April 2010 and April 2013 and registered in the ClinicalTrials.gov database. As such, this study assessed the compliance with, and impact of, prospective trial registration on publication bias. Analysis of the duration between trial start and publication dates was also provided in this study.

Because this study included trials that were registered before 2008, it allowed for further analysis to explore the adoption of the 2004 ICMJE mandate and the 2008 DoH on prospective trial registrations in ClinicalTrials.gov. The findings of this study informed the design and methodology of the subsequent study, presented in Chapter 4. The reported prevalence of publication bias and low compliance in prospective trial registration in this study suggested a broader scope of analysis beyond a single trial registry. Expanding the analysis to include trials in other registries will improve the generalizability of this study, and also help identify any differences in prospective trial registration across trial registries other than ClinicalTrials.gov. The qualitative formative analysis of reasons for trial discontinuation has informed the direction of this research with increased emphasis on recruitment compliance in the subsequent study. The analysis of time to publication suggested adjustment to inclusion criteria to include trials started in 2012, i.e. six years before the subsequent study was conducted. Chapter 4 – Addresses the Research Question 2 above. The primary research objective of this study was to examine the global prospective trial registration and publication rates of digital health RCTs registered in any of the seventeen WHO registries. With a global scope of analysis, this study complemented the findings and external validity of the previous study reported in Chapter 3 with respect to the degree of publication bias, low compliance with prospective trial registrations, recruitment compliance, and the duration between trial start and publication dates. This study also assessed the compliance in reporting the TRN in published trials, which is key to identify if the trial was registered, and to identify the registered trial protocol. Perhaps, one of the most significant contributions of this study was the detection, and critical assessment, of a new form of bias (I refer to as “Registration Bias” or “Selective Registration Bias”), which is defined as the increased (biased) tendency of the investigators to register a trial only when intending to publish the trial. The findings of this study have informed the design and methodology of the subsequent study, presented in Chapter 5. After concluding this and the earlier study, the advancement and maturity achieved in our own-developed search tools and techniques would enable the subsequent study to expand the search inquiry across all RCTs that were published recently (in 2018, marking 10 years after the adoption of the 7th revision of the DoH). Thus, we were compelled to provide a current, and empirical, validation of the compliance in prospective trial registration, reporting of TRN, and the newly detected registration bias in retrospectively registered trials.

Chapter 5 – Addresses the Research Questions 3 and 4 above. This study provides a current and empirical analysis of the adoption of the ICMJE and WHO guidelines for prospective trial registration and reporting of TRN ten years after the adoption of the 7th revision of the DoH. It includes all RCTs that were published in 2018 that were registered in any of the seventeen WHO trial registries and published in any PubMed-indexed journal, irrespective of the trial location, condition, phase, number of participants, or funding source. Chapter 6 – The Discussion (Direction for Future Research, Limitations) section provides a synthesis of key findings from this thesis and discusses how the collective knowledge obtained from the three subsequent studies (chapters 3-5) can inform future research surrounding the guidelines of publication and registration bias in RCTs. This chapter concludes with a summary of overarching limitations of this thesis. Chapter 7 – Chapter 7 provides the conclusions of this thesis.

Chapter 2 Background

Background

2.1 Publication Bias in Biomedical Research

In the realm of scientific research, study results should be made available and accessible to the broader research community to assess the evidence around the efficacy and potential harm of healthcare interventions. Researchers, clinicians, and patients will then be able to assess the evidence around the benefits and adversity of healthcare interventions [1]. The first requirement of the Emanuel et al. ethical framework for clinical research describes the value derived from disseminating research results as “Only if society will gain knowledge, which requires sharing results, whether positive or negative, can exposing human subjects to risk in clinical research be justified” [5]. Empirical observations indicate that not all results of biomedical research are successfully published in peer-reviewed journals. In 1959, Theodore D. Sterling examined 294 experimental research articles that used a statistical test of significance and were published in four psychology journals between 1955 and 1956 [2]. He found that 286 (97.28%) of those articles rejected the null hypothesis at the traditional alpha level (P ≤0.05), and concluded that in research where statistical tests of significance are commonly used, research with nonsignificant results is not published. In 1972, Bozarth and Roberts reviewed over a thousand of research articles that used a statistical test of significance and were published in three major counseling journals between January 1967 and August 1970 [41]. In corroboration with Sterling's findings, they found that 94% of these articles rejected the null hypothesis at the traditional alpha level (P ≤0.05), and that statistically significant results are directly related to the publication of research articles. Bozarth and Roberts also indicated that it was not clear whether this bias in publishing research with significant results was due to editorial policies or due to the unwillingness of authors to submit their manuscripts with nonsignificant results. Bozarth and Roberts postulated that the editorial policies are more likely to explain this publication bias. Their postulate was based on (1) the importance of publication for the academic career of the researchers which negates their unwillingness to submit their manuscripts with nonsignificant results, and (2) the high rejection rate (50% to 88%) of all manuscripts submitted

to various American Psychological Association journals in 1964 [42,43]. Another study from Mahoney in 1977 has shown that peer-reviewers are more inclined to recommend publication of a submitted manuscript depending on the direction of the study outcomes and results [44]. Robert Rosenthal introduced the term “file drawer problem,” in 1979, acknowledging the existence of selective publication bias in studies that reported positive and significant results [3]. A decade later, Kay Dickersin defined publication bias as the drive of the researchers, peer- reviewers, and journal editors to submit and accept manuscripts for publication based on the significance and direction of the research outcomes [45]. Clinical trials are key to running biomedical research on human participants and to advance evidence-based medical research. Investigators of clinical trials have ethical and scholarly commitments to share and publish the knowledge generated by their clinical trials, in particular, to disseminate findings with respect to the trial efficacy, safety, potential risk and adverse events [6]. Studies have shown that not all results of clinical trials are published and trials with favorable results, even when reporting on different outcomes than intended, are more likely to be published [46]. In 2008, a study of publication rates of clinical trials supporting successful new FDA drug applications found that over half of all included trials were unpublished 5 years after FDA approval, and that trials with statistically significant results were more likely to be published compared to trials with nonsignificant results [10]. Similar findings were reported by a different study of publication rates of NIH funded trials registered in ClinicalTrials.gov [7]. In that study, over half of all included clinical trials were unpublished in any peer reviewed journal indexed by Medline within 30 months of trial completion. Another study of publication rates of randomized trials on vaccines also reported that half of the included and completed clinical trials were ultimately published in peer-reviewed journals [9]. In Germany, a study of publication rates of a cohort of clinical studies approved between the year 2000 and 2002 by the research ethics committee of the University of Freiburg also found that half of the clinical research conducted at a large German university medical center remained unpublished [1]. On the other hand, a study of large RCTs with at least 500 enrolled participants reported higher publication rates and only 29% of the included trials remained unpublished [6]. In 2014, two different studies of discontinued RCTs reported trial discontinuation rates at 21% and 24.9% of all included trials, with the discontinued trials being more likely to remain unpublished [9,47]. The selective, or incomplete reporting of clinical trial outcomes constitutes a major concern and leads to biased

interpretation of the clinical evidence by overstating the benefits or understating the adversity of the trial results [48-50]. 2.2 Contributing Factors to Publication Bias

The phenomenon of publication bias in clinical trials was attributed to the tendency of investigators and journal editors to submit and publish manuscripts with positive results [11-13]. As explained by the Wager et al. study, a distinction should be made between significant results and favorable results when describing publication bias for trials with positive results [12]. Trials with significant results are trials that reported statistically significant results to reject the null hypothesis. Trials with favorable results are trials with results supporting the direction, or position, of the research inquiry, such as supporting a new treatment or sponsor's product. For example, favorable results could be evaluated based on the clinical implications of the intervention in addition to, or irrespective of, the statistical test. The same study also surveyed editors of medical journals that publish clinical trials, and found that editors agreed that under- publication of trials with nonsignificant results is likely higher than trials with unfavorable results - as there seems to be more interest and scientific value in trails with unfavorable, compared to nonsignificant, results. In 1959, Sterling acknowledged a general perception among readers of reported studies to expect the rejection of null hypothesis in published results [2]. Editors of publishing journals and their acceptance criteria have an influence on publishing studies with favorable results. In 1980, the British Medical Journal (BMJ) published a letter justifying the rejection of 4000 articles with the emphasis on the focus of the journal to accept submissions reporting on new findings with improved prognosis or simplified management of common diseases [51]. Announcements of well-established peer-reviewed journals, such as the BMJ, would discourage investigators of clinical trials to submit their trial results for publication when primarily reporting unfavorable results. In 1987, Dickersin et al. reported that the major driver for nonpublication of clinical trials was the lack of publication submissions by investigators of completed trials, rather than a rejection decision by the journal editors [45,52]. Investigators of clinical trials reported a lack of time as the key factor for not publishing their results in a peer-reviewed journal, along with other factors such as the lack of relevance and importance of their results and disagreement with coauthors [14,15]. Another reported risk factor for trial nonpublication was trial size. Larger trials reasonably require more resources and staff,

which may be an indicator for the study quality. As such, the magnitude of statistical significance of clinical trials is proportional to the trial sample size, i.e. controlling for all other variables, the P value of the same trial will be more significant with larger trial size [53]. The evidence in the literature indicates that clinical trials with larger sample size are more likely to be published [10,45,52]. Funding sources, study language (in particular non-English language) and study design (single center versus multi center studies) were also identified as contributing factors for potential bias [20,24]. Davidson reported a statically significant association (P=0.002) between the funding sources and study outcomes of clinical trials, with trials funded by the pharmaceutical industry favoring the efficacy of a new drug treatment [45,54]. The Nylenna et al. experiment showed that the English version of a test manuscript seemed to be accepted more easily than a non-English version of the same manuscript [20,55]. This experiment included two test manuscripts with a number of common methodological flaws that were sent to 180 Scandinavian reviewers, and each reviewer received one of the manuscripts in English and the other manuscript in the national language. A number of studies showed that single-center, compared to multi-center trials, were likely to report larger treatment effects, with the multicenter RCTs being more likely to be published [56-60]. 2.3 Trial Registration to Mitigate Publication Bias

The registration of clinical trials, first proposed by Simes in 1986, provides a means to mitigate publication bias through allowing researchers, scholars, and healthcare professionals to explore another source of findings and information of trials, that may or may not be published [11-13]. Several researchers and advocates in the field have since called for the mandatory registration of clinical trials [21-26]. The trial registries would also help mitigate selective reporting of positive outcomes by comparing outcomes reported in trial publication versus outcome measures indicated in the trial registration [14,16-20]. The timely and prospective registration of clinical trials, i.e. early in the enrollment phase and prior to data collection, will also help ensure that the pre-defined primary outcome remains unchanged after potential analysis of the preliminary trial results [29,31]. Started in the year of 2000 and maintained by the United States (US) National Library of Medicine (NLM) at the National Institutes of Health (NIH), ClinicalTrials.gov registry has become the world’s largest clinical trial registry to date [27]. In September 2004, the ICMJE mandated trial registration in a public registry at or before study enrollment as a prerequisite for

publication in any of the ICMJE member journals and that the public trial registry should be publicly accessible at no charge and managed by a not-for-profit organization [31]. In 2005, the (WHO) started an initiative to standardize trial registrations and trial registry datasets across multiple national and international registries with emphasis on the inclusion of the TRN in the abstracts of the published trials to enable linking the trial publication with the trial protocol in the trial registry [28]. In 2007, the US Food and Drug Administration (FDA) Amendments Act expanded the compliance requirement for clinical trials to be registered within 21 days of enrolling the first participant [61,62]. In October 2008, the 7th revision of the DoH was adopted by the World Medical Association’s General Assembly, with increasing emphasis on prospective registration of trials and the ethical obligation of researchers to publish their study results [32]. In 2013 the United Kingdom (UK) National Health Services (NHS), through its health research authority, mandated a new compliance requirement for the registration of clinical trials no later than six weeks after the enrollment of the first study participant [63]. In 2015, the WHO announced a new statement on public disclosure of clinical trial results with more guidelines on trial registration, publication of results, and the inclusion of TRN in respective publications to enable linking of trial reports with clinical trial registry information [33]. As of June 8th, 2019, the WHO International Clinical Trials Registry Platform (ICTRP) includes seventeen different national and international trial registries with a unified search and access to registration information of 493,357 unique clinical trials [64]. Several studies, that were published between 2011 and 2019, examined the compliance in prospective trial registration, as defined by the ICMJE, WHO and DoH recommendations (i.e. registering the trial on or before the enrollment of the first patient) [65-81]. The results of these studies reported that the compliance in prospective registration of clinical trials was not adequate and ranged between 3.6% and 71.1% of all registered and published trials. In 2012, Reveiz et al. analyzed 526 RCTs published in 2010 (PUBMED/LILACS) from Latin America and the Caribbean and found that only 19 RCTs were registered prospectively [72]. In 2018, El- Boghdadly et al. analyzed 90 RCTs published in a single anesthetic journal (Anaesthesia) over a three-year period (between 2014 and 2016) and found that only 64 RCTs were prospectively registered with a clinical trials registry [67]. Other studies reported on the compliance in prospective trial registration with a more flexible definition of the prospective registration criteria. The reported compliance in prospective trial registration was at 33% and 77% in the Dechartes et al. study and Gopal et al. study, respectively

[82,83]. However, both studies considered a trial registration prospective if registered up to one month after the enrollment date of the first participant. Huser et al. found that the compliance in prospective trial registration was at 56% and 72% for published RCTs that were registered in 2006 and 2011 respectively [84]. However, this study was limited to RCTs published in five ICMJE journals and considered a trial registration prospective if registered up to two months after the enrollment date of the first participant. Dal-Ré et al. analyzed the timely registration of trials published in the six highest-impact general medicine journals, and found that 72% of these trials were registered before the date of primary end point ascertainment [85]. Overall, the existing studies of compliance in prospective registration of clinical trials were limited in scope and included trials published in a specific journal or a group of journals, trials of specific treatment or condition, or trials registered in a specific trial registry or a small group of trial registries. To improve the external validity of the findings of the existing studies, a broader research is needed to appraise the current compliance in prospective registration of clinical trials registered in a larger, and a global, group of clinical trial registries, irrespective of the trial condition, location, treatment, size, and publishing journals (for published trials). Exploring the rationale and challenges behind non-compliance with the prospective trial registration, i.e. the delay in retrospective registration of clinical trials, would help (1) inform investigators of future trials to be more proactive in the design, conduct, and dissemination of their research inquiry, and (2) inform the development and mandate of clinical trials registration and publication guidelines by research and ethics committees. The inclusion of the TRN in published RCTs is required to establish the linkage to underlying RCT protocols. Future research, such as systematic reviews and meta-analysis, would benefit immensely from this linkage. It enables a high degree of automation in the processes of screening and matching published RCTs to their registered protocols. In 2015, the WHO announced a new statement on public disclosure of clinical trial results with a focus on the inclusion of the TRN in publications to enable linking of trial reports with clinical trial registry information [33]. The current ICMEJ recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals require submitting authors of registered clinical trials to include the TRN at the end of the paper abstract [34]. The adoption of the ICMJE and WHO recommendation to include the TRN in published trials was reported in a number of reviews published with varying results [63-65,85-87]. Perhaps the earliest published analysis of the inclusion of TRN in published trials was by McGee et al. in

2011, which reported that the TRN was included in 59% of trials in kidney transplantation published between October 2005 and December 2010 [88]. In 2012, Jones et al. and van de Wetering et al. reported similar findings and indicated that the TRN was included in 46% and 60% of published trials respectively [65,87]. The highest 95.8% inclusion of TRN was reported by Huser et al. in 2013 for trials that were registered in the ClinicalTrials.gov registry and published in five ICMJE journals between 2010 and 2011 [84]. Another two studies reported that the TRN was only included in 26% and 28% of published trials in 2015 and 2018 respectively [64,86]. In 2019, Ross et al. reported that the TRN was included in 52.5% of rhinosinusitis trials that were registered in ClinicalTrials.gov and published in any PubMed(MEDLINE) indexed journal between January 2015 and October 2017 [89]. The implications of not including the TRN in the published trials would hinder establishing the linkage between registered and published trials and thus, impeding the evaluation of the quality and bias in reporting trial outcomes. 2.4 The Case for Digital Health Trials

In the past two decades, the adoption of technology components in the delivery of clinical interventions has become increasingly prevalent [35,90]. These technology components include internet-based websites (eHealth), smart phones (mHealth and Tele-Health), and smart devices applications (Ubiquitous-Health) that became an integral part to a wide range of behavioral change, chronic diseases, educational and counseling interventions [36,91,92]. Digital health clinical trials bring new opportunities and advancement to the field of clinical research, such as (1) the access to larger patient groups (large sample size), (2) the remote and online recruitment of patients from different parts of the world (enhanced generalizability), (3) the electronic collection of quantitative and qualitative data (such as vital signs, online questionnaires, and geolocation telemetry) and (4) cost advantage when compared to administrative cost of traditional trials (feasibility) [93,94]. There are significant challenges that are unique to digital health trials. Unlike traditional drug and medical devices trials, it’s common in digital health trials to report on the healthcare services delivery and utilization, instead of patient outcomes, as a primary outcome measure. Investigators may be unclear with the registration requirement for trials that do not report on patient outcomes [93,95,96]. Eysenbach presented other methodological challenges to digital health trials such as (1) double-blinding of the participants may not be possible as participants would know if they were enrolled in a digital health arm, and (2) the potential contamination of

the control group in particular in educational digital health interventions – due to the exposure of the control group to other online resources beyond those provided by the intervention [93]. Many evaluations of digital health interventions have shown no positive or small effect size [36,37,97-106]. A potential difficulty in evaluating these interventions is to measure the exposure and engagement of the intervention participants. These evaluation and efficacy challenges could contribute to publication bias subject to digital health trials. The existing research has shown that Dose-Response analysis suggests that adequate exposure to the intervention improves the effect of these interventions [35,37,107]. Constructs of the diffusion of innovations theory and persuasive system design can be employed to improve the exposure and adherence to digital health interventions [35-37,108]. Within the context of digital health trials, dropout and attrition of trial participants are inevitable. Several studies reported high dropout and attrition rates of participants in digital health interventions. The AL-Asadi et al. study reported on 3107 patients who participated in five online self-help anxiety treatment programs and found that 88.83% of these participants did not complete the post-treatment assessment [109]. The results from the Habibović et al. study published in 2014, a web-based randomized controlled trial to improve psychological well-being in patients with an implantable cardioverter defibrillator, have shown that only (23.3%) of the 146 trial participants have completed the treatment as intended [110]. In 2012, Peets et al. published the results from their clustered RCT with the objective to promote physical activities among participants above fifty years of age. The RCT examined the difference in attrition rates and showed that the dropout rate was significantly higher in the Web-based intervention group (53%) compared to the print-delivered intervention (39%, P<0.001) [111]. The indicated high dropout and attrition rates naturally introduce a publication bias in digital health trials, as reporting such results would question the efficacy and generalizability (external validity) of such interventions, and hence, less compelling reasons for publication relevance [112].

Chapter 3 Publication Bias in Digital Health Trials Registered in the ClinicalTrials.gov Trial Registry

Published in the Journal of Medical Internet Research on December 18th, 2018 Cited as: Al-Durra M, Nolan RP, Seto E, Cafazzo JA, Eysenbach G. Nonpublication Rates and Characteristics of Registered Randomized Clinical Trials in Digital Health: Cross-Sectional Analysis. J Med Internet Res 2018 Dec 18.

Summary: this study addresses the first research question of this thesis “What is the prevalence of publication bias, and compliance in prospective trial registration, in digital health RCTs registered in ClinicalTrials.gov trial registry?” This study included 556 digital health RCTs that were completed between April 2010 and April 2013 and registered in the ClinicalTrials.gov database. The primary research objective was to evaluate the nonpublication rate of digital health randomized clinical trials registered in ClinicalTrials.gov. The secondary research objective was to determine whether industry funding contributes to nonpublication of digital health trials. Overall, 150 (27%) of all included RCTs remained unpublished five years after their completion date. The bivariate and multivariate analyses reported statistically significant differences in RCT characteristics between published and unpublished RCTs for the intervention target condition, country, trial size, trial phases, and recruitment. There are substantial differences in nonpublication rates between trials funded by industry and none-industry sponsors. This study was published in the Journal of Medical Internet Research on December 18th, 2018. This chapter of this thesis includes the complete and published manuscript of this study with two minor modifications; (1) the font styles were adjusted to the style of this thesis document, (2) the citations and references were re-organizing to align with the overall references of this thesis, (3) Fisher’s exact test was added where applicable (i.e. when the expected cell value is less than 5), and (4) a few reference group and undefined groups were modified in the binary logistic regression to avoid non-convergence in the regression results. These changes do not materially alter the findings and conclusions of the published manuscript and are only included here for further precision as recommended by the external examiner for this thesis.

Publication Bias in Digital Health Trials Registered in the ClinicalTrials.gov Trial Registry

3.1 Abstract

Background: Clinical trials are key to advancing evidence-based medical research. The medical research literature has identified the impact of publication bias in clinical trials. Selective publication for positive outcomes or nonpublication of negative results could misdirect subsequent research and result in literature reviews leaning toward positive outcomes. Digital health trials face specific challenges, including a high attrition rate, usability issues, and insufficient formative research. These challenges may contribute to nonpublication of the trial results. To our knowledge, no study has thus far reported the nonpublication rates of digital health trials. Objective: The primary research objective was to evaluate the nonpublication rate of digital health randomized clinical trials registered in ClinicalTrials.gov. Our secondary research objective was to determine whether industry funding contributes to nonpublication of digital health trials. Methods: To identify digital health trials, a list of 47 search terms was developed through an iterative process and applied to the “Title,” “Interventions,” and “Outcome Measures” fields of registered trials with completion dates between April 1, 2010, and April 1, 2013. The search was based on the full dataset exported from the ClinlicalTrials.gov database, with 265,657 trials entries downloaded on February 10, 2018, to allow publication of studies within 5 years of trial completion. We identified publications related to the results of the trials through a comprehensive approach that included an automated and manual publication-identification process. Results: In total, 6717 articles matched the a priori search terms, of which 803 trials matched our latest completion date criteria. After screening, 556 trials were included in this study. We found that 150 (27%) of all included trials remained unpublished 5 years after their completion date. In bivariate analyses, we observed statistically significant differences in trial characteristics between published and unpublished trials in terms of the intervention target condition, country, trial size, trial phases, recruitment, and prospective trial registration. In multivariate analyses, differences in trial characteristics between published and unpublished trials remained statistically

significant for the intervention target condition, country, trial size, trial phases, and recruitment; the odds of publication for non-US–based trials were significant, and these trials were 3.3 (95% CI 1.845-5.964) times more likely to be published than US–based trials. We observed a trend of 1.5 times higher nonpublication rates for industry-funded trials. However, the trend was not statistically significant. Conclusions: In the domain of digital health, 27% of registered clinical trials results are unpublished, which is lower than nonpublication rates in other fields. There are substantial differences in nonpublication rates between trials funded by industry and nonindustry sponsors. Further research is required to define the determinants and reasons for nonpublication and, more importantly, to articulate the impact and risk of publication bias in the field of digital health trials.

3.2 Introduction

3.2.1 Background

Empirical observations demonstrate that not all clinical studies successfully publish their results in peer-reviewed journals. Perhaps, the earliest indication of publication bias in the area of scientific research was in 1979 by Robert Rosenthal with the term “file drawer problem,” acknowledging the existence of selective publication bias for studies with positive and significant results [3]. A decade later, Kay Dickersin defined publication bias as “the tendency on the part of investigators, reviewers, and editors to submit or accept manuscripts for publication based on the direction or strength of the study findings.”[45] The phenomenon of publication bias in clinical trials was attributed to the tendency of primary investigators and editors to submit or publish findings that are strong or statistically significant [11-13]. In 2008, a study of publication rates of clinical trials supporting successful new FDA drug applications found that over half of all the included trials were unpublished 5 years after obtaining approval from the FDA [10]. Similar findings were reported by other studies, indicating that half of all clinical trials remain unpublished in any peer-reviewed journal [7,8,11]. In 2014, two studies on discontinued RCTs reported discontinuation rates of 21% and 24.9%. This presents an ethical concern when considering the scarce research resources invested in the respective trials without the dissemination of any findings [9,47]. The registration of clinical trials, first proposed by Simes in 1986 [13], provides a means to mitigate publication bias by allowing researchers, scholars, and healthcare professionals to explore another source of trial results and information that may not be published [11-13]. It also helps identify discrepancies in primary outcome reporting by comparing primary outcome measures, as indicated in the trial protocols and published primary outcomes, which poses a key risk to the validity of trials [48,113-117]. During the past two decades, this proposal triggered numerous calls demanding mandatory registration of clinical trials [21-26]. In September 2004, the ICMJE mandated trial registration in a public registry at or before study enrollment as a prerequisite for publication in any of the ICMJE member journals and that the public trial registry should be publicly accessible at no charge and managed by a not-for-profit organization [31,118]. Soon thereafter, major medical journals announced the adoption of this new policy, including the BMJ, the Lancet, and the Journal of Medical Internet Research (JMIR) [21,24,119]. In October 2008, the 7th revision of the DoH was adopted by the World Medical Association’s General Assembly, with increasing emphasis on

prospective registration of trials and the ethical obligation on researchers to publish their study results [32]. Since its establishment in the year 2000, the ClinicalTrials.gov website, which is maintained by the US NLM at the NIH, has become the world’s largest clinical trial registry, with 286,717 registered trials, 60% of which are non-US–based as of October 11, 2018 [118,120- 122]. A number of studies have analyzed and reported the characteristics of publication rates of clinical trials registered in ClinicalTrials.gov [6-8,47] and other data sources [9,10]. However, to our knowledge, no study has thus far analyzed and reported the characteristics of publication rates within the domain of digital health. Digital health RCTs face specific challenges, including a high attrition rate, usability issues, and insufficient prior formative research [21,35-40]. These challenges may contribute to nonpublication of trial results. This study aimed to examine the prevalence and characteristics of the nonpublication rate of digital health randomized controlled trials registered in ClinicalTrials.gov. 3.2.2 Research Objectives

The primary research objective was to examine the prevalence and characteristics of the nonpublication rate among digital health RCTs registered in the ClinicalTrials.gov database. The secondary research objective was to determine whether industry funding contributes to nonpublication of trials. Considering that the ClinicalTrials.gov registry is a US–based registry including 60% of non-US–based trials, we intended to explore differences in the nonpublication rate and trial size between US- and non-US–based trials [123]. We also aimed to report outcome discrepancy between prospective and published primary outcomes of the included trials. 3.3 Methods

3.3.1 Data Source

The ClinicalTrials.gov website provides free, global open access to the online registry database through a comprehensive website search page as well as download capabilities; for example, all registration information for a given trial can be downloaded in XML format via a Web service interface. For our study, we downloaded the entire ClinicalTrials.gov online database, with 265,657 registered clinical trials entries, on February 10, 2018. 3.3.2 Inclusion Criteria

The research included all eHealth-, mHealth-, telehealth-, and digital health-related RCTs that are registered in the ClinicalTrials.gov website and include any information and communication

technology component, such as cellular phones, mobile phones, smart phones; devices and computer-assisted interventions; internet, online websites, and mobile applications; blogs and social media components; and emails, messages, and texts. We also included interventional and behavioral trials with or without the results. We limited our inclusion criteria to trials with latest completion dates between April 1, 2010, and April 1, 2013. The latest date between trials’ primary completion date and completion date fields was considered the latest completion date. Details regarding the evaluation of the latest completion date of trials are described in Appendix 1 [124,125]. 3.3.3 Justification of the Completion Date

Our search allowed for almost 5 years of a “publication lag period” between the stated trial completion date (up to April 1, 2013) and the search date for published reports (February 10, 2018). This strategy allowed us to account for longer publication cycles that may take up to several years, as indicated in prior studies [120]. For example, a study from the Netherlands that investigated the effects of a mobile phone app on the quality of life in patients with type 1 diabetes was published on May 11, 2015 [126], but the underlying clinical trial (NCT01444534) was first received by ClinicalTrials.gov on September 26, 2011, and the last update in ClinicalTrials.gov was made on October 23, 2012. To keep our data sample relevant, representative, and manageable, we chose to focus our study on a 3-year cross-sectional analysis for trials completed between April 2010 and April 2013. 3.3.4 Exclusion Criteria

Our search excluded registered clinical trials that were not randomized or only focused on electronic record-management systems such as electronic medical records, electronic health records, and hospital information systems as well as back-end integration systems, middleware applications, and Web services. Registered clinical trials that only reported on internet, Web- based, online, and computer-based surveys as well as television or online advertisement were also excluded. In addition, the search excluded registered clinical trials that focused only on biotechnology, bioinformatics analysis, and sequencing techniques. Finally, trials on medical devices and those only related to diagnostic imaging device, computerized neuropsychological, cognition, and oxygen assessment tools were excluded.

3.3.5 Search Terms

The search terms and phrases were conceptually derived from the inclusion criteria. A complete list of included search terms and phrases was developed through an iterative process (Appendix 2 [98,127-136]). The following list presents the final list of the 47 search terms and phrases that were included in the search process: “smartphone,” “smart-phone,” “cellphone,” “cell-phone,” “cellular phone,” “cellular-phone,” “mobile phone,” “cell phone,” “messaging,” “sms,” “texting,” “text reminder,” “short message,” “email,” “e-mail,” “iphone,” “android,” “ipad,” “fitbit,” “on-line,” “online,” “e-Health,” “eHealth,” “mhealth,” “m-health,” “internet,” “e- therapies,” “social-media,” “social media,” “facebook,” “twitter,” “whatsapp,” “information technology,” “communication technology,” “app,” “information application,” “health application,” “mobile application,” “electronic application,” “phone application,” “touch application,” “well-being application,” “informatic,” “computer,” “digital,” “web,” and “wearable.” 3.3.6 Data Extraction

3.3.6.1 Conditions

The “condition” field in ClinicalTrials.gov was defined as “the disease, disorder, syndrome, illness, or injury that is being studied” [137]. We analyzed and consolidated a total of 487 unique conditions of the 556 included registered RCTs into eight different groups, as reported in Table 2. Details of the condition classifications are provided in Appendix 3 [138]. 3.3.6.2 Discontinuation Reasons

The data exported from the ClinicalTrials.gov database includes a field “Why_Stopped” that indicates the reasons for trial discontinuation. This field is populated for trials with a withdrawn, suspended, and terminated recruitment status. We extracted and evaluated the textual content of this field as part of our recruitment analysis. Details of classification of the reasons for trial discontinuations are indicated in Appendix 4. 3.3.6.3 Major Technology

We analyzed the descriptions of the 556 included RCTs to identify the major type of technology that was utilized within the respective interventions. Details of major technology classifications of the trials are indicated in Appendix 5.

3.3.6.4 Prospective Trial Registrations

The XML data exported from the ClinicalTrials.gov database did not include an explicit field to indicate whether the trial was registered prospectively. We compared each trial’s “study_first_submitted” field to the “start_date” field in order to determine if the trial was registered prospectively or retrospectively. The “study_first_submitted” field indicates the dates when the trial’s primary investigator first submitted the trial record to ClinicalTrials.gov, whereas the “start_date” field indicates the date when the first participant was enrolled in the study [137]. We considered the registration to be prospective if the “study_first_submitted” date was before the “start_date.” 3.3.6.5 Reporting of Study Results

The data exported from the ClinicalTrials.gov database includes a field “Has Results” to indicate whether results have been submitted for the underlying study. The XML export of the trial metadata also includes the field “FirstReceived_Results_Date,” which is the date on which the study’s first results were received. These fields are maintained by the primary investigators of the respective trials and, in many cases, as explained in the “Limitations” section, this field is updated voluntarily by the primary investigator and seems to be inconsistent. Our analysis showed that only 61 (11%) of all included 556 RCTs reported results in the ClinicalTrials.Gov database. 3.3.6.6 Lead Sponsor of Trials

We defined a comprehensive and specific categorization of the funding sources of trials. We analyzed the content of the “Lead_Sponsor” field, available in trials’ XML files exported from ClinicalTrials.gov, which comprises information regarding the entity or individual that sponsors the clinical study [139]. We were able to categorize the “Lead_Sponsor” field into six different groups, with a more specific breakdown for industry sponsors (Table 1). Table 1. Analysis of RCTs by their Lead Sponsor Information. Lead Sponsor Category (N=556) Trials, n (%)

Foundations; Institutes and Research Centers 72 (12.9%)

Hospitals & Medical Center 102 (18.3%)

US Fed 25 (4.5%)

University 301 (54.1%)

Other 18 (3.2%)

Industry 38 (6.8%)

Insurance 6 (15.8%)

Pharmaceuticals 2 (5.3%)

Technology and Services 29 (76.3%)

Telecommunication 1 (3.1%) 3.3.6.7 Identification of Publication

We exported all the contents of the 556 included registered RCTs from the ClinicalTrials.gov website in XML format and then identified existing publications by two processes: automated and manual identification processes. The automated identification process considered all publications referenced in the trial's registry record as well as a PubMed search according to each trial’s National Clinical Trial registration number. The manual identification process was a multistep process aimed to search trial publications by key trial attributes and author details in two major bibliographic databases (PubMed and Medline) as well as the Google search engine. We only considered the results of a clinical trial to be “published” if at least one of the primary outcome measures was reported. Complete details of the publication-identification processes are described in Appendix 6 [62,140-142]. 3.4 Results

3.4.1 Screening Process

We exported the entire ClinlicalTrials.gov database, with 265,657 registered clinical trials entries as of February 10, 2018, into a local Structured Query Language server database. The 47 indicated search terms and phrases were then applied in the Structured Query Language server database as follows: 1. For every search term and phrase, identify matching records by the [Title] OR [Interventions] OR [Outcome Measures] fields. We identified 6717 matching trials. 2. Apply the latest completion date criteria between April 1, 2010, and April 1, 2013. We obtained 803 matching trials. 3. After screening against all inclusion and exclusion criteria, 247 registered clinical trials were excluded as per the following breakdown: • 149 trials were not randomized. • 52 trials had false-positive matching terms. For example, the registered clinical trial (NCT01287377) examined the association between nicotine patch messaging

and smoking cessation. The trial term “messaging” was a false-positive match to one of our search terms. • 17 trials were only related to computerized neuropsychological, cognition, and oxygen assessment tools. • 11 trials focused only on internet, Web-based, online, and computer-based surveys. • 9 trials were limited to the phone call intervention component. • 5 trials were related to scanners and diagnostic imaging devices. • 3 trials were related to television or online advertisement. • 1 trial was related to electronic medical record systems. 4. Finally, 556 studies were included after screening. A summary of the search results is presented in Figure 1. Figure 1. Trials included from the search results.

Total number of trials registered in ClinicalTrials.gov (n=265,657)

Trials matching our 47 search terms and phrases (n=6717)

Trials identified within a 3-year period (April 2010 to April 2013) (n=803)

Excluded trials based on inclusion/exclusion criteria (n=247)

Included randomized clinical trials (n=556)

3.4.2 Publication Rates

In summary, 406 of 556 (73%) trials were associated with identified outcome publications and 150 of 556 (27%) trials did not have any identified publications or their identified publications did not report any of their primary outcomes. Only 6 of the 556 (1.1%) published trials did not report any of the primary outcome measures indicated in the trial’s registration protocols (Figure 2). Figure 2. Results of the publication-identification process. Included randomized clinical trials (N=556)

73% (n=406) 27% (n=150) Published with reported Not published or published primary outcomes without reported primary outcomes

40.6% (n=226) 32.4% (n=180) 9.2% (n=51) Published without reported primary outcomes Automated Manual • 3.6% (n=20) Design, evaluation, secondary analysis, or test study • 3.2% (n=18) Different study design, conditions, or populations publication identification publication identification • 1.3% (n=7) Published study protocol only • 1.1% (n=6) No primary outcome reported

4.9% (n=27) 35.8% (n=199) 17.8% (n=99) Identified in Identified in PubMed via Not published ClinicalTrials.gov references NCT* number *) NCT: National Clinical Trial 3.4.3 Analysis of Trial Characteristics

We conducted a statistical descriptive analysis, describing and summarizing the characteristics of all the 556 included registered RCTs by the following standard data elements exported from and defined by the ClinicalTrials.gov database: age group, condition, country, gender, intervention model, lead sponsor, masking, recruitment status, start date, study arms, study results, trial phase, and trial size [139]. To further our analysis, we added additional data fields that were extracted from the trial descriptions: follow-ups, latest completion date, major technology, primary outcome measure, and prospective trial registration. We examined the relationship between trial characteristics and the nonpublication rate using bivariate and multivariate analyses. For bivariate analysis, we used the Pearson Chi-square statistical test, and for multivariate analyses, we used binary logistics regression in SPSS (IBM Corporation, Armonk, NY). We used the Fisher’s exact test statistic for bivariate analyses where the expected cell value was less than 5. The Fisher’s exact test was performed in STAT Statistics version 14.2 (StataCorp LLC, TX). The results of this analysis are depicted in Table 2.

Table 2. Relationship between the characteristics of RCTs and nonpublication rate.

Pearson Binary logistic regression Chi- square Test Unpublished Trials P value RCTsa/Total Characteristics P RCTsa, n (%) Odds Ratio (95% CI) [Fisher’s value b Exact Test P value] b

Overall 150/556 (27%)

Age Groups 0.52 0.22 [0.576]

Adult 27/97 (27.8%) 0.47 0.689 [0.254-1.871]

Adult|Senior 90/312 (28.8%) 0.73 0.864 [0.337-1.984]

Child 0/2 (0%) c

Child|Adult 20/79 (25.3%) 0.29 1.738 [0.627-4.821]

Child|Adult|Senior 13/66 (19.7%) Reference

Condition 0.005 0.01 [0.005]

Cancer 14/31 (45.2%) 0.10 0.414 [0.740-16.173]

Chronic Pain and 24/81 (29.6%) 0.52 0.752 [0.317-1.784] Chronic Conditions (Including Diabetes, Asthma and COPD)

Heart Diseases, 15/53 (28.3%) 0.80 1.130 [0.436-2.931] Hypertension and Stroke

Mental Health, 14/78 (17.9%) 0.31 1.585 [0.648-3.877] Neurodevelopmental Disorders, Alzheimer, Dementia and Epilepsy

Multi-Conditions 23/53 (43.4%) 0.11 0.480 [0.197-1.165]

Obesity, Weight 17/60 (28.3%) 0.11 2.455 [0.810-7.438] Management, Nutrition and Physical Activity

Smoking, Alcohol 9/57 (15.8%) 0.12 3.458 [0.740-16.173] Consumption, Substance Abuse and Addiction

Others 34/143 (23.8%) Reference

Country <0.001 <0.001 [<0.001]

Non-US 39/218 (17.9%) <0.001 3.317 [1.845-5.964]

US 111/338 (32.8%) Reference

Enrollment <0.001 0.008 [<0.001]

<=5th Percentile (up 15/29 (51.7%) 0.007 0.235 [0.083-0.668] to 26 Participants)

Between the 5th and 58/244 (23.8%) Reference 50th Percentile (between 27 and 148 Participants)

Between the 50th and 59/246 (24%) 0.10 1.592 [0.917-2.767] 95th Percentile (between 149-1962 Participants)

> 95th Percentile 8/27 (29.6%) 0.82 1.138 [0.381-3.403] (more than 1962 Participant)

Undefined 10/10 (100%) c

Follow-Ups 0.14 0.21 [0.13]

1-3 Month 34/138 (24.6%) 0.44 1.436 [0.574-3.595]

4-6 Month 32/171 (18.7%) 0.62 0.792 [0.314-1.997]

6-12 Month 45/128 (35.2%) 0.39 0.670 [0.272-1.653]

12-24 Month 12/40 (30%) 0.89 1.085 [0.330-3.570]

>24 Month 5/17 (29.4%) 0.90 0.908 [0.200-4.124]

Undefined 9/60 (15%) 0.19 2.199 [0.673-7.185]

<1 Month 13/56 (23.2%) Reference

Gender 0.98 0.64 [0.97]

Both 132/491 (26.9%) 0.88 0.877 [0.168-4.567]

Female 15/55 (27.3%) 0.76 1.318 [0.225-7.738]

Male 3/10 (30%) Reference

Intervention Model 0.09 0.18 [0.10]

Single Assignment 14/33 (42.4%) Reference

Crossover 4/21 (19%) 0.11 3.954 [0.727-21.501] Assignment

Parallel Assignment 121/464 (26.1%) 0.18 1.888 [0.744-4.490]

Factorial Assignment 11/32 (34.4%) 0.81 0.851 [0.233-3.105]

Undefined 0/6 (0%) c

Latest Completion 0.07 0.06 Date by Yeard [0.07]

Before 2012 63/269 (23.4%) 0.06 1.636 [0.987-2.714]

On or after 2012 87/287 (30.3%) Reference

Lead Sponsor 0.07 0.30 (Industry)

No 135/518 (26.1%) 0.30 1.609 [0.650-3.986]

Yes 15/38 (39.5%) Reference

Major Technology 0.67 0.58 [0.65]

Computer-Based 27/97 (27.8%) 0.99 0.995 [0.119-8.299] Intervention (Offline)

Email Notifications 7/24 (29.2%) 0.88 0.834 [0.082-8.444]

Mobile Phone 5/14 (35.7%) 0.84 0.771 [0.058-10.204] Application

Telemonitoring 16/64 (25%) 0.54 1.950 [0.226-16.842] Devices

Text Messaging 9/53 (17%) 0.61 1.799 [0.188-17.215]

Web-Based 84/294 (28.6%) 0.93 0.914 [0.114-7.336] Intervention

Wii 2/10 (20%) Reference

Masking 0.41 0.22 [0.41]

Open Label 86/319 (26.7%) 0.03 15.267 [1.230-189.459]

Single Label 53/177 (29.9%) 0.06 11.675 [0.934-146.000]

Double Label 7/30 (23.3%) 0.04 16.902 [1.163-245.555]

Triple Label 1/16 (6.3%) c

Quadruple Label 1/7 (14.3%) 0.09 20.614 [0.661-642.804]

Undefined 2/7 (28.6%) Reference

Phases 0.01 0.004 [0.01]

0/I 5/31 (16.1%) 0.08 3.112 [0.876-11.054]

I/II, II 8/56 (14.3%) 0.01 3.882 [1.460-10.318]

II/III, III or IV 17/42 (40.5%) 0.13 0.512 [0.217-1.208]

Undefined 120/427 (28.1%) Reference

Primary Outcome 0.16 0.25 Measures [0.14]

Adherence to 11/26 (42.3%) 0.69 0.761 [0.202-2.868] Treatment

Clinical Evaluation 76/316 (24%) 0.42 1.386 [0.631-3.044]

Drug, Tobacco and 10/41 (24.1%) 0.81 0.813 [0.148-4.475] Alcohol use

Physical Activity and 9/30 (30%) 0.97 1.022 [0.330-3.161] Diet Intake

Process Evaluation 13/58 (22.4%) 0.04 2.924 [1.036-8.250]

Undefined 1/3 (33.3%) 0.30 1.341 [0.782-2.297]

Vital Measurement 30/82 (36.6%) Reference

Prospective 0.006 0.29 Registration [0.009]

Retrospective 93/393 (23.7%) 0.29 1.341 [0.782-2.297]

Prospective 57/163 (35%) Reference

Recruitment <0.001 <0.001 [<0.001]

Active, not recruiting 0/1 (0%) c

Completed 105/468 (22.4%) 0.002 3.303 [1.564-6.976)]

Suspended 3/4 (75%) 0.21 0.188 [0.014-2.497]

Terminated 11/17 (64.7%) 0.21 0.403 [0.098-1.656]

Unknown Status 21/56 (37.5%) Reference

Withdrawn 10/10 (100%) c

Start Date by Year e 0.71 0.92 [0.70]

After 2008 109/413 (26.4%) 0.99 1.033 [0.555-1.924]

On or Before 2008 41/142 (28.9%) Reference

Undefined 0/1 (0%) c

Study Arms 0.11 0.40 [0.11]

One 8/18 (44.4%) 0.17 0.240 [0.032-1.820]

Two 101/410 (24.6%) 0.63 1.486 [0.296-7.459]

Three 27/75 (36%) 0.74 0.756 [0.143-3.999]

Four or More 11/38 (28.9%) 0.78 1.295 [0.219-7.646]

Undefined 3/15 (20%) Reference

Study Results 0.87 0.79 [0.88]

No 133/495 (26.9%) 0.79 1.113 [0.512-2.420]

Yes 17/61 (27.9%) Reference a) RCT: randomized controlled trial. b) P value was provided from Fisher’s exact test when the expected cell size is less than 5. c) Logistic regression model was not appropriate for this variable level value; hence this value was not included in the model. d) The median of the latest completion date year was 2012. e) The cut-off point for the year of start date was set at 2008, the year when the 7th DoH was adopted. The Pearson Chi-square test and binary logistic regression test results reported significant relationships (P<0.05) between the nonpublication rate of trials and trial characteristics including

trial condition, country, prospective registration, recruitment, trial size, and trial phases. Both tests reported no significant relationships between the nonpublication rate of trials and the age group, follow-up period, gender, intervention model, latest completion date, lead sponsor, primary outcome measures, major technology, masking, start date, study arms, and updates of trials in ClinicalTrials.gov results database. 3.4.3.1 Conditions

The Pearson Chi-square test results showed a significant association (P=0.005) between the nonpublication rate and the eight different condition groups. The highest nonpublication rate was 45.2% for RCTs focusing on the “Cancer” condition. In contrast, the lowest nonpublication rate was 15.8% for RCTs focusing on “Smoking, Alcohol Consumption, Substance Abuse and Addiction” conditions. The binary logistic regression test results showed a significant association (P=0.01) between the nonpublication rate and intervention condition groups; however, trials on cancer or addiction/smoking conditions were not a significant predictor for nonpublication (P=0.10, odds ratio [OR]=0.414, 95% CI: 0.740-16.173 and P=0.12, OR=3.458, 95% CI: 0.740- 16.173, respectively). 3.4.3.2 Country

The Pearson Chi-square test results showed significant differences (P<0.001) in the nonpublication rates between the US and other countries; the highest nonpublication rate was observed for trials in the United States (US) (32.8%) as compared to non-US trials. The binary logistic regression test results showed a significant association between the nonpublication rate between the US and non-US trials. The odds of publication for non-US trials were significant, and these trials were 3.3 times more likely to be published than the reference group of the US– based trials (P<0.001, OR=3.317, 95% CI: 1.845-5.964). The global distribution of all 556 RCTs included is depicted in Appendix 7. 3.4.3.3 Lead Sponsors

Only 38 (6.8%) of the 556 included registered RCTs were funded by industry sponsors. We observed a trend of 1.5 times higher nonpublication rate for industry-funded trials than non- industry-funded trials. However, this trend was not statistically significant (P=0.07), which may be explained by the small sample size. We also found that the percentage of industry-funded trials in the US (12%) was five times higher than that in international non-US trials (2%).

3.4.3.4 Phases

Our Pearson Chi-square test results showed significant differences (P=0.01) between the nonpublication rate of trials and their respective study phases. Of the 556 RCTs, 427 (76.8%) had no information reported on trial phases. For 129 (23.2%) of the RCTs that reported a study phase, phase II trials (including trials registered for both phase I and II) were most commonly reported (56 trials) and had the lowest nonpublication rate (14.3%). There were 42 phase III/IV trials (including trials registered for both phase II and III), with the highest nonpublication rate of 40.5%. The binary logistic regression test results showed a significant relationship (P=0.004) between the nonpublication rate and trial size, and phase II trials (including trials registered for both phase I and II) were 3.9 times more likely to be published (P=0.01, OR=3.882, 95% CI: 1.460-10.318) than other phase trials. The odds of nonpublication showed a trend towards significance for phase III/IV trials (including trials registered for both phase II and III), and these trials were 3.1 times more likely to be published (P=0.08, OR=3.112, 95% CI: 0.876-11.054); however, the trend did not reach statistical significance. 3.4.3.5 Registration of Prospective Trials

We examined the relationship between prospective trial registrations and trial nonpublication rates. Results of the Pearson Chi-square test showed a statistically significant relationship (P=0.006) between prospective trial registrations and the nonpublication rates, with higher nonpublication rates for prospectively registered trials (11.3%) than retrospectively registered trials. Our analysis also showed that only 163 (29.3%) of all our included trials were registered prospectively. We advanced our analysis to explore the impact of the 2004 ICMJE mandate and the 2008 DoH on prospective trial registrations in ClinicalTrials.gov [31,32]. Results of the Pearson Chi-square test showed a statistically significant relationship (P<0.001) between prospective trial registration and the start date of trials, with a lower number of prospective registrations reported for trials that started after 2008 (29.7%; Table 3). Table 3. Results of the Pearson Chi-square test between start date of trials and prospective trial registration. Trials Start Date Prospective trial registrations/total, n (%) Pearson Chi-square Test (P value) Before or on 2008 73/142 (51.4%) <0.001 After 2008 90/414 (21.7%)

3.4.3.6 Recruitment

Results of the Pearson Chi-square test showed a statistically significant relationship (P<0.001) between the trial recruitment status and nonpublication rate. Similarly, the binary logistic regression test showed a significant relationship (P<0.001) between the trial recruitment status and nonpublication rate, and the completed trials were 3.3 times more likely to be published (P=0.002, OR=3.303, 95% CI: 1.564-6.976). Our results also showed that discontinued trials have higher nonpublication rates than completed or active trials. We referred to trials with withdrawn, suspended, and terminated recruitment statuses as discontinued trials. We extended our analysis to explore the reasons for trial discontinuation as potential contributors to higher nonpublication rate. We examined the reasons for discontinuation of 31 trials with withdrawn, suspended, and terminated recruitment statuses among the included trials (Table 4). Table 4. Summary of reasons for discontinuation. Reason for discontinuation Trials (N=31), n (%) Recruitment Challenges 9 (29%) Funding Challenges 6 (19.4%) New Study Priorities 3 (9.7%) Primary Investigator/Staff attrition 2 (6.5%) Drop out 2 (6.5%) Technical Challenges 2 (6.5%) Primary Investigator/Staff attrition + Funding Challenges 2 (6.5%) Not Provided 5 (16.1%) Our analysis showed that recruitment and funding challenges are major factors contributing to discontinuation of trials and their nonpublication rates. Details of the classification of discontinuation reasons are provided in Appendix 4. 3.4.3.7 Reporting of Study Results

Results of the Pearson Chi-square test showed no statistically significant relationship (P=0.86) between the primary investigators who reported the results in the ClinicalTrials.gov database and the publication of trial results. 3.4.3.8 Time to Publication

We aimed to analyze the duration required to publish trial results for the 556 included trials. We measured the time to publication as the duration in years between the start date of trials and their respective publication date, which we then reported along with the number of published trials and cumulative nonpublication rates on a biyearly scale (Table 5, Figure 3).

Table 5. Analysis of trial publication cycles (duration).

Time to publication Published trials (N=556), n Cumulative nonpublication rate (start date to publication date), years (%) (N=556), %

2 108 (19.4%) 80.8

4 148 (26.6%) 54

6 100 (18%) 36

8 37 (6.7%) 29.3

10 9 (1.6%) 27.7

<15 3 (1%) 27.2 Figure 3. Time to publication of registered clinical trials in digital health.

100% 90% 80% 70% 60% 50% 40% 30%

20% Percentage of Trials (N=556) Trials of Percentage 10% 0% 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Years between Trials Publication and Trials Start Date

Published Trials Comulative Non-Publication Rate

The majority of our 556 included trials were published within 6 and 8 years of the trial’s start date (356 [64%] and 393 [70.7%], respectively). A total of 148 (26.6%) trials were published in the fourth year of the trial. We also observed that half of our included trials were published between the fourth and fifth year after the trial start date. 3.4.3.9 Trial Size

No enrollment values were identified for ten trials in the ClinicalTrials.gov database, and we could not identify any publications for these trials. We stratified all trials into four strata by size at the 5th, 50th, and 95th percentiles and found a statistically significant difference between the nonpublication rate of trials and trial size. The highest nonpublication rate was 51.7% for small trials that enrolled no more than 26 participants (at the 5th percentile), whereas the lowest nonpublication rate was 23.8% for trials that enrolled between 27 and 148 participants (between 32

the 5th and 50th percentile). The Pearson Chi-square test showed a statistically significant relationship between the nonpublication rate and trial size (P<0.001). In addition, we found that half of the 546 randomized controlled trials that provided details of the trial size enrolled ≥148 participants (actual or intended). The cumulative enrolment in the 546 trials was 312,906 participants, split between 236,066 (75.44%) participants in published trials and 76,840 (24.56%) in unpublished trials. We found that the nonpublication rate was twice as high as that for trials below the 5th trial size percentile (≤26 participants) compared to other trials above the 5th trial size percentile (>26 participants). 3.5 Discussion

The research literature has identified the impact and risks of publication bias for researchers, clinicians, healthcare professionals, and health policy decision makers as well as a number of factors contributing to nonpublication and discontinuation of clinical trials [24,122,16-19]. Recruitment challenges were the most-frequently reported factor contributing to clinical trial discontinuation [9], and clinical trials with larger numbers of participants or statistically significant positive outcomes were more likely to be published [6,10,14, 20]. Funding sources, study language (in particular non-English language) and study design (single-center versus multicenter studies) were also identified as contributing factors for potential bias [20,24,]. Authors and primary investigators reported a lack of time as the key factor for not publishing their results in a peer-reviewed journal along with other factors such as the lack of relevance and importance of their results and disagreement with coauthors [14,15]. In the domain of digital health, we analyzed the nonpublication rate among 556 RCTs that were registered in ClinicalTrials.gov, with the latest completion date between April 2010 and April 2013. We found that 27% of all included trials remain unpublished 5 years after the latest completion date. Our finding is in line with a similar study of large RCTs, with at least 500 enrolled participants, that reported a 29% nonpublication rate [6]. However, our reported nonpublication rate (27%) was considerably less than that reported in a few other similar studies with nearly half of the trials remaining unpublished [8,10,11]. We postulate that this difference may be explained by two major factors. First, the fast-paced technology involved in digital health trials could provide more extrinsic motivation for primary investigators to share and publish their results in order to become leaders in the field and stay ahead of the digital innovation curve. Second, digital health trials are likely to be sponsored by academic entities, such as universities,

hospitals, and medical and research centers, that are more disciplined and obliged by scholarly ethics to publish their results. Industry sponsors and digital technology developers, on the other hand, are likely to be more driven by the scale and opportunity in the broader digital health marketplace, beyond the realm for academia and the complexity of randomized trials design. As part of our publication-identification process, we compared the published outcomes and primary outcomes of trials indicated in the trial registration entries in ClinicalTrials.gov. Only 6 of the 556 (1.1%) published trials did not report any of the primary outcome measures indicated in the trial registration protocols. Our finding is substantially different and should not be compared to findings from other studies that reported that 40%–62% of clinical trials had at least one change in primary outcome when comparing trial publications and protocols [48,113,115]. The difference lies in our focus on identifying trial publications with at least one reported primary outcome from the trial protocol without measuring whether all, or a subset, of the primary outcomes outlined in the trial protocol were reported or examining if secondary outcomes were reported. We reported a statistically significant relationship between the nonpublication rate and eight different condition groups in the Pearson Chi-square test (P=0.005) and the binary logistic regression test (P=0.01). The highest nonpublication rate was 45.2% for RCTs focusing on the “Cancer” condition. This relative underreporting suggests challenges in conducting digital health oncology trials. These challenges align with and may be explained by findings from other studies that reported several barriers to traditional oncology trials, such as recruitment, eligibility, follow-up, and oncologist and patient attitudes [143-145]. However, we suspect that there are explicit barriers to digital health oncology trials, in particular, at the pre-enrollment and recruitment stages of the trial. Oncologists may be more inclined to enroll their patients in other traditional, nondigital health, oncology trials, where experimental drug treatment could have more tangible outcomes for their patients. Patients’ perceptions and priorities to enroll in a trial could also be influenced by the preferences of their treating oncologists. In our study, only two trials were funded by the pharmaceutical industry: This clearly small number of pharmaceutical industry-funded trials supports our postulate of explicit pre-enrollment barriers to digital health oncology trials. We also found that half of our included trials enrolled ≥148 participants, which is similar to other findings from two different studies: 46% of trials included ≥160 participants, and 45% of trials included ≥100 participants [7,146]. On comparing trial enrollment between US–based and

international randomized controlled trials, we found that US–based trials had a cumulative enrolment of 228,479 participants as compared to 48,427 participants in international trials. This finding indicates that digital health trials within the US enroll 4.7 times more participants than international trials; this value is higher than that in all clinical trials reported in a different study, which showed that US–based trials enroll only two-thirds of the number of participants enrolled in international trials [143]. The nonpublication rate was twice as high for trials with a trial size below the 5th percentile(≤26 participants) as compared to trials with a trial size above the 5th trial size percentile (>26 participants), which is consistent with the findings of similar studies reporting that clinical trials with a larger number of participants are more likely to be published [6,10]. RCTs are usually conducted in a series of phases, 0 to IV, to examine the intervention efficacy, safety, and adverse events over various periods and sizes of population samples [137,147-150]. However, clinical studies focusing on medical devices or behavioral interventions might not be conducted in phases and did not report information in the phase field in the ClinicalTrials.gov database [139]. The finding of our study confirms this notion, as 427 (76.8%) of the 556 included RCTs reported no information on the trial phases in the ClinicalTrials.gov database. Our results showed that phase III/IV trials have the highest nonpublication rate (40.5%) among all other phase trials and are terminated and withdrawn four times more often than other phase trials. The fact that phase III/IV trials include a large group of participants may justify the higher nonpublication, termination, and withdrawal rates when considering recruitment and attrition challenges. In our study, we reported a statistically significant relationship between the trial recruitment status and trial nonpublication rate, and completed trials were 3.3 times more likely to be published (P=0.002, OR=3.303, 95% CI: 1.564-6.976). Our analysis of 31 discontinued trials (trials with withdrawn, suspended, and terminated recruitment statuses) showed that enrollment and funding challenges were major contributors to the higher nonpublication rate among our included trials. This finding is in line with that of another study indicating that recruitment challenges were the most-frequently reported factor contributing to discontinuation of clinical trials [9]. Another less-frequently reported reason for discontinuation of trials is new study priorities—when the primary investigator shifts his or her priority to a new trial. The fact that a primary investigator discontinues an existing registered trial to start another new, and perhaps, similar trial questions his or her commitment to the ethics of trial registration. It is important to

understand the motivation behind the discontinuation of the existing trial and the interest in starting a new trial. Primary investigators should explain if the shift in priorities to a new trial was driven by implementation challenges of the existing trial (such as insignificant outcomes and adverse events) or the research perspective of the new trial (such as a new funding or collaboration opportunity). We analyzed the nonpublication rate with regard to the start date year of trials, stratified according to their start before or after 2008, when the 7th revision of the DoH was adopted [32]. We found that the nonpublication rate for trials started in or before 2008 was 3% higher than that for trials started after 2008, although the difference was not statistically significant. We postulate that the nonpublication rate may be higher for trials registered prospectively, as the primary investigator would register a trial before the enrollment of any participant, without knowing if the trial would be completed successfully or the results would ultimately be published. The Pearson Chi-square test showed a statistically significant relationship (P=0.006) between prospectively registered trials and nonpublication rates, with a higher nonpublication rate for prospectively registered trials (11.3%). We also expected to see an incremental trend in the prospective registration of trials after 2008, when the 7th revision of the DoH was adopted to raise awareness of prospective trial registration within the scholar community [32]. Contrary to our expectation, the Pearson Chi-square test showed a statistically significant relationship (P<0.001) between the prospective trial registration and the trial start date, with a lower number of prospective registrations for trials starting after 2008 (29.6%). This significant decline in prospective registration, compared to the influx in retrospective registration, may be explained by the general emphasis on trial registration after 2008. It is possible that the primary investigators of unregistered trials were increasingly required to register their trials retrospectively prior to publication by the editors or the submission guidelines of the scholarly journals. However, there are two major limitations to this finding in our study: the majority (74.3%) of our included trials started after 2008, and the study scope was limited to digital health trials. These two limitations can impact the internal and external validity of our analysis to evaluate the general impact of adoption of the 7th revision of the DoH on the nonpublication rate of trials and prospective trial registrations. Most of our included trials were published within 6 to 8 years after the trial start date (356 [64%] and 393 [70.7%], respectively). We also observed that half of our included trials were published between the fourth and fifth year of the trial start date. The timelines of our findings are

comparable to those of a 2007 study that analyzed time to publication of clinical trials (also measured from the start to publication date) and reported that clinical trials with statistically significant positive results were published 4-5 years after their start date, whereas trials with negative results were published in 6-8 years [151]. When we analyzed the funding sources of trials, we found that only a small number of trials (38 [6.8%] of our included trials) were funded by the industry. This finding is in contrast with the results of other studies, in which most included trials were funded by the industry. A study of delayed and nonpublication of RCTs on vaccines reported that 85% of their included trials were funded by the industry [8]. Another cross-sectional study of nonpublication of large RCTs found that 80% of the included trials were funded by the industry [6], whereas an observational study of discontinuation and nonpublication of surgical randomized controlled trials reported that 42% of the included trials were funded by the industry [47]. In our study, a majority (76.3%) of the 38 industry-sponsored trials were funded by a technology and service industry sponsor, and only two trials were funded by a pharmaceutical industry sponsor. We observed a trend of 1.5 times higher nonpublication rates among industry-funded trials than among non-industry-funded trials. However, the trend was not statistically significant, which may be explained by the small sample size. We also found that the ratio of industry-funded trials in the US is five times higher than that of international trials. Although these findings may be interpreted by the predominantly privately funded healthcare system in the US, they could also be attributed to the scale of the digital health industry in the US compared to the rest of the world, with US–based digital health startups holding 75% of the global market shares between 2013 and 2017 [152-154]. 3.6 Limitations

Despite ICMJE–mandated trial registration since 2005, not all randomized trials are registered [66]. Therefore, in practice, the proportion of unreported trials, trials that failed, and publications that did not report the primary outcomes may be different. In this study, the ClinicalTrials.gov database was the sole data source of trial registrations. The choice was driven by feasibility challenges with limited research resources available for this study initiative and broader and global adoption of the ClinicalTrials.gov registry within the biomedical research enterprise. There are many other trials registries such as the European Clinical Trials Registry [155] and the International Standard Registered Clinical/Social Study

Number (ISRCTN) registry [156]. The exclusion of all trial registries other than ClinicalTrials.gov in our analysis may have impacted the external validity (generalizability) of our findings. Our publication-identification process was conducted between June 29, 2016, and February 10, 2018, for all included 556 RCTs. Therefore, our findings did not include studies published after February 10, 2018. This study includes trials based on their completion date and primary completion date declared in the registry record in ClinicalTrials.gov. When not provided, we considered the latest completion date as described in Appendix 1. These criteria assume that the primary investigators and study sponsors provided and updated trial details in the ClinicalTrials.gov database. However, this is a manual and voluntarily process that may not be fully complied with, given the competing priorities and limited resources available for the primary investigators and study sponsors. These limitations may impact the generalizability of our study results. 3.7 Conclusion

From our study of 556 RCTs in the field of digital health that are registered in the ClinicalTrials.gov database, we found that nonpublication of trials is prevalent, with almost a third (150, 27%) of all included trials remaining unpublished 5 years after their completion date. There are distinct differences in nonpublication rates between US- and non-US–based trials and according to the funding sources (industry sponsors vs non-industry sponsors). Further research is required to define the rationale behind the nonpublication rates from the perspectives of primary investigators and, more importantly, to articulate the impact and risk of publication bias in the field of digital health clinical trials. Future studies could also include nonrandomized trials such as projects published in protocols (such as JMIR Research Protocols). It is not clear whether the research or technology failed, or if the results were disappointing and scholars did not write up a report, or if reports were rejected by journals; however, given the multitude of potential publication venues, and increased transparency in publishing, the former seems more likely. Scholarly communication is evolving, and short reports of failed trials may not always be published in peer-reviewed journals, but may be found in preprint servers. With the growing popularity of preprints, future analyses may also include searches for draft reports on preprint servers (such as preprints.jmir.org) to include unpublished reports, which may further shed light on why trials failed or remained unpublished. In the meantime, a general

recommendation would be to conduct thorough formative research and pilot studies before conducting a full randomized controlled trial to reduce the risk of failure such as having insufficient power due to nonuse attrition [112]. 3.8 Conflicts of Interest

GE is the editor-in-chief of the JMIR (and publisher at JMIR Publications) but was not involved in the peer-review or decision-making process for this paper. The associate editor handling this manuscript and the reviewers were blinded and not aware of the co-authorship of GE. As owner of JMIR Publications, GE may benefit from increased publication rates of digital health trials. The other authors declare no conflicts of interests.

Chapter 4 Registration and Publication Bias in Digital Health Trials Registered in Global Trial Registries

Submitted to the npj Digital Medicine journal on August 9th, 2019 (Manuscript ID: NPJDIGITALMED-00521) as: Al-Durra M, Nolan RP, Seto E, Cafazzo JA. Prospective Trial Registration and Publication Rates of Randomized Clinical Trials in Digital Health: A Cross Sectional Analysis of Global Trial Registries.

Summary: The reported prevalence of publication bias and low compliance in prospective trial registration in the previous study (Chapter 3) suggested a broader scope of analysis beyond a single trial registry. Expanding the analysis to include RCTs in other registries will improve the generalizability of this study and help identify any differences in prospective registration of digital health RCTs across trial registries other than ClinicalTrials.gov. This study included 417 digital health RCTs that enrolled participants in 2012 and were registered in any of the seventeen WHO registries. It addresses the second research question of this thesis “What is the global prevalence of publication bias, and compliance in prospective trial registration, in digital health RCTs registered in any of the seventeen WHO registries?” The prospective registration and publication rates were at (38.4%) and (65.5%) respectively. A new form of bias was detected in this study, referred to as “Registration Bias” or “Selective Registration Bias”, with the vast majority 95.7% of digital health RCTs that were published within a year after trial registration, were registered retrospectively. There was a statistically significant relationship between prospective registration and funding sources, with industry funded RCTs having the lowest compliance with prospective registration at (14.3%). The manuscript of this study was submitted to the npj Digital Medicine journal on August 9th, 2019 (Manuscript ID: NPJDIGITALMED-00521). This chapter of this thesis includes the complete and submitted manuscript of this study with no modifications, other than adjusting the font styles and re-organizing the citations and references to align with the overall references of this thesis.

Registration and Publication Bias in Digital Health Trials Registered in Global Trial Registries 4.1 Abstract Registration of clinical trials was introduced to mitigate the risk of publication and selective reporting bias in the realm of clinical research. The prevalence of publication and selective reporting bias in trial results has been evidenced through scientific research. This bias may compromise the ethical and methodological conduct in the design, implementation and dissemination of evidence-based healthcare interventions. Principal investigators of digital health trials may be overwhelmed with challenges that are unique to digital health research, such as the usability of the intervention under test, participant recruitment, and retention challenges that may contribute to non-publication rate and prospective trial registration. Our primary research objective was to examine the prevalence of prospective registration and publication rates in digital health trials. We included 417 trials that enrolled participants in 2012 and were registered in any of the seventeen WHO registries. The prospective registration and publication rates were at (38.4%) and (65.5%) respectively. We identified a statistically significant (P<0.001) “Selective Registration Bias” with 95.7% of trials published within a year after registration, were registered retrospectively. We reported a statistically significant relationship (P=0.003) between prospective registration and funding sources, with industry-funded trials having the lowest compliance with prospective registration at (14.3%). The lowest non-publication rates were in the Middle East (26.7%) and Europe (28%), and the highest were in Asia (56.5%) and the U.S. (42.5%). We found statistically significant differences (P<0.001) between trial location and funding sources with the highest percentage of industry funded trials in Asia (17.3%) and the U.S. (3.3%).

4.2 Introduction In the realm of scientific research, study results should be made available and accessible to the broader research community to assess the evidence around the efficacy and potential harm of healthcare interventions [1]. Emanuel et al.’s framework defines seven ethical guidelines for clinical research involving human subjects [5]. The first of which describes the value derived from disseminating research results as “Only if society will gain knowledge, which requires sharing results, whether positive or negative, can exposing human subjects to risk in clinical research be justified”.

A number of studies have examined the ramification of publication bias and debated a number of contributing factors to non-publication of clinical trial results such as recruitment challenges, funding sources and study design [6,10,16-20,122] Other challenges pertaining to investigators were also identified as contributing factors to non-publication of clinical trial results, such as lack of time or disagreement with coauthors [14,15]. The notion of clinical trial registration was conceived in 1986, when Simes suggested that the act of registration would help control the risk of publication bias by providing a new data source for trial information and results[11-13]. The trial registries would also help mitigate selective reporting of positive outcome by comparing outcomes reported in trial publication versus outcome measures indicated in the trial registration [48,113-117]. In 2004, the ICMJE introduced a new mandate to promote prompt registration of all clinical trials [31]. In 2005, the WHO started an initiative to standardize trial registrations and trial registry datasets across multiple national and international registries [33]. In October 2008, the 7th revision of the DoH was adopted with new requirements focusing on the importance of prospective registration of clinical trials and reporting of their results [28]. The prospective registration of clinical trial means that investigators should register their trials prior to the enrollment date of the first trial participant, otherwise, the registration would be considered retrospective. In 2015, the WHO announced a new statement on public disclosure of clinical trial results with more guidelines on trial registration, publication of results, and the inclusion of the TRN in respective publications to enable linking of trial reports with clinical trial registry information [32]. As of September 2nd, 2018, the WHO ICTRP includes seventeen different national and international trial registries with a unified search and access to registration information of 441,033 unique clinical trials [74]. Two studies reported publication rates between 66% and 68% for clinical trials registered in the US-based clinical trials registry, ClinicalTrials.gov [7,47]. Another study reported publication rates at 73% for clinical trials registered in at least one of several clinical trial registries (ClinicalTrials.gov, Current Controlled Trials, WHO ICTRP, Clinical Study Register, and Indian, Australian-New Zealand, and Chinese Clinical Trial Registries) [8]. Three other studies investigated prospective trial registration and the quality of registration in the International Standard Randomized Controlled Trial Number (ISRCTN) registry and the WHO ICTRP registries [158-160]. The results of these studies reported that prospective trial registration was between 37.8% and 53.4%.

However, investigators could be overwhelmed with challenges that may be particularly problematic in digital health trials, such as the usability of the intervention under test, participant recruitment, and retention challenges that may contribute to non-publication rate and prospective trial registration [21,35-40] To our knowledge, this is the first review to analyze the nonpublication rate and prospective registration of digital health clinical trials. We sought to examine, at the global level, the non-publication and prospective trial registration rates in digital health trials across the seventeen WHO ICTRP registries. 4.3 Research Objectives The primary research objective was to examine the prospective trial registration and publication rates of digital health RCTs registered in any registry that is part of the WHO ICTRP registries. The secondary research objectives were (1) to investigate the compliance with recruitment and inclusion of the TRN in the published trials, (2) to explore the relationship between publication rates, the time to publication and trial size, and (3) to analyze the relationship between retrospective trial registration and the duration from trial registration to the publication of trial results. 4.4 Methods 4.4.1 Data Source

The WHO ICTRP is a free online portal that provides a unified access to trial registration information across different clinical trial registries [74]. As of September 2nd, 2018, the ICTRP database included 441,033 unique trial protocols from 17 different clinical trials registries. We utilized the advanced search feature in the ICTRP search portal to apply our search terms to the [Title] or [Intervention] field and downloaded the matching trials in XML format for further analysis. 4.4.2 Statistical Analysis

We used the Pearson Chi-square statistic for bivariate analyses. These statistical analysis was performed in SPSS Statistics version 24 (IBM Corporation, Armonk, NY). We used the Fisher’s exact test statistic for bivariate analyses where the expected cell value was less than 5. The Fisher’s exact test was performed in STAT Statistics version 14.2 (StataCorp LLC, TX).

4.4.3 Inclusion Criteria

We included all eHealth, mHealth, telehealth, and digital health related RCTs that are registered in any of the seventeen trial registries within the ICTRP database and include any information and communication technology component, such as: • Online web and mobile application • Internet, websites and personal computer application • Digital games and social media application • Telehealth and telemedicine components We included trials irrespective of their recruitment status or trial phases. We limited our inclusion criteria to trials that started in 2012. We considered the enrollment date in the trial registries to indicate the start date of the included trials. 4.4.4 Enrollment Date Justification

We aimed to allow for longer publication cycles to account for late publications of included trials. We reviewed existing studies that reported that investigators of clinical trials may take up to three years to complete their trials and another two to three years before they would publish their results [7,146,161,162]. A 2007 study indicated that clinical trials with positive results were published four to five years after their start date whereas trials with negative results would take six to eight years to publish their results [151]. We informed our design by this evidence and chose to include trials that started their enrollment in 2012, i.e. six years prior to conducting our research in September, 2018. We limited the scope of our analysis to one year only, i.e. 2012, to keep our study manageable and feasible. 4.4.5 Exclusion Criteria

We excluded registered clinical trials that were not randomized or did not include any digital components in their intervention. Trials that merely utilized short messages service (SMS), phone-calls, emails or video communication without any other interactive components were also excluded. We excluded trials that only reported on computerized or online surveys and questionnaires. 4.4.6 Search Terms

We developed a comprehensive list of search terms and phrases through an iterative process as explained in Appendix 8. Our final set of 86 search terms included:

“smartphone,” “smart-phone,” “cellphone,” “cell-phone,” “cellular phone,” “cellular-phone,” “cell phone,” “mobile phone,” “health application,” “mobile application,” “phone application,” “touch application,” “health app,” “mobile app,” “phone app,” “touch app,” “multimedia,” “multi-media,” “multi media,” “text reminder,” “short message,” “text message,” “messaging,” “texting,” “sms,” “email,” “e-mail,” “electronic mail,” “iphone,” “android,” “ipad,” “online,” “on-line,” “e-Health,” “eHealth,” “mhealth,” “m-health,” “internet,” “etherapy,” “e-therapy,” “e-therapies,” “information technology,” “communication technology,” “information application,” “electronic application,” “well-being application,” “informatic,” “computer,” “digital,” “Web,” “telehealth,” “tele-health,” “tele-monitoring,” “telemonitoring,” “telemedicine,” “telemedicine,” “tele-rehabilitation,” “telerehabilitation,” “tele-consult,” “video consult,” “video-consult,” “video conferenc,” “video-conferenc,” “skype,” “social media,” “social-media,” “social network,” “social-network,” “social app,” “facebook,” “twitter,” “tweet,” “whatsapp,” “wearable,” “fitbit,” “wii,” “nintendo,” “kinect,” “xbox,” “playstation,” “game,” “gamif,” “gaming,” “hololens,” “virtual reality,” “augmented reality,” “google glass.”

4.4.7 Data Extraction

We downloaded the XML files for the 417 matching clinical trials in the ICTRP search portal. We transformed these files into a tabular format and imported them into a local SQL Server Database for further data preparation and analysis.

4.4.7.1 Conditions The content of the condition field in the ICTRP dataset is a free-text description of the trial conditions. We were able to consolidate a total of 375 unique condition descriptions of the 417 included trials to 5 distinct condition groups as described in Appendix 9.

4.4.7.2 Prospective Trial Registration The data export from the ICTRP dataset did include a field to indicate whether a trial was registered retrospectively [163,164]. However, our analysis showed that this field did not include correct information for a substantial number of trials within our sample of 417 included trials. We therefore chose to evaluate prospective trial registration based on the actual difference between the registration and enrollment dates as described in Appendix 10.

4.4.7.3 Primary Sponsors We analyzed the primary sponsor field in the data extract from the ICTRP dataset [163]. Within our 417 included trials, we categorized the primary sponsors as 205 “University”, 137 as “Institute/Foundation/Research Center”, and grouped all the remaining 75 sponsors under “Others” as they had minimal representation within our included dataset. The remaining 75 sponsors, that we categorized as “Others”, included 14 industry, 18 government, and 44 hospital or clinic sponsors.

4.4.7.4 Major Technology We evaluated the digital components utilized within the included trials and provided a classification for major technology used within the respective interventions. Details of our classification approach are provided in Appendix 11.

4.4.8 Identification of Randomized Trials

We were able to identify randomized trials through a text match search in the following three fields in the data export from the ICTRP dataset: “Study_design,” “Public_title,” and “Scientific_title”. We searched for matching words that included the term “Random” and did not include the term “Non-Random”.

4.4.9 Identification of Publication

We identified existing publications through an automated and a manual publication identification process. The automated identification process included a PubMed search by every trial registration ID as well as a review of listed publication references and citations for trials registered in the ClinicalTrials.gov registry. The manual process included a pragmatic search in PubMed and Google based on a combination of search terms concatenated information from trial titles, investigators, location/city and institution. We only considered trial publications that reported at least one of the primary outcome measures of the underlying trials. Complete details of the publication identification processes are described in Appendix 12. 4.5 Results A full search of the WHO ICTRP database returned 441,033 unique clinical trials (507,455 in total including 66,442 duplicates) as of September 2nd, 2018. We utilized the advanced search functionality on the ICTRP search portal to apply our 86 search terms and phrases including all trial phases and recruitment status. There were 22,859 unique trials that matched our search 46

terms, within which, 15,096 trials were randomized, and 1,018 trials were enrolled in 2012. After screening against our inclusion and exclusion criteria, 417 trials were included, and 601 trials were excluded as shown in Figure 4:

Figure 4. Included trials from the search results.

All Trials available in ICTRP Database Duplicates Identified and Removed (n=507,455) (n=66,422)

Unique Trials available in ICTRP Database Excluded Based on Exclusion Criteria (n=441,033 ) (n=601) ------487 - RCTs not related to digital health trials

Trials Matched our 86 Search Terms – RCTs limited to short messages service (SMS) (n=22,859) intervention components

– RCTs limited to phone-calls intervention components

Randomized Trials (RCTs) – RCTs limited to email intervention components (n=15,096) – RCTs limited to video intervention components

– RCTs focused only on computerized or RCTs with Enrollment Started in 2012 online questionnaires and/or surveys (n=1,018)

Included RCTs (n=417)

4.5.1 Prospective Trial Registration and Publication Rates In summary, 273/417 (65.5%) trials were associated with identified outcome publications and 144/556 (34.5%) trials did not have any identified publications as shown in Figure 5:

Figure 5. Identification of trials publication.

Included Randomized Clinical Trials (N=417)

65.5% (n=273) 34.5% (n=144) Published Not Published

36% (n=150) 29.5% (n=123) Automated Manual Publications Identification Publications Identification

34.1% (n=142) 13.7% (n=57) Identified in PubMed Manually Identified in PubMed Trial Registration ID

1.9% (n=8) 15.8% (n=66) Identified in Manually Identified in ClinicalTrials.Gov References Google Search

We examined the relationship between trial characteristics and the non-publication rate and prospective registration of all included trials as shown in Table 6:

Table 6. Relationship between trial characteristics and prospective registration and nonpublication rates of included trials.

Pearson Pearson Chi- Chi- square square Number of Number of Test Test Non- Prospective P value P value Trial Characteristics Published Trials/ [Fisher’s Trials/ [Fisher’ n Total (%) Exact n Total (%) s Exact Test Test P value] P c value] c

Overall 160/417 (38.4%) 144/417 (34.5%)

0.31 0.40 Age Groups [0.26] [0.36]

Adult 47/128 (36.7%) 41/128 (32%)

Adult|Senior 69/174 (39.7%) 58/174 (33.3%)

Child 29/67 (43.3%) 28/67 (41.8%)

Child|Adult 5/14 (35.7%) 2/14 (14.3%)

Child|Adult|Senior 8/23 (34.8%) 10/23 (43.5%)

Senior 0/8 (0%) 4/8 (50%)

Undefined 2/3 (66.7%) 1/3 (33.3%)

Condition 0.02 0.042

Cancer 9/19 (47.4%) 12/19 (63.2%)

Chronic Conditions 20/46 (43.5%) 19/46 (41.3%)

Mental Health 67/149 (45%) 45/149 (30.2%)

Obesity 31/78 (39.7%) 29/78 (37.2%)

Others 33/125 (26.4%) 39/125 (31.2%)

<0.001 0.01 Continent a [<0.001] [0.01]

Africa 0/2 (0%) 2/2 (100%)

Americas (Non-US) 9/34 (26.5%) 11/34 (32.4%)

Asia 7/23 (30.4%) 13/23 (56.5%)

Australia & New Zealand 34/38 (89.5%) 12/38 (31.6%)

Europe 62/182 (34.1%) 51/182 (28%)

Global (Multi) 2/3 (66.7%) 0/3 (0%)

Middle East 2/15 (13.3%) 4/15 (26.7%)

US 44/120 (36.7%) 51/120 (42.5%)

Trial Size b 0.40 0.32

<=129 76/209 (36.4%) 77/209 (36.8%)

>129 84/208 (40.4%) 67/208 (32.2%)

Enrollment Year 0.13 0.88

First Half 2012 70/202 (34.7%) 69/202 (34.2%)

Second Half 2010 90/215 (41.9%) 75/215 (34.9%)

0.003 0.17 Funding Source [0.003] [0.16]

Government/ Authority 12/18 (66.7%) 9/18 (50%)

Hospitals and Clinics 20/44 (45.5%) 20/44 (45.5%)

Industry 2/14 (14.3%) 6/14 (42.9%)

Institute/Foundations/Research Center 61/137 (44.5%) 40/137 (29.2%)

University 65/204 (31.9%) 69/204 (33.8%)

0.07 0.56 Gender [0.06] [0.58]

Both 127/330 (38.5%) 119/330 (36.1%)

Female 10/41 (24.4%) 12/41 (29.3%)

Male 7/11 (63.6%) 4/11 (36.4%)

Undefined 16/35 (45.7%) 9/35 (25.7%)

0.06 0.78 Major Technology [0.06] [0.81]

Digital Games 13/36 (36.1%) 14/36 (38.9%)

Internet/Web 103/233 (44.2%) 77/233 (33%)

Mobile Apps 13/32 (40.6%) 11/32 (34.4%)

Offline 17/55 (30.9%) 22/55 (40%)

Social Media 1/7 (14.3%) 1/7 (14.3%)

Telehealth 11/39 (28.2%) 15/39 (38.5%)

Virtual Reality 2/15 (13.3%) 4/15 (26.7%)

0.44 0.86 Phase [0.45] [0.80]

Phase 0 3/5 (60%) 2/5 (40%)

Phase I 5/12 (41.7%) 4/12 (33.3%)

Phase II 4/12 (33.3%) 4/12 (33.3%)

Phase II/III 2/2 (100%) 1/2 (50%)

Phase III 8/18 (44.4%) 8/18 (44.4%)

Phase IV 2/3 (66.7%) 2/3(67%)

Undefined 136/365 (37.3%) 123/365 (33.7%)

Registration Year n/a 0.88

Before 2012 n/a 14/42 (33.3%)

In 2012 (Prospective) n/a 41/118 (34.7%)

In 2012 (Retrospective) n/a 46/124 (37.1%)

After 2012 n/a 43/133 (32.3%)

<0.001 0.13 Trials Registry [<0.001] [0.16]

Australian New Zealand 35/39 (89.7%) 12/39 (30.8%) Clinical Trials Registry (ANZCTR)

Brazilian Clinical Trials 0/6 (0%) 2/6 (33.3%) Registry (ReBEC)

Clinical Trials Registry - India 2/8 (25%) 5/8 (62.5%) (CTRI)

ClinicalTrials.Gov (US) 80/234 (34.2%) 85/234 (36.3%)

EU Clinical Trials Register 2/2 (100%) 2/2 (100%)

International Standard 15/65 (23.1%) 19/65 (29.2%) Randomized Controlled Trial Number (ISRCTN)

Internet Portal of the German 4/9 (44.4%) 2/9 (22.2%) Clinical Trials Register (DRKS)

Iranian Registry of Clinical 1/12 (8.3%) 3/12 (25%) Trials

Japan Primary Registries 5/8 (62.5%) 4/8 (50%) Network

Pan African Clinical Trial 0/2 (0%) 2/2 (100%) Registry (PACTR)

The Netherlands Trial Register 16/32 (50%) 8/32 (25%)

0.58 0.02 Number of Study Arms [0.59] [0.02]

Single 4/8 (50%) 6/8 (75%)

Two or More 14/43 (32.6%) 10/43 (23.3%)

Undefined 142/366 (38.8%) 128/366 (35%) a) A complete list of trial locations is provided in Appendix 14. b) The median target size of trial enrollment was 129. c) P value was provided from Fisher’s exact test when the expected cell size is less than 5. The Pearson Chi-square test results found significant relationships (P<0.05) between trial nonpublication rate and trial characteristics including trial conditions, locations, and number of study arms. There were no significant relationships between trial non-publication rate and trial age group, enrollment date, funding source, gender, major technology, registration date, trial phase, trial registry, and whether the trials were registered prospectively. Only 160(38.4%) of the included trials were registered prospectively. There were significant relationships (P<0.05) between prospective trial registration and trial characteristics, including trial conditions, funding source, location, and trial registry. Our results showed that 144(34.5%) of the included trials remain unpublished. There were significant relationships (P<0.05) between the non-publication rates and trial characteristics, including trial’s condition, location, and study arms. The highest non-publication rates were reported in Asia and the US at 56.5% and 42.5% respectively. We did not consider the 100% non-publication rate of digital health trials in Africa due to the limited number of two trials included in our study. The highest percentage of industry funded trials was reported in Asia and the US at 17.3% and 3.3% respectively. To interpret these results, we examined the relationship, and found statistically significant differences (P<0.001), between the trial location and funding sources as indicated in Appendix 13.

4.5.2 Enrollment-To-Publication Duration and Trial Size

We postulated that smaller trials would be easier to conduct as they have less recruitment and enrollment challenges, hence they would likely be completed and published in a shorter time compared to larger trials. To validate our postulate, we analyzed the relationship, and found statistically significant differences (P=0.002), between the trial size and the enrollment-to- publication duration. The enrollment-to-publication time was measured as the duration between the enrollment date and the publication date of the included and published trials, details provided in Appendix 15. We also explored the trend between enrollment-to-publication duration, trial size, number of published trials and the cumulative percentage of non-publication rate of all published trials. Details can be found in Appendix 16. The results are depicted in Figure 6 revealing an incremental trend of published trial until a critical time point at the 4th year, where fewer trials were published into the 5th and 6th year after enrollment. Figure 6. Relationship between Trial Enrollment-To-Publication Duration and Trial Size, Number of Published Trials and Non-Publication Rate.

120 100 80 60 40

Number of Trials of Number 20 0 1 2 3 4 5 6 Years between Trials Publication and Enrollment Date

Numbr of Published Trials Cumulative Percentage (%) of Non-Publication Rates

Numbr of Published Large Trials Numbr of Published Small Trials

4.5.3 Registration-To-Publication Duration and Retrospective Registration

We analyzed the registration-to-publication time as the duration between the registration date and the publication date of the included and published trials. We examined the prevalence of retrospective trial registration and its relationship to the registration-to-publication duration. We found a statistically significant relationship (P<0.001) between retrospective trial registration and the registration-to-publication duration. The vast majority 95.7% of digital health clinical trials that were published within a year after trial registration, were registered retrospectively as shown in Table 7.

Table 7. Relationship between trial registration-to-publication duration and retrospective trial registration.

Pearson Chi-square Test Registration to Number of Retrospectively Registered Trials/ P value Publication Number of Published Trials (%) [Fisher’s Exact Test (Years) P value] a

1 or less year 44/46 (95.7%)

2 30/48 (62.5%)

3 34/52 (65.4%) <0.001 4 30/57 (52.6%) [<0.001] 5 21/45 (46.7%)

6 9/22 (40.9%)

7 0/3 (0%) a) P value was provided from Fisher’s exact test when the expected cell size is less than 5. 4.5.4 Recruitment Compliance and Inclusion of the TRN in Published Trials

We sought to analyze the recruitment compliance in the 273 published trials. We compared the target trial size indicated in the registration information of the respective trials with their actual recruitment as reported in the identified publications. We found that 111(40.7%) of the published trials reported fewer subjects who were actually recruited than the target size indicated in the trial registry, 96(35.2%) published trials reported actual recruitment matching the trial target size, and 66(24.2%) published trial recruited more participants (over-recruitment) than the target size in the trial registry. Ensuring adequate participant recruitment is critical to support the statistical power and internal validity of the trial results. Enrollment of fewer participants could introduce type II errors in reported study results [179]. We suggest that over-recruitment could be appropriate and would empower further assessment of secondary hypotheses. Therefore, we considered a trial recruitment as compliant if the actual recruitment was equal to or more than the target size as defined in the trial registration. We also analyzed the inclusion of the TRN in published trials. We verified whether a reference to the TRN was indicated in the papers of the 273 published trials. We examined the relationship between trial characteristics and the recruitment compliance and inclusion of the TRN in published trials as shown in Table 8.

Table 8. Recruitment compliance and inclusion of TRN in published trials.

Number of Pearson Recruitment Chi- Pearson Compliant square Number of Chi-square Trials / Test Publications that Test Trials Number of All P value Included TRN/ P value

Characteristics Published [Fisher’s Number of All Trials (%) Exact Published Trials [Fisher’s Test (%) Exact Test P value] P value] a a

Overall 162/273 (59.3%) 142/273 (52%)

0.42 0.02 Age Groups [0.44] [0.009]

Adult 47/87 (54%) 56/87 (64.4%)

Adult|Senior 70/116 (60.3%) 49/116 (42.2%)

Child 23/39 (59%) 22/39 (56.4%)

Child|Adult 7/12 (58.3%) 8/12 (66.7%)

Child|Adult|Senior 11/13 (84.6%) 6/13 (46.2%)

Senior 2/4 (50%) 0/4 (0%)

Undefined 2/2 (100%) 1/2 (50%)

0.85 0.12 Condition [0.85] [0.12]

Cancer 3/7 (42.9%) 5/7 (71.4%)

Chronic Conditions 17/27 (63%) 16/27 (59.3%)

Mental Health 60/104 (57.7%) 45/104 (43.3%)

Obesity 31/49 (63.3%) 31/49 (63.3%)

Others 51/86 (59.3%) 45/86 (52.3%)

0.46 <0.001 Continent [0.46] [<0.001]

Americas (Non-US) 14/23 (60.9%) 12/23 (52.2%)

Asia 6/10 (60%) 1/10 (10%)

Australia & New 12/26 (46.2%) 11/26 (42.3%) Zealand

Europe 76/131 (58%) 90/131 (68.7%)

Global (Multi) 1/3 (33.3%) 0/3 (0%)

Middle East 9/11 (81.8%) 2/11 (18.2%)

US 44/69 (63.8%) 30/69 (43.5%)

Trial Size a 0.009 <0.001

<=129 89/132 (67.4%) 51/132 (38.6%)

73/141 (51.8%) 91/141 (64.5%) >129

Enrollment Year 0.61 0.003

81/133 (60.9%) 57/133 (42.9%) First Half 2012

81/140 (57.9%) 88/140 (62.9%) Second Half 2012

0.02 0.03 Funding Source [0.01] [0.03]

Government/Authority 4/9 (44.4%) 5/9 (55.6%)

Hospitals and Clinics 9/24 (37.5%) 11/24 (45.8%)

Industry 4/8 (50%) 4/8 (50%)

Institute/Foundations/R 52/97 (53.6%) 63/97 (64.9%) esearch Center

University 93/135 (68.9%) 59/135 (48.1%)

0.90 0.002 Gender [0.94] [0.001]

Both 124/211 (58.8%) 106/211 (50.2%)

Female 18/29 (62.1%) 10/29 (34.5%)

Male 5/7 (71.4%) 4/7 (57.1%)

Undefined 15/26 (57.7%) 22/26 (84.6%)

0.60 0.29 Major Technology [0.59] [0.28]

Digital Games 13/22 (59.1%) 11/22 (50%)

Internet/Web 96/156 (61.5%) 85/156 (54.5%)

Mobile Apps 15/21 (71.4%) 15/21 (71.4%)

Offline 15/33 (45.5%) 12/33 (36.4%)

Social Media 3/6 (50%) 3/6 (50%)

Telehealth 14/24 (58.3%) 11/24 (45.8%)

Virtual Reality 6/11 (54.5%) 5/11 (45.5%)

0.44 0.054 Phase [0.51] [0.03]

Phase 0 142/242 (58.7%) 1/3 (33.3%)

Phase I 3/3 (100%) 2/8 (25%)

Phase II 3/8 (37.5%) 1/8 (12.5%)

Phase II/III 6/8 (75%) 1/1 (100%)

Phase III 1/1 (100%) 3/10 (30%)

Phase IV 6/10 (60%) 1/1 (100%)

Undefined 1/1 (100%) 133/242 (55%)

Prospective Registration 0.28 0.69

Retrospective 104/168 (61.9%) 89/168 (53%)

Prospective 58/105 (55.2%) 56/105 (53.3%)

Registration Year 0.45 0.63

Before 2012 18/28 (64.3%) 12/28 (42.9%)

In 2012 (Prospective) 40/77 (51.9%) 41/77 (53.2%)

In 2012 (Retrospective) 47/78 (60.3%) 44/78 (56.4%)

After 2012 57/90 (63.3%) 45/90 (50%)

0.16 <0.001 Trials Registry [0.14] [<0.001]

Australian New 13/27 (48.1%) 11/27 (40.7%) Zealand Clinical Trials Registry

Brazilian Clinical Trials 2/4 (50%) 1/4 (25%) Registry (ReBEC)

Clinical Trials Registry 3/3 (100%) 0/3 (0%) - India (CTRI)

ClinicalTrials.Gov (US) 95/149 (63.8%) 71/149 (47.7%)

International Standard 21/46 (45.7%) 33/46 (71.7%) Randomized Controlled Trial Number (ISRCTN)

Internet Portal of the 4/7 (57.1%) 3/7 (42.9%) German Clinical Trials Register (DRKS)

Iranian Registry of 8/9 (88.9%) 2/9 (22.2%) Clinical Trials

Japan Primary 2/4 (50%) 0/4 (0%) Registries Network

The Netherlands Trial 14/24 (58.3%) 21/24 (87.5%) Register

0.32 0.002 Study Arms [0.41] [0.001]

Single 2/2 (100%) 2/2 (100%)

Two or More 17/33 (51.5%) 26/33 (78.8%)

Undefined 143/238 (60.1%) 114/238 (47.9%) a) The median target size of trial enrollment was 129. Results from logistic regression tests of recruitment compliance by trial size is provided in Appendix 8.28. b) P value from Fisher’s exact test is provided when the expected cell size is less than 5 The recruitment compliance rate was at 59.3% and nearly half of the trials 52% provided a reference to their respective trial identification number in their publications. The Pearson Chi-square test results reported significant relationships (P<0.05) between trial recruitment compliance and trial funding source and trial size.

There were significant relationships (P<0.05) between trials with included TRN and trial characteristics including trial age group, enrollment date, location, funding source, gender, trial size, trial registry, phase and study arms. 4.6 Discussion

The primary objectives of this study were to examine the non-publication rate and prospective registration of digital health RCTs registered in any of the seventeen WHO ICTRP registries. A total of 417 RCTs met our inclusion criteria, were in forty different countries, and were registered in eleven trial registries. We found that 34.5% of all included trials remain unpublished six years after the enrollment of trial participants. Nearly half of our included trials 56.1% were registered in the ClinicalTrials.gov registry, of which 36.3% remain unpublished. Our non-publication rate was higher compared to a 2018 study that reported 27% non-publication rate for digital health trials registered in ClinicalTrials.gov [166]. Although both studies focused on examining nonpublication rates in digital health trials, their study design, data source and approach were quite different, hence a direct comparison would not be valid. The current study has a broader scope including 417 trials that were registered within one year (2012) in any of the seventeen WHO registries. The former 2018 study included 556 trials that were registered in the ClinicalTrials.gov registry only and were completed within three years, between 2010 and 2013 [166]. Our finding is also higher than that reported in another study with a non-publication rate at 29% for large RCTs, i.e. trials with at least 500 enrolled participants [6]. Other studies reported that nearly half of the included trials were not published, which is considerably higher than our own findings [1,8,10]. To explain these differences in non-publication rates, we postulate that the rapidly evolving technology elements of digital health trials could introduce extrinsic motivation to investigators to publish their results and stay ahead of the technology innovation curve. Our results also showed that the vast majority 96.6% of digital health trials are funded by non- industry sponsors, such as universities, hospitals, medical institutes and research centers, that are more disciplined and obliged by scholarly ethics to publish their results. We speculate that industry sponsors would be more interested in the broader opportunity in the digital health marketplace beyond the realm of academia and best practices of randomized trials design.

We found statistically significant relationships between prospective trial registration and their funding sources. The lowest compliance with prospective trial registration was at 14.3% for industry-funded trials. In contrast to our present study, other studies assessing the prospective registration of clinical trials found that industry funded trials were more likely to be compliant with prospective trial registration compared to non-industry-funded trials [78,87,143-145,167- 171]. However, we also found that only 14 (3.4%) trials of the included 417 trials were industry- funded which may limit the generalizability of our finding in comparison to other studies. For the 273 trials with identified publication in our study, the compliance in trial recruitment was at 59.3%, which is comparable to results from other studies reporting nearly one third of the clinical trials recruited their original target size [172,173]. Our Pearson Chi-square test reported a statically significant relationship between the compliance with trial recruitment and the trial size (P=0.009) and funding source (P=0.02). Smaller trials were 1.3 times more compliant with recruitment than larger trials. We suspect that it would be easier to adequately recruit, manage and retain participants of smaller trials with a clear logistical, operational and feasibility advantages over larger trials. We also found that trials funded by university sponsors were 1.5 times more compliant with trial recruitment compared to trials funded by any other sponsors. This finding indicates that investigators of university sponsored trials are exceling at adopting best practices and strategies in trial design to improve participant recruitment, which is to be expected within the academic context of the university sponsors. 4.6.1 Retrospective Registration of Digital Health Clinical Trials

Despite the emphasis on prospective trial registration introduced by the 2004 ICMJE mandate and the 2008 DoH, we found that only 38.4% of all included trials were registered prospectively [28,31]. Similar findings were reported by two independent studies indicating that compliance with prospective trial registration was at 31% [66,79]. We hypothesize that investigators may be more inclined, or biased, to registering their trial only when submitting their results to peer- reviewed journals for publication. 4.6.2 Selective Registration Bias of Digital Health Clinical Trials We found a statistically significant relationship (P<0.001) between retrospective registration and the registration-to-publication duration in digital health clinical trials, for which we coined a new term as “Selective Registration Bias”, or simply “Registration Bias”.

Within our sample of 273 published trials, the vast majority 95.7% of trials registered within one year before publication were registered retrospectively. Our results showed that investigators, who did not register their trials promptly prior to enrollment, were required or motivated to do so only when they submit their results to scholarly journals. There may be a number of contributing factors to this selective registration bias. Firstly, the investigator may have deferred the trial registration task until the completion of the trial and only when the results are finalized and ready to be published. Secondly, the investigator may also be not aware of the ethical expectation to register their trial promptly. Lastly, journal editors and peer-reviewers are likely to suggest registration of the submitted trial publication prior to accepting the publication submission. It’s important to establish and broaden the adoption of prompt trial registration requirement in ethical approval and guidelines of clinical trial design at the institutional and academic level. 4.6.3 Challenges in Oncology Trials We found statistically significant relationships between 5 different condition groups and the nonpublication rate (P=0.042) as well as compliance with prospective trial registration (P=0.02). Our results indicate that investigators of oncology trials are the most compliant with prospective trial registration at 47.4% compared to investigators of trials of other conditions, but they seem to face more challenges to publish their oncology trial results with the highest non-publication rate reported in oncology trials at 63.2%. The significantly higher underreporting of oncology trial results suggests challenges in conducting oncology studies in the realm of digital health trials. These challenges may align with other studies indicating a number of barriers to conducting traditional oncology trials including eligibility, recruitment, follow-ups, and oncologist and patient attitudes [143-145]. However, we postulate that there may be a few other challenges that are specific to digital health oncology trials, particularly in the recruitment and enrollment stage of those trials. These explicit barriers may be explained by the treating oncologist, or patient, preferences to enroll in other non-digital health trials, such as experimental drug trials, with more measurable clinical outcomes. The results from our study validate this postulate through two distinct data points. First, we found only one oncology trial in our study that was funded by a pharmaceutical sponsor, indicating the lack of interest from pharmaceutical sponsors to invest in digital health trials. Second, we reported the lowest recruitment compliance rate for oncology trials at 42.9%, which is a clear indicator to enrollment barriers to digital health oncology trials.

4.6.4 Compliance with Prospective Trial Registration in The Australian New Zealand Clinical Trials Registry The Australian New Zealand Clinical Trials Registry (ANZCTR) leads with 89.7% compliance with prospective trial registration. Also, trials from Australia and New Zealand were leading with 89.5% compliance with prospective trial registration (the 0.2 percent decline is due to one trial that was registered in the ANZCTR registry and was not located in Australia or New Zealand). The high compliance with prospective trial registration in the ANZCTR registry was acknowledged by another study indicating an incremental trend in prospective trial registration from 48% to 63% for trials registered in the ANZCTR registry between 2006 and 2012 [167]. 4.6.5 Recruitment Compliance in the Middle East Digital health trials in the Middle East had the lowest non-publication rate and the highest recruitment compliance at 26.7% and 81.1% respectively. These results were driven by Iranian trials registered in the Iranian Registry of Clinical Trials (IRCT). Compared to other WHO registries in our study, the IRCT trials had the lowest non-publication rate and the highest recruitment compliance at 25% and 88.9% respectively. Several studies reported to the legacy of the IRCT and its role in upholding best practices and ethical guidelines in the field of clinical research in Iran [168-170]. However, we found that prospective trial registration was the lowest at 8.3% for trials registered in the IRCT. Our results support findings from another study that reported on the rationale behind the low prospective trial registration rate at 8.3% for trials registered in the IRCT between 2008 and 2011[78]. All trials registered in the IRCT were funded by university sponsors. This finding aligns with another study that analyzed all trials registered in the IRCT until the end of 2015 and reported that 97% of those trials were funded by university and other governmental institutions [98]. The academic and public sponsorship may be a contributing factor to low non-publication rate and high compliance in recruitment of trials registered in the IRCT as they would likely encourage and promote a culture of adherence with trial design best practices and ethical guidelines. We also found that inclusion of the TRN in respective trial publications was the lowest for trials registered in the IRCT at 22.2%. Combined with the lowest prospective trial registration of those trials, we postulate that investigators of trials registered in the IRCT may be unaware of the ICMJE recommendation for prospective trial registration and the WHO re-affirmation to include the TRN in the trial publication to enable the easy linkage between the publication and registry entry of the respective trial.

4.6.6 Recruitment Compliance in Low and Middle-Income Countries We found significant differences (P=0.002) in the relationship between recruitment compliance and trial location. We grouped Asia, the Middle East and the Americas without the US as low and middle-income countries, and grouped high-income countries as Europe, US, Australia and New Zealand. The average compliance with trial recruitment in low and middle-income countries was at 51%, which is significantly higher than that in high-income countries at 30%. The significantly higher compliance in trial recruitment in low and middle-income countries may be explained by differences in clinical, regulatory, and economical standards in these countries, such as (1) access to a large population of potential trial participants (often with lower socioeconomic status and medical literacy), (2) lower cost of research resources, (3) less enforced, or developed, regulatory standards, and (4) if participation in a trial would provide access to, otherwise unavailable or unaffordable, medical care for the participants [157,174,175] These differences raise ethical concerns in conducting clinical trials in these countries. These concerns are best described in article 20 of the DoH emphasizing on the ethics in participant recruitment, and on the relevance and tangible benefits of the research outcomes to the participants population [28,157]. Local regulatory agencies, and ethics committees in low and middle-income countries need to be cautious about these ethical concerns and ensure the scientific integrity of digital health trials in their respective countries. 4.6.7 Adherence to Best Practices in Clinical Trials from Europe Europe was the region with the largest number of digital health trials in our study. European trials constituted 182(43.6%) of our included trials, and had the highest compliance with the inclusion of the TRN at 68.7%, and the second lowest non-publication rate at 28% tied with the Middle East. The overall lead in European trials is influenced by strong publication compliance demonstrated by the investigators of digital health trials in the Netherlands. The Netherlands National Trial Register (NTR) had the second lowest non-publication rate at 25%, and the highest rate in compliance with inclusion of the TRN in their respective trial publication at 87.5%. The latter finding was higher than another study of trials registered in the NTR that indicated that the compliance with reporting the TRN in respective trial publications was at 60% [87]. 4.6.8 Time to Publication The majority (88.3%) of the published trials in our study were published within five years after enrollment. Our finding is comparable to the results of a 2007 study indicating that trials with

positive results were published within five years after enrollment and trials with negative results were published in six to eight years [163]. We postulated that small trials would be easier to conduct as they have fewer recruitment and enrollment challenges, hence they would likely be completed and published in a shorter time compared to large trials. To validate our postulate, we analyzed the relationship, and found statistically significant differences (P=0.002), between trial size and enrollment-to-publication duration. We observed a reversed trend in publication between large and small trials at the fourth-year mark after trial enrollment. Small trials published their results 1.3 times more than large trials within the first four years after enrollment, after which large trials published their results 2.2 times more than small trials. This trend is likely driven by the longer amount of time required for the investigator of the larger trials to complete the enrollment and intervention for a larger group of participants. 4.7 Limitations Our study is the first to our knowledge that analyzed global digital health clinical trials registered in all the WHO recognized trial registries [74]. Our study did not consider any other registries that are not part of the seventeen WHO primary registries, such as the federal office of public health’s portal for human research in Switzerland, and the Philippine health research registry [177,178]. We acknowledge that not considering trials registries other than the WHO primary registries may have impacted the external validity of our study results. Despite the ICMJE and WHO emphasis on trial registration, not all randomized trials are registered [66,174] We did not include unregistered trials in our analysis, which may impact the internal validity of our study results. We included trials that are registered and started the enrollment in 2012. We considered the enrollment and/or trial start date as provided in the registration information of the included trials. The registration information is provided manually and voluntarily by the registering investigators who are often overwhelmed with competing priorities and limited resources. Therefore, the enrollment date provided in the trial registries may not always be up-to-date or maintained promptly. Lastly, our study only included registered trials enrolled in 2012. Our findings predate the recent 2015 WHO calls for improving public disclosure of trial results and the linkages between these results and the respective trial registry entries [32].

4.8 Conclusion In the field of digital health RCTs, the adherence of investigators to the best practices of trial registration and result dissemination is still evolving. We analyzed digital health RCTs that were registered in the seventeen WHO recognized trials and started their enrollment in 2012. Within our included 417 trials, non-publication rate and retrospective trial registration were prevalent at 34.5% and 61.6% respectively. Our study indicated low compliance rate with recruitment and inclusion of the TRN in publication of the 273 trials, with identified publication, at 59.3% and 52% respectively. It would be advisable for the research community, from research ethics boards to journal editorial boards, to promote and advocate for better adherence to trial publication and registration. In particular, prospective trial registration could be mandated prior to obtaining institutional ethics approval and explicit reference to trial registration identification for all submitted trial manuscript for peer-reviewed journals could be enforced. Further research is required to identify contributing factors and mitigation strategies to low compliance rate with trial publication and prospective registration in digital health clinical trials. 4.9 Data availability Clinical trials registration data and material were downloaded from the WHO ICTRP database or directly from the website of the primary registries in the WHO registry network. The seventeen clinical trials registries included in this study are publicly available online: https://www.who.int/ictrp/network/primary/en/ 4.10 Code availability Computer code used in this study is available upon reasonable request to the corresponding author and under a collaboration agreement. 4.11 Preprint Statement This article was previously published as a preprint [260] in the free online preprint server medRxiv. 4.12 Authors Contribution The lead author (MAD) and senior author (JC) conceived the study idea and developed the study design with contributions from all authors. MAD executed the search queries, extracted data, analyzed the results, and drafted the manuscript. All authors interpreted results, were involved in the reporting of the study, reviewing and editing the manuscript, and approved the final version of the manuscript. All authors agree to take responsibility for the work.

4.13 Acknowledgements None declared. 4.14 Competing Interests None declared. 4.15 Funding This study was not funded.

Chapter 5 Prospective Registration and Reporting of Trial Number in Randomized Clinical Trials: Adoption of the ICMJE and the Declaration of Helsinki Recommendations

Submitted to the BMJ journal on August 9th, 2019 (Manuscript ID: BMJ-2019-052091) as: Al-Durra M, Nolan RP, Seto E, Cafazzo JA. Prospective Registration and Reporting of Trial Number in Randomized Clinical Trials: Adoption of the ICMJE and the Declaration of Helsinki Recommendations.

Summary: After concluding the former two studies (Chapters 3 and 4), the advancement and maturity achieved in the own-developed search tools and techniques enabled expanding the search inquiry beyond the boundaries of digital health RCTs. This study addresses the third and fourth research questions of this thesis. It provides a current and empirical analysis of the adoption of the ICMJE and WHO guidelines for prospective trial registration and reporting of TRN ten years after the adoption of the 7th revision of the DoH. This study also validates the newly detected registration bias in retrospectively registered trials. It included all RCTs that were published in 2018 that were registered in any of the seventeen WHO trial registries and published in any PubMed-indexed journal, irrespective of the trial location, condition, phase, number of participants, or funding source. Overall, the compliance with reporting the TRN and prospective trial registration was 71.2% and 41.7% respectively. A manuscript published in an ICMJE member journal was 5.9 times more likely to include the TRN, and a published trial was 1.6 times more likely to be registered prospectively when published in an ICMJE member journal compared to other journals. The new form of bias was validated in this study, referred to as “Registration Bias” or “Selective Registration Bias,” with a significantly (P<0.001) higher proportion (85.1%) of investigators registering their trials retrospectively within a year of submitting their manuscript for publication. The manuscript of this study was submitted to the BMJ journal on August 9th, 2019 (Manuscript ID: BMJ-2019-052091). This chapter of this thesis includes the complete and submitted manuscript of this study with no modifications, other than adjusting the font styles and re- organizing the citations and references to align with the overall references of this thesis.

Prospective Registration and Reporting of Trial Number in Randomized Clinical Trials: Adoption of the ICMJE and the Declaration of Helsinki Recommendations 5.1 Abstract OBJECITVES: To evaluate the compliance with prospective registration and the inclusion of the trial registration number (TRN) in published randomized controlled trials (RCTs). The secondary research objective was to analyze the rationale behind retrospective trial registration and to detect registration bias (selective registration bias) in retrospectively registered trials. DESIGN: Cross sectional analysis. DATA SOURCES: Five different data sources; PubMed database, the World Health Organization’s (WHO) International Clinical Trials Registry Platform (ICTRP) including seventeen trial registries, the University of Toronto library resources, the International Committee of Medical Journal Editors (ICMJE) list of member journals and journals that follow the ICMJE's recommendations, and the InCites Journal Citation Reports 2017. STUDY SELECTION CRITERIA: All RCTs that were registered in all seventeen WHO ICTRP registries and published in any PubMed-indexed journal in 2018. RESULTS: This study included 10,500 manuscripts that were published in 2,105 journals. The compliance with reporting the TRN and prospective trial registration was 71.2% and 41.7% respectively. The bivariate and multivariate analyses reported significant relationships (P<0.05) between the reporting of the TRN and the impact factor and the ICMJE membership of the publishing journal. A significant relationships (P<0.05) was also observed between the prospective trial registration and the trial registry, region, condition, funding, trial size, interval between paper registration and submission dates, impact factor, and ICMJE membership of the publishing journal. A manuscript published in an ICMJE member journal was 5.9 times more likely to include the TRN (P<0.001, OR=5.934, 95% CI: 4.144 to 8.496), and a published trial was 1.6 times more likely to be registered prospectively (P<0.001, OR=1.599, 95% CI: 1.309 to 1.953) when published in an ICMJE member journal compared to other journals. A new form of bias was detected in this study, referred to as “Registration Bias” or “Selective Registration Bias,” with a significantly (P<0.001) higher proportion (85.1%) of investigators registering their trials retrospectively within a year of submitting their manuscript for publication. When investigating the delayed registration of retrospectively registered RCTs, higher rates of RCT 66

registrations were observed within the first three to eight weeks after enrollment. Within the 286 trials registered retrospectively and published in an ICMJE member journal, only 8 (3%) of the authors, included a statement justifying the delayed registration of the published trials with the reasons that included lack of awareness, error of omission, and the registration process taking longer than planned. CONCLUSIONS: This analysis is a critical assessment of the adoption of the ICMJE and WHO recommendations for the registration of clinical trials. There was high compliance in reporting the TRN within trial papers published in ICMJE member journals; however, prospective trial registration was found to be low. This study challenges the viability of the ICMJE and WHO definition of prospective trial registration without allowing for a reasonable grace period after enrollment.

5.2 Introduction The selective or incomplete reporting of RCTs outcomes constitutes a major concern and leads to biased interpretation of clinical evidence; through overstating the benefits or understating the adversity in the trial results [11,46,48-50,64]. In 1986, Simes explained how the implementation of a trial registry would mitigate the bias in selective reporting of published trials [13]. Trial registries would equip the researcher with another source of trial information to verify if the reported outcomes in the published trials were indeed the a priori measures submitted in the trial protocols [11,113]. The success of trial registries is predicated upon the prompt registration of clinical trials. Early trial registration will mitigate the bias associated with modification of pre-defined trial outcome measures based on preliminary analysis of the trial results [29,30]. In 2004, the ICMJE promoted a new mandate that required clinical trials to be registered prospectively in a publicly accessible trial registry, i.e., before the enrollment of the first patient, as a prerequisite for accepting trial publication in any ICMJE member journal [31]. Shortly after, the WHO announced an initiative to standardize the trial registrations process across multiple international trial registries [28]. The 59th World Medical Association General Assembly in Seoul adopted the 7th revision of the DoH in October 2008 with emphasis (paragraph 19) on prospective trial registration prior to the enrollment of the first patient [32,180]. On April 14, 2015, the WHO introduced a new statement signifying the importance of including the TRN in the abstracts of the published trials to enable linking the trial publication with trial protocol in the trial registry [33,181]. The compliance with the inclusion of TRN in trial publication was found to be between 26% and 68% in a number of studies [64,65,75,86-89]. The compliance with the prospective registration of RCTs was reported to be between 3.6% and 77% in a number of studies [64-73,75-85]. These studies reported a low to moderate adoption of the ICMJE recommendation and were published between 2011 and 2019. However, these studies were limited in scope and included trials published in a specific group of journals, trials of a specific treatment or condition, or trials registered in a specific trial registry or a small group of trial registries. In our study, we aimed to provide an empirical appraisal of the current state of compliance with the ICMJE recommendation ten years after the adoption of the 7th revision of the DoH. We wished to investigate the compliance with the inclusion of TRN in published RCTs and the

compliance with the prospective registration of RCTs across all the seventeen WHO trial registries. To aid the generalizability of our study, we expanded our study to include all papers of RCTs published in all PubMed-indexed journals in 2018. We also sought to understand the rationale behind late registration of RCTs and validate our own hypothesis of a potential new form of bias, which we refer to as “Registration Bias” or “Selective Registration Bias.” This bias is defined by the tendency of investigators to selectively register their trials only when intending to publish the manuscript in a peer-reviewed journal. 5.3 Research Objectives The primary research objectives were to evaluate the compliance with the ICMJE recommendations (1) in reporting the TRN in the manuscript of RCTs published in all PubMed- indexed journals, and (2) in prospective registration of RCTs across all the seventeen WHO trial registries. The secondary research objectives were (1) to examine the rationale behind delayed registration for retrospectively registered RCTs and (2) to validate our hypothesis of “Registration Bias” or “Selective Registration Bias” for retrospectively registered RCTs, i.e., if the investigators of clinical trials would not consider registering their trial promptly before the enrollment of the trial participants, and only consider retrospective registration when planning to publish the trial results and nearing the submission date of RCT manuscript for publication. 5.4 Methods 5.4.1 Data Source We amalgamated five different data sources in the course of this study. First, we searched the PubMed database for trial papers and downloaded abstracts and meta-data of RCTs that were published in 2018 [182]. Our second data source was the WHO’s ICTRP, including seventeen trial registries, that we utilized to identify and download registration protocols of published trials [155,156,183-198]. Third, we accessed the University of Toronto library resources to download the full-text manuscripts of the included trial papers to help extract the TRN [199]. Our fourth data source was the ICMJE website to identify ICMJE member journals and journals that follow the ICMJE's recommendations for the conduct, reporting, editing and publication of scholarly work in medical journals [200,201]. Lastly, we considered the InCites Journal Citation Reports 2017 to cross-reference the impact factor of the publishing journal of our included trial papers [202]. We accessed the InCites Journal Citation Reports 2017 through the University of Toronto online library resources [203]. 69

5.4.2 Tools for Statistical Analysis and Data Preparation

We used the Pearson Chi-square statistic for bivariate analyses and binary logistics regression for multivariate analyses of our results. These statistical analyses were performed in SPSS Statistics version 24 (IBM Corporation, Armonk, NY). We used the Fisher’s exact test statistic for bivariate analyses where the expected cell value was less than 5. The Fisher’s exact test was performed in STAT Statistics version 14.2 (StataCorp LLC, TX). We also used other tools to aid in the data preparation process including Microsoft Excel (Microsoft Office 365 ProPlus Version 1906), Microsoft SQL Server 2017, Microsoft .NET 4.5/C#, Microsoft Visual Studio 2017, NuGet package iTextSharp.5.5.13, and PaperPile (Paperpile LLC, Cambridge, MA). 5.4.3 Inclusion Criteria

We included publications of all RCTs that were published in 2018 in all PubMed-indexed journals or databases. For our trial registration analysis, we considered seventeen clinical trial registries that are included in the WHO Registry Network. 5.4.4 Exclusion Criteria

We excluded publications of registered, but not randomized, clinical trials. We also excluded publications that were secondary analyses, editorials, letters, erratum, corrigendum, reviews, meta-analysis, and design papers. 5.4.5 Search Terms and Methodology

The following search terms were applied to the title field in PubMed: “Trial” OR “RCT” OR “Randomized” OR “Randomised.” Our study was not a systemic review or meta-analysis. It provides a critical evaluation of the registration of RCTs. However, we followed the recommendations of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement in the identification, screening, and inclusion of our search results [204].

5.4.6 Data Extraction

5.4.6.1 Identification of the TRN

We downloaded the publication metadata for the trial papers in Extensible Markup Language (XML) format from PubMed—a sample of the XML file export is provided in Appendix 17. We identified the TRN in the papers of published trials in three different steps. First, we obtained the TRN and trial registry name from two XML data elements; “AccessionNumber” and “DataBankName” respectively [205]. Second, for papers where the trial number was not indicated in these two XML data elements, we searched for the TRN in the paper abstract provided in the “Abstract” XML data element. Last, and for the remaining papers where the TRN was still not identified, we downloaded the full-text manuscripts in the Portable Document Format (PDF) format from the online library resources at the University of Toronto to complete our search. We utilized a commercially available reference management software, PaperPile, to instrument the downloading process of the full-text manuscripts in PDF format. We leveraged an existing open source programming library, NuGet package iTextSharp.5.5.13, to serialize and parse the full-text content from the PDF files. We developed our own computer program, in Microsoft .NET/C# programming language and used advanced text-pattern search techniques, also known as Regular Expressions (RegEx), to identify potential matches of TRN. Further details describing the search methodology to identify the TRN in published papers are provided in Appendix 18. Details describing the text patterns of the seventeen WHO trial registries are provided in Appendix 19. 5.4.6.2 Classification of Publishing Journals

We classified the publishing journals of the matched trials into three categories as follows: ICMJE Member Journal If the publishing journal of a matched trial was one of the 14 ICMJE member journals [200]. ICMJE Follower Journal If the publishing journal of a matched trial was not one of the ICMJE member journals and if the journal was one of the 5061 journals whose editors or publishers have explicitly indicated compliance with the ICMJE's recommendations for the conduct, reporting, editing and publication of scholarly work in medical journals [201,203].

None If the publishing journal of a matched trial was neither enlisted as an ICMJE member journal nor an ICMJE follower journal. 5.4.6.3 Trial Registration Information

We utilized the ICTRP search portal to search for and download the trial registration protocols of all included trials with an identified TRN. The trial registration protocols were downloaded in XML format. The XML file was then transformed and imported into a SQL Server database for further analysis. Twenty-four TRNs were not found in the ICTRP search portal. For those trials, we were able to download the trial protocols directly from their respective trial registries. Details are provided in Appendix 20. We used the data provided in the trial protocols to abstract and categorize study conditions, funding, and cohort as described in Appendices 21, 22, and 23. 5.4.6.4 Prospective Trial Registration

The ICMJE guidelines require all clinical trials to be publicly and prospectively registered to be considered for publication within any ICMJE member journal; i.e., investigators of clinical trials need to register their trials in one of the public trial registries recognized by the WHO before the enrollment of the first patient [34,201]. We evaluated the prospective registration of our included trials by comparing the trial registration and enrollment dates provided in the trial protocols. We categorized the trial registration to be prospective if the trial registration date was earlier, or the same as, the trial enrollment date. Otherwise, the trial registration was categorized as retrospective. 5.4.7 Patient and public involvement

This research was done without patient involvement. Patients were not invited to comment on the study design and were not consulted to develop patient relevant outcomes or interpret the results. Patients were not invited to contribute to the writing or editing of this document for readability or accuracy. 5.5 Results We applied our search terms and phrases to the title field in the PubMed advanced search portal on January 13th, 2019, as indicated in Figure 7: Figure 7. Search Results in PubMed.

Our preliminary search result included 11,857 trial papers, for which we screened the titles to exclude editorial, erratum and design papers. We reviewed the journal publication dates in the XML export from PubMed and excluded 19 trials because the journal publication date was not in 2018. More details on the screening process for the journal publication dates are provided in Appendix 24. We excluded 575 papers that did not have the TRN reported in the abstract or PubMed publication metadata, and for which we were not able to access the full-text manuscript. We included 10,500 trial papers after screening. We searched for the TRN as depicted in the following Figure 8: Figure 8. Screening and Identification of TRN.

782 11,857 Papers Excluded after Title Screening Papers Matched Search ------Terms in PubMed – had the word Trials – had the word Design Identification – Publication Dates were not 2018 – Editorials/Corrections/Reviews

11,075 Papers Screened for TRN 575 No TRN identified in PubMed Metadata or Abstracts, we had no access to the full text manuscript

Screening 10,500 Papers Included after Screening for TRN

3,779 6,721 2,931 Papers Excluded Papers with TRNs identified in Papers with Full Text ------the PubMed XML export Manuscript Found and 51 - Editorial and Reviews Downloaded 5 - Not a TIN (false positive) (2610 Papers with TRNs 2844 - no TRN found identified in PubMed Metadata 14 - TRN referencing other trials 17 - TRN in the references section + 3,790 1169 Papers with TRNs Papers with TRN identified in the PubMed Identified in the Full Text

Abstract) Manuscript Eligibility 109 Published TRNs with 7,390 Unique TRNs Identified in 7,569 Unique Papers No Protocols Identified

7,281 Unique Protocols Found and Downloaded (2504 in PubMed Metadata + 1218 in PubMed Abstracts + 3559 in Full Text Manuscripts)

7,218 63 Protocols Excluded 7,473 (Missing either Unique Protocols with Valid Unique Papers with Matching Registration or Registration and Enrollment Inclusion Protocols Enrollment dates) Dates

Initially, the TRN was identified in the PubMed publication metadata for 3,779 papers, and in the full-text manuscript for 3,790 papers. We were not able to access the full-text manuscripts for 575 papers through the University of Toronto online library resources. Further details describing the search methodology to identify the TRN in published papers are provided in Appendix 18.

Combined, we identified 7,390 unique TRNs across 7,569 published papers. We were able to download 7,281 trial protocols from the WHO search portal or directly from the respective trial registry when the protocol was not found in the WHO search portal. There was a remainder of 109 identified TRNs in published papers, for which we did not find any matching trial protocols in the WHO search portal and in their respective trial registries. In summary, we identified valid TRNs and downloaded 7,281 matching trial protocols for 7,473 of the 10,500 included papers. With respect to evaluating the rate and characteristics of prospective trial registration in our study, we only included 7,218 trial protocols with provided trial registration and participant enrollment dates. These two data fields were required to determine if the trial was registered prospectively. We excluded 63 protocols with missing trial registration or enrollment dates. Within our included 7,218 trial protocols, 287 trials were reported in 612 papers—i.e., when a trial is published in more than one paper. We needed to consider one paper only—for any given trial—to extract the journal impact factor, ICMJE membership, submission, and publication dates. In these cases, we considered the paper with the earlier publication date because it signifies the first publication of the registered trial. 5.5.1 Reporting of TRN

We identified valid TRNs for 7,473 (71.2%) of the 10,500 included papers. Based on the identified TRNs, we downloaded 7,281 unique and matching trial protocols that were registered in sixteen different WHO trial registries from 127 countries. We examined the relationship between the reporting of valid TRN in the trial papers and the impact factor as well as the ICMJE membership of the publishing journals, as shown in Table 9: Table 9. Journal Characteristics Relationship to Reporting of TRN.

Pearson Number of Papers Chi- Journal with Identified Binary Logistic Regression square Characteristics TRN / All Papers - Test within Groups (%) P value P value Odds Ratio (95% CI) Overall 7473/10500 (71.2%)

Journal Impact <0.001 <0.001 Factor a

Fourth Quartile 2011/2261 (88.9%) <0.001 3.617 [3.040 to 4.304] (4.876 or more) Third Quartile 1742/2250 (77.4%) <0.001 2.178 [1.884 to 2.519] (3.147 to 4.876) Second Quartile 1550/2250 (68.9%) <0.001 1.397 [1.215 to 1.605)] (2.21 to 3.147)

First Quartile 1246/2253 (55.3%) 0.001 0.800 [0.699 to 0.916] (less than 2.21) Undefined - - - Reference

Journal Type <0.001 <0.001

ICMJE Member 745/779 (95.6%) <0.001 5.934 [4.144 to 8.496] Journal

ICMJE 2327/3048 (76.3%) <0.001 1.373 [1.240 to 1.521] Following Journal Neither 4401/6673 (66%) - Reference a) Excluding 1486 papers from the bivariate analysis, for which the journal impact factor was not identified. The Pearson Chi-square test results and the binary logistic regression test results indicated significant relationships (P<0.05) between the characteristics of publishing journals and whether the published papers reported a valid TRN. 5.5.2 Prospective Registration of Clinical Trials We identified and downloaded a total of 7,218 unique trial protocols with valid trial registration dates and participant enrollment dates. We found that 3,013 (41.7%) of our identified trials were registered prospectively. The Pearson Chi-square test results and the binary logistic regression test results indicated significant relationships (P<0.05) between the prospective registration of clinical trials and the trial registry, region, condition, funding, trial size, interval between registration and paper submission dates, impact factor, and the ICMJE membership of the publishing journal as shown in Table 10: Table 10. Prospective Trial Registration Relationship to Trial Characteristics, Journal Impact Factors and Journal ICMJE Membership.

Number of Papers of Pearson Prospectively Trial Chi- Registered Trials / Binary Logistic Regression Characteristic square All Papers with s Test Valid TRN – within Groups (%) P value P value Odds Ratio (95% CI) Overall 3013/7218 (41.7%) Trial Registry <0.001 <0.001 EU Clinical 81/87 (93.1%) <0.001 10.775 [3.629 to 31.993] Trials Register (EU-CTR) Australian New 311/440 (70.7%) 0.003 2.523 [1.384 to 4.602] Zealand Clinical Trials Registry (ANZCTR) Japan Primary 135/220 (61.4%) 0.04 1.943 [1.038 to 3.637] Registries Network (JPRN) The Netherlands 115/188 (61.2%) 0.46 1.285 [0.657 to 2.512] National Trial Register (NTR)

Chinese Clinical 41/76 (53.9%) 0.23 1.568 [0.751 to 3.276] Trial Register (ChiCTR) International 256/586 (43.7%) 0.07 0.568 [0.307 to 1.054] Standard Randomised Controlled Trial Number (ISRCTN) Pan African 24/54 (44.4%) 0.62 0.801 [0.334 to 1.920] Clinical Trial Registry (PACTR) ClinicalTrials.go 1830/4712 (38.8%) 0.01 0.476 [0.266 to 0.854] v German Clinical 37/96 (38.5%) 0.42 0.738 [0.355 to 1.536] Trials Register (DRKS) Clinical Trials 36/124 (29%) 0.11 0.574 [0.289 to 1.137] Registry - India (CTRI) Clinical Research 15/55 (27.3%) 0.11 0.509 [0.221 to 1.168] Information Service (CRiS), Republic of Korea Iranian Registry 98/420 (23.3%) 0.04 0.474 [0.232 to 0.968] of Clinical Trials (IRCT) Brazilian Clinical 6/95 (6.3%) <0.001 0.092 [0.032 to 0.259] Trials Registry (ReBec) Sri Lanka 5/5 (100%) i - h Clinical Trials Registry (SLCTR) Cuban Public 0/3 (0%) i - h Registry of Clinical Trials (RPCEC) Thai Clinical 23/57 (40.4%) - Reference Trials Register (TCTR) Region a <0.001 <0.001 Multi 360/566 (63.6%) <0.001 2.892 [2.118 to 3.947]

Asia 725/1472 (49.3%) 0.01 1.472 [1.102 to 1.967]

US 548/1243 (44.1%) <0.001 2.196 [1.686 to 2.861]

Africa 108/251 (43%) 0.003 1.776 [1.209 to 2.611]

Europe 864/2086 (41.4%) <0.001 1.623 [1.246 to 2.114]

Americas (non- 170/554 (30.7%) <0.001 1.881 [1.381 to 2.561] US) Middle East 140/564 (24.8%) 0.07 1.499 [0.961 to 2.337]

Undefined - - Reference

Phase b 0.62 0.043 Phase I 57/121 (47.1%) 0.02 1.591 [1.078 to 2.349]

Phase II 316/711 (44.4%) 0.03 1.222 [1.018 to 1.468]

Phase III 532/1149 (46.3%) 0.37 1.074 [0.918 to 1.256]

Phase IV 281/648 (43.4%) 0.21 1.124 [0.935 to 1.352]

Undefined - - Reference

Condition c <0.001 <0.001 Addiction 71/134 (53%) 0.19 1.309 [0.871-1.967]

Eye Disease 38/72 (52.8%) 0.85 0.954 [0.579-1.572]

Skin Condition 168/326 (51.5%) 0.39 1.139 [0.848-1.529]

Pulmonary 149/293 (50.9%) 0.47 1.120 [0.827-1.515] Disease Diabetes 153/336 (45.5%) 0.90 0.981 [0.732-1.313]

Surgery, 271/609 (44.5%) - Reference Abdominal Disease and Intensive Care Oncology 311/720 (43.2%) 0.004 0.705 [0.554-0.897]

Reproduction and 253/603 (42%) 0.27 0.870 [0.678-1.116] Sexual Health

Heart Disease 293/701 (41.8%) 0.12 0.827 [0.652-1.048]

Mental Health 412/986 (41.8%) 0.17 0.856 [0.684-1.070]

Aging and 27/65 (41.5%) 0.47 0.811 [0.458-1.436] Palliative Care Arthritis and 335/842 (39.8%) 0.43 0.912 [0.725-1.148] Injuries Ear, Nose and 33/84 (39.3%) 0.85 0.954 [0.579-1.572] Throat Obesity and 163/423 (38.5%) 0.17 0.826 [0.628-1.088] Physical Activity

Kidney Disease 86/232 (37.1%) 0.16 0.784 [0.559-1.099]

Oral Health 29/191 (15.2%) <0.001 0.396 [0.250-0.625]

Others 218/599 (36.4%) 0.03 0.751 [0.584-0.967]

Undefined - - h

Funding d <0.001 <0.001 Industry 537/934 (57.5%) 0.058 2.512 [0.970 to 6.505]

Government 527/1068 (49.3%) 0.16 1.990 [0.770 to 5.145]

Foundation 422/918 (46%) 0.15 2.012 [0.778 to 5.201]

Not Funded 54/120 (45%) 0.39 1.564 [0.569 to 4.303]

Hospital 335/815 (41.1%) 0.15 2.003 [0.772 to 5.199]

University 1120/3329 (33.6%) 0.31 1.639 [0.637 to 4.217]

Undefined - - Reference

Trial Size e <0.001 <0.001 Q4 (more than 1000/1882 (53.1%) 0.56 1.339 [0.502 to 3.575] 300 participants)

Q3 (120 to 300 779/1813 (43.0%) 0.92 1.051 [0.394 to 2.805] participants)

Q2 (60-120 667/1896 (35.2%) 0.79 0.872 [0.327 to 2.329] participants) Q1 (less than 60 538/1587 (33.9%) 0.83 0.896 [0.336 to 2.394] participants)

Undefined - - Reference

Journal Type <0.001 <0.001 ICMJE Journal 458/744 (61.6%) <0.001 1.599 [1.309 to 1.953]

ICMJE 930/2249 (41.4%) 0.53 1.039 [0.924 to 1.168] Following Journal Neither 1625/4225 (38.5%) - Reference

Journal Impact <0.001 0.02 Factor f Fourth Quartile 861/1586 (54.3%) 0.02 1.273 [1.044 to 1.553] (more than 5.559) Third Quartile 683/1615 (42.3%) 0.10 1.168 [0.969 to 1.408] (3.463 to 5.559) Second Quartile 588/1558 (37.7%) 0.98 1.002 [0.831 to 1.209] (2.42 to 3.463) First Quartile 545/1568 (34.8%) 0.99 1.001 [0.829 to 1.209] (less than 2.42) Undefined - - Reference

Interval between <0.001 <0.001 Registration and Paper Submission Dates g More than 3 1280/2592 (49.4%) 0.03 1.143 [1.010 to 1.295] years 2 to 3 years 352/813 (43.3%) 0.56 1.053 [0.884 to 1.255]

1 to 2 years 264/710 (37.2%) 0.68 0.962 [0.798 to 1.159]

Within one year 109/731 (14.9%) <0.001 0.314 [0.247 to 0.398] Undefined - - Reference a) 482 protocols were excluded from the bivariate analysis because they did not provide the study location. b) 4589 protocols were excluded from the bivariate analysis because they did not provide trial phase. c) 3 protocols were excluded from the bivariate analysis because they did not provide the study condition. d) 34 protocols were excluded from the bivariate analysis because they did not provide the funding source. e) 40 protocols were excluded from the bivariate analysis because they did not provide the target trial size. f) 891 protocols were excluded from the bivariate analysis because no impact factor was identified for the publishing journals. g) 2372 protocols were excluded from the bivariate analysis because the submission date was not identified in the trial protocols. h) Logistic regression model was not appropriate for this variable level value; hence this value was not included in the model. i) Fisher’s exact test exceeded memory limits and did not complete. Given the large N size of this table, and only 8 RCTs combined in the Sri Lanka Clinical Trials Registry (SLCTR) and the Cuban Public Registry of Clinical Trials (RPCEC), the P value of the Fisher’s exact test is expected to remain statistically significant. 5.5.3 Authors Explanation for Delayed Registration of Clinical Trials The ICMJE recommendations emphasize that in the exceptional cases of retrospective registration of clinical trials, the investigators should explicitly indicate the rationale behind the delayed registration in the paper publication [34]. The ICMJE recommendations also signify that editors of ICMJE member journals should publish a statement explaining why an exception was made to accept a publication for a retrospectively registered trial [34]. In our study, 286 trials were published in an ICMJE member journal and were registered retrospectively. To investigate the compliance of the submitting authors and the journal editors with the ICMJE recommendations, we screened the full-text content of the 286 trials publications and found that authors of only 8 (2.8%) of these trial publications did indicate an explicit explanation for the delayed registration; i.e., registering the trial retrospectively after the enrollment of the first patient has commenced. We did not identify any published statement from the editors of the publishing ICMJE member journals. The details of this analysis are provided in Appendix 25.

Based on the authors’ explanations, there seems to be a case where the authors were committed to register prospectively; however, the registration was not completed until during, or after, the enrollment due to an extended revision process with the registry or attempting to validate the feasibly of recruitment prior to registration [206,207]. We sought to measure the interval between the enrollment date and the date at which the trial was registered for the 4,205 retrospectively registered trials in our study with a weekly breakdown as shown in Figure 9 below (the source data is also provided in Appendix 23): Figure 9. Delayed Trial Registration after Enrollment (Measured in Weeks).

a) The analysis included 2,730 trials that were registered retrospectively up to 52 weeks after enrollment. There was a slightly changing trend after the 52 weeks with an average of 4 trials registration per week, hence excluded. We observed a clear trend of more trial registrations in the early weeks after enrollment. In particular, the number of trials registered retrospectively in the first three weeks after enrollment was in the top 95th percentile of the number of trials registered retrospectively at any given week up to 52 weeks after enrollment. The US FDA requires all submitted trials to be registered no later than 21 days after enrollment [61,62]. We suspect that the increased trend in retrospective trial registration within the first three weeks after enrollment may be associated with the FDA requirement for US-based trials. We compared the retrospective trial registrations within three weeks after enrollment between US-based trials and trials from other countries, with the assumption that US-based trials are more likely to acknowledge and follow the FDA requirements. We found that US-based, compared to international, trials were more likely (P<0.001) to be registered within the first three weeks after enrollment, as shown in Table 11: Table 11. Relationship of Retrospective Trial Registration within the First Three Weeks after Enrollment between US based Trials and International Trials.

Number of Trials Registered Retrospectively within the First Pearson Chi- Three Weeks after Enrollment / Number of All Trials Registered Trial Location square Test Retrospectively later than Three Weeks after Enrollment – within (P value) Groups (%)

US based Trials 178/699 (25.5%) <0.001 International Trials 633/3506 (18.1%) To conclude our analysis, we also aimed to report if the delayed registration occurred during or after the enrollment period. We measured the interval between the enrollment start date and the enrollment end date in 330 trials that were registered retrospectively in the ISRCTN registry. We chose this sample because the WHO trial registration dataset does not include the enrollment end date [163]. The enrollment start date and end date are provided in the ISRCTN trial registration dataset. We found that 70.9% of the retrospectively registered trials were registered within the enrollment interval. The Pearson Chi-square test reported statistically significant (P=0.04) differences between the enrollment interval (M=693.5, SD=910.18) and the journal’s impact factor, as shown in Table 12: Table 12. Relationship between Enrollment Interval, Journal Types, and Journals Impact Factors of Trial Registered Retrospectively in the ISRCTN Registry.

Number of Trials Registered Retrospectively within the Pearson Journal Enrollment Interval / Number of Chi-square Characteristics All Trials Registered Test Retrospectively – within Groups (P value) (%) Overall 234/330 (70.9%) Journal Type 0.142 ICMJE Member Journal 31/39 (79.5%) ICMJE Follower Journal 58/91 (63.7%) Neither 145/200 (72.5%) Journal Impact 0.04 Factors Fourth Quartile (more than 5.151) 57/72 (79.2%) Third Quartile (3.229 to 5.151) 53/71 (74.6%) Second Quartile (2.413 to 3.229) 54/77 (70.1%) First Quartile (less than 2.413) 38/66 (57.6%)

5.6 Discussion This study evaluated the compliance with the ICMJE recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals ten years after the 7th revision of the DoH in 2008 [32,180]. Our primary research objectives were to examine compliance with the ICMJE recommendation in reporting the TRN in the manuscript of

published trials as well as in prospective registration of RCTs across all the seventeen WHO trial registries. 5.6.1 Reporting of TRN

Our study showed higher overall compliance (71.2%) with reporting the TRN in published trial papers compared to seven other studies reporting a compliance range from 26% to 68% [64,65,75,86-89]. As indicated in Appendix 26, these five studies had a limited scope, such as a focus on a specific condition or a group of journals, with a substantially smaller size: between 54 and 317 trials included only. We also found that the compliance with the inclusion of TRN in published RCTs was higher (95.6%) for RCT papers published in ICMJE member journals, compared to (76.3%) and (66%) for RCT papers published in ICMJE following journals and other journals respectively. Our results align with the Huser et al. study that reported nearly the same compliance (95.8%) with the inclusion of TRN in 698 papers of trials that were registered in the ClinicalTrials.gov registry and were published in a purposive set of five ICMJE journals [84]. A unique and valid TRN was identified in the PubMed metadata for 2,504 (34.4%), in the paper abstract for 1,218 (16.7%), and otherwise in the paper full-text manuscript for 3,559 (48.9%) of the 7,281 unique trials. We emphasize the existing ICMJE and WHO recommendations to include the TRN at the end of the paper abstract [33,34,181]. Surfacing the TRN in the paper abstract is important for systematic reviews and meta-analysis, as future research could automate the screening and pre-identification of the underpinning trial protocols without the need to skim over the full-text manuscript. 5.6.2 Prospective Registration of Clinical Trials

The WHO registry criteria indicate that the seventeen member registries should enter clinical trials into their databases prospectively (that is, before the first participant is recruited) [74]. We reported a 41.7% compliance in prospective registration of our included unique 7,218 randomized trials with valid registration and enrollment dates. Our results are different from other studies that reported the compliance in prospective trial registration to be between 3.6% and 77% [64-73,75-85]. As indicated in Appendix 26, these twenty-one studies had a limited scope, such as a focus on a specific condition, a group of journals, or a specific trial registry. The Pearson Chi-square test results and the binary logistic regression test results indicated significant relationships (P<0.05) between the prospective registration of clinical trials and the

trial registry, region, condition, funding, trial size, interval between paper registration and submission dates, impact factor, and the ICMJE membership of the publishing journal. The European Clinical Trials Register (EU-CTR) reported the highest compliance with prospective trial registration at 93.1%. Trials that were registered in the EU-CTR were 10.8 times more likely to be registered prospectively before the enrollment date (P<0.001, OR=10.775, 95% CI: 3.629-31.993) compared to trials registered in other WHO trial registries. The EU-CTR has a two-step registration process with the investigator initiating the registration process and the member state regulatory authority to complete and authorize the trial with details from the respective ethics committees [208]. Furthermore, the EU-CTR provides a detailed explanation of this registration and authorization process and how it may result in a seemingly delayed registration date due to its technical implementation—over which the investigators have no control—although their registration should still be validated as a prospective registration in accordance with the ICMJE guidelines [209]. This level of rigor and coordination between the EU-CTR registry and the member state authorities may have contributed to the significantly higher compliance in prospective trial registration compared to the rest of the WHO trial registries. In fact, the authors of a publication of a retrospectively registered trial included in our study reported that after their initial trial registration at the EU-CTR was declined, they were able to register in ClinicalTrials.gov successfully [210]. This process may also explain the tendency for European trials not to register in the EU-CTR, as we identified 2086 European trials, of which only 87 were registered in the EU-CTR. Given the scale, investment and impact of large and global trials, investigators would likely practice more diligence in the design phase of these trials. Global trials, i.e., trials with study locations across multiple countries, are 2.9 times more likely to be registered prospectively (P<0.001, OR=2.892, 95% CI: 2.118-3.947). The bivariate analysis reported a statistically significant (P<0.05) relationship between trial size and prospective trial registration with the highest compliance (53.1%) in prospective registration reported for larger trials, i.e., trials with a top quartile trial size). However, our multivariate analysis reported that the top quartile trial size is not a predictor for prospective trial registration. We observed a wide range of compliance in prospective registration between different study conditions, from oral health trials at 15.2% to addiction trials at 53%. We also found that mental health trials constitute the largest condition group, with 986 trials at 41.8% compliance. Addiction interventions face unique recruitment and retention challenges, such as staff

skepticism of the research efficacy, lack of patient motivation, and fear of randomization into a placebo group [211,212]. We postulate that these recruitment challenges may lead to increased awareness and thoughtfulness of the investigators in adhering to best practices during the design phase of addiction RCTs. This postulate may explain why investigators of addiction RCTs are more inclined to comply with the ICMJE recommendation and register their trials prospectively. Another 2015 oral health trials study reported 9% compliance of prospective trial registration in fifteen high impact factor journals [64]. None of the fifteen journals was an ICMJE journal, and only 2 were ICMJE follower journals. As such, none of the 191 oral health trials in our study were published in an ICMJE journal (61 published in ICMJE follower journals and 130 were published in journals that are not ICMJE or ICMJE follower journals). We also confirm that none of the 14 ICMJE member journals is focused on dentistry or oral health studies. The lack of ICMJE member journals and the under-representation of ICMJE follower journals in the cohort of journals where oral health trials are published may explain the low compliance in prospective registration of oral health trials. The odds of prospective trial registration showed a trend towards significance for industry- funded trials with 2.5 times greater likelihood to be registered prospectively before enrollment (P=0.058, OR=2.512, 95% CI: 0.970-6.50); however, the trend did not reach statistical significance. Aligned with findings from similar studies, our finding suggests that investigators of industry-funded trials are more adherent to the ICMJE trial registration recommendations [145,167,171]. Perhaps, this adherence is driven by editors and peer-reviewers of scholarly journals exercising more rigor when reviewing paper submissions of industry-funded trials, as the funding source may warrant a bias to favorable results. 5.6.3 Compliance within the ICMJE and High Impact Factor Journals

The Pearson Chi-square test and the binary logistic regression test reported statistically significant (P<0.05) differences in compliance with reporting the TRN and with prospective trial registration in trial papers published in ICMJE member journals at 95.6% and 61.6% respectively. A manuscript published in an ICMJE member journal was 5.9 times more likely to include the TRN (P<0.001, OR=5.934, 95% CI: 4.144-8.496), and a published trial was 1.6 times more likely to be registered prospectively (P<0.001, OR=1.599, 95% CI: 1.309-1.953) when published in an ICMJE member journal compared to other journals. The high compliance in reporting TRN is aligned with two other studies from 2012 and 2013 reporting a 97% and 95.8% compliance with including the TRN in papers published in ICMJE 84

member journal [84,87]. Our reported compliance in prospective registration of trials published in ICMJE member journals was lower than two other studies reporting the compliance at 77% and 72% [82,84]. Compared to our study, these two studies were different in size and scope and allowed for a grace period in the prospective registration of RCTs. We included over ten times more trials in our study (7,218 versus 486 and 698, respectively) [82,84]. The Gopal et al. study was limited to RCTs published in the ten highest-impact US medical specialty society journals and considered a trial registration prospective if registered up to one month after the enrollment date of the first participant [82]. The Huser et al. study was limited to RCTs published in five ICMJE journals and considered a trial registration prospective if registered up to two months after the enrollment date of the first participant [84]. These differences may have contributed to the lower compliance rate in prospective trial registration in our study. Given the advocacy of the ICMJE community in promoting the compliance in reporting TRN and prospective trial registration, the higher compliance within its group of journals would not be surprising but would nevertheless be reassuring to set the path for the rest of the scholarly journals to follow suit. Compared to trial papers published in ICMJE member journals, we reported lower compliance in reporting the TRN and in prospective trial registration in trial papers published in ICMJE- following journals at 76.6% and 41.4% respectively. Our finding is aligned with a 2014 study that surveyed editors of ICMJE-following journals and showed that at least half of these journals did not adhere to the ICMJE recommendation and that 67% of editors of these journals would consider the publication of retrospectively registered trials [213]. The same study also reported that only 18% of surveyed journal editors crosschecked submitted papers against registered trial protocols to identify any discrepancies, which may justify the identified 109 trial publications in our study with a reported TRN, for which we could not locate an actual trial protocol in the respective trial registry. The Pearson Chi-square test and the binary logistic regression test reported statistically significant (P<0.05) differences in compliance with reporting the TRN and with prospective trial registration in trial papers published in high impact journals at 88.9% and 54.3% respectively. Trials with papers published in a journal with a top quartile impact factor were 1.3 times more likely to be registered prospectively (P=0.02, OR=1.273, 95% CI: 1.044-1.553), and the published papers are 3.6 times more likely to include the TRN (P<0.001, OR=3.617, 95% CI: 3.040-4.304). Our finding could be explained by the rigorous revision process of journals with

higher impact factors, but also by findings from a 2011 study signifying that journals with higher impact factors are more likely (P=0.015) to provide editorial advice and recommendations to necessitate trial registration in the author instructions [80]. 5.6.4 Registration Bias (Selective Registration Bias)

The adoption of the ICMJE recommendations for trial registration among editors of medical journals may have inaugurated another form of bias within the researcher community of clinical trials. We hypothesize that investigators of clinical trials may be biased to selectively register their trials when intending to submit the trial paper for publication. The retrospective and late registration would increase the acceptance chances of the submitted manuscript by the journal editors and peer-reviewers. We aimed to examine the relationship between the timely trial registration and the interval between the trial registration date and the date at which the trial paper was submitted for publication. Our Pearson Chi-square test and binary logistic regression test showed a statistically significant relationship (P<0.001) between retrospective trial registration and the register-to- submit interval, with 14.9% of the trials registered within one year from submitting the trial paper for journal publication were registered prospectively. This result shows that 85.1% of the trials registered within one year from submitting the trial manuscript for journal publication were registered retrospectively, for which we coined a new term: “Registration Bias” or “Selective Registration Bias.” 5.6.5 Authors’ Explanation for Delayed Registration of Clinical Trials

The ICMJE recommendations signify that submitting authors of retrospectively registered trials as well as editors of publishing journals should include a statement in the publication explaining the rationale behind the delayed registration and the editors’ justification in accepting the submission of a retrospectively registered trial [34]. We conducted a cross-sectional analysis for the 286 trials that were registered retrospectively and published in an ICMJE member journal, as we expected higher compliance within the group of ICMJE member journals. We did not find any editor statement in the 286 papers. Only 8 authors included a statement explaining the reasons behind the delayed registration of the published trials, as described in Appendix 25. In summary, the reasons for delayed registration were lack of awareness or error of omission in five papers, registration process took longer time than planned in two papers, and the authors of one

paper intended to test the feasibility of recruitment prior to registration [95,96,206,207,210,214- 216]. Two of the eight studies emphasized that the registration occurred during the enrollment process and before the collection of any outcome data [206,207]. With this notion, we thought to challenge the ICMJE definition of prospective registration to be, perhaps, during enrollment and not strictly before enrollment. In the scoping analysis described in Appendix 26, we identified four other studies that reported on prospective trial registration and considered a trial registration as prospective up to two months after enrollment, and in one study even up to the date of primary endpoint ascertainment [82-85]. The ICMJE recommendation does not allow for any grace period for prospective registration (i.e., considering a trial registration prospective if the trial was registered after enrollment within a defined period). Allowing a grace period for prospective registration after enrollment is also indicated in the recommendations of the US FDA and the UK NHS through its health research authority that requires trial registration no later than three and six weeks respectively after enrollment [61-63]. To help determine what the right grace period might be—or if there should be any—we analyzed the interval from enrollment to registration of the 2,730 trials registered up to 52 weeks after enrollment in our study as depicted in Figure 2 and Appendix 23. We observed a clear trend of more trial registration in the early weeks after enrollment. The number of trials registered retrospectively in the first three weeks after enrollment was in the top 95th percentile of the number of trials registered retrospectively at any given week up to 52 weeks after enrollment. There was a sharp drop in the two-period moving average trend line that started at the third-week mark until the eighth-week mark after enrollment. We postulated that this trend of higher trial registration rate in the first three weeks after enrollment might be explained by the three-week grace period requirement for a clinical trial registration to be considered by the FDA [61,62]. To support our postulate, we found that US-based trials were more likely (P<0.001) to be registered within the first three weeks after enrollment compared to international trials, as shown in Table 11 (with the assumption that US-based trials would adhere more to the FDA requirements). As such, the trend of higher trial registration continued until the eighth week after enrollment with 1443 (34.3%) of all retrospectively registered trials registered within the first eight weeks, nearly two months, after enrollment. We suspect that this eight-week delay may be driven by protocol registration, revision, and ethics approvals in multi-site trials, for example when the trial

is approved, and enrollment commenced in one site while awaiting approval for other sites before being able to register the trial protocol. These delays are likely to be beyond the control of the investigator [209,206]. Therefore, our findings would support the notion of perhaps including a grace period, between three weeks up to two months, to the existing ICMJE definition of prospective trial registration. We wished to augment our analysis with validation of whether the late registration of the retrospective registered trials occurred within the enrollment period (i.e., if the registration date was before the enrollment end date). We analyzed 330 trials that were registered retrospectively in the ISRCTN registry. We found that 70.9% of these trials were registered before the end of the enrollment date and are more likely (P=0.04) to be published in a high impact journal. However, we could not validate if the trials were registered before any primary outcome data is collected. The primary outcome data collection start date is not a designated field in either the WHO trial registration dataset or the major trial registries: ClinicalTrials.gov and ISRCTN [163]. This date is only provided voluntarily inline within the textual description of the primary outcome field. With the large size of 4,205 retrospective trials included in our study, such analysis would be extensive and not feasible with our limited resources. If the investigator registers the trial after the primary outcome data collection has started, the investigator would have the chance to adjust the trial protocol based on the preliminary finding of the collected data. However, registering the trial prior to the primary outcome data collection would eliminate this type of bias. Perhaps, the definition of prospective trial registration based on registering the trial before primary outcome data collection would be more meaningful and applicable to the field. However, we foresee a few challenges with this definition: (1) it requires schema modifications to the WHO trial registration data set to include a dedicated and mandatory start date field for primary outcome data collection, (2) the primary outcome measure may not be captured within a predefined timepoint for all study subjects—for example, in surgery-related trials, or in trials with outcomes measured by a frequency of relapses or episodes, and (3) it does not preclude the bias for protocol modification after enrollment that may be driven by other factors, such as feasibly of recruitment, resources, and pre-intervention baseline measures [82,163]. 5.7 Implications for Editors, Investigators and Policy Makers Mitigating bias in clinical trial design through timely trial registration is important and should be endorsed by all the stakeholders within the research enterprise. Supported by our findings, we

propose a few recommendations to the ICMJE, journal editors, WHO, ethics committees, trial investigators, and authors, with the objective to aid the broader adoption of prospective trial registration and lay the groundwork for future compliance studies to measure the impact and adherence with the timely trial registration. 5.7.1 Recommendations to the ICMJE

Redefining the prospective registration for a clinical trial to include a grace period after enrollment is compelling based on the findings of our study. The grace period could be between three weeks (as per the FDA requirement), six weeks (as per the NHS requirement), or up to two months based on our findings. 5.7.2 Recommendations to Journal Editors

It may be possible to decrease the “Registration Bias” or “Selective Registration Bias" if journal editors provided a statement that includes the definition, merits, and the requirement of prospective trial registration, as well as the inclusion of the TRN in the instructions for authors within their journals. Editors and peer-reviewers should promote more compliance with these requirements, validate submissions of trial papers with the underpinning trial protocols to confirm the timely registration and validity of the TRN. If the submission was otherwise not compliant, the editors should adhere to the ICMJE recommendations and provide a statement justifying their editorial decision. System developers of publishing journals should also leverage the PubMed and Medline designated metadata for disclosing the clinical trial registration information, namely the ”DatabankName” for the trial registry name and the “AccessionNumber” for the TRN [205]. 5.7.3 Recommendations to the WHO

The WHO ICTRP portal has come a long way to promote the global adoption of clinical trial registration. To continue that journey, we propose that it is advisable to provide a publicly accessible Application Programmable Interface (also known as RESTFul APIs) to look up and validate any given TRN programmatically. This will help publishing journals to automate the validation of the TRN as part of the paper submission process. To provide a critical appraisal for the timely registration of clinical trials, the WHO should consider adding the following two data fields to the existing trial registration dataset [163]: the end date for enrollment and dedicated date fields to capture the start and end dates of the assessment of the primary outcome endpoints. The primary outcome timepoints are currently

embedded within the text field, including all other primary outcome information, which causes the abstraction process to be very manual and error-prone. We also noticed a limited search capability on the ICTRP portal. Based on our experience, any search with an expected result of more than 50 trials would fail on the portal. When the search is for less than 50 trials, the portal took on average 25 seconds to provide the results, and another 20 seconds to download the results in XML format. This is not the case with other major global registries, such as the ClinicalTrial.gov and ISRCTN portal, where the search could include up to a few hundred trials within 3 seconds. We acknowledge the WHO effort to provide a crawling option, however, and given the underpinning structured database of the ICTRP portal, it will be more applicable to provide public access to the backend database or provide a full download for content analysis (similar to the user experience with ClinicalTrialso.gov) [217]. 5.7.4 Recommendations to Ethics Committees

National, academic and institutional ethics committees should promote the prospective registration of clinical trials and perhaps endorse it as a requirement for ethics approval. In September 2013, the UK Health Research Authority set the example by mandating trial registration as a condition for winning ethics approval [63]. 5.7.5 Recommendations to the Investigators and Authors

The trial registration process varies across the different trial registries and may include additional validation and confirmation steps. These steps are beyond the control of the investigators and could require a few days or weeks to complete. Therefore, investigators should further their efforts in registering their trials prospectively and account for a sufficient lead time to promptly conclude the trial registration process before the enrolment date of the first trial participant. The registration is applicable to RCTs, but also to non-randomized trials, and trials with other study designs including clinical and process outcomes. Trial registries also allow for updating the trial protocol should there be any changes after registration. Trial registries will keep a change log for all the changes and updates with the commitment to remain transparent and control potential bias by disclosing these changes with full traceability. Submitting authors should publish a statement including the TRN and the trial registry name at the end of the abstract page. The TRN should be provided in its original format; if uncertain, the authors should validate the trial number by running a search in the respective registry and confirm the format. It’s important not to omit any trailing or leading zero from the original TRN.

The adherence to these steps will improve the searchability of the published paper and the direct linkage to its trial protocol. It will also enable researchers to identify publication of registered trials in PubMed without the need to access, download, skim, or crawl the full-text manuscript, scouting for a matching TRN. 5.8 Limitations There are a few limitations to our study. Our inclusion criteria were limited to randomized trials published in PubMed indexed journal and did not consider other relevant journals that are not indexed in PubMed, such as npj Digital Medicine [218]. Because we based our trials registration on the WHO registry network, we only considered the seventeen registries to search and obtain the trial protocols. We did not report on publications of trials that may be registered in a non-WHO registry, such as the Philippine Health Research Registry and the Federal Office of Public Health’s portal for human research in Switzerland [177,219]. We could not report on trial registration of 3,027 published papers that did not include the TRN. There is a chance that some of these trials were registered and the authors were not aware of the significance of including the trial number in the published paper. A manual search for multiple thousands of potentially matching trials was not feasible with our scarce research resources. Hence, we were not able to report on the rate of non-registration of clinical trials within our study. We excluded 575 papers, as we did not have access to the full-text paper through our university online resources. Analysis of trials’ characteristics, registration, and enrollment dates was based on trial protocols that we downloaded from the respective seventeen trial registries. The protocol information is provided manually, and in many cases voluntarily, by the investigators when registering their trials. There may exist some discrepancies in the trial protocols due to entry error or lack of interest and resources on the researcher team to promptly maintain and update the protocol. These discrepancies may have impacted the internal validity of our finding. We acknowledge that these limitations may have impacted the external validity of our study results.

5.9 Conclusion

Our analysis of 10,500 papers of RCTs published in 2018 contributes a critical assessment to the adoption of the ICMJE recommendations for the prospective registration of, and the inclusion of TRN in the published trials ten years after the 7th revision of the DoH in 2008. Compliance with including the TRN in published trials was at 71.2% overall, and significantly (P<0.001) higher at 95.6% for papers published in an ICMJE member journal. The ICMJE member journals should be commended for the strong adherence to their mandate and driving the adoption within the broader community of scholarly medical journals. We found low compliance with prospective trial registration, as defined by the ICMJE (i.e., when the trial is registered before the enrollment of the first patient) at 41.7% overall and 61.6% for trials published in one of the ICMJE member journals. The empirically low adoption, even within the ICMJE member journals, ten years after the 7th revision of the DoH, questions the adaptability and viability of this recommendation. We detected a new form of bias, to which we refer as “Registration Bias” or “Selective Registration Bias,” with a significantly (P<0.001) high proportion (85.1%) of investigators registering their trials retrospectively within one year of submitting the trial paper for journal publication. Within the cohort of trials that were registered retrospectively and published in one of the ICMJE member journals, we measured the compliance with publishing a statement justifying the late registration by the authors and the reasons behind accepting the submission of the retrospectively registered trial by the journal editor. There was no compliance with reporting editor’s statement. The compliance with the reporting author’s statement was very low at 3%. Reasons for late registration were lack of awareness, error of omission, or the registration process took a longer time than planned. Some of the authors emphasized that the late registration was completed before any data analysis or collection. Supported by the findings of this research, we proposed several recommendations to inform the future development of publication and registration guidelines of RCTs. These recommendations were addressed to policy makers, investigators, and journal editors. 5.10 Contributors

The lead author (MAD) and senior author (JC) conceived the study idea and developed the study design with contributions from all authors. MAD executed the search queries, extracted data,

analyzed the results, and drafted the manuscript. All authors interpreted results, were involved in the reporting of the study, reviewing and editing the manuscript, and approved the final version of the manuscript. All authors agree to take responsibility for the work. MAD is the guarantor. The authors followed the recommendations of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement in the identification, screening, and inclusion of our search results. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. 5.11 Funding

None received. 5.12 Competing Interests

All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organization for the submitted work in the previous three years; no financial relationships with any organizations that might have an interest in the submitted work; no other relationships or activities that could appear to have influenced the submitted work. 5.13 Ethical Approval

Not required. 5.14 Data Sharing

The citations of the published RCT manuscripts were downloaded from the PubMed database and the respective RCT protocols were downloaded from the publicly available WHO ICTRP. The data set for the reported results of this research is available upon request. 5.15 Transparency Statement

The lead author (MAD) affirm that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained. This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms,

provided the original work is properly cited and the use is noncommercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Chapter 6 Discussion Discussion 6.1 Summary of Research Finding In this thesis, four major research questions are presented with respect to publication bias and compliance in prospective registration of RCTs. These research questions were addressed in three interconnected and subsequent studies (chapters 3-5). Within each study, a comprehensive reporting, analysis and discussion of the underlying results was provided. The focus of this section is to provide a synthesis of overarching research results that were reported across the three studies.

6.1.1 Prevalence of Publication Bias in Digital Health RCTs

Based on the findings from the first two studies (chapters 3 and 4), nearly one third of all digital health RCTs remain unpublished (27% and 34.5% of included digital health RCTs remain unpublished in the first and second study respectively). This finding is disconcerting due to (1) the loss of finite resources that were allocated to these RCTs, and (2) the outcomes and learnings from these RCTs that went unreported and could have informed other investigators and future research. In this thesis, the reported non-publication rates were the first to be published in the domain of digital health RCTs, hence no direct comparison with any similar study was possible. However, the reported non-publication rates (between 27% and 34.5% in chapters 3 and 4 respectively) are in line with another study (not specific to digital health) of large RCTs that reported a 29% non-publication rate [6]. Other studies (not specific to digital health) reported higher non-publication rates with nearly 50% of all included trials remaining unpublished [7,8,10]. I postulate that this difference (higher publication rate in digital health RCTs) may be explained by two major factors. First, the fast-pace and volume of technology development generates more opportunities for investigators to publish their results in order to become leaders in the field and stay ahead of the digital innovation curve. The second factor relates to the funding source of digital health RCTs. Results from this thesis found that only a small portion (6.8% in the first study, and 3.4% in the second study) of digital health RCTs were funded by industry sponsors. Pham et al. reported that only 7.8% of mHealth trials were funded by industry sponsors [161]. Therefore, the majority (over 90%) of digital health RCTs are sponsored by

government and academic entities, such as universities, hospitals, and medical and research centers, that are more disciplined and accountable by scholarly requirements to promptly publish their results in peer-reviewed journals. Industry sponsors and digital technology developers, on the other hand, are likely to be more driven by the scale and opportunity in the broader digital health marketplace, beyond the realm for academia, peer-review process, and the complexity of randomized trials design. Significant differences in non-publication rates were also identified between digital health RCTs in the US and international trials conducted elsewhere in the world. The non-publication rates of US based digital health RCTs were significantly higher when compared to the international rates (P<0.001 in the first study, and P=0.03 in the second study) as depicted in table 13. Table 13. Results of the Pearson Chi-square test between Non-Publication Rates and Location of Digital Health RCTs Number of Unpublished RCTs/Number of Pearson Chi- Study in this Location all RCTs within Group (Non-Publication square Thesis Rate %) Test (P value) First Study US based RCTs 111/338 (32.8%) <0.001 (Chapter 3) International RCTs 39/218 (17.9%) Second Study US based RCTs 51/120 (42.5%) 0.03 (Chapter 4) International RCTs 93/297 (31.3%) The predominantly privately funded healthcare system in the US may have contributed to the higher non-publication rates of US based digital health RCTs. As such, the scale of the digital health industry in the US compared to the rest of the world could be another factor, with US based digital health startups holding 75% of the global market shares between 2013 and 2017 [153,154]. It might be postulated that the interest of the investigators of US based digital health RCTs may be influenced by the commercial intent of the underpinning RCT (such as attracting further investments or collaboration with industry sponsors) with less focus on knowledge generation and dissemination. Findings of this thesis support linking recruitment challenges to non-publication rates in digital health RCTs. The first study (chapter 3) showed that completed RCTs were 3.3 times more likely to be published (P=0.002, OR=3.303, 95% CI: 1.564-6.976) compared to uncompleted RCTs (withdrawn, suspended, and terminated RCTs). A qualitative analysis of uncompleted RCTs showed that enrollment challenges were the most reported reasons for trial discontinuation, and therefore a major contributor to the higher non-publication rate. This finding is in line with that of another study indicating that recruitment challenges were the most frequently reported factors contributing to discontinuation of clinical trials [9]. Recruitment challenges were also validated

in the second study of this thesis (chapter 4) with 41% of the published RCTs reporting fewer subjects recruited than the target size indicated in the RCT protocol. Not only would recruitment challenges contribute to lower publication rates, but they would weaken the statistical power and internal validity of the RCT results and could introduce type II errors in reported results [179]. In the first two studies, over a half of all the digital health RCTs were categorized as web-based intervention (53% and 56% in study 1 and 2 respectively). The proportion of mobile apps interventions was low; however, it nearly tripled between the first and second study (3% and 8% respectively). This is a clear indication of an increasing adoption of mobile apps interventions in digital health RCTs. It will be interesting to see how this shift in technology trends would impact the rates of publication bias in digital health RCTs. Based on the finding of this thesis, the major technology variable was not significantly associated with the publication bias in the both studies. A current analysis may not be feasible at this point of time as RCTs may take 5 to 8 years from start to publication as discussed in Chapter 3. Hence a 2019 analysis would still include RCT started around 2012 timeframe, which is similar to our second study. The highest non-publication rates were reported for oncology studies in digital health with nearly a half remaining unpublished (45.2% and 47.4% in study 1 and 2 respectively). The differences in non-publication rates of oncology RCTs in digital health were significant different (P<.05) when compared to digital health RCTs of other conditions. While the hypothetical rationales were explained in the discussion section of the second study (Chapter 4), these findings suggest a more specific analysis is needed of potential publication challenges in digital health oncology RCTs.

6.1.2 Compliance in Prospective Registration of RCTs

The adoption of the 7th revision of the DoH in 2008 has significantly stimulated more compliance in prospective trial registration since 2008. The third study (chapter 5) found statistically significant (P=0.009) differences in prospective registration of RCTs before and after 2008 at 35.7% and 42.1% respectively. However, the overall compliance in prospective registration of RCTs remains low across all the three studies of this thesis (29.3%, 38% and 41.7% in the first, second and third study respectively). A Pearson Chi-square test in the second and third study (chapters 4 and 5) showed a significant relationship (P=0.003 and P<0.001 respectively) between prospective registration of RCTs and the funding sources. In the second study (chapter 4), the lowest compliance in prospective

registration was reported at 14% in industry-funded digital health RCTs, which may be a result of several factors. These include the following: (1) the lack of the investigators’ awareness with respect to the importance of prospective registration of RCTs, (2) in the earlier stages of industry-funded digital health RCTs, i.e. prior to enrollment, the investigators are likely to have competing priorities other than promptly registering their RCT, such as ensuring the timely launch and completion of the RCTs to remain current in a rapidly evolving digital health marketplace, and (3) the investigators may have opted to not register the RCT, and to iteratively modify the study design and outcomes to maintain the commercial intent of the digital health technology component. In contrast to the studies on digital health RCTs in chapters 3 and 4, the third study (chapter 5) reported the highest overall compliance, 57.5%, in prospective registration of industry-funded RCTs (compared to 29.3% and 14.3% in chapters 3 and 4 respectively). The differences in the design and inclusion criteria of these studies may have contributed to the differences in the reported compliance in prospective trial registration. In chapter 3, the study was limited to digital health RCTs registered in one registry, ClinicalTrials.gov, where 142 (25.5%) of the included RCTs were registered before 2008, i.e. before the adoption of the 7th revision of the DoH. The results of that study reported a significant decline in prospective registration after 2008 that may be explained by the general emphasis on trial registration after 2008 [166]. Compared to chapter 3, the study in chapter 4 included a more diverse and globally representative sample of digital health RCTs registered in eleven different trial registries and started enrollment of their participant in 2012, i.e. 4 years after the DoH. The low representation of industry-funded digital health RCTs in the first two studies may be explained by the interest of the industry sponsors in the broader opportunity in the digital health marketplace beyond the realm of academia and best practices of randomized trials design. The study in chapter 5 was not limited to digital health RCTs only and included a larger sample of 934 industry funded RCTs. In chapter 5, the multivariate analysis of prospective registration of RCTs suggested that industry-funded RCTs are 2.5 times more likely to be registered prospectively (P=0.058, OR=2.512, 95% CI: 0.970-6.50); however, the trend did not reach statistical significance. As with similar studies, these findings suggest that investigators of industry-funded trials are more adherent to the ICMJE trial registration recommendations [145,167,171]. Consistent and significant differences between compliance in prospective registration of RCTs registered across different trial registries were observed. Findings from the second and third

study (chapters 4 and 5) showed the highest compliance in prospective registration was for RCTs registered in the EU Clinical Trials Register (EU-CTR) followed by the ANZCTR. RCTs registered in the EU-CTR registry were 10.8 times more likely to be registered prospectively before the enrollment date (P<0.001, OR=10.775, 95% CI: 3.629-31.993). These differences suggest a varying level of rigor with adhering to best practices in trial registration among the seventeen WHO trial registries. Kock et al. has indicated that after failing to register their RCTs in the EU-CTR registry, they were able to successfully register their RCT in ClinicalTrials.gov registry retrospectively [210]. It is unclear why the registration of that particular RCT, and perhaps many other RCTs, was denied in one WHO registry and was accepted in another WHO registry. There may be a case for the WHO registries to consider the disclosure of the reasons behind declining such registrations. A public record of all submitted and not accepted trial registrations would be helpful to (1) inform prospective investigators and evidence-based researchers of potential trial registration challenges, to (2) inform policy makers and ethics committees to appraise the feasibility of existing trial registration best practices, and to (3) ultimately promote higher adherence with best practices in clinical research.

6.1.3 Compliance in Reporting the TRN within Published RCTs

In 2015, the WHO introduced a new statement signifying the importance of including the TRN in the abstracts of the published trials to enable linking the trial publication with trial protocol in the trial registry [33,181]. The ICMJE recommends that journals publish the TRN at the end of the abstract [34]. The PubMed metadata provides two dedicated data elements for disclosing the TRN as well as the trial registry, “AccessionNumber” and “DatabankName” respectively [205]. PubMed also provisioned a publicly available web-service (e-Utils) “http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi? db=pubmed&term=TRN” that allows for automated lookup of TRN within any field of the PubMed metadata, including the abstract and the “AccessionNumber”. The compliance in reporting the TRN in published RCTs was low across all three studies of this thesis; 49%, 52% and 71.2% in chapters 3, 4, and 5, respectively. This finding is aligned with several other studies indicating a compliance range from 26% to 68% [64,65,75,86-89]. The inclusion of the TRN in the published papers of RCTs is important to establish the linkage to underlying RCT protocols. Future research, such as systematic reviews and meta-analysis, would

benefit immensely from this linkage. It enables a high degree of automation in the processes of screening and matching published RCTs to their registered protocols. In the third study (chapter 5), a specific analysis on the inclusion of TRN in published RCTs was provided. The objective was to measure the compliance in reporting the TRN in the paper abstracts and the PubMed designated fields as per the WHO and ICMJE recommendations. This analysis showed that the TRN was neither included in the PubMed designated field nor in the PubMed abstract for half of the 7,569 published RCTs with included TRN (15.4% in PubMed designated fields, 34.5% in the PubMed abstract, and 50% elsewhere in the manuscript). This finding has implications on the design and feasibility of future evidence-based research. It is best to include the TRN in the PubMed designated field as it will enable the automation of identifying the TRN from the PubMed metadata directly without the need for any further analysis. This is a joint responsibility for the submitting authors and the journal editors. Irrespective of this integration with the PubMed designated field, the authors of RCT papers should always include the TRN in the abstract. The PubMed abstract is available as a text field in the PubMed metadata, which could be downloaded directly from PubMed without the need to access the full-text manuscript. Also, the PubMed abstract is indexed and searchable through the publicly available PubMed e-Utils API as a web service, which enables the online search and identification of RCT publications by their TRN. When authors of RCT papers include the TRN elsewhere in the manuscript, other than in the abstract, they introduce challenges for future research. These challenges are described in a later section in this chapter, “6.3.3 Challenges in Identifying the TRN from the Full-Text RCT ”.

6.2 Conceptual Contribution

6.2.1 Articulating a New Form of Bias in RCTs (Registration Bias)

First published in May 1994, the Cochrane Handbook for Systematic Reviews of Interventions presented seven major types of bias in the dissemination of research findings "Publication bias, Time lag bias, Multiple (Duplicate) Publication Bias, Location bias, Citation bias, Language bias, and Outcome reporting bias" [222,223]. In RCTs, the randomization between the experimental groups and the control groups was regarded as the gold-standard in research design as it would be less susceptible to bias than other study designs. However, there exists several biases inherit to RCTs such as the lack of intention-to-treat analysis and the poor allocation

100

concealment [261]. A number of studies provided a critical analysis of non-publication bias and selective reporting bias with a focus on RCTs and the potential of trial registration to mitigate selective reporting bias [24,224]. The ICMJE, WHO and the 7th revision of the DoH recommended that RCTs should be registered prospectively (before the enrollment of the first participant) [28,31,32]. Prospective registration of RCTs will mitigate the bias associated with preferential modification of pre-defined trial outcome measures due to preliminary analysis of the trial results [29,30]. This thesis offers a critical contribution with respect to the field of publication bias in RCTs. This contribution could inform the development of registration and publication guidelines by the Cochrane Review Groups, the ICMJE and the WHO. Specifically, a new form of bias in the registration of RCTs (I refer to this as “Registration Bias” or “Selective Registration Bias”) was detected. This bias is defined by the tendency of investigators to selectively register their trials only when intending to publish the manuscript in a peer-reviewed journal. Findings from the second and the third study of this thesis (chapters 4 and 5) consistently reported a significant (P<0.001) relationship between the retrospective registration of RCTs and the interval between the RCT registration date and the date when the RCT manuscript was submitted for journal publication. The majority of RCTs (95.7% in chapter 4 and 85.1% in chapter 5), that were registered retrospectively, were registered within one year before submitting the RCT manuscript for journal publication. The selective registration bias is likely a result of several factors. These include: (1) the investigators may not be aware of the ICMJE and WHO recommendation to register the trial promptly before enrollment, (2) with the finite research resources available to investigators, and the amount of operational and administrative activities involved in the early stages of RCTs, the trial registration may become a low priority task that is often deferred until a later point of time, (3) the desire of the investigators to iterate on, and modify the RCT design, informed by ongoing recruitment feasibility and preliminary findings of study outcomes, and (4) the retrospective registration of an RCT (nearing the submission date of an RCT manuscript for publication) would improve the likelihood of acceptance of the submitted manuscript. Findings of the qualitative analysis of eight papers of retrospectively registered RCTs in the third study of this thesis (chapter 5) support the contributing factors to retrospective trial registration. However, these findings are of limited generalizability as only a small portion (2.8%) of authors

101

of retrospectively registered RCTs published in an ICMJE journal provided their explanation of delayed registration as per the ICMJE recommendation [34]. To mitigate the risk of selective registration of RCTs, the following recommendations are proposed: (1) ethics boards of research institutions should mandate the registration of an RCT prior to approving its research protocol; (2) the WHO and the trial registries should introduce a new mandatory field in the trial registration dataset to capture the reasons of delayed registration in the case when the trial is registered after the enrollment date; (3) editors of peer-reviewed journals should clearly indicate, and adhere to, the ICMJE recommendations for prospective trial registration in the “Instructions for Authors” section in their journal guidelines; (4) in cases where the editors grant an exemption for an exceptional manuscript of a retrospectively registered RCT, the editors should require the authors to publish a statement explaining the rationale behind the late registration; and 5) the editors should also publish a statement indicating why an exception was made to publish a retrospectively registered RCT.

6.2.2 Extending the Definition of Prospective Registration

The reporting of prospective registration of RCTs across all studies of this thesis was based on the strict definition of registering the RCT on or before the enrollment of the first participant. This definition was endorsed by the ICMJE, WHO and the 7th revision of the DoH recommendations for prospective trial registration [28,31,32]. However, the US FDA and the UK NHS allow for a grace period for trial registration up to three and six weeks after enrollment respectively [61-63]. Other studies that have reported on prospective trial registration, considered a trial registration as prospective up to two months after enrollment, and in one study as late as the date of primary end point ascertainment [82-85]. Even ten years after the 7th revision of the DoH, the compliance in prospective registration of RCTs on or before enrollment remains to be low across all three studies of this thesis (29.3%, 38% and 41.7% in the chapters 3, 4 and 5 respectively). To date, fourteen years after the 2004 ICMJE recommendations for prospective trial registration, the compliance in prospective registration of RCTs published in ICMJE member journals remains to be low at 61.6% as

102

reported in the third study of this thesis (chapter 5). The empirically low compliance in the adoption of prospective trial registration, defined as registration on or before enrollment, questions the practicality. This thesis supports an extension of the existing definition of the prospective registration to allow for a defined grace period after enrollment. Findings from the third study (chapter 5) presented higher rates of RCT registrations in the first three to eight weeks after enrollment. The higher rates in registering the RCTs just a few weeks after enrollment may be explained by the adherence of the investigators to the prospective trial registration guidelines of the US FDA, or the UK NHS, that allow for a few weeks of grace period [61-63]. It may also be influenced by administrative or operational challenges that caused the completion of the trial registration to fall short of the enrollment date in some of the WHO trial registries. For example, the EU-CTR has a two-step registration process with the investigators initiating the registration process and the member state regulatory authority completing the registration process with details from the respective ethics committees. This administrative procedure may result in a seemingly delayed registration date - due to its technical implementation for which the investigators have no control, and their registration should still be validated as prospective registration in accordance with the ICMJE guidelines [208,209]. The Clinical Trials Registry - India (CTRI) has a similar process where submitted trials are reviewed by CTRI and required modifications and/or clarifications are subsequently addressed by the investigators [225]. After this review process is completed, a third-party validation process is required which includes contacting the trial site investigator, trial contact person (scientific and public queries) and the overall trial coordinator (if applicable) via email or telephone. The trial registration is only completed after a response is received from at least one investigator [226]. Some academic and research institutions have implemented an internal trial registration system which introduced further delays in the registration process. For example, the University of Waterloo in Canada requires all investigators to initiate the trial registration in the university's internal Protocol Registration and Results System within up to three weeks after recruitment of the first participant. An internal revision process will then commence, and when completed, the trial registration will be submitted to ClinicalTrials.gov and become publicly available within two to five days [227].

103

6.3 Methodological Contribution

6.3.1 Challenges and Opportunities in Identifying Unpublished RCTs

ClinicalTrials.gov provides a section within the trial registration record to include bibliographic citations of trial results publications. In theory, this section could be very useful to establish the linkage between the registered trial protocol and the trial results publications. However, findings from the first two studies of this thesis (chapters 3 and 4) substantiated that only a small portion of trial registration records in ClinilcalTrials.gov included citations to publications of trial results (4.9% and 2% respectively). In some cases, the citations in the trial registration records were referring to publications other than for the results of the trial that was registered. In such cases, these publications were cited because they were similar studies, or other related studies published by the investigators. For example, a trial registered in ClinicalTrials.gov (NCT00311948) has 99 citations listed in the trial registration record, but none of these refer to the actual publication of the trial. Therefore, a researcher would need to conduct a bibliographic search to identify potential RCT publications. Searching by the TRN in PubMed would be very helpful, however it is not sufficient, as I found the inclusion of TRN in the published RCTs to be at 49% and 52% of all included RCTs in chapters 3 and 4, respectively. The rate of inclusion of the TRN in PubMed abstracts and metadata of all included RCTs was at 36% in the third study of this thesis (chapter 5). Therefore, a manual search would be necessary to identify potential RCT publications. In essence, as indicated in the methods of the first two studies of this thesis (chapters 3 and 4), the RCT registration information, such as the RCT title, registration date and the name of the investigator(s), were searched in PubMed and other bibliographic databases. Once the publication(s) were identified, I verified if these publications reported at least one of the primary outcomes as indicated in the RCT registered protocol. This process, of identifying and validating publications of registered RCTs with reported primary outcomes, is onerous and prone to error. Some authors publish more than one paper pertaining to a single RCT. For example, in the context of one registered RCT, the investigator could publish a primary paper reporting on the primary outcomes of the RCT, a design paper, and a feasibility or secondary analysis paper [228-230]. Other matching publications for a registered RCT could also include erratum [231,232], editorial [233,234], and protocol papers [235,236]. The title of the RCT publication is likely not to be an exact match to the title of the RCT in the registry. Therefore, a text search based on the exact title would not be conclusive and subject to the

104

probabilistic (fuzzy) search algorithm of the search engine used in the bibliographic database. For example, the RCT (NCT03298919) that was registered in ClinicalTrials.gov with the title “Efficacy of Exercise Videogames for Physical Activity Adoption and Maintenance” was published with this title “Exercise Videogames, Physical Activity, and Health: Wii Heart Fitness: A Randomized Clinical Trial.” [229]. Some investigators register and conduct multiple RCTs in the same field, with seemingly similar titles but different design, methodology and outcomes [231,237]. It is also possible that the investigator of the RCT, enlisted in the trial registry, may not be the first author, or even a co-author, in the publications [229,230]. Some RCTs are registered with an entity name, not a person name, as the investigator (NCT00699374), which then requires additional diligence in identifying the potential publication without being able to match any of the paper authors to the investigator of the RCT. Because of these challenges, identifying RCT publications based on the registration information would be impractical, particularly for a larger scale empirical analysis of publication bias in hundreds or thousands of RCTs. The design and methodology of the first two studies of this thesis (chapters 3 and 4) provide a pragmatic approach with some level of automation to identify potential publications of registered RCTs, and to help inform the feasibly of future research of publication bias in RCTs. This methodology is summarized in the following three major steps as depicted in Figure 10. Figure 10. Identifying Unpublished Trials.

Step I Step II Step III TRN Search in Publication Pragmatical PubMed Search in the Search Trial Registries

105

Step I – TRN Search in PubMed: Identify potential RCT publications through a TRN search in PubMed, utilizing the publicly available PubMed e-Utils API (Web Service). This step can be automated and run with a scripted computer program. Step II – Publication Search in the Trial Registries: If no publication is identified, refer to the RCT publications (if) referenced in the trial registration protocol. Step III – Pragmatical Search: If no publication identified, conduct a pragmatic search in bibliographic databases. Initiate this search by screening papers that were authored by the investigator(s) of the RCT and published after the registration date of the RCT. The search query could be augmented by including key terms from the RCT title in the registry. For example, key terms from the RCT “Efficacy of Exercise Videogames for Physical Activity Adoption and Maintenance” (NCT03298919) would be “Exercise”, “Physical Activity” and “Videogames”. Acknowledging the subjectivity in determining these terms, including unique and less common terms could be more conclusive in identifying matching publications. For example, the specificity of the term “Videogames” is likely to be higher than “Exercise” or “Physical Activity” within the context of PubMed. Trial registries can play an important role in aiding the feasibility of identifying potential publications of the registered RCTs. The ClinicalTrials.gov registry now indexes all PubMed publications of a given RCT based on the TRN [238], for example RCT (NCT03298919). The ISRCTN trial registry includes two additional data fields “Publication and dissemination plan” and “Intention to publish date” that could be very helpful in determining when to expect the publication of the registered RCT [156]. The “Intention to publish date” in the ISRCTN registry could be useful in future analysis of publication bias. For example, if a researcher is currently (2019) examining the rate of publication bias of RCTs, and no results publication were identified for a few RCTs of the included sample. Before identifying these RCTs as unpublished, the researcher should verify if the “Intention to publish date” has expired, i.e. before 2019. Otherwise, these RCTs, with no identified results publication, should not be considered unpublished as they were planned to be published at a later point of time after 2019. These enhancements to extend the trial registration dataset to (1) include a planned date for publication, and to (2) automatically index matching publications of registered RCTs by their TRN in PubMed, should be mandated and implemented across all the WHO trial registries. Advancement in machine learning, artificial intelligence, and natural language processing techniques have the potential to revolutionize the identification process of published trials. Early

106

strides in this field have shown encouraging indicators for success in adopting these techniques for classifying RCT, versus non-RCT publications [239,240]. The use case for machine learning to identify a potential publication of a given RCT would be (1) predicting the degree of similarities between the RCT title in the registry and the publication title, (2) classifying primary RCT papers with reported outcomes and secondary RCT papers such as protocol, design, feasibility and editorial papers, and (3) assessing selective reporting bias in identified published RCT by predicting the degree of similarity between the description of the RCT primary outcomes as registered in the trial registry versus the primary outcome reported in the publication. The machine learning results will always be probabilistic, and not deterministic, with a certain level of confidence in the matching publication. Trial registries could implement these techniques and provide predicted publications for the registered RCT with disclosing the confidence score of the matching algorithm to inform the reader, or researcher, accordingly. Wallace et al. introduced a unique hybrid approach to provide a more accurate RCT identification strategy of 158,000 Embase publications by combining machine learning techniques and crowdsourcing, http://crowd.cochrane.org [241]. Perhaps the Wallace et al. strategy could be extended to help identify potential publications of registered RCTs. Hypothetically, trial registries could require an investigator of an RCT to review a reasonable number, perhaps a dozen, of algorithmically pre-identified publications of RCTs as a prerequisite to complete the RCT registration. Likewise, editors of peer-reviewed journals could also implement the same strategy as a prerequisite to publishing an RCT. Not only will this practice help improve the overall linkage between RCTs registration and publications, but it would also raise the awareness, among investigators of RCTs, about the significance of linking their RCT registration record to the RCT publication.

6.3.2 Automation Strategy for Identifying the TRN in Published RCTs

A semi-automated process was developed to establish the linkage between published RCTs and their respective publications. This process is repeatable and could be utilized in future research with the objective to extract TRN of RCTs, that were registered in any of the 17 WHO trial registries, and published in any PubMed indexed journals. This process is valuable to automate many of the steps involved in identifying the TRN in published RCTs, and therefore, to aid the feasibility of conducting empirical reviews and meta-analysis of measuring the compliance, or variance, between published and registered RCTs, such as publication bias, registration bias,

107

recruitment compliance, funding bias and selective outcome reporting. In summary, this process consisted of 5 major steps as depicted in Figure 11. Figure 11. Identifying TRN in Published RCTs

Step I Step II Step III Step IV Step V Identify RCT Extract the Extract the Extract the Identify Publications TRN from TRN from TRN from Registration in PubMed PubMed PubMed the Full-Text Records of Metadata Abstracts Manuscripts Published RCTs

Step I – Identify RCT Publications in PubMed: Conduct a search in PubMed to identify publications of RCTs within defined inclusion criteria utilizing the advanced search functionality in the PubMed portal [182]. The results of this search can then be exported in a structured file format (XML). The XML file includes the abstracts, the designated fields for TRN and trial registry “AccessionNumber” and “DataBankName”, publishing journals, and other metadata of the matching publications [205]. Step II – Extract the TRN from PubMed Metadata: Develop a computer program (or script) to iterate through every publication item in the XML file. The script will verify if the “AccessionNumber” data element is provided and, when provided, extract the text value from the “AccessionNumber” data field. Because the “AccessionNumber” may include identifiers other than the TRN, the script should verify if the “AccessionNumber” value represents a TRN. Because every registry of the seventeen WHO trial registries has a unique TRN pattern, as described in Appendix I, the script should compare and identify if the pattern of the “AccessionNumber” value is a match for any valid TRN of the seventeen WHO trial registries. Many, if not all, of the major computer programming languages offer advanced text-pattern search techniques (RegEx), which can be utilized to validate the pattern match for the TRN. For example, the TRN in ClinicalTrials.gov always starts with the three characters “NCT” followed

108

by 8 digits number, such as (NCT03639220), and the respective RegEx pattern is “NCT[0-9] [0- 9] [0-9] [0-9] [0-9] [0-9] [0-9] [0-9]”. Step III – Extract the TRN from PubMed Abstracts: Not all TRNs of published RCTs are properly indexed in the PubMed metadata fields, “AccessionNumber”. For the remaining RCTs, with no TRN identified in Step II, The computer program was extended to search and extract matching TRN from the abstract field (text field) in the PubMed XML file. The PubMed abstract contains the original textual content of the published RCTs, in which, the ICMJE recommendations require authors of RCT papers to always include the TRN [34]. Step IV – Extract the TRN from the Full-Text Manuscripts: Not all authors follow the ICMJE recommendation. The third study (chapter 5) found that half (50%) of the authors who included the TRN in the published RCTs, did not include the TRN in the PubMed abstract or metadata, and included the TRN elsewhere in the manuscript. Therefore, for the remaining RCTs with no TRN identified in the previous steps, the full-text manuscripts need to be accessed and downloaded. As this last step, a computer program was developed to search and extract matching TRN from the full-text manuscripts. One of the major challenges in Step III and Step IV, is that the authors of published RCTs tend to report the TRN in different formats and patterns, that often deviate from the original TRN pattern as defined in the trial registry. In other cases, the TRN was reported with a typing error or unidentified TRN (not matching any TRN in the respective trial registry). A representative, not comprehensive, sample of different formats of patterns in reporting the TRN in published RCTs is shown in Table 14. Table 14. Differences in Formats and Patterns of Reported TRN in Published RCTs

The Correct RCT Paper TRN as Reported in the RCT Paper Challenges TRN Format

Different Format Maki et al.250 UMIN/2017/000027650 UMIN000027650 and Number Pegington et al.251 ISRCTN number 68576140 ISRCTN68576140 Different Format Seebacher et al.252 ISRCTN registration number = 77,829,558 ISRCTN77829558 Different Format ISRCTN website (ref: 10726798) Different and Houchen-Wolloff et al.253 ISRCTN10726798 Trial registration number 10726798 Incomplete Format Weiß et al.254 ISRCTN ID: 02335796 ISRCTN02335796 Different Format Nair et al.255 ISRCTN, no. 58128724 ISRCTN58128724 Different Format McKenna et al.256 ISRCTN (Registration number: 26302774) ISRCTN26302774 Different Format

109

Boff et al.257 NTC02455973 NCT02455973 Wrong Number Unidentified Gnanendran et al.258 ISRCTN Registry RCT 0076-3 Not a TRN Number Li et al.259 ISRCTNCT02703935 NCT02703935 Wrong Number There is no immediate solution for this type of challenge. An iterative approach with a text processing technique was developed to mitigate much of this challenge. Instead of extracting the full pattern of a TRN, I captured the lines (or sentences) in the manuscript, or abstracts, that have the acronym of the trial registry identifier and/or the prefix of the TRN. For example, identifying a sentence (or a line in the text) that contains the term “ISRCTN” or “ NCT” (with a leading space before the letter N to avoid matching terms like function, junction…etc.). By iteratively analyzing these sentences, I observed few re-occurring patterns. For example, authors often inject redundant terms and punctuation in the TRN such as “ID”, “No.”, “Number” and “:”. Accordingly, I developed a text processing script to automate many, but not all, of the required text manipulation and cleaning to produce the correct TRN. For the remaining of the cases, that were not accounted for by the automated text processing scripts, a manual revision is required to extract the correct TRN. However, these remaining cases are generally exceptional and present only a fraction of the full dataset of the included RCTs. Step V – Identify Registration Records of Published RCTs: After extracting the TRN of published RCTs, the registration protocols can be accessed and downloaded from the WHO ICTRP search portal [183]. The portal allows a search for registered trials by their TRN across the seventeen WHO trial registries, with the option to download the registration protocols in XML file format. Once downloaded, the linkage between published and registered RCTs can be established with a comprehensive dataset from PubMed and WHO trial registries combined. Minor limitations in the WHO ICTRP portal were encountered. Based on the findings from the second and third study (chapters 4 and 5), it was observed that any search with an expected result of more than fifty trials would fail on the portal. When the result set is less than fifty trials, the portal took on average 25 seconds to present the results, and another 20 seconds to download the results in XML format. This is not the case with other major global registries, such as the ClinicalTrial.gov and ISRCTN portal, that return up to hundreds of trials within only few seconds. Given the underpinning structured database behind the WHO ICTRP search portal, it will be more practical to provide public access to the backend database, or provide a full download for content analysis (similar to the user experience with ClinicalTrialso.gov) [217].

110

6.3.3 Challenges in Identifying the TRN from the Full-Text RCT Manuscripts

When authors include the TRN elsewhere in the manuscript, other than the abstract, they introduce challenges for future research. First, not all published RCTs are fully and freely accessible, and many open access papers have a delayed period before they become freely accessible [242-244]. Second, terms and conditions of many publishers obligate academic and educational institutions to block any bulk download of articles, or limit the rate of download to protect the propriety content from any malicious or web crawlers downloads [245,246]. This limitation will impede accessing and downloading a large number of trial manuscripts for text mining. Third, searching the full-text manuscript for the TRN will be limited to only verifying if a published RCT was registered, and identifying the relevant RCT registration protocol, as in the third study (chapter 5). It will not enable identifying relevant publications of registered RCTs, as in the first two studies (chapters 3 and 4). Last, there is an increased complexity in the text mining process when considering the cases of false-positive identification of valid TRNs in the full-text manuscripts. The false-positive identification of a TRN occurs when the author of a given trial includes a TRN of another trial in the manuscript, commonly in the introduction, discussion or references sections, to cite another registered trial for relevance only. For example, when considering the Cluver et al. RCT that examined whether esomeprazole could prolong pregnancy in women who have received a diagnosis of preterm preeclampsia [247]. The RCT was registered with the Pan African Clinical Trials Registry with the TRN (PACTR201504000771349) as indicated in the “Methods” section of the published manuscript. However, in the discussion section of the same manuscript, the authors pointed to three other TRNs of other trials; “Antithrombin was assessed to treat preterm preeclampsia in the PRESERVE-1 trial that enrolled 120 women from 23 tertiary hospitals in the United States over 28 months (ISRCTN23410175). There was no difference in prolongation of pregnancy or composite neonatal outcomes. Trials that have assessed serelaxin (NCT01566630), pravastatin, high doses of antithrombin, and celecoxib (NCT00442676) have been attempted, but all were terminated, perhaps because of poor recruitment.” In another example, the Klonoff et al. RCT, that evaluated the efficacy and safety of fast-acting insulin aspart versus insulin aspart used in continuous subcutaneous insulin infusion in participants with type 1 diabetes, was registered in ClinicalTrials.gov with the TRN

111

(NCT02825251) as indicated in the "Methods" section of the published manuscript [248]. However, in the discussion section of the same manuscript, the authors cited four other trials, for which four TRNs were included in the references section of that manuscript; “The prospective use of ultra-fast-acting insulin in closed-loop therapy is of great clinical interest and studies are underway.15–18 15. ClinicalTrials.gov. NCT03212950. https://clinicaltrials.gov/ct2/show/NCT03212950. Accessed November 30, 2018.

16. ClinicalTrials.gov. NCT03579615. https://clinicaltrials.gov/ct2/show/NCT03579615. Accessed November 30, 2018.

17. ClinicalTrials.gov. NCT03554486. https://clinicaltrials.gov/ct2/show/NCT03554486. Accessed November 30, 2018.

18. ClinicalTrials.gov. NCT03262116. https://clinicaltrials.gov/ct2/show/NCT03262116. Accessed November 30, 2018.” In these two examples, the identification process should only consider the first identified TRN and ignore the other identified TRNs as there were indicated for cross-referencing other trials. There exist other cases where the RCT is registered in multiple trial registries with multiple TRNs indicated in the published manuscripts. For example, the Izadi et al. RCT that examined the effect of high protein diets on the anthropometric indexes and cardiometabolic risk factors among children with excess weight [249]. The RCT was registered in Clinical trial.gov (NCT01886482) and in the Iran Clinical Trials Center (IRCT201307232839N6) as indicated in the “Ethical approval“ section of the published manuscript. The nuances and patterns demonstrated in these challenges introduce significant complexities to the automation process to identify the TRN in the manuscripts of published RCTs. The automated process needs to be augmented with a manual screening for such cases, where multiple valid TRNs are identified in a published manuscript of a given RCT.

6.4 Limitations

The limitations related to each study of this thesis were acknowledged in their respective chapters. There exist four major overarching limitations of this thesis. The first limitation is related to the scope of this thesis that was limited to RCTs only. While RCTs are regarded one of the most rigorous methods in trial design, there exist other design methods for conducting clinical research, such as quasi-experimental design, that were not included in this thesis. Mohr et al. and the mHealth Evidence Workshop have emphasized on the importance of trial designs, other than RCTs, in digital health research to remain relevant in the fast pace domain of technology innovation [262,263]. Pham et al. found that 80% of mHealth trials are RCTs however, which supports the generalizability of the finding of the first two

112

studies (Chapters 3 and 4) of this thesis despite the general limitations of the method as it applies to digital health evaluation. The second limitation is related to the analysis and reporting of non-publication bias in the first two studies (chapters 3 and 4). The findings of publication rates of registered RCTs were based on case studies of digital health RCTs. Therefore, the external validity of these findings would be limited beyond the realm of digital health RCTs. The lack of properly documented linkage between the published RCT manuscripts and the registered RCT protocols, and the considerable amount of effort involved in manually identifying unpublished RCTs were key challenges impeding a larger scale non-publication bias study. The third limitation is related to reporting the prospective registration of RCTs across all the studies of this thesis (chapters 3-5). The determination of prospective registration was based on the definition of the WHO, ICMJE and the 7th revision of the DoH, i.e. when the trial is registered on or before the enrollment date. This thesis contributes a formative assessment to the existing definition of prospective registration of RCTs with the recommendation to allow for a meaningful grace period after enrollment. However, this thesis did not provide an analysis of prospective registration of RCTs on or before commencing the data collection of primary outcomes. In essence, the compliance in registering the RCT on or before the enrollment date is proposed to mitigate the bias associated with preferential modification of pre-defined trial outcome measures due to preliminary analysis of the trial results. Therefore, the definition may be extended to registering the trial on or before the start date of data collection. There were a number of feasibility and methodological challenges with conducting this type of analysis. The start date of RCT data collection is not a defined field within the WHO trial registries. The data collection timelines are often indicated declaratively in free-form text within the primary outcome field of the trial registration protocol. Thus, extracting the actual start date of data collection would require an extensive text mining process. The other challenge was that some primary outcomes were based on clinical events that are episodic, and could not be described with a pre-defined date, such as trial outcomes related to rehospitalization or psychotic relapses. The fourth limitation is related to the search and screening processes of trial registries, bibliographic databases (PubMed), and published manuscripts of RCTs. There was only one reviewer (the author) who conducted the search and screening throughout all studies of this thesis. The absence of the second reviewer was due limited resources. Few measures were taken to mitigate this limitation, (1) as indicated in Appendix 2, an iterative process was employed to

113

identify the search terms for the first study. The process included 16 published reviews to identify the initial set of search terms, which were then validated and refined iteratively through further search in ClinicalTrials.gov in first study and in the WHO registries in the second study, (2) details related to the search and screening processes were fully described in the methods and appendices sections. These details were provided to ensure that these processes are transparent and repeatable, (3) the data sets of registered RCTs were utilized to automate extracting trial characteristics from the trial protocols, such as the TRN, start date, principal investigator, etc., and lastly, (4) the identification of TRNs from published manuscripts was automated through a computer program, with a rigorous revision and validation process as described in appendix 18.

114

Chapter 7 Conclusion Conclusion Clinical trials are key to advancing evidence-based medical research. The literature has identified the impact of publication bias in clinical trials. Selective reporting of positive outcomes or nonpublication of negative results could misdirect subsequent research and result in literature reviews leaning toward positive outcomes. Registration of clinical trials was introduced to mitigate the risk of publication and selective reporting bias in the realm of clinical research. Furthermore, fulfilling the potential objectives of trial registration necessitates the prompt and prospective registration of clinical trials in advance of the trial start date. Thus, the prospective registration of clinical trials could mitigate the bias associated with preferential modification of pre-defined trial outcome measures due to preliminary analysis of the trial results. In the case of digital health RCTs, the publication bias was prevalent with nearly a third (between 27% and 34.5%) of those trials remaining unpublished. The high non-publication rate in digital health RCTs presents a major concern due to (1) the loss of finite resources that were allocated to the unpublished RCTs, and (2) the unreported outcomes and learnings from these RCTs, which could have informed other researchers and advanced the design and methodology for future RCTs. Investigators of digital health RCTs may be overwhelmed with unique challenges such as the usability of the intervention under evaluation, participant recruitment, and retention challenges that may contribute to this high non-publication rate. Findings from this thesis acknowledge the potential impact of the challenges on the non-publication of digital health RCTs. Completed RCTs were 3.3 times more likely to be published compared to uncompleted RCTs. For the uncompleted RCTs (withdrawn, suspended, and terminated), enrollment challenges were the most reported reasons for trial discontinuation, and therefore a major contributor to a higher nonpublication rate. The publication of discontinued or failed digital health RCTs, perhaps in the form of editorials or discussion papers, could be of great value to inform the design and implementation of future RCTs. The overall compliance in prospective registration of RCTs was low. Only less than half of RCTs were registered prospectively before the enrollment of the first trial participant (between 29.3% and 41.7%). The low compliance with the prospective registration of RCTs was not

115

consistent across all the seventeen WHO trial registries. RCTs registered in the EU Clinical Trials Register (EU-CTR), compared to the other registries, were 10.8 times more likely to be registered prospectively before the enrollment date. These differences suggest a varying level of rigor with adhering to best practices in prospective trial registration among the seventeen WHO trial registries. The compliance in prospective registration of RCTs, published in any of the ICMJE member journals, was found to be also low (61.6%). The low adoption, even within the ICMJE member journals, ten years after the 7th revision of the DoH, or fourteen years after the announcement of the ICMJE recommendations for prospective trial registration, questions the adaptability and viability of the ICMJE, and the WHO, definition of prospective registration to be on or before the enrollment date. This thesis included a critical analysis and provided further recommendations to extend this definition with a grace period, between three and eight weeks after registration. The extension of this definition will account for administrative or operational challenges that may have caused the completion of the trial registration to fall shortly behind the enrollment date. It will also be aligned with the grace period guidelines of the US FDA and the UK NHS that consider a trial registration prospective up to three and eight weeks, respectively, after enrollment. The inclusion of the TRN in the published papers of RCTs is important to establish the linkage to underlying RCT protocols in the trial registries. The overall compliance with including the TRN in the published RCTs was 71.2%. It remains unknown if the remaining 29.8% of published RCTs were never registered, or registered without disclosing the linkage to the trial registry which, in either case, requires an extensive and manual effort to establish. This thesis also reported on the compliance in including the TRN in the paper abstracts and the PubMed designated fields as per the ICMJE and the WHO recommendations. The TRN was published elsewhere in the manuscript, other than the PubMed designated field or the PubMed abstract, for half (50%) of the published RCTs. Extracting the TRN in the paper abstract is important for systematic reviews and meta-analysis, as future research could automate the screening and pre- identification of the underpinning trial protocols without the need to access the full-text manuscript. In the field of prospective registration and publication bias of RCTs, a major contribution of this thesis is the detection of a new form of bias in the registration of RCTs; “Registration Bias” or “Selective Registration Bias”. I observed a tendency of investigators to consider the registration

116

of their RCTs only when intending to publish the RCT paper in peer-reviewed journals. The majority of RCTs (between 85.1% and 95.7%), that were registered retrospectively, were registered within one year before submitting the RCT paper for journal publication. The compliance with prospective registration of RCTs and the inclusion of TRN in the published trials is paramount to mitigate publication and registration bias in the field of evidence-based medical research. The adoption of best practices in the registration and publication of RCTs should be endorsed by all the stakeholders within the research enterprise. This thesis proposed several recommendations to inform the future development of publication and registration guidelines of RCTs. I hope these recommendations will receive the attention of the ICMJE, the Cochrane Review Groups, the WHO, journal editors, ethics committees and investigators of RCTs.

117

References

1 Blümle A, Meerpohl JJ, Schumacher M, von Elm E, editors P. Fate of Clinical Research Studies after Ethical Approval – Follow-Up of Study Protocols until Publication. Sampson M, ed. PLoS One. 2014;9(2):e87184. doi:10.1371/journal.pone.0087184. 2 Sterling TD. Publication Decisions and Their Possible Effects on Inferences Drawn from Tests of Significance--Or Vice Versa. J Am Stat Assoc. 1959;54(285):30. doi:10.2307/2282137 3 Rosenthal R. The file drawer problem and tolerance for null results. Psychol Bull. 1979. doi:10.1037/0033-2909.86.3.638 4 Bhatt A. Evolution of clinical research: a history before and beyond james lind. Perspect Clin Res. 2010 Jan-Mar;1(1):6-10. PubMed PMID: 21829774; PubMed Central PMCID: PMC3149409. 5 Emanuel EJ. What Makes Clinical Research Ethical? Jama. 2000;283(20):2701. doi:10.1001/jama.283.20.2701. 6 Jones CW, Handler L, Crowell KE, Keil LG, Weaver MA, Platts-Mills TF. Non- publication of large randomized clinical trials: cross sectional analysis. Bmj. 2013;347(oct28 9):f6104-f6104. doi:10.1136/bmj.f6104. 7 Ross JS, Tse T, Zarin DA, Xu H, Zhou L, Krumholz HM. Publication of NIH funded trials registered in ClinicalTrials.gov: Cross sectional analysis. BMJ. 2012. doi:10.1136/bmj.d7292 PMID: 22214755 8 Manzoli L, Flacco ME, D’Addario M, et al. Non-publication and delayed publication of randomized trials on vaccines: Survey. BMJ. 2014. doi:10.1136/bmj.g3058 PMID: 24838102 9 Kasenda B, Von Elm E, You J, et al. Prevalence, characteristics, and publication of discontinued randomized trials. JAMA - J Am Med Assoc. 2014. doi:10.1001/jama.2014.1361 PMID: 24618966 10 Lee K, Bacchetti P, Sim I. Publication of clinical trials supporting successful new drug applications: A literature analysis. PLoS Med. 2008. doi:10.1371/journal.pmed.0050191 PMID: 18816163 11 Harrison BA, Mayo-Wilson E. Trial Registration: Understanding and Preventing Reporting Bias in Social Work Research. Res Soc Work Pract. 2014. doi:10.1177/1049731513512374 12 Wager E, Williams P. “Hardly worth the effort”? Medical journals’ policies and their editors’ and publishers’ views on trial registration and publication bias: Quantitative and qualitative study. BMJ. 2013. doi:10.1136/bmj.f5248 PMID: 24014339 13 Simes RJ. Publication bias: the case for an international registry of clinical trials. J Clin Oncol. 1986. PMID: 3760920 14 Dickersin K, Min YI. NIH clinical trials and publication bias. Online J Curr Clin Trials. 1993;Doc No 50:[4967 words; 53 paragraphs]. 15 Weber EJ, Callaham ML, Wears RL, Barton C, Young G. Unpublished research from a medical specialty meeting: why investigators fail to publish. JAMA. 1998;280(3):257- 259. 16 Liebeskind DS, Kidwell CS, Sayre JW, Saver JL. Evidence of publication bias in reporting acute stroke clinical trials. Neurology. 2006;67(6):973-979. doi:10.1212/01.wnl.0000237331.16541.ac.

118

17 Moscati R, Jehle D, Ellis D, Fiorello A, Landi M. Positive-outcome bias: comparison of emergency medicine and general medicine literatures. Acad Emerg Med. 1(3):267-271. 18 Sterling TD, Rosenbaum WL, Weinkam JJ. Publication Decisions Revisited: The Effect of the Outcome of Statistical Tests on the Decision to Publish and Vice Versa. Am Stat. 1995;49(1):108-112. doi:10.1080/00031305.1995.10476125. 19 Parekh-Bhurke S, Kwok CS, Pang C, et al. Uptake of methods to deal with publication bias in systematic reviews has increased over time, but there is still much scope for improvement. J Clin Epidemiol. 2011;64(4):349-357. doi:10.1016/j.jclinepi.2010.04.022. 20 Song F, Parekh, S, Hooper L, et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess (Rockv). 2010;14(8):iii, ix- xi, 1-193. doi:10.3310/hta14080. 21 Eysenbach G. Tackling publication bias and selective reporting in health informatics research: Register your eHealth trials in the international eHealth studies registry. J Med Internet Res. 2004. doi:10.2196/jmir.6.3.e35 PMID: 15471761 22 Dickersin K, Rennie D. Registering Clinical Trials. J Am Med Assoc. 2003. doi:10.1001/jama.290.4.516 PMID: 12876095 23 Chalmers I. Underreporting research is scientific misconduct. JAMA. 1990. doi:10.1001/jama.1990.03440100121018 PMID: 2304220 24 Turner L, Boutron I, Hróbjartsson A, Altman DG, Moher D. The evolution of assessing bias in Cochrane systematic reviews of interventions: celebrating methodological contributions of the Cochrane Collaboration. Syst Rev. 2013. doi:10.1186/2046-4053-2- 79 PMID: 24059942 25 Horton R, Smith R. Time to register randomised trials. BMJ. 1999;319(7214):865. http://www.bmj.com/content/319/7214/865.abstract. doi: https://doi.org/10.1136/bmj.319.7214.865 26 Lapierre YD, Mohr E. Clinical-trial registration. Can Med Assoc J. 1994;150(12):1939. http://www.cmaj.ca/content/150/12/1939.abstract. 27 Trends, Charts, and Maps - ClinicalTrials.gov. https://ClinicalTrials.gov/ct2/resources/trends. Accessed May 13, 2019. 28 Gülmezoglu AM, Pang T, Horton R, Dickersin K. WHO facilitates international collaboration in setting standards for clinical trial registration. Lancet (London, England). 2005;365(9474):1829-1831. doi:10.1016/S0140-6736(05)66589-0 29 Antes G, Chalmers I. Under-reporting of clinical trials is unethical. Lancet (London, England). 2003;361(9362):978-979. doi:10.1016/S0140-6736(03)12838-3 30 Dyer C. UK clinical trials must be registered to win ethics committee approval. BMJ. 2013;347:f5614. doi:10.1136/bmj.f5614 31 De Angelis C, Drazen JM, Frizelle FA, et al. Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors. Ann Intern Med. 2004;141(6):477. doi:10.7326/0003-4819-141-6-200409210-00109 PMID: 15364170 32 Krleza-Jerić K, Lemmens T. 7th revision of the Declaration of Helsinki: good news for the transparency of clinical trials. Croat Med J. 2009;50(2):105-110. doi:10.3325/CMJ.2009.50.105 PMID: 19399942 33 Moorthy VS, Karam G, Vannice KS, Kieny M-P. Rationale for WHO’s New Position Calling for Prompt Reporting and Public Disclosure of Interventional Clinical Trial Results. PLOS Med. 2015;12(4):e1001819. doi:10.1371/journal.pmed.1001819

119

34 International Committee of Medical Journal Editors (ICMJE). Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals. http://www.icmje.org/icmje-recommendations.pdf. Accessed April 24, 2019. 35 Donkin L, Christensen H, Naismith SL, Neal B, Hickie IB, Glozier N. A systematic review of the impact of adherence on the effectiveness of e-therapies. J Med Internet Res. 2011. doi:10.2196/jmir.1772 PMID: 21821503 36 Brouwer W, Kroeze W, Crutzen R, et al. Which intervention characteristics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? A systematic review. J Med Internet Res. 2011. doi:10.2196/jmir.1639 PMID: 21212045 37 Kelders SM, Kok RN, Ossebaard HC, Van Gemert-Pijnen JEWC. Persuasive system design does matter: A systematic review of adherence to web-based interventions. J Med Internet Res. 2012. doi:10.2196/jmir.2104 PMID: 23151820 38 Crutzen R, de Nooijer J, Brouwer W, Oenema A, Brug J, de Vries NK. Strategies to facilitate exposure to internet-delivered health behavior change interventions aimed at adolescents or young adults: a systematic review. Health Educ Behav. 2011. doi:10.1177/1090198110372878 PMID: 21189422 39 Clough BA, Casey LM. Technological adjuncts to increase adherence to therapy: A review. Clin Psychol Rev. 2011. doi:10.1016/j.cpr.2011.03.006 PMID: 21497153 40 Chib A, Van Velthoven MH, Car J. MHealth adoption in low-resource environments: A review of the use of mobile healthcare in developing countries. J Health Commun. 2015. doi:10.1080/10810730.2013.864735 PMID: 24673171 41 Bozarth JD, Roberts RR. Signifying significant significance. Am Psychol. 1972;27(8):774-775. doi:10.1037/h0038034 42 Newman SH. Improving the Evaluation of Submitted Manuscripts. Am Psychol. 1966;21(10):980-981. doi:10.1037/h0021046 43 American Psychological Association. Summary report of journal operations: Calendar 1964: Council of Editors. Am Psychol. 1965;20(12):1093-1093. doi:10.1037/h0021104 44 Mahoney MJ. Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognit Ther Res. 1977;1(2):161-175. doi:10.1007/BF01173636 45 Dickersin K. The existence of publication bias and risk factors for its occurrence. Jama. 1990. doi:10.1001/jama.263.10.1385 PMID: 2406472 46 Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev. 2009;(1):MR000006. doi:10.1002/14651858.MR000006.pub3 47 Chapman SJ, Shelton B, Mahmood H, Fitzgerald JE, Harrison EM, Bhangu A. Discontinuation and non-publication of surgical randomised controlled trials: observational study. BMJ. 2014;349(dec09 1):g6870. doi:10.1136/bmj.g6870 48 Chan A-W, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical Evidence for Selective Reporting of Outcomes in Randomized Trials. JAMA. 2004;291(20):2457. doi:10.1001/jama.291.20.2457 49 Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B. Evidence b(i)ased medicine-- selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. BMJ. 2003;326(7400):1171-1173. doi:10.1136/bmj.326.7400.1171 50 Wager E. The need for trial identifiers. Curr Med Res Opin. 2004;20(2):203-206. doi:10.1185/030079903125002883

120

51 The editor regrets. Br Med J. 1980;280(6213):508. http://www.ncbi.nlm.nih.gov/pubmed/20793055. Accessed June 8, 2019. 52 Dickersin K, Chan S, Chalmersx TC, Sacks HS, Smith H. Publication bias and clinical trials. Control Clin Trials. 1987;8(4):343-353. doi:10.1016/0197-2456(87)90155-3 53 McGough JJ, Faraone S V. Estimating the size of treatment effects: moving beyond p values. Psychiatry (Edgmont). 2009;6(10):21-29. http://www.ncbi.nlm.nih.gov/pubmed/20011465. Accessed June 8, 2019. 54 Davidson RA. Source of funding and outcome of clinical trials. J Gen Intern Med. 1(3):155-158. doi:10.1007/bf02602327 55 Nylenna M, Riis P, Karlsson Y. Multiple Blinded Reviews of the Same Two Manuscripts. JAMA. 1994;272(2):149. doi:10.1001/jama.1994.03520020075021 56 von Elm, Erik; Röllin, Alexandra; Blümle, Anette; Huwiler, Karin; Witschi, Mark; Egger, Matthias (2008). Publication and non-publication of clinical trials: longitudinal study of applications submitted to a research ethics committee. Swiss medical weekly, 138(13-14), pp. 197-203. Muttenz: EMH Schweizerischer Ärzteverlag 57 Bafeta A, Dechartres A, Trinquart L, Yavchitz A, Boutron I, Ravaud P. Impact of single centre status on estimates of intervention effects in trials with continuous outcomes: meta-epidemiological study. BMJ. 2012;344:e813. doi:10.1136/bmj.e813 58 Dechartres A, Boutron I, Trinquart L, Charles P, Ravaud P. Single-Center Trials Show Larger Treatment Effects Than Multicenter Trials: Evidence From a Meta- epidemiologic Study. Ann Intern Med. 2011;155(1):39. doi:10.7326/0003-4819-155-1- 201107050-00006 59 Unverzagt S, Prondzinsky R, Peinemann F. Single-center trials tend to provide larger treatment effects than multicenter trials: a systematic review. J Clin Epidemiol. 2013;66(11):1271-1280. doi:10.1016/j.jclinepi.2013.05.016 60 Berlin JA, Begg CB, Louis TA. An Assessment of Publication Bias Using a Sample of Published Clinical Trials. J Am Stat Assoc. 1989;84(406):381. doi:10.2307/2289921 61 U.S. Food and Drug Administration. Section 801 of the Food and Drug Administration Amendments Act of 2007. https://www.govinfo.gov/content/pkg/PLAW- 110publ85/pdf/PLAW-110publ85.pdf. Accessed April 27, 2019. 62 ClinicalTrials.gov. FDAAA 801 Requirements. https://ClinicalTrials.gov/ct2/manage- recs/fdaaa. Archived at: http://webcitation.org/6OyiQKYxq 63 National Health Service - Health Research Authority. Research registration and research project identifiers. https://www.hra.nhs.uk/planning-and-improving-research/research- planning/research-registration-research-project-identifiers/. Accessed April 30, 2019. 64 Smaïl-Faugeron V, Fron-Chabouis H, Durieux P. Clinical Trial Registration in Oral Health Journals. J Dent Res. 2015;94(3_suppl):8S-13S. doi:10.1177/0022034514552492 65 Jones CW, Platts-Mills TF. Quality of Registration for Clinical Trials Published in Emergency Medicine Journals. Ann Emerg Med. 2012;60(4):458-464.e1. doi:10.1016/j.annemergmed.2012.02.005 66 Harriman SL, Patel J. When are clinical trials registered? An analysis of prospective versus retrospective registration. Trials. 2016;17(1):187. doi:10.1186/s13063-016-1310- 8 67 El-Boghdadly K, Wiles MD, Atton S, Bailey CR. Adherence to guidance on registration of randomised controlled trials published in Anaesthesia. Anaesthesia. 2018;73(5):556- 563. doi:10.1111/anae.14103

121

68 Bradley HA, Rucklidge JJ, Mulder RT. A systematic review of trial registration and selective outcome reporting in psychotherapy randomized controlled trials. Acta Psychiatr Scand. 2017;135(1):65-77. doi:10.1111/acps.12647 69 Farquhar CM, Showell MG, Showell EAE, et al. Clinical trial registration in fertility trials – a case for improvement? Hum Reprod. 2017;32(9):1827-1834. doi:10.1093/humrep/dex251 70 Mann E, Nguyen N, Fleischer S, Meyer G. Compliance with trial registration in five core journals of clinical geriatrics: a survey of original publications on randomised controlled trials from 2008 to 2012. Age Ageing. 2014;43(6):872-876. doi:10.1093/ageing/afu086 71 Cybulski L, Mayo-Wilson E, Grant S. Improving transparency and reproducibility through registration: The status of intervention trials published in clinical psychology journals. J Consult Clin Psychol. 2016;84(9):753-767. doi:10.1037/ccp0000115 72 Reveiz L, Bonfill X, Glujovsky D, et al. Trial registration in Latin America and the Caribbean’s: study of randomized trials published in 2010. J Clin Epidemiol. 2012;65(5):482-487. doi:10.1016/j.jclinepi.2011.09.003 73 Boccia S, Rothman KJ, Panic N, et al. Registration practices for observational studies on ClinicalTrials.gov indicated low adherence. J Clin Epidemiol. 2016;70:176-182. doi:10.1016/j.jclinepi.2015.09.009 74 WHO | Primary Registries. WHO. 2018. https://www.who.int/ictrp/network/primary/en/. Accessed June 8, 2019. 75 Rokhsefat S, Morra DE, Offringa M, Askie LM, Kelly LE. Trial registration in pediatric surgery trials. J Pediatr Surg. 2018;53(7):1273-1279. doi:10.1016/j.jpedsurg.2017.10.049 76 Huić M, Marušić M, Marušić A. Completeness and Changes in Registered Data and Reporting Bias of Randomized Controlled Trials in ICMJE Journals after Trial Registration Policy. Siegfried N, ed. PLoS One. 2011;6(9):e25258. doi:10.1371/journal.pone.0025258 77 Østervig RM, Sonne A, Rasmussen LS. Registration of randomized clinical trials - a challenge. Acta Anaesthesiol Scand. 2015;59(8):986-989. doi:10.1111/aas.12575 78 Solaymani-Dodaran M, Khalili D, Hosseini H, et al. Iranian Registry of Clinical Trials two years on and the timing of registrations. J Evid Based Med. 2011;4(3):168-171. doi:10.1111/j.1756-5391.2011.01146.x 79 Scott A, Rucklidge JJ, Mulder RT. Is Mandatory Prospective Trial Registration Working to Prevent Publication of Unregistered Trials and Selective Outcome Reporting? An Observational Study of Five Psychiatry Journals That Mandate Prospective Clinical Trial Registration. Wicherts JM, ed. PLoS One. 2015;10(8):e0133718. doi:10.1371/journal.pone.0133718 80 Kunath F, Grobe HR, Keck B, et al. Do urology journals enforce trial registration? A cross-sectional study of published trials. BMJ Open. 2011;1(2):e000430. doi:10.1136/bmjopen-2011-000430 81 Papageorgiou SN, Antonoglou GN, Sándor GK, Eliades T. Randomized clinical trials in orthodontics are rarely registered a priori and often published late or not at all. Milgrom PM, ed. PLoS One. 2017;12(8):e0182785. doi:10.1371/journal.pone.0182785 82 Gopal AD, Wallach JD, Aminawung JA, et al. Adherence to the International Committee of Medical Journal Editors’ (ICMJE) prospective registration policy and implications for outcome integrity: a cross-sectional analysis of trials published in high-

122

impact specialty society journals. Trials. 2018;19(1):448. doi:10.1186/s13063-018- 2825-y 83 Dechartres A, Ravaud P, Atal I, Riveros C, Boutron I. Association between trial registration and treatment effect estimates: a meta-epidemiological study. BMC Med. 2016;14(1):100. doi:10.1186/s12916-016-0639-x 84 Huser V, Cimino JJ. Evaluating adherence to the International Committee of Medical Journal Editors’ policy of mandatory, timely clinical trial registration. J Am Med Inform Assoc. 2013;20(e1):e169-74. doi:10.1136/amiajnl-2012-001501 85 Dal-Ré R, Ross JS, Marušić A. Compliance with prospective trial registration guidance remained low in high-impact journals and has implications for primary end point reporting. J Clin Epidemiol. 2016;75:100-107. doi:10.1016/j.jclinepi.2016.01.017 86 Rayhill ML, Sharon R, Burch R, Loder E. Registration status and outcome reporting of trials published in core headache medicine journals. Neurology. 2015;85(20):1789- 1794. doi:10.1212/WNL.0000000000002127 87 van de Wetering FT, Scholten RJPM, Haring T, Clarke M, Hooft L. Trial Registration Numbers Are Underreported in Biomedical Publications. Scherer RW, ed. PLoS One. 2012;7(11):e49599. doi:10.1371/journal.pone.0049599 88 McGee RG, Su M, Kelly PJ, Higgins GY, Craig JC, Webster AC. Trial Registration and Declaration of Registration by Authors of Randomized Controlled Trials. Transplantation. 2011;92(10):1094-1100. doi:10.1097/TP.0b013e318232baf2 89 Ross A, George D, Wayant C, Hamilton T, Vassar M. Registration Practices of Randomized Clinical Trials in Rhinosinusitis: A Cross-sectional Review. JAMA Otolaryngol Head Neck Surg. March 2019. doi:10.1001/jamaoto.2019.0145 90 Eysenbach G. What is e-health? J Med Internet Res. 2001;3(2):E20. doi:10.2196/jmir.3.2.e20 91 Henny KD, Wilkes AL, McDonald CM, Denson DJ, Neumann MS. A Rapid Review of eHealth Interventions Addressing the Continuum of HIV Care (2007–2017). AIDS Behav. 2018;22(1):43-63. doi:10.1007/s10461-017-1923-2 92 Gaggioli A, Raspelli S, Grassi A, et al. Ubiquitous health in practice: the interreality paradigm. Stud Health Technol Inform. 2011;163:185-191. http://www.ncbi.nlm.nih.gov/pubmed/21335786. Accessed July 14, 2019. 93 Eysenbach G. Issues in evaluating health websites in an Internet-based randomized controlled trial. J Med Internet Res. 2002;4(3):E17. doi:10.2196/jmir.4.3.e17 94 Barak A, Klein B, Proudfoot JG. Defining Internet-Supported Therapeutic Interventions. Ann Behav Med. 2009;38(1):4-17. doi:10.1007/s12160-009-9130-7 95 Goralnick E, Chaudhary MA, McCarty JC, et al. Effectiveness of Instructional Interventions for Hemorrhage Control Readiness for Laypersons in the Public Access and Tourniquet Training Study (PATTS): A Randomized Clinical Trial. JAMA Surg. 2018;153(9):791-799. doi:10.1001/jamasurg.2018.1099 96 White DB, Angus DC, Shields A-M, et al. A Randomized Trial of a Family-Support Intervention in Intensive Care Units. N Engl J Med. 2018;378(25):2365-2375. doi:10.1056/NEJMoa1802637 97 Norman GJ, Zabinski MF, Adams MA, Rosenberg DE, Yaroch AL, Atienza AA. A review of eHealth interventions for physical activity and dietary behavior change. Am J Prev Med 2007 Oct;33(4):336-345. 98 van den Berg MH, Schoones JW, Vliet Vlieland TP. Internet-based physical activity interventions: a systematic review of the literature. J Med Internet Res 2007;9(3):e26

123

99 Vandelanotte C, Spathonis KM, Eakin EG, Owen N. Website-delivered physical activity interventions a review of the literature. Am J Prev Med 2007 Jul;33(1):54-64. 100 Walters ST, Wright JA, Shegog R. A review of computer and Internet-based interventions for smoking behavior. Addict Behav 2006 Feb;31(2):264-277. 101 Weinstein PK. A review of weight loss programs delivered via the Internet. J Cardiovasc Nurs 2006;21(4):251-8; quiz 259. 102 Neve M, Morgan PJ, Jones PR, Collins CE. Effectiveness of web-based interventions in achieving weight loss and weight loss maintenance in overweight and obese adults: a systematic review with meta-analysis. Obes Rev 2010 Apr;11(4):306-321. 103 Black AD, Car J, Pagliari C, Anandan C, Cresswell K, Bokun T, et al. The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med 2011;8(1):e1000387 104 Kelders SM, Van Gemert-Pijnen JE, Werkman A, Nijland N, Seydel ER. Effectiveness of a Web-based intervention aimed at healthy dietary and physical activity behavior: a randomized controlled trial about users and usage. J Med Internet Res 2011;13(2):e32 105 van Gemert-Pijnen JE, Nijland N, van Limburg M, Ossebaard HC, Kelders SM, Eysenbach G, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res 2011;13(4):e111 106 Clarke G, Reid E, Eubanks D, et al. Overcoming depression on the Internet (ODIN): a randomized controlled trial of an Internet depression skills intervention program. J Med Internet Res. 2002;4(3):E14. doi:10.2196/jmir.4.3.e14 107 Manwaring JL, Bryson SW, Goldschmidt AB, Winzelberg AJ, Luce KH, Cunning D, et al. Do adherence variables predict outcome in an online program for the prevention of eating disorders? J Consult Clin Psychol 2008 Apr;76(2):341-346. 108 Rogers EM. Diffusion of innovations. 4th edition. New York: Free Press; 1995. 109 Al-Asadi AM, Klein B, Meyer D. Posttreatment attrition and its predictors, attrition bias, and treatment efficacy of the anxiety online programs. J Med Internet Res. 2014;16(10):e232. doi:10.2196/jmir.3513 110 Habibović M, Cuijpers P, Alings M, et al. Attrition and adherence in a WEB-Based Distress Management Program for Implantable Cardioverter defibrillator Patients (WEBCARE): randomized controlled trial. J Med Internet Res. 2014;16(2):e52. doi:10.2196/jmir.2809 111 Peels DA, Bolman C, Golsteijn RHJ, et al. Differences in reach and attrition between Web-based and print-delivered tailored interventions among adults over 50 years of age: clustered randomized trial. J Med Internet Res. 2012;14(6):e179. doi:10.2196/jmir.2229 112 Eysenbach G, The Law of Attrition, J Med Internet Res 2005;7(1):e11, URL: http://www.jmir.org/2005/1/e11, DOI: 10.2196/jmir.7.1.e11, PMID: 15829473, PMCID: PMC1550631 113 Dwan K, Altman DG, Arnaiz JA, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008. doi:10.1371/journal.pone.0003081 PMID: 18769481 114 Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010. doi:10.1136/bmj.c869 PMID: 22036893 115 Dwan K, Gamble C, Williamson PR, Kirkham JJ. Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias - An Updated Review. PLoS One. 2013. doi:10.1371/journal.pone.0066844 PMID: 23861749

124

116 Jones CW, Keil LG, Holland WC, Caughey MC, Platts-Mills TF. Comparison of registered and published outcomes in randomized controlled trials: A systematic review. BMC Med. 2015. doi:10.1186/s12916-015-0520-3 PMID: 26581191 117 Ioannidis JP, Caplan AL, Dal-Ré R. Outcome reporting bias in clinical trials: Why monitoring matters. BMJ. 2017. doi:10.1136/bmj.j408 PMID: 28196819 118 ClinicalTrials.gov. Trends, Charts, and Maps. https://ClinicalTrials.gov/ct2/resources/trends. Archived at: http://www.webcitation.org/71XIYK31a 119 Abbasi K. Compulsory registration of clinical trials. BMJ. 2004;329(7467):637. http://www.bmj.com/content/329/7467/637.abstract. PMID: 15374895 120 Huser V, Cimino JJ. Linking ClinicalTrials.gov and PubMed to Track Results of Interventional Human Clinical Trials. PLoS One. 2013. doi:10.1371/journal.pone.0068409 PMID: 23874614 121 ClinicalTrials.gov. Home. https://ClinicalTrials.gov/. Archived at: http://webcitation.org/5Eymd12jE 122 Carter AO, Griffin GH, Carter TP. A survey identified publication bias in the secondary literature. J Clin Epidemiol. 2006. doi:10.1016/j.jclinepi.2005.08.011 PMID: 16488354 123 ClinicalTrials.gov. Map of All Studies on ClinicalTrials.gov. https://clinicaltrials.gov/ct2/search/map?map= Archived at: http://www.webcitation.org/736HlK7Vd 124 ClinicalTrials.gov. How Are the Primary Completion Date and the Study Completion Date Different?. https://medschool.duke.edu/sites/medschool.duke.edu/files/field/ attachments/ctgov_what are completion dates Tip Sheet_20150519_1.pdf. Archived at: http://www.webcitation.org/74OubDzNj 125 ClinicalTrials.gov. Help. https://ClinicalTrials.gov/ct2/help/last_updated. Archived at: http://www.webcitation.org/71XIsSnsZ. 126 Drion I, Pameijer LR, Van Dijk PR, Groenier KH, Kleefstra N, Bilo HJG. The effects of a mobile phone application on quality of life in patients with type 1 diabetes mellitus: A randomized controlled trial. J Diabetes Sci Technol. 2015. doi:10.1177/1932296815585871 PMID: 25963412 127 Cugelman B, Thelwall M, Dawes P. Online interventions for social marketing health behavior change campaigns: A meta-analysis of psychological architectures and adherence factors. J Med Internet Res. 2011. doi:10.2196/jmir.1367 PMID: 21320854 128 Kohl LFM, Crutzen R, De Vries NK. Online prevention aimed at lifestyle behaviors: A systematic review of reviews. J Med Internet Res. 2013. doi:10.2196/jmir.2665 PMID: 23859884 129 Schubart JR, Stuckey HL, Ganeshamoorthy A, Sciamanna CN. Chronic health conditions and internet behavioral interventions: a review of factors to enhance user engagement. Comput Inform Nurs. 2011. doi:10.1097/NCN.0b013e3182065eed PMID: 21164337 130 Källander K, Tibenderana JK, Akpogheneta OJ, et al. Mobile health (mHealth) approaches and lessons for increased performance and retention of community health workers in low and middle-income countries: a review. J Med Internet Res. 2013. doi:10.2196/jmir.2130 PMID: 23353680

125

131 Clough BA, Casey LM. Technological adjuncts to enhance current psychotherapy practices: A review. Clin Psychol Rev. 2011. doi:10.1016/j.cpr.2010.12.008 PMID: 21382535 132 Davies K. The information-seeking behaviour of doctors: A review of the evidence. Health Info Libr J. 2007. doi:10.1111/j.1471-1842.2007.00713.x PMID: 17584211 133 Lehto T, Oinas-Kukkonen H. Persuasive features in web-based alcohol and smoking interventions: A systematic review of the literature. J Med Internet Res. 2011. doi:10.2196/jmir.1559 PMID: 21795238 134 Revere D, Dunbar PJ. Review of computer-generated outpatient health behavior interventions: clinical encounters “in absentia”. J Am Med Inform Assoc. 2001. doi:10.1136/jamia.2001.0080062 PMID: 11141513 135 Younger P. Internet-based information-seeking behaviour amongst doctors and nurses: A short review of the literature: Review Article. Health Info Libr J. 2010. doi:10.1111/j.1471-1842.2010.00883.x PMID: 20402799 136 ClinicalTrials.gov. Advanced Search Field Definitions. https://ClinicalTrials.gov/ct2/help/how-find/advanced/field-defs. Archived at: http://www.webcitation.org/71XIvs08o 137 ClinicalTrials.gov. Glossary of Common Site Terms. https://ClinicalTrials.gov/ct2/about-studies/glossary#C. Archived at: http://webcitation.org/71XIdKBQK 138 NCBI. Diabetes Mellitus Mesh Term - National Center for Biotechnology Information, U.S. National Library of Medicine. https://www.ncbi.nlm.nih.gov/mesh/68003920. Archived at: http://www.webcitation.org/71XJF8VvK 139 ClinicalTrials.gov. Protocol Registration Data Element Definitions for Interventional and Observational Studies. https://prsinfo.ClinicalTrials.gov/definitions.html. Archived at: http://www.webcitation.org/71XIov38G 140 Wood AJJ. Progress and Deficiencies in the Registration of Clinical Trials. N Engl J Med. 2009. doi:10.1056/NEJMsr0806582 PMID: 19228628 141 National Institutes of Health (NIH). HHS takes steps to provide more information about clinical trials to the public. https://www.nih.gov/news-events/news-releases/hhs-takes- steps-provide-more-information-about-clinical-trials-public. Archived at: http://webcitation.org/71XJ5CroN. 142 U.S. National Library of Medicine. Clinical Trial Registry Numbers in MEDLINE®/PubMed® Records. https://www.nlm.nih.gov/bsd/policy/clin_trials.html. Archived at: http://webcitation.org/71XJAHi35 143 Institute of Medicine. Transforming Clinical Research in the United States: Challenges and Opportunities: Workshop Summary (Forum on Drug Discovery, Development, and Translation). Washington, DC: National Academies Press (US); 2010. 144 Unger JM, Cook E, Tai E, Bleyer A. The Role of Clinical Trial Participation in Cancer Research: Barriers, Evidence, and Strategies. Am Soc Clin Oncol Educ Book 2016;35:185-198. doi: 10.1200/EDBK_156686. Medline: 27249699 145 Ajithkumar T V, Gilbert DC. Modern Challenges of Cancer Clinical Trials. Clin Oncol. 2017;29(12):767-769. doi:10.1016/j.clon.2017.10.006 PMID: 29066171 146 Ross JS, Mulvey GK, Hines EM, Nissen SE, Krumholz HM. Trial publication after registration in ClinicalTrials.gov: A cross-sectional analysis. PLoS Med. 2009. doi:10.1371/journal.pmed.1000144 PMID: 19901971

126

147 Kummar S, Rubinstein L, Kinders R, et al. Phase 0 clinical trials: Conceptions and misconceptions. Cancer J. 2008. doi:10.1097/PPO.0b013e318172d6f3 PMID: 18536551 148 Fromer MJ. FDA Introduces New Phase 0 for Clinical Trials: Some Enthusiastic, Some Skeptical. Oncol Times. 2006;28(15). doi: 10.1097/01.COT.0000293387.34124.a8 149 U.S. Food and Drug Administration. The Drug Development Process - Step 3: Clinical Research. https://www.fda.gov/forpatients/approvals/drugs/ucm405622.htm. Archived at: http://webcitation.org/6u5fLb6O8 150 Umscheid CA, Margolis DJ, Grossman CE. Key Concepts of Clinical Trials: A Narrative Review Conflict of Interest Statement. Postgr Med. 2011. doi:10.3810/pgm.2011.09.2475 doi: 10.3810/pgm.2011.09.2475 PMID: 21904102 151 Hopewell S, Clarke MJ, Stewart L, Tierney J. Time to publication for results of clinical trials. Cochrane Database Syst Rev. 2007. doi:10.1002/14651858.MR000011.pub2 PMID: 17443632 152 National Health Expenditure Data, 2016 - Centers for Medicare & Medicaid Services. https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and- Reports/NationalHealthExpendData/NHE-Fact-Sheet.html. Archived at: http://www.webcitation.org/71XJOPDlw 153 OECD Health Data: Health expenditure and financing: Health expenditure indicators. Peterson-Kaiser Health System Tracker. https://www.healthsystemtracker.org/chart- collection/health-spending-u-s-compare-countries/#item-u-s-similar-public-spending- private-sector-spending-triple-comparable-countries. Archived at: http://webcitation.org/71XJSKRb6 154 Healthy Planet: Mapping Digital Health Startups Across the Globe. CB INSIGHTS: Research Briefs. https://www.cbinsights.com/research/digital-health-startups-world- map/. Archived at: http://webcitation.org/71XIAP8TX 155 The European Union Clinical Trials Register. The European Union Clinical Trials Register. https://www.ClinicalTrialsregister.eu. Archived at http://webcitation.org/736HlK7Vd 156 The International Standard Randomised Controlled Trial Number Register. The International Standard Randomised ControlledTrial Number Register. https://www.isrctn.com. Archived at http://webcitation.org/74NiwWo30 157 Glickman SW, McHutchison JG, Peterson ED, et al. Ethical and Scientific Implications of the Globalization of Clinical Research. N Engl J Med. 2009;360(8):816-823. doi:10.1056/NEJMsb0803929 158 Faure H, Hrynaszkiewicz I. The ISRCTN Register: achievements and challenges 8 years on. J Evid Based Med. 2011;4(3):188-192. doi:10.1111/j.1756-5391.2011.01138.x 159 Viergever RF, Ghersi D. The Quality of Registration of Clinical Trials. PLoS One. 2011;6(2):e14701. doi:10.1371/JOURNAL.PONE.0014701 160 Viergever RF, Karam G, Reis A, Ghersi D. The quality of registration of clinical trials: still a problem. PLoS One. 2014;9(1):e84727. doi:10.1371/journal.pone.0084727 161 Pham Q, Wiljer D, Cafazzo JA. Beyond the Randomized Controlled Trial: A Review of Alternatives in mHealth Clinical Trial Methods. JMIR mHealth uHealth. 2016;4(3):e107. doi:10.2196/mhealth.5720 162 Huser V, Cimino JJ. Translational Science on Trialome : Retrolective Study of Publishing Results of Clinical Trials (podium presentation). AMIA Annu Symp Proc. 2012.

127

163 WHO Trial Registration Data Set (Version 1.3.1). World Health Organization. http://www.who.int/ictrp/network/trds/en/. Accessed November 16, 2018. 164 International Clinical Trials Registry Platform (ICTRP) Glossary. World Health Organization. http://www.who.int/ictrp/glossary/en/. Accessed November 16, 2018. 165 HHS take steps to provide more information about clinical trials to the public. National Institutes of Health (NIH). https://www.nih.gov/news-events/news-releases/hhs-takes- steps-provide-more-information-about-clinical-trials-public. Accessed November 17, 2018. 166 Al-Durra M, Nolan RP, Seto E, Cafazzo JA, Eysenbach G. Publication Rates and Characteristics of Registered Randomized Clinical Trials in Digital Health: A Cross Sectional Analysis (Preprint). J Med Internet Res. August 2018. doi:10.2196/11924 167 Hunter KE, Seidler AL, Askie LM. Prospective registration trends, reasons for retrospective registration and mechanisms to increase prospective registration compliance: descriptive analysis and survey. BMJ Open. 2018;8(3):e019983. doi:10.1136/bmjopen-2017-019983 PMID: 29496896 168 Solaymani-Dodaran M, Vasei M, Ghanei M. Iranian Registry of Clinical Trials: a four- year steady progress. Arch Iran Med. 2013;16(11):671-674. doi:0131611/AIM.0012 169 Solaymani-Dodaran M, Ostovar A, Khalili D, Vasei M. Iranian Registry of Clinical Trials: path and challenges from conception to a World Health Organization primary register. J Evid Based Med. 2009;2(1):32-35. doi:10.1111/j.1756-5391.2009.01002.x 170 Feizabadi M, Fahimnia F, Mosavi Jarrahi A, Naghshineh N, Tofighi S. Iranian clinical trials: An analysis of registered trials in International Clinical Trial Registry Platform (ICTRP). J Evid Based Med. 2017;10(2):91-96. doi:10.1111/jebm.12248 171 Zarin DA, Tse T, Williams RJ, Rajakannan T. Update on Trial Registration 11 Years after the ICMJE Policy Was Established. N Engl J Med. 2017;376(4):383-391. doi:10.1056/NEJMsr1601330 172 McDonald AM, Knight RC, Campbell MK, et al. What influences recruitment to randomised controlled trials? A review of trials funded by two UK funding agencies. Trials. 2006;7:9. doi:10.1186/1745-6215-7-9 173 Campbell MK, Snowdon C, Francis D, et al. Recruitment to randomised trials: strategies for trial enrollment and participation study. The STEPS study. Health Technol Assess. 2007;11(48):iii, ix-105. http://www.ncbi.nlm.nih.gov/pubmed/17999843. Accessed December 7, 2018. 174 Drain PK, Robine M, Holmes KK, Bassett I V. Trial watch: global migration of clinical trials. Nat Rev Drug Discov. 2014;13(3):166-167. doi:10.1038/nrd4260 175 Homedes N, Ugalde A. Health and Ethical Consequences of Outsourcing Pivotal Clinical Trials to Latin America: A Cross-Sectional, Descriptive Study. PLoS One. 2016;11(6):e0157756. doi:10.1371/journal.pone.0157756 176 Hackshaw A. Small studies: strengths and limitations. Eur Respir J. 2008;32(5):1141- 1143. doi:10.1183/09031936.00136408 177 The Federal Office of Public Health’s (FOPH) portal for human research in Switzerland. https://www.kofam.ch/en/swiss-clinical-trials-portal.html. Accessed December 8, 2018. 178 The Philippine Health Research Registry (PHRR). http://registry.healthresearch.ph/. Accessed December 8, 2018. 179 Page SJ, Persch AC. Recruitment, retention, and blinding in clinical trials. Am J Occup Ther. 2013;67(2):154-161. doi:10.5014/ajot.2013.006197

128

180 THE WORLD MEDICAL ASSOCIATION I. DECLARATION OF HELSINKI - Ethical Principles for Medical Research Involving Human Subjects. https://www.wma.net/wp- content/uploads/2018/07/DoH-Oct2008.pdf. Accessed April 29, 2019. 181 World Health Organization. WHO Statement on Public Disclosure of Clinical Trial Results. https://www.who.int/ictrp/results/reporting/en/. Published 2017. Accessed April 28, 2019. 182 National Center for Biotechnology Information USNL of M. Advanced search - PubMed - NCBI. https://www.ncbi.nlm.nih.gov/pubmed/advanced. Accessed April 24, 2019. 183 World Health Organization. International Clinical Trials Registry Platform (ICTRP) Search Portal. http://apps.who.int/trialsearch/. Accessed April 24, 2019. 184 Australian New Zealand Clinical Trials Registry (ANZCTR). http://www.anzctr.org.au/. Accessed April 24, 2019. 185 The Chinese clinical test registration center. The Chinese Clinical Trial Registry (ChiCTR). http://www.chictr.org.cn/enIndex.aspx. Accessed April 24, 2019. 186 U.S. National Library of Medicine. Advanced Search - ClinicalTrials.gov. https://clinicaltrials.gov/ct2/search/advanced. Accessed April 24, 2019. 187 Cochrane Netherlands, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht. The Netherlands National Trial Register. https://www.trialregister.nl/trials. Accessed April 24, 2019. 188 Fundação Oswaldo Cruz. The Brazilian Clinical Trials Registry (ReBec). http://www.ensaiosclinicos.gov.br/. Accessed April 24, 2019. 189 National Institute of Medical Statistics. The Clinical Trials Registry - India (CTRI). http://ctri.nic.in/Clinicaltrials/login.php. Accessed April 24, 2019. 190 The Korea Centers for Disease Control and Prevention (KCDC). Clinical Research Information Service (CRIS) - Republic of Korea. https://cris.nih.go.kr/cris/en/use_guide/cris_introduce.jsp. Accessed April 24, 2019. 191 The National Coordinating Center of Clinical Trials (CENCEC). The Cuban Public Registry of Clinical Trials (RPCEC). http://registroclinico.sld.cu/en/home. Accessed April 24, 2019. 192 Bundesministerium für Bildung und Forschung (Federal Ministry of Education and Research). The German Clinical Trials Register (DRKS). https://www.drks.de. Accessed April 24, 2019. 193 The Ministry of Health and Medical Education (MOHME). The Iranian Registry of Clinical Trials (IRCT). https://www.irct.ir/. Accessed April 24, 2019. 194 The National Institute of Public Health. Japan Primary Registries Network (JPRN) - NIPH Clinical Trials Search of Japan. https://rctportal.niph.go.jp/en/. Accessed April 24, 2019. 195 Cochrane South Africa (Cochrane SA) at the South African Medical Research Council (SAMRC). The Pan African Clinical Trial Registry (PACTR). https://pactr.samrc.ac.za/Default.aspx. Accessed April 24, 2019. 196 Sri Lanka Medical Association. The Sri Lanka Clinical Trials Registry (SLCTR). http://www.slctr.lk/. Accessed April 24, 2019. 197 The Clinical Research Collaboration Network (CRCN). The Thai Clinical Trials Registry (TCTR). http://www.clinicaltrials.in.th/. Accessed April 24, 2019. 198 The National Institute of Health. The Peruvian Clinical Trials Registry (REPEC). https://ensayosclinicos-repec.ins.gob.pe/en/. Accessed April 24, 2019.

129

199 The University of Toronto. University of Toronto Libraries. https://onesearch.library.utoronto.ca/. Accessed April 24, 2019. 200 International Committee of Medical Journal Editors (ICMJE) | About ICMJE | FAQ | ICMJE Membership. http://icmje.org/about-icmje/faqs/icmje-membership/. Accessed April 1, 2019. 201 International Committee of Medical Journal Editors (ICMJE). Journals stating that they follow the ICMJE Recommendations. http://icmje.org/journals-following-the-icmje- recommendations/. Accessed April 24, 2019. 202 Clarivate Analytics. InCites Journal Citation Reports. https://incites.clarivate.com. Accessed April 24, 2019. 203 The University of Toronto. The Gerstein Science Information Centre. https://gerstein.library.utoronto.ca/. Accessed April 24, 2019. 204 Moher D, Liberati A, Tetzlaff J, Altman DG, Group TP. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med. 2009;6(7):e1000097. doi:10.1371/journal.pmed.1000097 205 U.S. National Library of Medicine. MEDLINE®/PubMed® Data Element (Field) Descriptions. https://www.nlm.nih.gov/bsd/mms/medlineelements.html#si. Accessed May 23, 2019. 206 Thorn BE, Eyer JC, Van Dyke BP, et al. Literacy-Adapted Cognitive Behavioral Therapy Versus Education for Chronic Pain at Low-Income Clinics: A Randomized Controlled Trial. Ann Intern Med. 2018;168(7):471-480. doi:10.7326/M17-0972 207 Geldsetzer P, Francis JM, Sando D, et al. Community delivery of antiretroviral drugs: A non-inferiority cluster-randomized pragmatic trial in Dar es Salaam, Tanzania. Newell M-L, ed. PLoS Med. 2018;15(9):e1002659. doi:10.1371/journal.pmed.1002659 208 EU Clinical Trials Register. National medicine regulatory authorities. https://www.clinicaltrialsregister.eu/natauthorities.html. Accessed April 26, 2019. 209 EU Clinical Trials Register. Clarification note on the date of registration in the register. https://www.clinicaltrialsregister.eu/doc/14-03-26 FLS - Response to Paolo Daniele Siviero - to whom it may concern.pdf. Accessed April 26, 2019. 210 Koch G, Bonnì S, Casula EP, et al. Effect of Cerebellar Stimulation on Gait and Balance Recovery in Patients With Hemiparetic Stroke. JAMA Neurol. 2019;76(2):170. doi:10.1001/jamaneurol.2018.3639 211 Thomson C, Morley K, Teesson M, Sannibale C, Haber P. Issues with recruitment to randomised controlled trials in the drug and alcohol field: a literature review and Australian case study. Drug Alcohol Rev. 2008;27(2):115-122. doi:10.1080/09595230701829561 212 Ashery RS, McAuliffe WE. Implementation issues and techniques in randomized trials of outpatient psychosocial treatments for drug abusers: recruitment of subjects. Am J Drug Alcohol Abuse. 1992;18(3):305-329. http://www.ncbi.nlm.nih.gov/pubmed/1329493. Accessed April 26, 2019. 213 Hooft L, Korevaar DA, Molenaar N, Bossuyt PMM, Scholten RJPM. Endorsement of ICMJE’s Clinical Trial Registration Policy: a survey among journal editors. Neth J Med. 2014;72(7):349-355. http://www.ncbi.nlm.nih.gov/pubmed/25178768. Accessed April 25, 2019. 214 Ogedegbe G, Plange-Rhule J, Gyamfi J, et al. Health insurance coverage with or without a nurse-led task shifting strategy for hypertension control: A pragmatic cluster

130

randomized trial in Ghana. Rahimi K, ed. PLoS Med. 2018;15(5):e1002561. doi:10.1371/journal.pmed.1002561 215 Källander K, Alfvén T, Funk T, et al. Universal versus conditional day 3 follow-up for children with non-severe unclassified fever at the community level in Ethiopia: A cluster-randomised non-inferiority trial. Persson LÅ, ed. PLoS Med. 2018;15(4):e1002553. doi:10.1371/journal.pmed.1002553 216 Rabin JC, Karunathilake N, Patrizi K. Effects of Milk vs Dark Chocolate Consumption on Visual Acuity and Contrast Sensitivity Within 2 Hours. JAMA Ophthalmol. 2018;136(6):678. doi:10.1001/jamaophthalmol.2018.0978 217 U.S. National Library of Medicine. Downloading Content for Analysis - ClinicalTrials.gov. https://clinicaltrials.gov/ct2/resources/download. Accessed April 28, 2019. 218 Springer Nature Publishing AG. NPJ Digital Medicine. Nature Publishing Group; 2018. 219 The Philippine Health Research Registry (PHRR). http://registry.healthresearch.ph/. Accessed April 27, 2018. 220 Kazi AM, Ali M, Zubair K, et al. Effect of Mobile Phone Text Message Reminders on Routine Immunization Uptake in Pakistan: Randomized Controlled Trial. JMIR public Heal Surveill. 2018;4(1):e20. doi:10.2196/publichealth.7026 221 Abdalmageed OS, Farghaly TA, Abdelaleem AA, Abdelmagied AE, Ali MK, Abbas AM. Impact of Metformin on IVF Outcomes in Overweight and Obese Women With Polycystic Ovary Syndrome: A Randomized Double-Blind Controlled Trial. Reprod Sci. January 2018:1933719118765985. doi:10.1177/1933719118765985 222 Sterne JAC, Egger M, Moher D. In: Cochrane handbook for systematic reviews of intervention. Version 5.1.0 (Updated march 2011) edition. Higgins JPT, Green S, editor. The Cochrane Collaboration; 2011 223 The Cochrane Collaboration. Reporting Biases. https://methods.cochrane.org/bias/reporting-biases. Accessed May 23, 2019 224 Kirkham Jamie J, Dwan Kerry M, Altman Douglas G, Gamble Carrol, Dodd Susanna, Smyth Rebecca et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews BMJ 2010; 340 :c365 225 Bhaskar SB. Clinical trial registration: A practical perspective. Indian J Anaesth. 2018;62(1):10-15. doi:10.4103/ija.IJA_761_17 226 Pandey A, Aggarwal A, Maulik M, Gupta J, Juneja A. Challenges in Administering a Clinical Trials Registry: Lessons from the Clinical Trials Registry-India. Pharmaceut Med. 2013;27(2):83-93. doi:10.1007/s40290-013-0009-3 227 The University of Waterloo. Steps for registering a clinical trial on clinicaltrials.gov | Research | University of Waterloo. https://uwaterloo.ca/research/office-research- ethics/research-human-participants/pre-submission-and-training/human-research- guidelines-and-policies-alphabetical-list/steps-registering-clinical-trial-clinicaltrialsgov. Accessed June 29, 2019. 228 Bock BC, Thind H, Dunsiger SI, et al. Exercise videogames for physical activity and fitness: Design and rationale of the Wii Heart Fitness trial. Contemp Clin Trials. 2015;42:204-212. doi:10.1016/j.cct.2015.04.007 229 Bock BC, Dunsiger SI, Ciccolo JT, et al. Exercise Videogames, Physical Activity, and Health: Wii Heart Fitness: A Randomized Clinical Trial. Am J Prev Med. 2019;56(4):501-511. doi:10.1016/j.amepre.2018.11.026

131

230 Serber ER, Ciccolo J, Palmer K, Cobb V, Tilkemeier PL, Bock BC. The feasibility of exercise videogames for cardiovascular risk reduction among adults: a pilot for "Wii heart fitness". J Sports Med Phys Fitness. 2016;56(3):319-327. http://www.ncbi.nlm.nih.gov/pubmed/25692860. Accessed May 25, 2019. 231 Adams MA, Hurley JC, Todd M, Bhuiyan N, Jarrett CL, Tucker WJ, Hollingshead KE, Angadi SS. Adaptive goal setting and financial incentives: a 2 × 2 factorial randomized controlled trial to increase adults' physical activity. BMC Public Health. 2017 Mar 29;17(1):286. doi: 10.1186/s12889-017-4197-8. Erratum in: BMC Public Health. 2017 Apr 6;17 (1):303. PubMed PMID: 28356097; PubMed Central PMCID: PMC5372290. 232 Adams MA, Hurley JC, Todd M, Bhuiyan N, Jarrett CL, Tucker WJ, Hollingshead KE, Angadi SS. Erratum to: Adaptive goal setting and financial incentives: a 2 × 2 factorial randomized controlled trial to increase adults' physical activity. BMC Public Health. 2017 Apr 6;17(1):303. doi: 10.1186/s12889-017-4231-x. Erratum for: BMC Public Health. 2017 Mar 29;17 (1):286. PubMed PMID: 28385148; PubMed Central PMCID: PMC5382665. 233 Froessler B, Palm P, Weber I, Hodyl NA, Singh R, Murphy EM. The Important Role for Intravenous Iron in Perioperative Patient Blood Management in Major Abdominal Surgery. Ann Surg. 2016;264(1):41-46. doi:10.1097/SLA.0000000000001646 234 Young MJ, Noon AP. Re: The Important Role for Intravenous Iron in Perioperative Patient Blood Management in Major Abdominal Surgery: A Randomized Controlled Trial. Eur Urol. 2017;71(4):690-691. doi:10.1016/j.eururo.2016.11.013 235 Collins PW, Cannings-John R, Bruynseels D, et al. Viscoelastometric-guided early fibrinogen concentrate replacement during postpartum haemorrhage: OBS2, a double- blind randomized controlled trial. Br J Anaesth. 2017;119(3):411-421. doi:10.1093/bja/aex181 236 Aawar N, Alikhan R, Bruynseels D, et al. Fibrinogen concentrate versus placebo for treatment of postpartum haemorrhage: study protocol for a randomised controlled trial. Trials. 2015;16(1):169. doi:10.1186/s13063-015-0670-9 237 Adams MA, Sallis JF, Norman GJ, Hovell MF, Hekler EB, Perata E. An adaptive physical activity intervention for overweight adults: a randomized controlled trial. PLoS One. 2013 Dec 9;8(12):e82901. doi: 10.1371/journal.pone.0082901. PubMed PMID: 24349392; PubMed Central PMCID: PMC3857300 238 U.S. National Library of Medicine. How to Find Results of Studies - ClinicalTrials.gov. https://clinicaltrials.gov/ct2/help/how-find/find-study-results. Accessed May 25, 2019. 239 Marshall IJ, Noel-Storr A, Kuiper J, Thomas J, Wallace BC. Machine learning for identifying Randomized Controlled Trials: An evaluation and practitioner’s guide. Res Synth Methods. 2018;9(4):602-614. doi:10.1002/jrsm.1287 240 Cohen AM, Smalheiser NR, McDonagh MS, et al. Automated confidence ranked classification of randomized controlled trial articles: an aid to evidence-based medicine. J Am Med Informatics Assoc. 2015;22(3):707-717. doi:10.1093/jamia/ocu025 241 Wallace BC, Noel-Storr A, Marshall IJ, Cohen AM, Smalheiser NR, Thomas J. Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach. J Am Med Inform Assoc. 2017;24(6):1165-1168. doi:10.1093/jamia/ocx053 242 Nature. Radical open-access plan could spell end to journal subscriptions. https://www.nature.com/articles/d41586-018-06178-7. Published September 4, 2018. Accessed May 23, 2019.

132

243 Universities UK. MONITORING THE TRANSITION TO OPEN ACCESS .https://www.universitiesuk.ac.uk/policy-and- analysis/reports/Documents/2017/monitoring-transition-open-access-2017.pdf. Published December 2017. Accessed May 23, 2019. 244 Laakso M, Björk B-C. Delayed open access: An overlooked high-impact category of openly available scientific literature. J Am Soc Inf Sci Technol. 2013;64(7):1323-1329. doi:10.1002/asi.22856 245 Institute of Electrical and Electronics Engineers (IEEE). What is the IEEE policy on robots or other intelligent agents? https://www.ieee.org/publications/subscriptions/info/licensing.html#8. Accessed May 23, 2019 246 SAGE Publications. Text and Data Mining on SAGE Journals. https://journals.sagepub.com/page/policies/text-and-data-mining. Accessed May 23, 2019 247 Cluver CA, Hannan NJ, van Papendorp E, et al. Esomeprazole to treat women with preterm preeclampsia: a randomized placebo controlled trial. Am J Obstet Gynecol. 2018;219(4):388.e1-388.e17. doi:10.1016/j.ajog.2018.07.019 248 Klonoff DC, Evans ML, Lane W, et al. A randomized, multicentre trial evaluating the efficacy and safety of fast‐acting insulin aspart in continuous subcutaneous insulin infusion in adults with type 1 diabetes (onset 5). Diabetes, Obes Metab. December 2018:dom.13610. doi:10.1111/dom.13610 249 Izadi V, Esmaillzadeh A, Hashemipour M, Surkan PJ, Azadbakht L, Kelishadi R. High protein diets do not affect anthropometric indexes and cardiometabolic risk factors among children with excess weight: A randomized controlled trial. J Cardiovasc Thorac Res. 2018;10(2):95-10. doi:10.15171/jcvtr.2018.15 250 Maki N, Sakamoto H, Takata Y, et al. Effect of respiratory rehabilitation for frail older patients with musculoskeletal disorders: a randomized controlled trial. J Rehabil Med. 2018;50(10):908-913. doi:10.2340/16501977-2490 251 Pegington M, Adams JE, Bundred NJ, et al. Recruitment to the "Breast-Activity and Healthy Eating After Diagnosis" (B-AHEAD) Randomized Controlled Trial. Integr Cancer Ther. 2018;17(1):131-137. doi:10.1177/1534735416687850 252 Seebacher B, Kuisma R, Glynn A, Berger T. Effects and mechanisms of differently cued and non-cued motor imagery in people with multiple sclerosis: A randomised controlled trial. Mult Scler. August 2018:1352458518795332. doi:10.1177/1352458518795332 253 Houchen-Wolloff L, Gardiner N, Devi R, et al. Web-based cardiac REhabilitatioN alternative for those declining or dropping out of conventional rehabilitation: results of the WREN feasibility randomised controlled trial. Open Hear. 2018;5(2):e000860. doi:10.1136/openhrt-2018-000860 254 Weiß KT, Zeman F, Schreml S. A randomized trial of early endovenous ablation in venous ulceration: a critical appraisal: Original Article: Gohel MS, Heatly F, Liu X et al. A randomized trial of early endovenous ablation in venous ulceration. N Engl J Med 2018; 378:2105-114. Br J Dermatol. 2019;180(1):51-55. doi:10.1111/bjd.17237 255 Nair SK, Sudarshan CD, Thorpe BS, et al. Mini-Stern Trial: A randomized trial comparing mini-sternotomy to full median sternotomy for aortic valve replacement. J Thorac Cardiovasc Surg. 2018;156(6):2124-2132.e31. doi:10.1016/j.jtcvs.2018.05.057

133

256 McKenna G, Allen PF, Hayes M, DaMata C, Moore C, Cronin M. Impact of oral rehabilitation on the quality of life of partially dentate elders in a randomised controlled clinical trial: 2 year follow-up. Li X, ed. PLoS One. 2018;13(10):e0203349. doi:10.1371/journal.pone.0203349 257 Boff R de M, Dornelles MA, Feoli AMP, Gustavo A da S, Oliveira M da S. Transtheoretical model for change in obese adolescents: MERC randomized clinical trial. J Health Psychol. August 2018:1359105318793189. doi:10.1177/1359105318793189 258 Gnanendran S, Porrett J, Woods C, et al. A randomised controlled trial of consumption of dark chocolate in pregnancy to reduce pre-eclampsia: Difficulties in recruitment, allocation and adherence. Aust N Z J Obstet Gynaecol. 2018;58(3):358-361. doi:10.1111/ajo.12694 259 Li WHC, Ho KY, Lam KKW, et al. Adventure-based training to promote physical activity and reduce fatigue among childhood cancer survivors: A randomized controlled trial. Int J Nurs Stud. 2018;83:65-74. doi:10.1016/J.IJNURSTU.2018.04.007 260 Al-Durra M, Nolan RP, Seto E, Cafazzo J. Prospective Trial Registration and Publication Rates of Randomized Clinical Trials in Digital Health: A Cross Sectional Analysis of Global Trial Registries. medRxiv. August 2019:19004390. doi:10.1101/19004390 261 Lewis SC, Warlow CP. How to spot bias and other potential problems in randomised controlled trials. J Neurol Neurosurg Psychiatry. 2004;75(2):181-187. doi:10.1136/JNNP.2003.025833 262 Kumar S, Nilsen WJ, Abernethy A, et al. Mobile health technology evaluation: the mHealth evidence workshop. Am J Prev Med. 2013;45(2):228-236. doi:10.1016/j.amepre.2013.03.017 263 Mohr DC, Cheung K, Schueller SM, Hendricks Brown C, Duan N. Continuous Evaluation of Evolving Behavioral Intervention Technologies. Am J Prev Med. 2013;45(4):517-523. doi:10.1016/j.amepre.2013.06.006

134

Appendices 8.1 Appendix 1 - Evaluation of Trial Latest Completion Date We evaluated the Latest Completion Date as the later of the trials’ Primary Completion Date and the Completion Date fields. The Primary Completion Date is “The date on which the last participant in a clinical study was examined or received an intervention and that data for the Primary Outcome Measure were collected.”[123] Whereas the Completion Date is “The date on which the final data for a clinical study were collected because the last study participant made the final visit to the study location (that is, ‘last subject, last visit’).”[124] Both above date fields could also be tagged as either anticipated or actual dates. “A Type menu is also included, with options Anticipated and Actual.”[125] Our evaluation will consider the actual dates when provided or otherwise will consider the anticipated dates. In cases where Completion Date is not provided, we considered the Primary Completion Date as the Latest Completion Date and vice versa. This is a best effort approach to indicate the study completion date. In the cases where both date fields, Completion Date and Primary Completion Date, are not provided, we considered the Last Updated date as the Latest Completion Date. The ”Last Updated” date field represents the most recent date when changes to a study record were submitted to ClinicalTrials.gov [126].

135

8.2 Appendix 2 - Determination of Search Terms and Phrases We started by summarizing all MeSH terms and authors’ keywords from nine literature reviews that we identified in PubMed with the focus on the adoption and usage of eHealth, mHealth, and digital health intervention [98,128-135]. We then employed forward analysis by following relevant recommendations of the initial nine reviews in PubMed (first-degree recommendations) to explore other reviews’ MeSH terms and authors’ keywords of relevance. Lastly, we went further to a second-degree forward recommendation analysis (recommended reviews of the recommended reviews in PubMed) iteratively until we reached saturation with the terms and keywords findings. In total, we considered 16 reviews to derive our final list of MesH terms and authors’ keywords [16,62,98,128,137,140-142]. The full list of all identified MeSH terms and authors’ keywords are listed below:

[Computer-Assisted Instruction], [Evidence-Based Medicine], [Information Dissemination], [Information Services], [Internet], [Online Systems], [Therapy, Computer-Assisted], [Humans], [Counseling], [Feedback], [Health Promotion], [Life Style], [Peer Group], [Social Support], [Telecommunication], [Microcomputers], [Guideline Adherence], [Persuasive Communication], [Behavior Therapy], [Behavioral Mesh Terms Medicine], [Age Groups], [Female], [Male], [Intervention Studies], [Motivation], [Psychology], [Social Marketing], [Chronic Disease], [Behavior], [Health Education], [Technology], [Medical Informatics], [Community Health Services], [Software], [Self Care], [Attitude to Health], [Health Status], [Attitude to Computers], [Physical Fitness], [Motor Activity], [Patient Care Planning], [Decision Support Techniques], [Computer Systems], [Access to Information]

[e-therapy], [e-Therapies], [eTherapies], [eTherapy], [Computer-Assisted Therapy], [Computer-Assisted Therapies], [Blogging], [Blog], [Blogs], [Messaging], [Text], [Texting], [Social], [Facebook], [Twitter], Authors’ [Tweets], [Persistence], [Electronic Mail], [eMail], [e-Mail], [Electronic Mails], [eMails], [e-Mails], [Promotional Health], [Life Styles], [Life Style], [Lifestyle], [Lifestyles], [Life-Styles], [Life-Style], Keywords [Behavior], [Behavioral], [Feedback], [Digital], [Interventions], [Intervention], [Promoting Health], [Persuasive], [Motivation], [Motivational], [Prevention], [Informatics], [Informatic], [Chronic], [Technology], [Biofeedback], [Wearables], [Monitoring], [Exercise] Verification of Preliminary Set in Clinical Trials.Gov The preliminary set of Mesh terms and authors’ keywords were applied by searching the ClinlicalTrials.gov database. The Mesh terms and authors’ keywords were validated on the [Title] fields of the ClinlicalTrials.gov database. The existing search functionality in ClinlicalTrials.gov website doesn’t allow limiting the search by the title field only “Search Terms—Use this field to specify words or phrases related to the studies you want to find. This performs a general search of the study information, including the title, brief description, conditions, interventions, and locations. This field is the same as the Search box at the top of every page.”[17] Therefore, the complete dataset of ClinicalTrials.gov database was exported by running an unlimited search (i.e., empty string search) for all trials. The result set included a total of 265,657 trials as of February 10th, 2018. The complete results

136

set was then imported into a local SQL Server database to enable a fully controlled and parameterized search. After we successfully applied the Mesh terms and authors’ keywords to the [Title] field search of the ClinicalTrials.gov database, we analyzed the preliminary matching trials. We considered the title field to enhance and enrich our search terms and phrases iteratively until a final list was concluded. Throughout this process and to systematically qualify every search term and phrase, and after validating each search term and phrase against the [Title] field of the ClinicalTrials.gov database, we expanded this iterative process to validate the complete set of terms with the [Interventions] and [Outcome Measures] fields of the ClinicalTrials.gov database as well. The [Interventions] field “…specifies] a process or action that is the focus of a clinical study”[17], whereas the [Outcome Measures] field “…specifies] a measurement, decided on in advance, that is used to determine the effect of interventions on participants in a clinical trial.”[17] An example to demonstrate how the [Interventions] and [Outcome Measures] were helpful to enhance our search terms is Trial (NCT02701998) below, where the term ”mHealth” was identified in both, [Interventions] and [Outcome Measures], fields but not in the [Title] field:

NCT Trial Number NCT02701998 Title The Stroke and Exercise Program (StEP) Interventions Behavioral: mHealth-Enhance Physical Activity Intervention

Outcome Measures Feasibility and acceptability of a mHealth-enhanced PA intervention via questionnaires…

137

8.3 Appendix 3 – Classification of Trial Condition Groups

We analyzed and found 487 unique entries in the condition field for the 556 included registered RCTs. We identified no particular naming convention, nomenclature or coding system used for specifying the study conditions. The condition field is a free-text entry field which offers the trials’ primary investigators complete flexibility to specify the trials’ conditions. For example, we identified nine different terms declaring diabetes (combined for diabetes I and II) as the study’s condition; “Diabetes Mellitus,” “Diabetes Mellitus, Type 2,” “Diabetes Mellitus Type 2,” “Type 2 Diabetes,” “Type 1 Diabetes,” “Diabetes,” “Diabetes (Insulin-requiring, Type 1 or Type 2),” “Type 2 Diabetes Mellitus,” “Type II Diabetes Mellitus.” Providing more systemic clinical coding and/or classification to the condition field leads to better quality information within the ClinicalTrials.gov database registry and enables more effective condition-driven secondary statistical analysis for researchers, clinicians, and healthcare professionals. The ClinicalTrials.gov database would have benefited from existing clinical condition classification systems, such as the International Classification of Diseases (ICD). However, the semantics of the condition fields in the ClinicalTrials.gov database extend to a broader range of information which may not be necessarily classified as clinical conditions: “On ClinicalTrials.gov, conditions may also include other health-related issues, such as lifespan, quality of life, and health risks”[36]. Hence, the ICD classification system may be a less appropriate option. Perhaps, a more generic classification system would better serve the context of the condition field in the ClinicalTrials.gov database. With regards to the earlier example of nine different terms for declaring diabetes (combined for diabetes I and II), we looked into the Medical Subject Headings (MeSH) browser, the US NLM controlled vocabulary thesaurus used for indexing articles for PubMed, and we found three distinct and systemic terms, “Diabetes Mellitus,” “Diabetes Mellitus, Type 1” and “Diabetes Mellitus, Type 2,” that could potentially substitute the nine different terms to describe diabetes in the ClinicalTrials.gov database [143]. The MeSH terms offers a wider range of terminology that would probably best serve the semantics of the condition field in the ClinicalTrials.gov database.

138

8.4 Appendix 4 – Classification of Trial Discontinuation Reasons

We have identified 7 different categories for trials’ discontinuation reasons for our 31 discontinued trials, i.e., trials with withdrawn, suspended, and terminated recruitment status. The classification is described in the following table: NCT Recruitment Discontinuation Reason # Classification Trial Number Status (Full Text from ClinicalTrials.gov) 1 NCT00641849 Terminated Terminated by PI r/t lack of funding and Both Funding and PI Challenges retirement of study personnel. 2 NCT01329692 Terminated The PI has left Duke—the primary study Both Funding and PI Challenges site—and the sponsor rescinded the support. 3 NCT01245907 Terminated Unacceptable high drop-out rate. Drop out 4 NCT01671033 Terminated The drop-out rate was too high. Drop out 5 NCT00864630 Terminated The study upon which this project Funding Challenges depended for subjects and the intervention was terminated prematurely due to lack of funds. 6 NCT00891631 Withdrawn Grant application was not funded. Funding Challenges 7 NCT00926003 Suspended Awaiting additional funding. Funding Challenges 8 NCT01112969 Terminated Project ended after 3 years. Funding Challenges 9 NCT01156324 Withdrawn Loss of funding. Funding Challenges 10 NCT01347528 Withdrawn Logistic and financial accounting Funding Challenges reasons. 11 NCT01044368 Suspended To focus on a similar study New Study/Project (NCT01175408) prior to recruiting participants. 12 NCT01060241 Suspended Other projects warranted more attention. New Study/Project 13 NCT01162694 Suspended Study was expanded and included in New Study/Project another study. 14 NCT00973635 Withdrawn NULL NULL 15 NCT01067963 Terminated NULL NULL 16 NCT01226238 Terminated NULL NULL 17 NCT01226641 Terminated NULL NULL 18 NCT01503008 Withdrawn NULL NULL 19 NCT00371462 Terminated PI no longer has an appointment and has PI/Staff attrition separated from Hines VAH and project was not transferred to another PI? 20 NCT01439334 Withdrawn PI left UF. PI/Staff attrition 21 NCT00606554 Terminated Slow recruitment of subjects. Recruitment Challenges 22 NCT00858559 Terminated Stopped due to low enrollment. Patients Recruitment Challenges will be followed up for 3 months. 23 NCT00877318 Terminated insufficient recruitment. Recruitment Challenges 24 NCT00878202 Terminated Insufficient recruitment. Recruitment Challenges 25 NCT01007643 Terminated Difficulty in recruitment of study Recruitment Challenges participants in allotted time and funding. 26 NCT01228890 Withdrawn No one enrolled. Recruitment Challenges 27 NCT01383278 Terminated Insufficient sample size to complete Recruitment Challenges study and merged with parent grant funded study. 28 NCT01302938 Terminated Stop date for randomization: 31/5/2012. Recruitment Challenges Recruitment terminated due to lack of recruitment. No new safety issues were identified. 29 NCT01960062 Withdrawn No potential participants met the Recruitment Challenges inclusion criteria.

139

30 NCT01141868 Withdrawn The study never opened due to technical Technical Challenges upgrades that were needed for the SHUTi computer system. 31 NCT01532258 Withdrawn Online program required re-design. Technical Challenges

140

8.5 Appendix 5 – Classification of Trial Major Technology

We have identified seven different categories for trials’ major technology as follows: 1- Web-Based Intervention: We categorized 294 (53%) of the 556 included RCTs as “Web- Based Intervention” where a computer software application was utilized to connect and synchronize data with a server computer and/or in the case of a web application accessible through the internet. For example, the clinical trial (NCT01096888) applied an internet-based weight loss intervention to participants in the Women, Infants, and Children program to enhance weight loss post-partum. 2- Computer-Based Intervention (Offline): We categorized 97 (17%) of the 556 included RCTs as “Computer-Based Intervention (Offline)” where the underlying trials utilized a stand-alone software application installed offline on a computer workstation. For example, the clinical trial (NCT00947947) introduced a computerized HIV/STD prevention program delivered on a laptop computer to increase positive perceptions of condoms and increase skills to use condoms. 3- Telemedicine Device: we identified 64 (12%) of the 556 included RCTs as “Telemedicine Device” where the underlying trials used telemedicine device in their interventions, such as the clinical trial (NCT00237692) that investigated a telemedicine intervention to improve blood pressure control in hypertensive patients. 4- Text Messaging: we identified 53(10%) of the 556 included RCTs as “Text Messaging” as these trials used texting as the major digital component in their interventions, such as the clinical trial (NCT00675389) where peer health workers send real-time text messages containing clinical and adherence data back to the central clinic to be reviewed by clinical staff. 5- Email Notifications: we identified 24 (4%) of the 556 included RCTs as “Email Notifications,” as these trials used emails as the major digital component in their interventions, such as the clinical trial (NCT01876680) where participants were provided physiotherapist support through e-mails. 6- Wii: Email Notifications: we identified 10 (2%) of the 556 included RCTs as “Wii,” as these trials used the Nintendo gaming console (Wii) in their interventions, such as the clinical trial (NCT01876680) where Wii was used for aerobics, strength training, and balance activities.

141

7- Mobile Phone Application: We also categorized 14 (3%) of the 556 included RCTs as “Mobile Phone Application” where the underlying trials utilized a mobile application delivered via smart devices, such as mobile phones and/or iPads. For example, the clinical trial (NCT01444534) investigated the utility of a diabetes mobile phone application to record the blood glucose values, dose of insulin injection, daily carbohydrate intake, amount of physical activity, and blood pressure.

142

8.6 Appendix 6 – Identification of Publication

We exported the complete content of the 556 included registered RCTs in XML format from the ClinicalTrials.gov website and then identified existing publications through two identification processes; automated and manual, as described in the following sections. Automated Publication Identification Results Every registered clinical trial in ClinicalTrials.gov has a unique registration number that starts with the three-letter prefix ”NCT” followed by eight numeric digits, for example: (NCT00906321). Publication citations and references are usually, but not always, provided in the reference element of the XML file exported from the ClinicalTrials.gov database. For example, the trial with the identifier (NCT00906321) has the following three references in the ClinicalTrials.gov exported XML file that are not publication citations of that clinical trial, as they were external relevant references to the underlying trial’s subject matter with publication years that predate the trial’s start date.

We automated the publication identification process by developing a simple computer software routine in the C#/.NET programming language to iterate over all exported 556 XML files of the included studies and generate the number of identified references and citations for every trial. There were only 141(25%) of the 556 included registered RCTs with reported references and/or publication citations in the ClinicalTrials.gov exported XML files. When we analyzed the data, we found that only 27(5%) of the 556 included registered RCTs that cited a publication related to the registered RCTs that reported at least one of the study primary outcome measures. Our finding is six times lower than that of another study by Ross et al., where the reported references rate was 31% of all included trials [18]. We acknowledge that the reported references and/or publication citations in the ClinicalTrials.gov database may be limited and would not suffice to identify existing publications for included trials. We postulate that the primary investigators of

143

those trials might have indeed published the findings of their trials but didn’t have the chance or interest to update their trials’ records in the ClinicalTrials.gov database. In fact, the registration and reporting requirements of the FDA Amendments Act, Section 801, also known as the FDAAA 801, has excluded feasibility trials focusing on testing prototype devices with the primary outcome measure relates to feasibility and not to health outcomes, as well as behavioral interventional trials that didn’t include a drug, biologics or devices [19,20]. In September 2016, the final rule for Clinical Trials Registration and Results Information Submission (42 CFR Part 11) was released and will take effect in the ClinicalTrials.gov database as of January 18, 2017. While the final rule clarifies and expands the legal requirements for submitting results information for registered clinical trials, it does not require the primary investigators to update their trials registry with publication information [14,20]. We, therefore, expanded our automated search approach to consider major bibliographic databases including Medline and PubMed. The clinical trials’ NCT registration number is included in the Medline record when published as part of the original paper, whereas in PubMed, the NCT number is included in the Secondary Source ID (SI) field and searchable using the [si] tag [15]. For Medline, there was no web-service or any other interface to query the Medline database by means of trials’ NCT numbers. Hence, we will consider Medline in the manual identification process only. For PubMed, we utilized the publicly available web-service (e-Utils) “http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi? db=pubmed&term=NCTxxxxxxxx [si]” to check for existing publications based on the registered clinical trials’ NCT number. The API will return a structured response in an XML format to indicate whether a publication exists for the given trial. However, we noticed that some cases exist where the NCT numbers were not indicated under the Secondary Source ID (SI) field and could only be identified by expanding the search beyond the [si] field. For example, publications of the registered trial number (NCT00003375) were not found when searched under the [si] field and could only be found when the search query was expanded to include all fields: “http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi? db=pubmed&term=NCT00003375.” Therefore, we extended our software routine to loop over all 556 included registered RCTs NCT numbers, and for each NCT number, the software routine submitted two API requests to the e- Utils API: one to search only by the [si] field and the other request for a comprehensive search across all fields. When the API response is received, the software will report the count of existing publications from the two responses for every NCT Number. We identified 228 (41%), and 233 (42%) trials of the 556 that included registered RCTs with matching publication in

144

PubMed when searched by the Secondary Source ID (SI) [si] field and all fields, respectively. When we analyzed the finding, we found that 199 (36%) of the 556 included registered RCTs did include a reference to relevant publication in PubMed with reported study primary outcome measures. Our finding is higher than another study by Huser in 2013, that indicated that the reported publication references rate was 23% of all included trials [119]. In total, the automated publication identification process identified publications for 226 (41%) of the 556 included registered RCTs with reported study primary outcomes measure, of which 199 were identified in PubMed and 27 ClinicalTrials.gov references. Manual Publication Identification Process Our automated publication identification process did not identify any publications for the remaining 330 (59%) registered RCTs. Therefore, we continued and augmented our automated publication identification process with two more systematic manual identification iterations. First Iteration (Trials Search): included identifying existing publication by some of the trials’ data fields: 1. Search Title in Medline by the trials’ titles fields; [brief_title] and [official_title], If no publication identified, then: 2. Search Title in PubMed by the trials’ titles fields; [brief_title] and [official_title], If no publication identified, then: 3. Google Search by the trials’ titles fields; [brief_title] and [official_title] 4. If any matches are identified in any of the above three steps, verify the following data elements for the top five matches only: • Authors’ Details: Principal Investigator and/or Responsible Party in the ClinicalTrials.gov page being one of the publication authors • Location Details: Institution/Facility/Affiliation (Investigators address) being the same in the publication and ClinicalTrials.gov page. • Study Context: Study descriptions and context as indicated in the Trial’s descriptive information in Clinical Trials.gov such as the following fields as needed: Brief Summary, Detailed Description, Intervention and Enrollment. Second Iteration (Authors Search): after completing the first iteration, and for the remaining trials with no publication identified, search by the Authors’ (Primary Investigator, PI) biographies to explore any relevant publications. If no PI was provided in ClinicalTrials.gov, the

145

search will consider the Responsible Party if provided. To identify any relevant publication for the author: 1. Google search to find the author’s online profile and bibliography. 2. If not available, search by the author’s name in PubMed. 3. For all listed publication and reference for the given author, search all references titles by the term ”Trial.” 4. If no publication identified, search by key terms abstracted from the trial’s title, for example, in clinical trial (NCT00974467), the title was “Evaluating a Website for Parents of Injured Children.” The key terms were ”Parents” and ”Web.“ The matching title was “After the injury: initial evaluation of a web-based intervention for parents of injured children.” The two iterations of the manual publication identification process have identified publications for 180 (32%) of the 556 included registered RCTs with a reported study primary outcomes measure.

146

8.7 Appendix 7 – Global Distribution of All Included Trials

Rank (By # of Trials) Country #RCTs (% of Total Number of RCTs)

1 United States 338 (61%)

2 Canada 34 (6%)

3 Sweden 31 (6%)

4 Germany 15 (3%)

5 United Kingdom 13 (2%)

6 Netherlands 11 (2%)

7 France 10 (2%)

8 Denmark 9 (2%)

9 Australia 7 (1%)

10 Norway 7 (1%)

11 Taiwan 5 (1%)

12 Uganda 4 (1%)

13 Spain 4 (1%)

14 Israel 4 (1%)

15 Italy 4 (1%)

16 Kenya 4 (1%)

17 Republic of Korea 4 (1%)

18 Brazil 4 (1%)

19 Finland 4 (1%)

20 China 3 (1%)

21 India 3 (1%)

22 Islamic Republic of Iran 2 (0.4%)

23 Ireland 2 (0.4%)

24 Switzerland 2 (0.4%)

25 United States & Canada 2 (0.4%)

26 United States & United Kingdom 2 (0.4%)

27 Tanzania 1 (0.2%)

28 Turkey 1 (0.2%)

29 Chile 1 (0.2%)

30 Czech Republic 1 (0.2%)

31 Cameroon 1 (0.2%)

32 Belgium 1 (0.2%)

147

33 Botswana 1 (0.2%)

34 Argentina 1 (0.2%)

35 Ghana 1 (0.2%)

36 Grenada 1 (0.2%)

37 Hong Kong 1 (0.2%)

38 Iceland 1 (0.2%)

39 Malaysia 1 (0.2%)

40 Pakistan 1 (0.2%)

41 Peru 1 (0.2%)

42 Philippines 1 (0.2%)

43 Poland 1 (0.2%)

44 Portugal 1 (0.2%)

45 Romania 1 (0.2%)

46 Singapore 1 (0.2%)

47 Slovenia 1 (0.2%)

48 South Africa 1 (0.2%)

49 Netherlands & Spain 1 (0.2%)

50 Mexico & Honduras 1 (0.2%)

51 United States & Puerto Rico 1 (0.2%)

52 United States & South Africa 1 (0.2%)

53 United Kingdom & Sweden 1 (0.2%)

54 Denmark & Estonia & Germany 1 (0.2%) & Italy & Spain & Sweden

148

8.8 Appendix 8 – Determination of Search Terms and Phrases We identified MeSH terms and author keywords from nine literature reviews with the focus on the adoption and usage of eHealth, mHealth, and digital health interventions in PubMed [35- 40,127-129]. To explore further terms and keywords, we considered a forward citation analysis by following relevant recommendations of the initial nine reviews in PubMed. We analyzed first- degree and second-degree recommendations (recommended reviews of the recommended reviews in PubMed) iteratively until we reached saturation with the best representative set of search terms from seven additional literature reviews [98,130-135]. Up to this point, we identified 47 preliminary search terms as follows:

“smartphone,” “smart-phone,” “cellphone,” “cell-phone,” “cellular phone,” “cellular-phone,” “mobile phone,” “cell phone,” “messaging,” “sms,” “texting,” “text reminder,” “short message,” “email,” “e-mail,” “iphone,” “android,” “ipad,” “fitbit,” “on-line,” “online,” “e-Health,” “eHealth,” “mhealth,” “m-health,” “internet,” “e-therapies,” “social-media,” “social media,” “facebook,” “twitter,” “whatsapp,” “information technology,” “communication technology,” “app,” “information application,” “health application,” “mobile application,” “electronic application,” “phone application,” “touch application,” “well-being application,” “informatic,” “computer,” “digital,” “web,” “wearable.”

Verification of Preliminary Set in ICTRP Search Portal We used the advanced search feature of the ICTRP search portal to verify the relevance of the preliminary set to the ICTRP dataset. We applied the search terms to the [Title] or [Intervention] search fields. These fields were limited to 256 characters only, hence we needed to break-down our preliminary search terms in 7 different group to be able to run the search. We ran the search on September 2nd, 2018 with 53,064 matched trials as shown below:

Group Matching Search Terms Trials (Preliminary Set) 1 814 smartphone OR smart-phone OR cellphone OR cell-phone OR cellular phone 2 4177 cellular-phone OR mobile phone OR cell phone OR messaging OR sms OR texting text reminder OR short message OR email OR e-mail OR iphone OR android OR ipad OR fitbit 3 2008 OR on-line online OR e-Health OR eHealth OR mhealth OR m-health OR internet OR etherapy OR e-therapy 4 8141 OR e-therapies OR social media OR social-media facebook OR twitter OR whatsapp OR information technology OR communication technology 5 209 information application OR electronic application well-being application OR app OR health application OR mobile application OR phone 6 36894 application OR touch application OR informatic OR computer OR digital OR web OR wearable telehealth OR tele-health OR tele-monitoring OR telemonitoring OR tele-medicine OR 7 821 telemedicine OR tele-rehabilitation OR telerehabilitation Total = 53064

149

We screened a random sample of the matching trials iteratively to verify and enhance our search terms. Informed by our screening process, we removed the term “App” because it added many false positive matches that did not refer to a mobile or computer App, such as “Applied”, “Applicable”…etc. We also added the following new terms to improve our search results: wii, Nintendo, Kinect, xBox, playstation, tele-consult, video consult, video-consult, video conferenc, video-conferenc, skype, social network, social-network, social app, tweet, game, gamif, gaming, virtual reality, augmented reality, google glass, multi media, multi-media, multi media, health app, mobile app, phone app and touch app.

We concluded our verification process with a total of 86 search terms that returned 29,011 (22859 unique trials across the 8 different groups) matched trials on September 2nd, 2018 as follows:

Group Matching Search Terms Trials (Final Set) 1 836 smartphone OR smart-phone OR cellphone OR cell-phone OR cellular phone 2 1517 cellular-phone OR cell phone OR mobile phone OR health application OR mobile application OR phone application OR touch application OR health app OR mobile app OR phone app OR touch app OR multimedia OR multi-media OR multi media 3 5610 text reminder OR short message OR text message OR messaging OR texting OR sms OR email OR e-mail OR electronic mail OR iphone OR android OR ipad 4 8301 online OR on-line OR e-Health OR eHealth OR mhealth OR m-health OR internet OR etherapy OR e-therapy OR e-therapies 5 8146 information technology OR communication technology OR information application OR electronic application OR well-being application OR informatic OR computer OR digital OR Web 6 927 telehealth OR tele-health OR tele-monitoring OR telemonitoring OR tele-medicine OR telemedicine OR tele-rehabilitation OR telerehabilitation OR tele-consult OR video consult OR video-consult OR video conferenc OR video-conferenc OR skype 7 741 social media OR social-media OR social network OR social-network OR social app OR facebook OR twitter OR tweet OR whatsapp 8 2933 wearable OR fitbit OR wii OR nintendo OR kinect OR xbox OR playstation OR game OR gamif OR gaming OR hololens OR virtual reality OR augmented reality OR google glass Total = 29011

150

8.9 Appendix 9 – Classification of Trials Condition Groups The content of the condition field in the ICTRP dataset is an unstructured free-text description of the trial conditions. There were 375 unique condition descriptions of the 417 included trials. We consolidated this diverse list of unstructured condition descriptions to the following 5 major condition groups:

1. Cancer: for all oncology and cancer related trials

2. Chronic Disease: for all combinations of chronic conditions, such as diabetes, asthma and COPD

3. Mental Health: including all mental or neurological diseases, such as addiction, anxiety, depression and sleep disorder

4. Obesity and physical activity: including all condition related to obesity and physical activity, such as arthritis, fracture, healthy eating, movement disorder, orthopedic conditions, rehabilitation and weight management.

There was no naming convention, nomenclature or coding system used for specifying the study conditions. This is likely because the underlying trial registries did not implement any data lookup or data validation system and allowed the investigators to freely enter the trial conditions in any format. For example, we identified sixteen different terms declaring diabetes (combined for diabetes I and II) as the study’s condition; “Diabetes,” “Diabetes Mellitus,” “Diabetes Mellitus Type 1,” “Diabetes Mellitus type II,” “Diabetes Mellitus, Type 1,” “Diabetes Mellitus, Type 2,” “Gestational Diabetes,” “Gestational Diabetes Mellitus,” “Insulin-Dependent Diabetes Mellitus,” “Non-Insulin-Dependent Diabetes Mellitus,” “Non-Insulin-Dependent Type 2 Diabetes,” “Pre-Diabetes,” “Type 1 Diabetes,” “Type 2 Diabetes,” “Type 2 Diabetes Mellitus,” “Type II Diabetes”.

Providing more systemic clinical coding and/or classification to the condition field leads to better quality information within the underlying trial registries and enables more effective condition- driven secondary statistical analysis for researchers, clinicians, and healthcare professionals. The trial registries would have benefited from existing clinical condition classification systems, such as the ICD. However, the semantics of the condition fields in the trial registries may extend to a broader range of information which may not be necessarily classified as clinical conditions. For example, the condition “Physical activity promotion” as described in the clinical trial

151

(ISRCTN20042348) registered in the ISRCTN registry. Hence, the ICD classification system may be a less appropriate option. Perhaps, a more generic classification system would better serve the context of the condition field in the trial registries. With regards to the earlier example of sixteen different terms for declaring diabetes (combined for diabetes I and II), we looked into the MeSH browser, the US NLM controlled vocabulary thesaurus used for indexing articles for PubMed, and we found three distinct and systemic terms, “Diabetes Mellitus,” “Diabetes Mellitus, Type 1,” and “Diabetes Mellitus, Type 2,” that could potentially substitute the sixteen different terms to describe diabetes in the trial registries. The MeSH terms offers a wider range of terminology that would probably best serve the semantics of the condition field in any of the clinical trial registries.

152

8.10 Appendix 10 – Identification of Prospective Trial Registration We analyzed the following three fields for the 417 included trials based on their XML data export from the ICTRP search portal: “Date_Registration”, “Date_Enrollment” and “Prospective_Registration”. The “Prospective_Registration” field includes a text value (“Yes” or “No”) indicating whether the trial was registered prospectively [163,164]. We compared the provided “Prospective_Registration” values with the differences between the other two fields, enrollment date and registration date. We found that all retrospective trials were reported appropriately in the ICTRP dataset, i.e. registration date was always later than the enrollment date when “Prospective_Registration” was set to “No”. However, we identified 124 trials that were tagged as prospective trials (“Prospective_Registration”=”Yes”) although the registration date was later than the enrollment date for those trials as shown in the table below:

Prospective Enrollment Registration Discrepancy* Trial ID Registration Date Date (Variance in Days)

IRCT201009114728N1 Yes 2/9/2012 12/6/2012 -301 NCT01746667 Yes 2/1/2012 11/19/2012 -292 NTR3659 Yes 2/1/2012 10/9/2012 -251 NCT01689675 Yes 1/1/2012 9/7/2012 -250 ISRCTN91944190 Yes 4/1/2012 11/29/2012 -242 ISRCTN16472838 Yes 1/2/2012 8/22/2012 -233 NCT01652222 Yes 1/1/2012 7/25/2012 -206 NTR3688 Yes 4/1/2012 10/18/2012 -200 NCT01630759 Yes 1/1/2012 6/22/2012 -173 NCT01710774 Yes 5/1/2012 10/17/2012 -169 ISRCTN45945396 Yes 3/1/2012 8/16/2012 -168 DRKS00004239 Yes 2/13/2012 7/25/2012 -163 NCT01709552 Yes 3/1/2012 8/10/2012 -162 NTR3692 Yes 6/1/2012 11/8/2012 -160 DRKS00004520 Yes 6/8/2012 11/14/2012 -159 CTRI/2012/12/003228 Yes 7/17/2012 12/17/2012 -153 NCT01618097 Yes 1/1/2012 5/29/2012 -149 NCT01602536 Yes 1/1/2012 5/17/2012 -137 ISRCTN15635876 Yes 4/15/2012 8/28/2012 -135 NCT01893749 Yes 2/1/2012 6/5/2012 -125 NTR3405 Yes 1/1/2012 4/19/2012 -109 NCT01688778 Yes 6/1/2012 9/17/2012 -108 NCT01645033 Yes 4/1/2012 7/17/2012 -107 IRCT201201094641N5 Yes 1/21/2012 4/27/2012 -97

153

NCT01596179 Yes 2/1/2012 5/8/2012 -97 NCT01681147 Yes 6/1/2012 9/4/2012 -95 NTR3643 Yes 7/1/2012 10/1/2012 -92 NCT01711944 Yes 7/1/2012 9/29/2012 -90 NCT01570374 Yes 1/1/2012 3/31/2012 -90 NCT01720641 Yes 8/1/2012 10/29/2012 -89 NCT01719120 Yes 8/1/2012 10/29/2012 -89 NCT01608191 Yes 3/1/2012 5/25/2012 -85 NCT01560130 Yes 1/1/2012 3/20/2012 -79 NCT01752608 Yes 10/1/2012 12/16/2012 -76 ISRCTN34757603 Yes 10/5/2012 12/19/2012 -75 NCT01731717 Yes 9/1/2012 11/12/2012 -72 NCT01578512 Yes 2/1/2012 4/12/2012 -71 NCT01702727 Yes 6/1/2012 8/8/2012 -68 NCT01663649 Yes 6/1/2012 8/8/2012 -68 ISRCTN33172094 Yes 1/18/2012 3/16/2012 -58 NCT01608932 Yes 4/1/2012 5/29/2012 -58 NCT01565629 Yes 2/1/2012 3/26/2012 -54 NCT01661894 Yes 4/1/2012 5/17/2012 -46 NCT01650675 Yes 6/1/2012 7/13/2012 -42 NCT01577290 Yes 3/1/2012 4/11/2012 -41 NCT01991782 Yes 6/1/2012 7/11/2012 -40 NCT01639079 Yes 6/1/2012 7/10/2012 -39 ACTRN12612000620820 Yes 5/1/2012 6/8/2012 -38 ISRCTN06220098 Yes 5/1/2012 6/7/2012 -37 ISRCTN67684181 Yes 9/3/2012 10/9/2012 -36 ISRCTN89287773 Yes 3/1/2012 4/5/2012 -35 NCT01573130 Yes 3/1/2012 3/31/2012 -30 NCT03308058 Yes 10/1/2012 10/30/2012 -29 NCT01655264 Yes 7/1/2012 7/30/2012 -29 NCT01545414 Yes 2/1/2012 3/1/2012 -29 NCT01635582 Yes 6/1/2012 6/29/2012 -28 NCT01610882 Yes 5/1/2012 5/29/2012 -28 NCT01563887 Yes 2/1/2012 2/29/2012 -28 ACTRN12612001271897 Yes 11/9/2012 12/6/2012 -27 NCT01675687 Yes 8/1/2012 8/28/2012 -27 NCT01544153 Yes 2/1/2012 2/28/2012 -27 IRCT201211204490N2 Yes 11/21/2012 12/17/2012 -26 NCT01654458 Yes 7/1/2012 7/27/2012 -26 ISRCTN54451987 Yes 6/15/2012 7/10/2012 -25 NCT01521078 Yes 1/1/2012 1/26/2012 -25

154

NCT01586949 Yes 4/1/2012 4/25/2012 -24 ISRCTN02293102 Yes 6/25/2012 7/18/2012 -23 NCT01694303 Yes 9/1/2012 9/24/2012 -23 NCT01694316 Yes 9/1/2012 9/24/2012 -23 NCT01652963 Yes 7/1/2012 7/23/2012 -22 ISRCTN67521059 Yes 9/13/2012 10/5/2012 -22 ISRCTN98176068 Yes 5/1/2012 5/23/2012 -22 NCT01522287 Yes 1/1/2012 1/23/2012 -22 ISRCTN95538913 Yes 6/25/2012 7/16/2012 -21 NCT01672398 Yes 8/1/2012 8/21/2012 -20 NCT01672385 Yes 8/1/2012 8/21/2012 -20 CTRI/2012/09/003002 Yes 8/28/2012 9/17/2012 -20 ISRCTN55991918 Yes 7/1/2012 7/20/2012 -19 NCT01571011 Yes 3/1/2012 3/20/2012 -19 NCT01712607 Yes 10/1/2012 10/20/2012 -19 NCT01634451 Yes 6/1/2012 6/20/2012 -19 ACTRN12612000613808 Yes 5/22/2012 6/8/2012 -17 NCT01541540 Yes 2/1/2012 2/17/2012 -16 NCT01738256 Yes 8/1/2012 8/17/2012 -16 ISRCTN08439795 Yes 5/16/2012 5/31/2012 -15 NCT01536509 Yes 2/1/2012 2/16/2012 -15 NCT01669031 Yes 8/1/2012 8/16/2012 -15 NCT01729715 Yes 11/1/2012 11/15/2012 -14 NCT01600534 Yes 5/1/2012 5/15/2012 -14 ISRCTN82857232 Yes 10/1/2012 10/15/2012 -14 NCT01692743 Yes 9/1/2012 9/13/2012 -12 NTR3396 Yes 4/1/2012 4/13/2012 -12 ISRCTN79967470 Yes 3/1/2012 3/13/2012 -12 NCT01619592 Yes 6/1/2012 6/12/2012 -11 NCT01752192 Yes 12/1/2012 12/12/2012 -11 IRCT201202204242N3 Yes 12/15/2012 12/26/2012 -11 ISRCTN09270496 Yes 5/5/2012 5/15/2012 -10 NCT01656213 Yes 7/1/2012 7/11/2012 -10 ISRCTN79652741 Yes 6/1/2012 6/11/2012 -10 NCT01577303 Yes 4/1/2012 4/11/2012 -10 NCT01639196 Yes 7/1/2012 7/10/2012 -9 NCT01664026 Yes 8/1/2012 8/9/2012 -8 NCT01532219 Yes 2/1/2012 2/9/2012 -8 NCT01551108 Yes 3/1/2012 3/8/2012 -7 IRCT2012121211736N1 Yes 12/20/2012 12/27/2012 -7 NCT01581983 Yes 4/1/2012 4/7/2012 -6

155

NCT01614275 Yes 5/30/2012 6/5/2012 -6 NCT01661569 Yes 8/1/2012 8/7/2012 -6 NCT02057952 Yes 4/1/2012 4/6/2012 -5 NCT01615471 Yes 6/1/2012 6/6/2012 -5 NCT01724632 Yes 11/1/2012 11/6/2012 -5 NCT01726153 Yes 11/1/2012 11/6/2012 -5 NCT01533402 Yes 2/1/2012 2/6/2012 -5 ISRCTN41694007 Yes 4/1/2012 4/5/2012 -4 NCT01732653 Yes 11/1/2012 11/5/2012 -4 NCT01636752 Yes 7/1/2012 7/5/2012 -4 NTR3737 Yes 12/1/2012 12/5/2012 -4 ISRCTN48086713 Yes 10/22/2012 10/26/2012 -4 NTR3483 Yes 6/1/2012 6/4/2012 -3 NCT01746472 Yes 12/1/2012 12/3/2012 -2 NCT01572896 Yes 4/1/2012 4/3/2012 -2 NTR3599 Yes 9/3/2012 9/5/2012 -2 NCT01700218 Yes 10/1/2012 10/2/2012 -1 ISRCTN20042348 Yes 10/11/2012 10/12/2012 -1 *) Registration date should be earlier than enrollment date for prospective trials. Our negative value in variance indicates the number of days that the enrollment date was earlier that the registration date. We therefore did not consider the provided values in the “Prospective_Registration” field from the ICTPR data export. Instead, we reevaluated prospective and retrospective trial registration based on the actual differences between the registration and enrollment dates of all included 417 trials.

156

8.11 Appendix 11 – Classifications of Trials’ Major Technology We have identified seven different categories for trials’ major technology as follows: Digital Games: We categorized 36 (9%) of the 417 included RCTs as “Digital Games” where the underlying trials utilized a computer, internet, or mobile phone game within their interventions. For example, the clinical trial (ACTRN12612000124831) introduced interactive games with a reward system to assist improving memory in preterm children. Virtual Reality: we identified 15 (4%) of the 417 included RCTs under this major technology type as these trials used virtual reality devices in their interventions, such as the clinical trial (ISRCTN70130279) where virtual-reality video-game dance training was used to increase cognitive function and gait parameters for their patients. Social Media: we classified 7 (2%) of the 417 included RCTs under this major technology type as these trials leveraged social media platforms for their interventions. For example, the clinical trial (NCT01994005) used Facebook as a platform to provide contraceptive counseling for patient at an urban obstetrics gynecology clinic at New York-Presbyterian Hospital-Weill Cornell Medical Center. Mobile Apps: We also categorized 32 (8%) of the 417 included RCTs as “Mobile Apps” where the underlying trials utilized a mobile phone application delivered via smart devices, such as mobile phones and/or iPads. For example, the clinical trial (ACTRN12612000978864) reported how a multicomponent school-based intervention using mobile phone applications could improve muscular fitness, movement skills, and key weight-related behaviors among low-income adolescent boys. Telehealth: we identified 39 (9%) of the 417 included RCTs as “Telehealth” where the underlying trials used tele-medicine device in their interventions, such as the clinical trial (ISRCTN89287773) that investigated the effectiveness and feasibility of a telemedicine intervention for discharged COPD patient in the UK. Internet/Web: We categorized 233 (59%) of the 417 included RCTs as “Internet/Web” where a computer software application was utilized to connect and synchronize data with a server computer and/or in the case of a web application accessible through the internet. For example, the clinical trial (IRCT2015072123279N1) studied the effect of web-based anger management program on mother-female adolescents' conflict. Offline: We categorized 55 (12%) of the 417 included RCTs as “Offline” where the underlying trials utilized a stand-alone software application installed offline on a computer workstation. For

157

example, the clinical trial (DRKS00003564) that investigated the feasibility of a computer-based program for depression patients in Germany.

158

8.12 Appendix 12 – Identification of Publication We identified existing publications through two identification processes; automated and manual, as described in the following sections. Automated Publication Identification Results Every clinical trial registered in any of the seventeen WHO ICTRP registries has a unique registration ID, referred to as the “Trial ID” in the ICTRP dataset. For example, (NCT01652222) is the trial registration ID within the ClinicalTrials.gov registry. Many authors and journals explicitly indicate the trial registration ID in the published manuscripts to reference the original trial registration. PubMed provides a publicly available web-service (e-Utils) that enables searching existing publications by all fields, including the trial registration ID. Here is an example of utilizing the PubMed web-service to check if there is any existing publication for the trial (NCT01577290). The http link in the example below can be copied and pasted to any internet browser to verify the result of this example for trial (NCT01577290): “http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term= NCT01577290”. The web-service will return a structured response in an XML format to indicate whether a publication exists for the given trial. Here is the sample result of the above web-service request for the trial (NCT01577290):

We automated the publication identification process by developing a simple computer software routine in the C#/.NET programming language to iterate over every trial registration ID and verify if there was a matching publication indexed in PubMed through utilizing the (e-Utils) web-service. The results will include matching publications by the registration ID of the included trials, though they will not conclude if the matched publications reported trial results versus protocol papers. The PubMed web-service runs a full text match based on the trial registration

159

ID. Therefore, we needed to screen these results and verify if the identified publication included trial results and reported primary outcome measures as described in the trial registration information. We found 142 (34%) trials with valid publications of the included 417 trials through utilizing the PubMed web-service. Our finding is higher than another study by Huser in 2013, that indicated that the reported publication references rate was 23% of all included trials [135]. Furthermore, we looked at 234 trials as a subset of the 417 included trials that were registered in the ClinicalTrials.gov registry. Publication citations and references are usually, but not always, provided in the reference element of the XML file exported from the ClinicalTrials.gov dataset. For example, the trial (NCT01535352) has the following reference in the ClinicalTrials.gov- exported XML file.

We automated the publication identification process by developing a simple computer software routine in the C#/.NET programming language to iterate over the exported 234 XML files from ClinicalTrials.gov registry and extract the list of included references and citations. There were only 56 of the 234 trials registered in the ClinicalTrials.gov registry that provided references and citations in the exported XML files. We analyzed these references and citations to determine if they published results and primary outcome measures of the underlying trials. There were only 8 (3%) of the 234 trials registered in the ClinicalTrials.gov registry that cited a relevant publication with reported results and primary outcome measures. Our finding is 10 times less than that of another study by Ross et al., where the reported references rate was 31% of all included trials [7]. Combined, we identified 150 (36%) of the 417 included trials through the automated process by utilizing the PubMed (e-Utils) web-service and reviewing publication citations and references provided in the subset of the trials that were registered in the ClinicalTrials.gov registry. We acknowledge that our automated process is limited and would not suffice to identify existing publications for all included trials for three main reasons. First, authors of trial publications may have not provided an explicit reference to their trial registration ID as they did not foresee the

160

value, or they were not required to do so by the publishing journal. Second, papers including trial results may have been published with explicit trial registration ID, though in a journal that is not indexed in PubMed. Third, when considering the subset of trials that were registered in the ClinicalTrials.gov registry, we postulate that the investigators of those trials might have indeed published the findings of their trials but didn’t have the chance or interest to update their trials’ records in the ClinicalTrials.gov database. In fact, the registration and reporting requirements of the FDA Amendments Act, Section 801, also known as the FDAAA 801, has excluded feasibility trials focusing on testing prototype devices with the primary outcome measure relating to feasibility and not to health outcomes, as well as behavioral interventional trials that did not include a drug, biologics or devices [62,140]. In September 2016, the final rule for Clinical Trials Registration and Results Information Submission (42 CFR Part 11) was released and took effect in the ClinicalTrials.gov database on January 18, 2017. While the final rule clarifies and expands the legal requirements for submitting results information for registered clinical trials, it does not require the investigators to update their trials registry with publication information [62,165]. Manual Publication Identification Process Our automated publication identification process did not identify any publications for the remaining 267 (64%) registered RCTs. Therefore, we augmented our automated publication identification process with a manual publication identification process. We searched in PubMed and Google by the public title of the trial as provided in ICTRP dataset and by the investigator name as authors. If not found, we build up a more pragmatic search string with a combination of key words from trial’s public title, the city/location, and the institution. For example, we identified the publication for trial (ACTRN12612000587808) by building the following search string “'Chronic Spinal Cord Injury' AND Australia AND 'Centre for Developmental Psychiatry & Psychology'” that included some key words from the public title of the trial, the institution of the investigator and the country, as shown below:

161

The manual publication identification process has identified publications for 123 (29.5%) of the 417 included trials with a reported study primary outcomes measure. There were 57 and 66 trial publications identified through pragmatic search in PubMed and Google respectively.

162

8.13 Appendix 13 – Funding Sources by Trial Registry and Location We analyzed the relationship, and found statistically significant differences (P<0.001), between

funding sources and trial registries as shown below:

India

Trial

)

German

Trial

Clinical

(

Trials

the

National

Zealand

Registries

Registry Registry Register

(NTR)

Standard

Controlled

Clinical

Trials

New

Clinical

(ReBEC) (PACTR)

(ISRCTN)

Portal

Trials Trials Trials

Registry

Primary

African

Netherlands

Clinical

Total

Clinical (ANZCTYR) Brazilian Registry Clinical (CTRI) ClinicalTrials.Gov EU International Randomized Number Internet Clinical (DRKS) Iranian Trials Japan Network Pan Registry The Trial Australian Government/ 0 0 0 10 1 3 0 0 3 0 1 18 Authority Hospitals and 2 0 2 37 0 2 0 0 0 1 0 44 Clinics Industry 1 0 2 5 1 2 0 0 1 0 2 14 Institute/ 26 3 1 41 0 40 1 0 2 0 23 137 Foundations/R esearch Center University 10 3 3 141 0 18 8 12 2 1 6 204 Total 39 6 8 234 2 65 9 12 8 2 32 417

We also found statistically significant differences (P<0.001), between funding sources and trial

locations as shown below:

East

(Multi)

US)

Zealand

Americas (Non Asia Australia New Europe Global Middle US Total Africa Government/Auth 0 1 3 0 9 0 0 5 18 ority Hospitals and 1 8 3 2 9 1 0 20 44 Clinics Industry 0 0 4 1 5 0 0 4 14 Institute/ 0 7 2 26 73 1 2 26 137 Foundations/Resea rch Center University 1 18 11 9 86 1 13 65 204 Total 2 34 23 38 182 3 15 120 417

163

8.14 Appendix 14 –Global Distribution of Registered Clinical Trials

Rank (By # of Trials) Country #Trials (% of Total Number of Trials N=417)

1 United States 120 (28.8%)

2 United Kingdom 41 (9.8%)

3 Australia 36 (8.6%)

4 The Netherlands 32 (7.7%)

5 Canada 22 (5.3%)

6 Sweden 21 (5%)

7 Germany 20 (4.8%)

8 Iran 12 (2.9%)

9 Spain 11 (2.6%)

10 Europe (Multi) 9 (2.2%)

11 India 9 (2.2%)

12 Switzerland 8 (1.9%)

13 Japan 7 (1.7%)

14 Brazil 6 (1.4%)

15 Italy 6 (1.4%)

16 Netherlands 6 (1.4%)

17 Denmark 4 (1%)

18 Finland 4 (1%)

19 Norway 4 (1%)

20 Austria 3 (0.7%)

21 Ireland 3 (0.7%)

22 Portugal 3 (0.7%)

23 Republic of Korea 3 (0.7%)

24 Belgium 2 (0.5%)

25 Chile 2 (0.5%)

26 France 2 (0.5%)

27 Hong Kong 2 (0.5%)

28 Israel 2 (0.5%)

29 New Zealand 2 (0.5%)

30 Argentina 1 (0.2%)

31 Bolivia 1 (0.2%)

32 Global (Multi) 1 (0.2%)

33 Iceland 1 (0.2%)

164

34 Kenya 1 (0.2%)

35 Mexico 1 (0.2%)

36 Peru 1 (0.2%)

37 Poland 1 (0.2%)

38 Romania 1 (0.2%)

39 Singapore 1 (0.2%)

40 Taiwan 1 (0.2%)

41 Turkey 1 (0.2%)

42 Uganda 1 (0.2%)

43 US/Canada (Multi) 1 (0.2%)

44 US/UK (Multi) 1 (0.2%)

165

8.15 Appendix 15 – Relationship between Trial Enrollment-To- Publication Duration and Trial Size

Enrollment-To-Publication Number of Small Trials/Number of Pearson Chi-square Test (Years) Large Trials a P value b

1 or less year 13/4 2 19/21 3 28/24 .002 4 46/34 5 17/35

6 9/23 a) We stratified the included trials into two strata, small and large trials. The stratification was based on the target size of the included trials compared to the median target size of 129 participants. b) Pearson Chi-square test examining the relationship between Enrollment-To-Publication duration and the differences in number of small and large trials.

166

8.16 Appendix 16 – Summary of Trial Enrollment-To-Publication Duration, Number of Published Trials, and the Cumulative Percentage of Non-Publication Rates

Enrollment-To-Publication Number of Trials Cumulative Percentage (%) of Non-Publication (Years) Published a Rates

1 or less year 17 95.9%

2 40 86.3%

3 52 73.9%

4 80 54.7%

5 52 42.2%

6 32 34.5% a) Number of trials published each year after enrollment in 2012.

167

8.17 Appendix 17 – Sample XML Export File from PubMed Results The PubMed website provides the option to download the metadata of one or many papers indexed in the PubMed database in XML format. The metadata also include the title and abstract of the indexed papers. The following example shows a snapshot of the downloaded metadata for the Kazi et al. trial paper included in our study [220].

168

8.18 Appendix 18 – Details Describing the Search Methodology to Identify the TRN in Published Papers We downloaded the publication metadata of the included 10,500 trials in XML format from PubMed. We identified the TRN in the papers of published trials in three different steps described in the following sections. Publication Metadata in PubMed We developed a computer program in the C#/.NET programming language to parse the TRN when identified in the “AccessionNumber” data element within the XML file. With this approach, we were able to identify the TRN in 2,610 of the included papers. We searched the abstract of the remaining papers as described in the following section. Abstract Search The abstract of the remaining papers was also provided in the XML file export from PubMed. We developed a computer program in the C#/.NET programming language to scan the abstract of the included papers and parse the TRN. The TRN has a unique pattern for each of the seventeen WHO trial registries, as shown in Appendix 19. We used a standard programming library, RegEx in C#, to define our text-pattern based search. For example, the TRN for the ClinicalTrials.gov registry always starts with the three characters “NCT,” followed by eight digits, such as (NCT03639220). The regular expression search for this pattern would be “NCT[0- 9] [0-9] [0-9] [0-9] [0-9] [0-9] [0-9] [0-9].” We ran our computer program and iteratively screened our matching results to optimize our search algorithm and account for more variations in the published content. For example, some authors or editors insert a space between the three characters “NCT” and the following 8-digit number in the ClinicalTrials.gov registry (see how the trial number NCT02910817 was reported in the Abdalmageed et al. trial paper) [221]. With this approach, we were able to identify the TRN in 1,169 of the included papers. Full Paper Search After completing our search in the abstract section, and for the remaining of the papers for which we were not able to identify the TRN in the abstract section, we needed to expand our search to include the full-text content of the published papers. We utilized a commercially available reference management software, PaperPile, to instrument the downloading process of the full- text articles in PDF format. We downloaded the full-text manuscript for 6,721 articles in PDF format from the online library resources at the University of Toronto. We used an open source

169

programming library, NuGet package iTextSharp.5.5.13, to serialize and parse the full-text content from the PDF files. Initially, we ran the same computer program we used to parse the abstract section, to parse the full-text content of the 6,721 downloaded manuscripts. In reviewing the results of our initial iteration, we came across a few nuances that challenged our pattern-search algorithm: (1) the TRN was reported in the methods or discussion section to refer to another trial (not the actual trial for which the paper was published), (2) the TRN was indicated in the references section as a citation for another trial, and (3) the published paper was, in fact, an editorial or review paper. To account for these new challenges, we optimized our computer program to capture the page number where the TRN was found, and where the references section is—through searching for the word “references.” If the TRN was found in a page after the references page, then we can conclude with confidence that the trial registration in this case was only indicated as a cited reference. We were able to screen and exclude 17 papers with this approach. We also optimized our computer program to look for few key words in the first page of the paper that could indicate a review or an editorial, such as "To the Editor," "Letter," "In Reply," "Erratum," "Commentary," "Correspondence," "Corrigendum," and “Review.” We were able to exclude 47 papers that were editorials and four papers that were reviews. We needed to screen the remaining papers with identified TRNs to confirm that the trial number was referring to the actual trial of the published paper and not as a cross-reference to another trial. To streamline our screening process, we enhanced our computer program to also capture the paragraphs (up to 300 characters) before and after the TRN. This helped us in expediting the manual revision process by focusing on the content surrounding the TRN. In summary, we reviewed and validated a total of 3,872 papers that did report a TRN in the full- text manuscript as in the following table: Number of Papers Reviewed Screening Results for Inclusion or Exclusion (N=3,872)

3,790 Included papers with an identified valid and relevant TRN in the full-text manuscript

47 Editorial papers excluded

4 Review papers excluded

Excluded papers with the TRN indicated in the introduction or discussion section as a 14 reference to another trial – other than the trial for which the paper was published

170

Excluded papers with the TRN cited in the references section as a reference to another 17 trial – other than the trial for which the paper was published

Our computer program did not identify any TRN for the remaining 2,849 papers. We screened a random sample of 300 papers to determine if we missed any search pattern to identify potential TRN. We did not find any TRN and concluded the end of our search iteration at this level.

171

8.19 Appendix 19 – Trial Registration Numbers Examples of the WHO Trial Registries In April 2015, the WHO launched the ICTRP project with the objectives to provide international standards for trial registrations [183]. The ICTRP portal has 17 primary registries and data providers that provide unique trial numbers for their registered trials as well as a link to a universal trial number (UTN) for those trials that registered across multiple registries [155,156,184-198]. We identified the patterns of TRN of every trial registry within the WHO ICTRP portal, as shown in the following table: # WHO ICTRP Data Provider (Registry) TRN Example 1 Australian New Zealand Clinical Trials Registry (ANZCTR) ACTRN12613000410752 2 Brazilian Clinical Trials Registry (ReBec) RBR-823rst

ChiCTR-TRC-13003468 3 Chinese Clinical Trial Register (ChiCTR) ChiCTR1800017987

4 Clinical Research Information Service (CRiS), Republic of Korea KCT0002710 5 Clinical Trials Registry - India (CTRI) CTRI/2010/091/000531 6 ClinicalTrials.gov NCT03202771 7 Cuban Public Registry of Clinical Trials (RPCEC) RPCEC00000125 EudraCT Number: 8 EU Clinical Trials Register (EU-CTR) 2016-004545-91 EUCTR2009-011725-14-IT 9 German Clinical Trials Register (DRKS) DRKS00013040 10 Iranian Registry of Clinical Trials (IRCT) IRCT2016102810804N8 International Standard Randomised Controlled Trial Number – UK 11 ISRCTN85596558 (ISRCTN)

JPRN-UMIN000000820 JPRN-JapicCTI-080632 12 Japan Primary Registries Network (JPRN) JPRN-JMA-IIA00322 JPRN-C000000266

13 Pan African Clinical Trial Registry (PACTR) PACTR201307000583416 14 Peruvian Clinical Trials Registry (REPEC) PER-035-18 15 Sri Lanka Clinical Trials Registry (SLCTR) SLCTR/2018/038 16 Thai Clinical Trials Register (TCTR) TCTR20181221002 17 The Netherlands National Trial Register (NTR) NTR1907

172

8.20 Appendix 20 – Trial Protocols not Found in the ICTRP Search Portal The for the following 24 trial protocols were not found in the ICTRP search portal. The trial registration protocols were only found through a direct search in their respective trial registries:

# TRN Direct Trial Registry 1 ACTRN12611000845932 Australian New Zealand Clinical Trials Registry (ANZCTR) 2 ACTRN12612000611820 Australian New Zealand Clinical Trials Registry (ANZCTR) 3 ACTRN12613001114730 Australian New Zealand Clinical Trials Registry (ANZCTR) 4 ACTRN12615000215527 Australian New Zealand Clinical Trials Registry (ANZCTR) 5 CHICTR1800019027 Chinese Clinical Trial Register (ChiCTR) 6 CHICTR-INR-1701194 Chinese Clinical Trial Register (ChiCTR) 7 ChiCTR-TTRCC-12002516 Chinese Clinical Trial Register (ChiCTR) 8 CTRI/2014/12/005301 Clinical Trials Registry - India (CTRI) 9 DRKS00003444 German Clinical Trials Register (DRKS) 10 DRKS00007939 German Clinical Trials Register (DRKS) 11 EUCTR2010-023414-31 EU Clinical Trials Register (EU-CTR) 12 EUCTR2017-001376-28 EU Clinical Trials Register (EU-CTR) 13 EUCTR2012-002857-41 EU Clinical Trials Register (EU-CTR) 14 EUCTR2013-000863-94 EU Clinical Trials Register (EU-CTR) 15 EUCTR2008-000586-26 EU Clinical Trials Register (EU-CTR) 16 EUCTR2011-006215-56 EU Clinical Trials Register (EU-CTR) 17 EUCTR2016-002854-19 EU Clinical Trials Register (EU-CTR) 18 EUCTR2004-003836-77 EU Clinical Trials Register (EU-CTR) 19 EUCTR2015-003943-20 EU Clinical Trials Register (EU-CTR) 20 EUCTR2009-017422-39 EU Clinical Trials Register (EU-CTR) 21 EUCTR2013-004002-25 EU Clinical Trials Register (EU-CTR) 22 EUCTR2012-003626-24 EU Clinical Trials Register (EU-CTR) 23 EUCTR2009-012947-40 EU Clinical Trials Register (EU-CTR) 24 NCT00433251 ClinicalTrials.gov

173

8.21 Appendix 21 – Classification of Trial Condition We classified the trial condition information provided in the trial protocols into the following seventeen different categories:

# Trial Condition Included Conditions 1 Addiction Smoking, alcohol, substance use and opioid studies 2 Arthritis and Injuries Musculoskeletal diseases, spine injuries, fracture, pain, rehabilitation, bone diseases, hip and knee surgeries 3 Diabetes Diabetes and Glucose related studies 4 Ear, Nose and Throat Otorhinolaryngology and all ENT conditions 5 Eye Disease All vision and eye related conditions 6 Heart Disease Cardiovascular, hypertension and blood diseases 7 Kidney Disease Kidney, bladder and renal diseases 8 Mental Health Mental illnesses, sleep related conditions, and nervous system diseases 9 Obesity and Physical Activity Including obesity, weight management, physical activity and nutrition studies 10 Oncology Cancer and tumors studies 11 Oral Health Dental, jaw and alveolar bone loss conditions 12 Aging and Palliative Care Aging, degenerative diseases, end-of-life support and elderly care. 13 Pulmonary Disease Including asthma, COPD and other respiratory conditions 14 Reproduction and Sexual Pregnancy, birth, infants, child behavior, parenting practices and sexual diseases Health 15 Skin Condition Skin conditions, scars and burn injuries 16 Surgery, Abdominal Disease, Surgeries, wounds, intensive and emergency care, brain and abdominal conditions Emergency and Intensive Care 17 Others All other conditions

174

8.22 Appendix 22 – Classification of Trial Funding Sources We classified the funding sources for the included trials into the following seven different categories as follows:

# Trial Funding Source Included Funding Entities 1 Foundation Charity foundations, councils, research institutes, funds, trusts, grants, associations, programs, alliances, not-for-profit organizations, groups, societies, federations and consortiums. 2 Government Included government entities, national institutes, ministries, authority, municipalities, department of health, US Department of Veterans Affairs and their affiliated agencies. 3 Hospital Such as teaching hospitals, college hospitals, clinics, health center, and health systems. 4 Industry Private companies and businesses. 5 University Including universities, colleges, academies and postgraduate institutions. 6 Not Funded Where the investigator of the clinical trials explicitly indicated that the trial was not funded and/or self-funded by investigator. 7 Not Identified No funding information was provided in the trial protocol.

175

8.23 Appendix 23 – Interval between Enrollment and Registration Dates in Retrospectively Registered Clinical Trials The following table provides the number of RCTs registered retrospectively after enrollment with a weekly breakdown within the group of 4,205 retrospectively registered trials in our study: Number of Weeks when a Trial was Number of Retrospectively Registered Trials within that Interval Registered After Enrollment (% of All Retrospectively Registered Trials N=4,205) 1 267 (6.3%) 2 284 (6.8%) 3 260 (6.2%) 4 194 (4.6%) 5 159 (3.8%) 6 126 (3.0%) 7 81 (1.9%) 8 72 (1.7%) 9 63 (1.5%) 10 68 (1.6%) 11 60 (1.4%) 12 59 (1.4%) 13 36 (0.9%) 14 42 (1.0%) 15 46 (1.1%) 16 36 (0.9%) 17 37 (0.9%) 18 46 (1.1%) 19 38 (0.9%) 20 37 (0.9%) 21 32 (0.8%) 22 27 (0.6%) 23 29 (0.7%) 24 31 (0.7%) 25 24 (0.6%) 26 31 (0.7%) 27 30 (0.7%) 28 23 (0.5%) 29 26 (0.6%) 30 30 (0.7%) 31 21 (0.5%) 32 18 (0.4%) 33 20 (0.5%) 34 24 (0.6%) 35 19 (0.5%) 36 13 (0.3%) 37 19 (0.5%) 38 33 (0.8%)

176

39 17 (0.4%) 40 23 (0.5%) 41 25 (0.6%) 42 16 (0.4%) 43 27 (0.6%) 44 22 (0.5%) 45 14 (0.3%) 46 18 (0.4%) 47 19 (0.5%) 48 19 (0.5%) 49 18 (0.4%) 50 20 (0.5%) 51 18 (0.4%) 52 13 (0.3%) 53 or more a 4 (0.0%) a) There were 1475 trials registered retrospectively after 53, or more, weeks after Enrollment Date with an average of 4.3 trials per week

177

8.24 Appendix 24 – Screening of Journal Publication Dates The XML file export from PubMed includes the journal publication dates as well as PubMed and Medline publication dates. For our study, we considered the journal publication date reported under the following XPath in the XML file export from PubMed: “PubMedArticle/MedlineCitation/Article/Journal/JournalIssue/PubDate” (also depicted in Appendix 17). We extracted the journal publication dates from the XML file downloaded from PubMed that included the publication metadata for all the 11,857 papers. We screened the journal publication date as follows: Number of Screening Results Papers 1,376 No journal publication date was provided in the following XPath: PubMedArticle/MedlineCitation/Article/Journal/JournalIssue/PubDate 275 Journal publication date was in 2019 3 Journal publication date was in 2017 4,463 Journal publication date was in 2018. The date was a valid date with a day, month and year values. Total: 11,857 When the journal publication date was missing, or not in 2018, we considered the PubMed publication date instead. The PubMed publication date is also provided in the XML file export as shown as depicted below:

178

If the PubMed date was also missing or incomplete, we viewed the paper abstract online in PubMed to document the publication date as indicated in the footer or header note of the published manuscript. If the publication date was still incomplete, we considered the following steps to default the dates in a best-effort approach:

Incomplete Publication Date Suggested Defaults The day was missing, as in “2018-7-” Default to the first day of the months, as in “2018-7-1” Month and Day are missing, as in “2018” Default to mid-year date, as in “2018-07-01” Month is not specific, as in “2018 Jan/Mar” Default to the first day of the earliest months, as in “2018-1-1” Seasonal journal issues, as in “2018 spring” Default as follows: - Summer to June 1 - Fall to October 1 - Winter to Dec.31 - Spring to April.1 After we completed our screening and the verification process, there were 19 papers with the publication date not in 2018 (they were published in 2017 or 2019). Therefore, we excluded these 19 papers, as explained in the results sections.

179

8.25 Appendix 25 – Reasons for Delayed Registration The table provides a summary of authors’ explanation for RCTs published in an ICMJE member journal and were registered retrospectively. Authors' Explanation for Registration Enrollment Papers Identified in Retrospective Trial Registration Date as in Date as in TRN PubMed as Indicated in the Published the Trial the Trial Papers Protocol Protocol Ogedegbe et al. 214 “...trial registration occurred during NCT01802372 2/28/2013 5/1/2012 the recruitment phase, prior to any analyses. The late registration was due to an error of omission. We hereby state that all future trials will be registered prospectively.” Kock et al. 210 “After the trial was completed, the NCT03456362 3/5/2018 3/1/2014 trial was registered late on March 7, 2018, in the ClinicalTrials.gov site following an initial failed attempt to get approval for the European Clinical Trials Database platform through the Italian regulatory authority....Registration was completed after patient recruitment started but before data analysis began.” Källander et al. 215 “The study protocol has been NCT02926625 10/5/2016 12/1/2015 published and was registered (ClinicalTrials.gov; identifier NCT02926625) after the first participant was randomised due to a miscommunication between study investigators.” Thorn et al. 206 “Although the trial registration was NCT01967342 10/17/2013 9/12/2013 submitted to ClinicalTrials.gov in June 2013, required revisions delayed its final publication. Sixteen of 290 patients (5.5% of the sample) were enrolled before trial registration on 21 October 2013, but no outcome data were collected.” Geldsetzer et al. 207 “There was a delay of 11 calendar NCT02711293 3/16 /2016 3/1/2016 days between study start and registration of the study on ClinicalTrials.gov because the first 9 days of the trial were used to verify whether the enrollment process was feasible. This informal pilot period did not lead to any changes in any of the planned study processes and was thus considered to be part of the main trial period. No outcome data were collected before registration of the trial.” White et al. 96 “Because of an administrative error NCT01844492 4/26/2013 7/23/2012 by the research team regarding whether quality-improvement projects could be registered on ClinicalTrials.gov, trial registration was finalized after enrollment commenced.”

180

Rabin et al. 216 “However, the first participant was NCT03326934 10/25/2 017 6/25/2017 enrolled June 25, 2017; all subjects were enrolled before clinical trial registration. The request for clinical trial registration was sent to clinicaltrials.gov on October 19, 2017, and the trial was first posted on October 31, 2017, after we realized that the study design met the definition of a clinical trial.” Goralnick et al. 95 “As this trial only examined the effect NCT03479112 3/20/2018 5/22/2017 of interventions on clinicians and not on patients, the institutional review board’s recommendation was to not preregister this trial on clinicaltrials.org, as this research did not meet the US Food and Drug Administration Amendments Act or International Committee of Medical Journal Editors clinical trials registration requirements. However, to comply with journal requirements, we registered this trial with ClinicalTrials.gov (NCT03479112) prior to publication.”

181

8.26 Appendix 26 – Scoping Analysis of Studies Reporting on Compliance in Prospective Trial Registration and Inclusion of TRN in Published Trial Papers We reviewed a total of 25 studies that assessed the compliance with prospective trial registration or the compliance with reporting the TRN within the published papers as depicted in the table below:

Compliance with Compliance Definition of Published Year Trial Reporting with Prospective Inclusion Criteria Studies Published Size TRN in Prospective Registration Published Registration Papers

Huić et al. 76 2011 152 - 20.4% When registered Adequacy of before the registration of enrollment of randomized controlled the first trial with participant ClinicalTrials.gov registration number published from September 2005 to April 2008 registration in ICMJE journals during 2.5 years after registration requirement policy. McGee 2011 307 59% - Trials in kidney et al. 88 transplantation published between October 2005 and December 2010 Solaymani- 2011 1355 - 45%, 42%, When registered To investigate the Dodaran 38% in 2009, before the timing of the et al. 78 2010 and 2011 enrollment of registrations in the respectively the first Iranian Registry of participant Clinical Trials (IRCT)

Jones et al. 65 2012 123 46% 21% When registered Randomized trials before the involving human enrollment of subjects and published the first between June 1, 2008, participant and May 31, 2011 in the 5 emergency medicine journals with the highest impact factors were included. Reveiz 2012 526 - 3.6% When registered Trial registration in et al. 72 before the randomized controlled enrollment of trials (RCTs) published the first in 2010 participant (PUBMED/LILACS) from Latin America and the Caribbean's (LAC)

182

van de 2012 312 60% - RCTs registered in the wetering Netherland Trial et al. 87 Registry (NTR) with a latest expected end date of 31 August 2008. Huser et al. 84 2013 698 95.8% 56% and 72% When registered We analyzed a in 2006 and before, or up to purposive set of five 2011 two months ICMJE respectively after, the founding journals enrollment date looking at all trial articles published in those journals during 2010-2011, and data from the ClinicalTrials.gov (CTG) trial registry. Mann et al. 70 2014 220 - 38.6% When registered To assess the before the proportion of registered enrollment of randomised controlled the first trials in five core participant clinical geriatric journals

Østervig 2015 200 - 17.1% and When registered The proportion of et al. 77 63.2% for trials before the correctly registered enrolled before enrollment of randomized controlled and after 2010 the first trials (RCTs) published respectively participant in Acta Anaesthesiologica Scandinavica (Acta) from 2009 to 2014.

Rayhill 2015 225 26% - RCTs published in core et al. 86 journals (Headache, Cephalalgia, and the Journal of Headache and Pain) from 2005 through 2014 Scott et al. 81 2015 181 - 33.1% When registered Any clinical trial (as before the defined by ICMJE) enrollment of published between 1 the first January 2009 and 31 participant July 2013 in the top five psychiatry journals adhering to ICMJE guidelines. Smaïl- 2015 317 68% 9% When registered For the 5 journals with Faugeron before the the highest 2012 impact et al. 64 enrollment of factors in each group, the first assessing registration of participant dentistry and oral surgery RCTs with results published in 2013 Boccia 2016 1109 - 13.8% When registered Cancer trials with et al. 79 before the prospective recruitment enrollment of of participants that the first were registered from participant February 2000 to December 2011 in ClinicalTrials.gov

183

Cybulski 2016 165 - 34.2% When registered Analysis of the 25 et al. 71 before the highest-impact clinical enrollment of psychology journals the first that published at least 1 participant RCT of a health-related psychological intervention in 2013. Dal-Ré 2016 144 - 72% When registered All articles reporting et al. 85 before the date trial results published in of primary end the six highest-impact point general medicine ascertainment journals in January- June 2014 that were registered in a public trial registry. Dechartres 2016 322 - 33% When registered Cochrane reviews et al. 83 before, or up to published between one month after, March 2011 and the enrollment September 2014 with date meta-analyses of a binary outcome including three or more randomised controlled trials published after 2006 Harriman 2016 108 - 31% When registered All clinical trials et al. 66 before the published in the BMC enrollment of series in 2013 were the first identified participant

Bradley 2017 112 - 24.1% When registered Randomized controlled et al. 68 before the psychotherapy trials enrollment of published between the first 2010 and 2014 were participant selected from the five highest impact factor journals in clinical psychology Farquhar 2017 693 - 21.7% When registered The Cochrane et al. 69 before the Gynaecology and enrollment of Fertility Group's the first specialised participant register was searched on 5 November 2015 for randomised fertility trials published from January 2010 to December 2014 fertility trials Papageorgiou 2017 80 - 24% When registered We searched el al. 80 before the ClinicalTrials.gov and enrollment of ISRCTN for registered the first randomized clinical participant trials in orthodontics that had been completed up to January 2017 and judged the publication status and date of registered trials using a systematic protocol

184

El-Boghdadly 2018 422 - 71.1% When registered Randomised controlled et al. 67 before the trials published in enrollment of Anaesthesia over a 3- the first year period (2014– participant 2016)

Gopal et al. 82 2018 486 - 77% When registered Analysis of the 50 most before, or up to recently published one month after, clinical trials that the enrollment reported primary results date in each of the ten highest-impact US medical specialty society journals between 1 January 2010 and 31 December 2015. Rokhsefat 2018 54 28% 60% When registered Adequacy of clinical et al. 75 before the trial registration in enrollment of pediatric surgery trials the first published in 2014. participant

Zarin et al. 73 2018 49,751 - 67.2% When registered ClinicalTrials.gov only. before the Not reporting on enrollment of registration of the first published trials, rather a participant registry analysis only. Ross et al. 89 2019 179 52.5% - The frequency of reported RCT registry numbers by authors of rhinosinusitis RCTs.

185

8.27 Appendix 27 –Trial Registration Numbers and Patterns In April 2015, the WHO launched the ICTRP project with the objective to provide international standards for trial registrations [183]. The ICTRP portal has 17 primary registries and data providers that provide unique trial numbers for their registered trials as well as a link to a UTN for those trials that registered across multiple registries [155,156,184-198]. The patterns of TRN of every trial registry within the WHO ICTRP portal were identified as shown in the following table:

# WHO ICTRP Data Provider (Registry) TRN Example 1 Australian New Zealand Clinical Trials Registry (ANZCTR) ACTRN12613000410752 2 Brazilian Clinical Trials Registry (ReBec) RBR-823rst 3 Chinese Clinical Trial Register (ChiCTR) ChiCTR-TRC-13003468 ChiCTR1800017987

4 Clinical Research Information Service (CRiS), Republic of Korea KCT0002710 5 Clinical Trials Registry - India (CTRI) CTRI/2010/091/000531 6 ClinicalTrials.gov NCT03202771 7 Cuban Public Registry of Clinical Trials (RPCEC) RPCEC00000125 8 EU Clinical Trials Register (EU-CTR) EudraCT Number: 2016-004545-91 EUCTR2009-011725-14-IT 9 German Clinical Trials Register (DRKS) DRKS00013040 10 Iranian Registry of Clinical Trials (IRCT) IRCT2016102810804N8 11 International Standard Randomised Controlled Trial Number – UK ISRCTN85596558 (ISRCTN) 12 Japan Primary Registries Network (JPRN) JPRN-UMIN000000820 JPRN-JapicCTI-080632 JPRN-JMA-IIA00322 JPRN-C000000266

186

8.28 Appendix 28 – Logistic Regression Test of Recruitment Compliance by Trial Size Below are the results of logistic regression test of recruitment compliance and the trial size, included as a continues independent variable in original values, log, mean-centered, squared and cubed:

- Original values: P=0.09, odds ratio [OR]=1, 95% CI: 0.999-1.001

- Mean-centered: P=0.46, odds ratio [OR]=1, 95% CI: 1-1

- Mean-centered-squared: P=0.36, odds ratio [OR]=1, 95% CI: 0.999-1

- Mean-centered-cubed: P=0.49, odds ratio [OR]=1, 95% CI: 1-1

- Log: P=0.04, odds ratio [OR]=0.604, 95% CI: 0.376-0.971

The follow plot is provided to depict the log (non-linear) effect between the log of the trial size and the recruitment compliance (P=0.04):

Probability of being compliant 0.8 0.75 0.7 0.65 0.6 0.55 0.5 0.45 0.4 0.35 0.3

0 2000 4000 6000 8000 10000 12000 14000 Estimated probability compliant probability of studybeing Estimated Trial size

187