Rothwell et al. Trials (2018) 19:544 https://doi.org/10.1186/s13063-018-2886-y RESEARCH Open Access A study of target effect sizes in randomised controlled trials published in the Health Technology Assessment journal Joanne C. Rothwell1,3* , Steven A. Julious1 and Cindy L. Cooper2 Abstract Background: When designing a randomised controlled trial (RCT), an important consideration is the sample size required.Thisiscalculatedfromseveralcomponents;oneofwhichisthetargetdifference.Thisstudyaimsto review the currently reported methods of elicitation of the target difference as well as to quantify the target differences used in Health Technology Assessment (HTA)-funded trials. Methods: Trials were identified from the National Institute of Health Research Health Technology Assessment journal. A total of 177 RCTs published between 2006 and 2016 were assessed for eligibility. Eligibility was established by the design of the trial and the quality of data available. The trial designs were parallel-group, superiority RCTs with a continuous primary endpoint. Data were extracted and the standardised anticipated and observed effect size estimates were calculated. Exclusion criteria was based on trials not providing enough detail in the sample size calculation and results, and trials not being of parallel-group, superiority design. Results: A total of 107 RCTs were included in the study from 102 reports. The most commonly reported method for effect size derivation was a review of evidence and use of previous research (52.3%). This was common across all clinical areas. The median standardised target effect size was 0.30 (interquartile range: 0.20–0.38), with the median standardised observed effect size 0.11 (IQR 0.05–0.29). The maximum anticipated and observed effect sizes were 0.76 and 1.18, respectively. Only two trials had anticipated target values above 0.60. Conclusion: The most commonly reported method of elicitation of the target effect size is previous published research. The average target effect size was 0.3. A clear distinction between the target difference and the minimum clinically important difference is recommended when designing a trial. Transparent explanation of target difference elicitation is advised, with multiple methods including a review of evidence and opinion-seeking advised as the more optimal methods for effect size quantification. Keywords: Randomised controlled trial, Target difference, Effect size, HTA, Health technology assessment Background including randomised controlled trials (RCTs) of clin- The major funder of research into clinical interven- ical interventions in the UK [1, 2]. tions in the United Kingdom (UK) is the National In- One of the conditions of funding from the HTA is that stitute of Health Research (NIHR), and the biggest all studies must write a HTA report to be published in programme within that is the Health Technology As- the Healthy Technology Assessment (HTA) journal. Many sessment Programme (HTA). The HTA funds commis- trials which are funded by the HTA are also published in sioned and researcher-led health-related research journals such as the Lancet, the British Medical Journal and the New England Journal of Medicine. However, the * Correspondence: [email protected]; [email protected] HTA publishes all reports for trials it funds, irrespective 1School of Health and Related Research, University of Sheffield, Sheffield, UK of the statistical significance achieved, and these reports 3The Medical Statistics Group, School of Health and Related Research (ScHARR), University of Sheffield, Regent Court, 30 Regent Street, Sheffield S1 have greater detail than journal articles can include. 4DA, UK Therefore, journals published in the HTA journal are Full list of author information is available at the end of the article © The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Rothwell et al. Trials (2018) 19:544 Page 2 of 13 suitable for review as they are published in detail, are of trials irrespective of their resulting statistical significance. high scientific standard and are published regardless of This ensures that reporting bias is not thought to be an im- the positive or negative nature of the results. portant problem in this study. Without the implications of A key component when designing a clinical trial is the reporting bias, and the high level of detail that is included sample size justification. If there are too few participants in HTA journal reports, the choice of the HTA journal al- then the trial may not result in statistical significance even lows greater understanding and transparency. if there is a true effect [3]. Conversely, having too many The search criteria consisted of including only RCTs participants could result in unethical practice; for ex- with a parallel-group design which had the objective to ample, randomising unnecessary numbers of participants assess superiority. The reason for this decision was due to a treatment which could may be shown to be inferior to the parallel-group design being the most commonly or harmful earlier and delaying the results of the study [3]. undertaken. This was confirmed by an initial scoping of The most sensitive part of the traditional sample size the HTA report. calculation is the anticipated difference or effect size be- The scoping consisted of assessing volumes 19 and 18 tween treatments. This difference can be categorised as for the number of reported RCTs and their designs. The either a clinically meaningful difference or a target dif- proportion of reports which were concerned with RCTs ference. A clinically meaningful difference is the value in these volumes were 23.9 and 20.6% for volumes 18 above which you would accept that one treatment is and 19, respectively. Of these RCTs, the percentage of clinically superior to another. However, it may not al- parallel-group superiority RCTs was 78% for volume 19 ways be desirable to use a clinically meaningful differ- and 80% for volume 18. ence. It could be that we need to demonstrate a Further exclusions were trials which did not contain difference greater than the minimum clinically meaning- the enough information for appropriate analyses to be ful difference to influence medical practice or policy. performed, trials with more than three arms due to the The target difference may then be set higher than the additional complexities involved in co-primary endpoints minimum clinically meaningful difference. Throughout and vaccination trials which also had multiple primary this paper we will use target difference when talking endpoints. These multiple primary endpoints resulted in about the effect size. more than one target difference in the various sample The elicitation of this target difference is a widely dis- size calculations, making data extraction complex. cussed issue, with a large review being performed in 2014 by Cook et al. which showed that a variety of Data extraction methods are used in establishing a target effect size [4, Each trial included had a unique identifier the Inter- 5]. This study draws from the findings of the DELTA national Standard Randomised Controlled Trial Number project, a Medical Research Council (MRC)-funded (ISRCTN). Data that could not be extracted from the in- study which resulted in the publication by Cook et al., cluded trials were denoted as ‘Missing’. and has been performed as part of the DELTA2 project, Data extraction was completed using a series of Microsoft also funded by the Medical Research Council (MRC). Excel spreadsheets with a large variety of variables and The purpose of the DELTA2 project is to formulate guid- free-text boxes for further information if required. A full list ance on choosing the target difference for RCTs, aiming of extracted variables can be seen in the Appendix.The to assist trialists in the design of trials. This study uses the extraction was carried out by one reviewer over a period of definitions of target difference elicitation methods devel- 9 months. All categorical variables were coded prior to oped by the original DELTA project in the review. completion of data extraction, with further additions to the This study aims to assess the currently reported coding if this provided clarity for various design features. methods of elicitation of the target difference as well as For example, the clinical areas and elicitation methods were quantify the target differences used in HTA-funded trials. amended during data extraction to provide more informa- tion, as described in the next section. Methods Trial identification Categorisation of variables A review of RCTs published in the HTA journal between In the event of a categorical variable being subjective in 2006 and 2016 was performed. This time frame was chosen nature, or outside the immediate understanding of the primarily because based on an initial scoping study to as- reviewer, further advice was sought. This occurred for sess if there were sufficient eligible reports, as well as being two variables, the clinical area of the trial and the target recent and manageable for the author in the time frame. effect size elicitation method. The use of the HTA journal as the data source for this For the clinical categorisation, data were initially cate- study means that both statistically significant and gorised into 15 clinical areas. At an interim assessment non-significant trials are included, since the journal reports point, however, a large number of trials fell into the Rothwell et al. Trials (2018) 19:544 Page 3 of 13 ‘Other’ category (18.7%).