Randomised Controlled Trials in the Social Sciences Conference Missing Data in Randomised Trials — Overview and Strategies James R. Carpenter London School of Hygiene & Tropical Medicine & MRC CTU at UCL [email protected] www.missingdata.org.uk September 7, 2018 Missing Data in Clinical Trials – 1 / 45 Acknowledgements Overview Harvey Goldstein (LSHTM) • Acknowledgements • Outline Rachael Hughes (Bristol) Mike Kenward (LSHTM) Towards a principled Geert Molenberghs (Limburgs University, Belgium) approach Critique of common Mike Kenward, James Roger (LSHTM) methods Sara Schroter (BMJ, London) Missing At Random Michael Spratt (Bristol) Multiple Imputation under Missing At Random Jonathan Sterne (Bristol) MI: example Stijn Vansteelandt (Ghent University, Belgium) Multiple Imputation under Missing Not At Random Tim Morris, Ian White (MRC CTU at UCL) Discussion Background to this session is in Chs 1 & 2 of ‘Missing data in clinical trials — a practical guide’ (joint with Mike Kenward), commissioned by the NHS, available free on-line at www.missingdata.org.uk. Missing Data in Clinical Trials – 2 / 45 Outline Overview 1. Missing data in trials— the need for a principled approach • Acknowledgements • Outline 2. Completers analysis Towards a principled 3. Imputation of simple mean approach Critique of common 4. Imputation of regression mean methods 5. Last Observation Carried Forward Missing At Random 6. Multiple Imputation Multiple Imputation under Missing At Random • assuming data are Missing At Random MI: example assuming data are Missing Not At Random Multiple Imputation under • Missing Not At Random 7. Discussion Discussion Missing Data in Clinical Trials – 3 / 45 Why is this necessary? Overview Missing data are common. Towards a principled approach • Why is this necessary? However, they are usually inadequately handled in both epidemiological • Further... and experimental research. • The E9 guideline, 1999 • Study validity and sensible analysis For example, [14] reviewed 71 recently published BMJ, JAMA, Lancet • Why there can be no universal method: and NEJM papers. • Example: Trial of training to improve the quality of peer review • 89% had partly missing outcome data. • Key points for analysis • Towards a systematic • In 37 trials with repeated outcome measures, 46% performed approach • A systematic approach complete case analysis. Critique of common • Only 21% reported sensitivity analysis. methods Missing At Random Unfortunately, an update in 2014 [1] showed relatively little had changed, Multiple Imputation under Missing At Random although Multiple Imputation [12] was much more commonly used. MI: example Multiple Imputation under Missing Not At Random Discussion Missing Data in Clinical Trials – 4 / 45 Further... Overview CONSORT guidelines state that the number of patients with missing data Towards a principled approach should be reported by treatment arm. • Why is this necessary? • Further... But [5] estimate that 65% of studies in PubMed journals do not report the • The E9 guideline, 1999 • Study validity and handling of missing data. sensible analysis • Why there can be no universal method: [14] identified serious weaknesses in the description of missing data and • Example: Trial of training to improve the the methodology adopted. quality of peer review • Key points for analysis • Towards a systematic approach • A systematic approach Critique of common methods Missing At Random Multiple Imputation under Missing At Random MI: example Multiple Imputation under Missing Not At Random Discussion Missing Data in Clinical Trials – 5 / 45 The E9 guideline, 1999 Overview • Missing data are a potential source of bias Towards a principled approach • Avoid if possible (!) • Why is this necessary? • With missing data, a trial may still be regarded as valid if the methods • Further... • The E9 guideline, 1999 are sensible, and preferably predefined • Study validity and sensible analysis • There can be no universally applicable method of handling missing • Why there can be no universal method: data • Example: Trial of training to improve the • The sensitivity of conclusions to methods should thus be investigated, quality of peer review • Key points for analysis particularly if there are a large number of missing observations • Towards a systematic approach Guidelines downloadable from www.ich.org • A systematic approach Critique of common methods The question is, how do we apply these principles in practice? Missing At Random Multiple Imputation under Missing At Random MI: example Multiple Imputation under Missing Not At Random Discussion Missing Data in Clinical Trials – 6 / 45 Study validity and sensible analysis Overview Missing data are observations we intended to make but did not. Towards a principled approach • Why is this necessary? The sampling process involves both the selection of the units, and the • Further... process by which observations become missing — the missingness • The E9 guideline, 1999 • Study validity and mechanism. sensible analysis • Why there can be no universal method: Thus for sensible inference, we need to take account of the missingness • Example: Trial of training to improve the mechanism quality of peer review • Key points for analysis • Towards a systematic By sensible we mean: approach • A systematic approach • Frequentist: nominal properties hold. Eg: Critique of common methods Estimators consistent; confidence intervals attain nominal coverage. Missing At Random • Bayesian: Multiple Imputation under Posterior distribution is unbiased, correctly reflects loss of information Missing At Random MI: example due to missingness mechanism. Multiple Imputation under Missing Not At Random Discussion Missing Data in Clinical Trials – 7 / 45 Why there can be no universal method: Overview In contrast with the sampling process, which is usually known, the Towards a principled approach missingness mechanism is usually unknown. • Why is this necessary? • Further... The data alone cannot usually definitively tell us the sampling process. • The E9 guideline, 1999 • Study validity and sensible analysis Likewise, the missingness pattern, and its relationship to the • Why there can be no universal method: observations, cannot identify the missingness mechanism. • Example: Trial of training to improve the quality of peer review With missing data, extra assumptions are thus required for analysis to • Key points for analysis • Towards a systematic proceed. approach • A systematic approach Critique of common The validity of these assumptions cannot be determined from the data at methods hand. Missing At Random Multiple Imputation under Assessing the sensitivity of the conclusions to the assumptions should Missing At Random therefore play a central role. MI: example Multiple Imputation under Missing Not At Random Discussion Missing Data in Clinical Trials – 8 / 45 Example: Trial of training to improve the quality of peer review Overview The graph below shows selected results from a RCT of training to Towards a principled approach improve the quality of peer review of medical articles [10]. Background • Why is this necessary? details are given on your practical sheet. • Further... • The E9 guideline, 1999 • Study validity and no training self−taught package sensible analysis 5 • Why there can be no universal method: • Example: Trial of training to improve the quality of peer review 4 • Key points for analysis • Towards a systematic approach • A systematic approach 3 Critique of common methods Missing At Random 2 Second paper mean RQI Multiple Imputation under Missing At Random MI: example 1 Multiple Imputation under 1 2 3 4 5 1 2 3 4 5 Missing Not At Random First (baseline) paper mean RQI Graphs by Training package Discussion Missing Data in Clinical Trials – 9 / 45 Key points for analysis Overview • the question (i.e. the hypothesis under investigation) — missing data Towards a principled approach usually does not change this; • Why is this necessary? • the information in the observed data, and • Further... • The E9 guideline, 1999 • the reason for missing data. • Study validity and sensible analysis • Why there can be no universal method: We will consider the impact of various assumptions about the missing • Example: Trial of training to improve the data. quality of peer review • Key points for analysis • Towards a systematic Importantly, the data themselves do not tell us which assumptions are approach • A systematic approach true. Critique of common methods We therefore want to explore whether our conclusions are robust to a Missing At Random range of plausible assumptions about the distribution of the missing Multiple Imputation under Missing At Random values. MI: example Multiple Imputation under Note, we can’t get back the missing values themselves! Missing Not At Random Discussion Missing Data in Clinical Trials – 10 / 45 Towards a systematic approach Overview Therefore, although with missing data some information is irretrievably Towards a principled approach lost, we can often salvage a lot. • Why is this necessary? • Further... The success of the salvage operation depends on: • The E9 guideline, 1999 • Study validity and sensible analysis 1. whether we can identify plausible reasons for the data being missing • Why there can be no universal method: (called missingness mechanisms), and • Example: Trial of training to improve the 2. the sensitivity of the conclusions to different missingness quality of peer review • Key points for analysis mechanisms. • Towards a systematic approach A possible systematic