Missing Data in Randomised Trials — Overview and Strategies James R

Randomised Controlled Trials in the Social Sciences Conference Missing Data in Randomised Trials — Overview and Strategies

James R. Carpenter London School of Hygiene & Tropical Medicine & MRC CTU at UCL [email protected]

www.missingdata.org.uk

September 7, 2018

Missing Data in Clinical Trials – 1 / 45 Acknowledgements

Overview Harvey Goldstein (LSHTM) • Acknowledgements • Outline Rachael Hughes (Bristol) Mike Kenward (LSHTM) Towards a principled Geert Molenberghs (Limburgs University, Belgium) approach

Critique of common Mike Kenward, James Roger (LSHTM) methods Sara Schroter (BMJ, London) Missing At Random Michael Spratt (Bristol) Multiple Imputation under Missing At Random Jonathan Sterne (Bristol) MI: example Stijn Vansteelandt (Ghent University, Belgium) Multiple Imputation under Missing Not At Random Tim Morris, Ian White (MRC CTU at UCL)

Discussion

Background to this session is in Chs 1 & 2 of ‘Missing data in clinical trials — a practical guide’ (joint with Mike Kenward), commissioned by the NHS, available free on-line at www.missingdata.org.uk.

Missing Data in Clinical Trials – 2 / 45 Outline

Overview 1. Missing data in trials— the need for a principled approach • Acknowledgements • Outline 2. Completers analysis Towards a principled 3. Imputation of simple mean approach

Critique of common 4. Imputation of regression mean methods 5. Last Observation Carried Forward Missing At Random 6. Multiple Imputation Multiple Imputation under Missing At Random • assuming data are Missing At Random MI: example assuming data are Missing Not At Random Multiple Imputation under • Missing Not At Random 7. Discussion Discussion

Missing Data in Clinical Trials – 3 / 45 Why is this necessary?

Overview Missing data are common. Towards a principled approach • Why is this necessary? However, they are usually inadequately handled in both epidemiological • Further... and experimental research. • The E9 guideline, 1999 • Study validity and sensible analysis For example, [14] reviewed 71 recently published BMJ, JAMA, Lancet • Why there can be no universal method: and NEJM papers. • Example: Trial of training to improve the quality of peer review • 89% had partly missing outcome data. • Key points for analysis • Towards a systematic • In 37 trials with repeated outcome measures, 46% performed approach • A systematic approach complete case analysis. Critique of common • Only 21% reported sensitivity analysis. methods Missing At Random Unfortunately, an update in 2014 [1] showed relatively little had changed, Multiple Imputation under Missing At Random although Multiple Imputation [12] was much more commonly used.

MI: example

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 4 / 45 Further...

Overview CONSORT guidelines state that the number of patients with missing data Towards a principled approach should be reported by treatment arm. • Why is this necessary? • Further... But [5] estimate that 65% of studies in PubMed journals do not report the • The E9 guideline, 1999 • Study validity and handling of missing data. sensible analysis • Why there can be no universal method: [14] identiﬁed serious weaknesses in the description of missing data and • Example: Trial of training to improve the the methodology adopted. quality of peer review • Key points for analysis • Towards a systematic approach • A systematic approach

Critique of common methods

Missing At Random

Multiple Imputation under Missing At Random

MI: example

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 5 / 45 The E9 guideline, 1999

Overview • Missing data are a potential source of bias Towards a principled approach • Avoid if possible (!) • Why is this necessary? • With missing data, a trial may still be regarded as valid if the methods • Further... • The E9 guideline, 1999 are sensible, and preferably predeﬁned • Study validity and sensible analysis • There can be no universally applicable method of handling missing • Why there can be no universal method: data • Example: Trial of training to improve the • The sensitivity of conclusions to methods should thus be investigated, quality of peer review • Key points for analysis particularly if there are a large number of missing observations • Towards a systematic approach Guidelines downloadable from www.ich.org • A systematic approach

Critique of common methods The question is, how do we apply these principles in practice?

Missing At Random

Multiple Imputation under Missing At Random

MI: example

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 6 / 45 Study validity and sensible analysis

Overview Missing data are observations we intended to make but did not. Towards a principled approach • Why is this necessary? The sampling process involves both the selection of the units, and the • Further... process by which observations become missing — the missingness • The E9 guideline, 1999 • Study validity and mechanism. sensible analysis • Why there can be no universal method: Thus for sensible inference, we need to take account of the missingness • Example: Trial of training to improve the mechanism quality of peer review • Key points for analysis • Towards a systematic By sensible we mean: approach • A systematic approach • Frequentist: nominal properties hold. Eg: Critique of common methods Estimators consistent; conﬁdence intervals attain nominal coverage. Missing At Random • Bayesian: Multiple Imputation under Posterior distribution is unbiased, correctly reﬂects loss of information Missing At Random

MI: example due to missingness mechanism.

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 7 / 45 Why there can be no universal method:

Overview In contrast with the sampling process, which is usually known, the Towards a principled approach missingness mechanism is usually unknown. • Why is this necessary? • Further... The data alone cannot usually deﬁnitively tell us the sampling process. • The E9 guideline, 1999 • Study validity and sensible analysis Likewise, the missingness pattern, and its relationship to the • Why there can be no universal method: observations, cannot identify the missingness mechanism. • Example: Trial of training to improve the quality of peer review With missing data, extra assumptions are thus required for analysis to • Key points for analysis • Towards a systematic proceed. approach • A systematic approach

Critique of common The validity of these assumptions cannot be determined from the data at methods hand. Missing At Random Multiple Imputation under Assessing the sensitivity of the conclusions to the assumptions should Missing At Random therefore play a central role. MI: example

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 8 / 45 Example: Trial of training to improve the quality of peer review

Overview The graph below shows selected results from a RCT of training to Towards a principled approach improve the quality of peer review of medical articles [10]. Background • Why is this necessary? details are given on your practical sheet. • Further... • The E9 guideline, 1999 • Study validity and no training self−taught package sensible analysis 5 • Why there can be no universal method: • Example: Trial of training to improve the quality of peer review 4 • Key points for analysis • Towards a systematic approach

• A systematic approach 3

Critique of common methods

Missing At Random 2 Second paper mean RQI Multiple Imputation under Missing At Random

MI: example 1

Multiple Imputation under 1 2 3 4 5 1 2 3 4 5 Missing Not At Random First (baseline) paper mean RQI Graphs by Training package Discussion

Missing Data in Clinical Trials – 9 / 45 Key points for analysis

Overview • the question (i.e. the hypothesis under investigation) — missing data Towards a principled approach usually does not change this; • Why is this necessary? • the information in the observed data, and • Further... • The E9 guideline, 1999 • the reason for missing data. • Study validity and sensible analysis • Why there can be no universal method: We will consider the impact of various assumptions about the missing • Example: Trial of training to improve the data. quality of peer review • Key points for analysis • Towards a systematic Importantly, the data themselves do not tell us which assumptions are approach • A systematic approach true.

Critique of common methods We therefore want to explore whether our conclusions are robust to a Missing At Random range of plausible assumptions about the distribution of the missing Multiple Imputation under Missing At Random values.

MI: example

Multiple Imputation under Note, we can’t get back the missing values themselves! Missing Not At Random

Discussion

Missing Data in Clinical Trials – 10 / 45 Towards a systematic approach

Overview Therefore, although with missing data some information is irretrievably Towards a principled approach lost, we can often salvage a lot. • Why is this necessary? • Further... The success of the salvage operation depends on: • The E9 guideline, 1999 • Study validity and sensible analysis 1. whether we can identify plausible reasons for the data being missing • Why there can be no universal method: (called missingness mechanisms), and • Example: Trial of training to improve the 2. the sensitivity of the conclusions to different missingness quality of peer review • Key points for analysis mechanisms. • Towards a systematic approach A possible systematic approach is the following: • A systematic approach

Critique of common methods

Missing At Random

Multiple Imputation under Missing At Random

MI: example

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 11 / 45 A systematic approach

Overview Investigators discuss possible missingness mechanisms, say A–E Towards a principled approach possibly informed by a blind review of the data, and consider their • Why is this necessary? plausibility. Then • Further... • The E9 guideline, 1999 • Study validity and 1. Under most plausible mechanism A, perform valid analysis, draw sensible analysis • Why there can be no conclusions universal method: • Example: Trial of 2. Under similar mechanisms, B–C, perform valid analysis, draw training to improve the quality of peer review conclusions • Key points for analysis 3. Under least plausible mechanisms, D–E, perform valid analysis, • Towards a systematic approach draw conclusions • A systematic approach

Critique of common Investigators discuss the implications, and arrive at a valid interpretation methods of the trial. Missing At Random

Multiple Imputation under Missing At Random This approach broadly agrees with the E9 guideline.

MI: example

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 12 / 45 Completers analysis

Overview Variables Towards a principled approach Unit 1 2 Critique of common methods 1 3.4 5.67 • Completers analysis deletes all units • Completers analysis with incomplete data. • Simple (marginal) 2 3.9 4.81 mean imputation • In RCTs with single follow-up, OK if • Reviewer trial 3 2.6 4.93 • Regression mean baseline variables predictive of out- imputation 4 1.9 6.21 • Reviewer Trial: 5 2.2 6.83 come & missing data included. regression mean imputation 6 3.3 5.61 • With longitudinal follow-up, likely both • Last (Baseline) Observation Carried 7 1.7 5.45 biased and inefﬁcient. Forward (LOCF) • Reviewer trial: BOCF 8 2.4 4.94 • General comments 9 2.8 5.73 Missing At Random 10 3.6 · Multiple Imputation under Missing At Random

MI: example

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 13 / 45 Simple (marginal) mean imputation

Overview Variables Towards a principled approach Unit 1 2 Critique of common methods 1 3.4 5.67 Missing observations replaced by ob- • Completers analysis • • Simple (marginal) 2 3.9 4.81 served mean for that variable mean imputation 3 2.6 4.93 • Reviewer trial • Inappropriate for categorical variables • Regression mean imputation 4 1.9 6.21 • Reviewer Trial: 5 2.2 6.83 regression mean • Reduces associations in data imputation 6 3.3 5.61 • Last (Baseline) • Variance underestimated Observation Carried 7 1.7 5.45 Forward (LOCF) • Reviewer trial: BOCF 8 2.4 4.94 • General comments 9 2.8 5.73 Missing At Random 10 3.6 5.58 Multiple Imputation under Missing At Random

MI: example

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 14 / 45 Reviewer trial

Overview Towards a principled no training self−taught package approach 5 Critique of common methods • Completers analysis • Simple (marginal) mean imputation 4 • Reviewer trial • Regression mean imputation • Reviewer Trial: regression mean 3 imputation • Last (Baseline) Observation Carried Forward (LOCF) 2 • Reviewer trial: BOCF Second paper mean RQI • General comments

Missing At Random

Multiple Imputation under 1 Missing At Random 1 2 3 4 5 1 2 3 4 5 MI: example First (baseline) paper mean RQI Multiple Imputation under Graphs by Training package Missing Not At Random Discussion Treatment effect: 0.220, SE 0.059, p < 0.001 Missing Data in Clinical Trials – 15 / 45 Regression mean imputation

Overview Variables Towards a principled • Use regression, here OLS: approach Unit 1 2 Critique of common V2 = α + βV1 + e methods 1 3.4 5.67 Using units 1–9 we get: • Completers analysis • • 2 3.9 4.81 Simple (marginal) Vˆ2 =6.56 − 0.366 × (V1). mean imputation 3 2.6 4.93 • Reviewer trial • For unit 10 this gives • Regression mean 4 1.9 6.21 imputation 6.56 − 0.366 × (3.6)=5.24. • Reviewer Trial: 5 2.2 6.83 regression mean • Now obtain unbiased estimators of imputation 6 3.3 5.61 • Last (Baseline) means, associations, under MAR Observation Carried 7 1.7 5.45 Forward (LOCF) • Variance still (often markedly) under- • Reviewer trial: BOCF 8 2.4 4.94 estimated • General comments 9 2.8 5.73 Missing At Random 10 3.6 5.24 Multiple Imputation under Missing At Random

MI: example

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 16 / 45 Reviewer Trial: regression mean imputation

Missing At Random

Multiple Imputation under 1 Missing At Random 1 2 3 4 5 1 2 3 4 5 MI: example First (baseline) paper mean RQI Multiple Imputation under Graphs by Training package Missing Not At Random Discussion Effect estimate: 0.237; SE: 0.0575, p < 0.001 Missing Data in Clinical Trials – 17 / 45 Last (Baseline) Observation Carried Forward (LOCF)

Overview Towards a principled • Makes strong, implausible as- approach Unit sumptions Critique of common Time 1 2 methods • Imputes a degenerate distribu- • Completers analysis 1 2.1 3.4 • Simple (marginal) tion mean imputation 2 3.8 3.9 • Reviewer trial • Means and variances wrong • Regression mean 3 3.8 ←· 2.6 imputation • In general neither conservative • 4 3.8 ←· 1.9 Reviewer Trial: or liberal; bias depends on un- regression mean 5 3.8 ←· 2.2 imputation known treatment effect! • Last (Baseline) 6 3.8 ←· 2.2 ←· Observation Carried • See [8, 6, 2] Forward (LOCF) . . . • Reviewer trial: BOCF . . . • General comments

Missing At Random

Multiple Imputation under Missing At Random Baseline carried forward is a (worse) variant of LOCF MI: example

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 18 / 45 Reviewer trial: BOCF

Missing At Random

Multiple Imputation under 1 Missing At Random 1 2 3 4 5 1 2 3 4 5 MI: example First (baseline) paper mean RQI Multiple Imputation under Graphs by Training package Missing Not At Random Discussion Intervention effect: 0.153, SE=0.061, p =0.013 Missing Data in Clinical Trials – 19 / 45 General comments

Overview 1. Single imputation methods generally put the ‘cart before the Towards a principled approach horse’—that is they adopt a simple approach to ‘complete’ the Critique of common dataset, and then look for arguments to justify this. methods • Completers analysis • Simple (marginal) Instead, we should — in consultation with those involved in the trial mean imputation • Reviewer trial (particularly those collecting the data) — make contextually • Regression mean imputation appropriate assumptions and then use a statistically principled • Reviewer Trial: regression mean method to draw inferences under those assumptions. imputation • Last (Baseline) 2. Further, when we use a single imputation method, our analysis Observation Carried Forward (LOCF) cannot distinguish between observed and imputed data — they have • Reviewer trial: BOCF the same status. • General comments Missing At Random The consequence is that the standard error is underestimated, and Multiple Imputation under Missing At Random may often be smaller than if we had no missing values!

MI: example

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 20 / 45 A better starting assumption: Missing At Random (MAR)

Overview A more reasonable starting assumption is that, given baseline and early Towards a principled approach follow-up data, the distribution of outcome data is the same whether or Critique of common not we observe it. methods

Missing At Random • A better starting In Rubin’s (1976) taxonomy of missing data [9], this is called ‘Missing At assumption: Missing At 1 Random (MAR) Random’ . • Example: Income and Job Type; true mean £45,000 It corresponds to saying that given the observed data the probability of • A note on the consequence of Missing data being missing does not depend on the value of that data. At Random in regression

Multiple Imputation under Missing At Random

MI: example

Multiple Imputation under Missing Not At Random

Discussion

1If the chance of the data being missing is unrelated to our inferential question (e.g. intervention effect) the data can be viewed as Missing Completely At Random (MCAR)

Missing Data in Clinical Trials – 21 / 45 Example: Income and Job Type; true mean £45,000

Overview Towards a principled 68/100 observed approach observed missing Critique of common methods

Missing At Random • A better starting assumption: Missing At Random (MAR) • Example: Income and 89/100 observed Job Type; true mean £45,000 • A note on the consequence of Missing At Random in regression

Multiple Imputation under Income (thousand pounds) Missing At Random

MI: example Mean observed: £60,927 Mean observed: £29,566 Multiple Imputation under 35 45 55 65 75 85 95 Missing Not At Random Banker Academic Job type Discussion Observed income: £43, 149. 100 × 60, 927 + 100 × 29, 566 MAR estimate: = £45, 246 200 Missing Data in Clinical Trials – 22 / 45 A note on the consequence of Missing At Random in regression

Overview • If the dependent variable is MAR given the covariates in a regression, Towards a principled approach then we get valid inference (the reverse is not true).

Critique of common methods • In a RCT with longitudinal follow-up, if all observed repeated outcome Missing At Random measures are included in the model alongside the covariates, again • A better starting assumption: Missing At we get valid inference under the MAR assumption. Random (MAR) • Example: Income and Job Type; true mean • However, we need to choose the model carefully (cf Ch 2, [4], and £45,000 • A note on the impose minimal structure on the mean and covariance of the data. consequence of Missing At Random in regression • In many cases (e.g. GLMs), we may not want to include all the Multiple Imputation under Missing At Random variables needed for a plausible MAR assumption in our model. MI: example • In these settings, and when we move beyond MAR, of Multiple Multiple Imputation under Missing Not At Random Imputation (MI) provides a practical approach.

Discussion

Missing Data in Clinical Trials – 23 / 45 Intuition for MI

Overview Suppose our data set has variables X,Y with some Y values MAR Towards a principled approach given X. Using only subjects with both observed we can get valid Critique of common estimates of the regression of Y on X. methods

Missing At Random We now show how MI works in this setting to also arrive at valid Multiple Imputation under Missing At Random inference. • Intuition for MI • The idea: The attraction of MI for trials is that • The key idea • Intuition behind 1. we can include additional variables, not in our main analysis model, multiple imputation: 1 • Intuition behind to improve the plausibility of MAR, and multiple imputation: 2 2. we can explore the robustness of our inferences to departures from MI: example MAR. Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 24 / 45 The idea:

Overview 1. Fit the regression of Y on X Towards a principled approach

Critique of common 2. Use this to impute the missing Y methods Missing At Random 3. With this completed data set, calculate our statistic of interest (e.g. Multiple Imputation under Missing At Random sample mean, variance, regression of X on Y , regression of Y on • Intuition for MI X). • The idea: • The key idea • Intuition behind multiple imputation: 1 As we can only ever know the distribution of missing data (given • Intuition behind multiple imputation: 2 observed), steps 2 & 3 have to be repeated, and the results averaged in MI: example some way. Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 25 / 45 The key idea

Overview The key idea is to use the data from individuals where both Y and X are Towards a principled approach observed to learn about the relationship between Y and X. Critique of common methods Then, if X˜ represents the vector of X values from individuals with Missing At Random missing Y ’s, we use this relationship to complete the data set by drawing Multiple Imputation under Missing At Random the missing observations from YM |X.˜ • Intuition for MI • The idea: • The key idea We do this K (typically >> 5) times, giving rise to K complete data sets. • Intuition behind multiple imputation: 1 • Intuition behind We analyse each of these data sets in the usual way. multiple imputation: 2 MI: example We combine the results using particular rules. Multiple Imputation under Missing Not At Random

Discussion Suppose the analysis of interest is calculating the marginal mean of Y , or regressing X on Y.

Missing Data in Clinical Trials – 26 / 45 Intuition behind multiple imputation: 1

Overview Model observed pairs, denoted (YO,X). Towards a principled approach

Critique of common methods

Missing At Random

Multiple Imputation under Missing At Random • Intuition for MI • The idea: • The key idea • Intuition behind multiple imputation: 1 Y • Intuition behind multiple imputation: 2

MI: example

Multiple Imputation under Missing Not At Random

Discussion

0.0 0.5 1.0 1.5??? 2.0

0 1 2 3 4

Missing Data in Clinical Trials – 27 / 45 Intuition behind multiple imputation: 2

Overview Draw YM by (i) drawing from distribution of regression line (this gives us Towards a principled approach the solid (red) line below) (ii) then drawing from variability about that line. Critique of common methods

Missing At Random

Multiple Imputation under Missing At Random • Intuition for MI • The idea: • The key idea • Intuition behind multiple imputation: 1 • Intuition behind

multiple imputation: 2 Y

MI: example

Multiple Imputation under Missing Not At Random

Discussion 0.0 0.5 1.0 1.5 2.0

0 1 2 3 4

Missing Data in Clinical Trials – 28 / 45 Results of multiple imputation:

Overview Data Imputation 1 Imputation 2 Imputation 3 Imputation 4 Towards a principled Y X Y X Y X Y X Y X approach Unit Critique of common 1 1.1 3.4 1.1 3.4 1.1 3.4 1.1 3.4 1.1 3.4 methods 2 1.5 3.9 1.5 3.9 1.5 3.9 1.5 3.9 1.5 3.9 Missing At Random 3 2.3 2.6 2.3 2.6 2.3 2.6 2.3 2.6 2.3 2.6 Multiple Imputation under 4 3.6 1.9 3.6 1.9 3.6 1.9 3.6 1.9 3.6 1.9 Missing At Random 5 0.8 2.2 0.8 2.2 0.8 2.2 0.8 2.2 0.8 2.2 MI: example • Results of multiple 6 3.6 3.3 3.6 3.3 3.6 3.3 3.6 3.3 3.6 3.3 imputation: 7 3.8 1.7 3.8 1.7 3.8 1.7 3.8 1.7 3.8 1.7 • Notation for analyses of K imputed datasets 8 ? 0.8 0.2 0.8 0.8 0.8 0.3 0.8 2.3 0.8 • Rubin’s MI rules 9 ? 2.0 1.7 2.0 2.4 2.0 1.8 2.0 3.5 2.0 • MI variance rule 10 ? 3.2 2.7 3.2 2.5 3.2 1.0 3.2 1.7 3.2 • Inference for θ • Video of multiple imputation • MI under MAR for the reviewer data • Comparison of MI and Complete Cases

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 29 / 45 Notation for analyses of K imputed datasets

Overview Analysing each imputed (i.e. ‘completed’) dataset the usual way (i.e. Towards a principled approach using the model intended if there were no missing data) gives us K Critique of common estimates of the original quantity of interest, say θ. Denote these methods estimates ˆ1 ˆ Missing At Random θ ,..., θK .

Multiple Imputation under Missing At Random The analysis of each imputed data set will also give an estimate of the 2 MI: example variance of the estimate θˆk, say σˆ . Again, this is the usual variance • Results of multiple k imputation: estimate from the model. • Notation for analyses of K imputed datasets • Rubin’s MI rules • MI variance rule • Inference for θ • Video of multiple imputation • MI under MAR for the reviewer data • Comparison of MI and Complete Cases

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 30 / 45 Rubin’s MI rules

Overview Rubin’s MI rules combine these quantities to get our overall estimate and Towards a principled approach its variance using certain rules.

Critique of common methods Let the multiple imputation estimate of θ be θˆMI . Then Missing At Random Multiple Imputation under K Missing At Random 1 θˆMI = θˆk. MI: example K • Results of multiple Xk=1 imputation: • Notation for analyses of K imputed datasets • Rubin’s MI rules • MI variance rule • Inference for θ • Video of multiple imputation • MI under MAR for the reviewer data • Comparison of MI and Complete Cases

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 31 / 45 MI variance rule

Overview Further deﬁne the within-imputation and between-imputation Towards a principled approach components of variance by

Critique of common methods K K 2 1 2 2 1 2 Missing At Random ˆ ˆ σˆw = σˆk, and σˆb = (θk − θMI ) , Multiple Imputation under K =1 K − 1 =1 Missing At Random Xk Xk

MI: example • Results of multiple imputation: Then • Notation for analyses of 2 1 2 2 K imputed datasets σˆMI = 1 + σˆb +ˆσw, • Rubin’s MI rules K • MI variance rule • Inference for θ • Video of multiple imputation so the estimated standard error of θˆMI is σˆMI . • MI under MAR for the reviewer data • Comparison of MI and Complete Cases

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 32 / 45 Inference for θ

Overview To test the null hypothesis θ = θ0, compare Towards a principled approach Critique of common θˆMI − θ0 methods to tν, σˆ Missing At Random MI Multiple Imputation under where Missing At Random 2 2 MI: example σˆw • Results of multiple ν =(K − 1) 1 + 2 . imputation: (1+1/K)ˆσb • Notation for analyses of K imputed datasets Thus, if tν,0.975 is the 97.5% point of the t distribution with ν degrees of • Rubin’s MI rules • MI variance rule freedom, the 95% conﬁdence interval is • Inference for θ • Video of multiple imputation (θˆMI − σˆMI tν,0.975, θˆMI +ˆσMI tν,0.975) • MI under MAR for the reviewer data • Comparison of MI and Complete Cases

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 33 / 45 Video of multiple imputation

Overview This video illustrates imputation of missing data using an asthma study. Towards a principled approach

Critique of common The outcome is lung function 12 weeks after randomisation, and we methods adjust for baseline and estimate the treatment effect. Missing At Random

Multiple Imputation under Missing At Random

MI: example • Results of multiple imputation: • Notation for analyses of K imputed datasets • Rubin’s MI rules • MI variance rule • Inference for θ • Video of multiple imputation • MI under MAR for the reviewer data • Comparison of MI and Complete Cases

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 34 / 45 MI under MAR for the reviewer data

Overview Towards a principled no training self−taught package approach 5 Critique of common methods

Missing At Random

Multiple Imputation under 4 Missing At Random

MI: example • Results of multiple imputation: 3 • Notation for analyses of K imputed datasets • Rubin’s MI rules • MI variance rule

• θ 2 Inference for Second paper mean RQI • Video of multiple imputation • MI under MAR for the reviewer data

• Comparison of MI and 1 Complete Cases 1 2 3 4 5 1 2 3 4 5 Multiple Imputation under First (baseline) paper mean RQI Missing Not At Random Graphs by Training package Discussion

Missing Data in Clinical Trials – 35 / 45 Comparison of MI and Complete Cases

Overview Towards a principled approach Analysis Est SE p-value 95% CI Critique of common methods

Missing At Random Multiple Imputation under Missing at random Missing At Random

MI: example • Results of multiple Observeddata 0.24 0.070 0.001 (0.10,0.38) imputation: • Notation for analyses of MI under MAR, K = 20 0.23 0.071 0.002 (0.09,0.37) K imputed datasets • Rubin’s MI rules • MI variance rule • Inference for θ • Video of multiple imputation • MI under MAR for the reviewer data • Comparison of MI and Complete Cases

Multiple Imputation under Missing Not At Random

Discussion

Missing Data in Clinical Trials – 36 / 45 MNAR in the reviewer study

Overview The practical sheet outlines why it is plausible that the missing RQIs in Towards a principled approach the reviewer trial are not MAR.

Critique of common methods An attractive option is to elicit expert opinion on the difference between Missing At Random the observed and missing values. This can be viewed as attempting to Multiple Imputation under Missing At Random quantify how experts would implicitly adjust their interpretation of the MI: example study due to the missing data. See [11, 7]. Multiple Imputation under Missing Not At Random • MNAR in the reviewer study • Prior elicitation for reviewer trial • Elicitation of expert opinion • Model • Using this information with MI • One of the MNAR imputed datasets • Comparison of results

Discussion

Missing Data in Clinical Trials – 37 / 45 Prior elicitation for reviewer trial

Overview For the peer review study, [13] devised a questionnaire, designed to elicit Towards a principled approach experts’ prior belief about the difference, δ, between the average missing Critique of common and average observed review quality index in this study. methods Missing At Random This was completed by 2 investigators and 20 editors and staff at the Multiple Imputation under Missing At Random British Medical Journal.

MI: example

Multiple Imputation under The resulting distribution is negatively skewed, with mean −0.21 and SD Missing Not At Random • MNAR in the reviewer 0.46. study • Prior elicitation for reviewer trial Unfortunately, it was not possible to collect information about how this • Elicitation of expert opinion was inﬂuenced by the randomised intervention (i.e. whether δN and δS • Model • Using this information have the same mean, and what their correlation is). with MI • One of the MNAR imputed datasets • Comparison of results

Discussion

Missing Data in Clinical Trials – 38 / 45 Elicitation of expert opinion

Overview We adopt a bivariate normal model approximation to the prior: Towards a principled approach

Critique of common δN −0.21 2 1 ρ methods ∼ N , 0.46 . (1) δS −0.21 ρ 1 Missing At Random

Multiple Imputation under Missing At Random Because it was not possible to elicit a prior on ρ from the experts, we

MI: example analyse the data with ρ =0; assuming ρ ≥ 0, this choice gives the Multiple Imputation under largest standard error for the intervention effect. Missing Not At Random • MNAR in the reviewer study • Prior elicitation for reviewer trial • Elicitation of expert opinion • Model • Using this information with MI • One of the MNAR imputed datasets • Comparison of results

Discussion

Missing Data in Clinical Trials – 39 / 45 Model

Overview Given a draw (δN ,δS) from this distribution the pattern mixture model is Towards a principled approach

Critique of common methods RQIi = β0 + β1self-taughti + β2baseline RQIi + ei if RQIi observed, Missing At Random RQIi =(β0 + δN )+(β1 + δS − δN )self-taughti Multiple Imputation under Missing At Random + β2baseline RQIi + ei if RQIi missing, MI: example iid 2 Multiple Imputation under ei ∼N(0,σ ). (2) Missing Not At Random • MNAR in the reviewer study Thus the mean review quality, relative to that in the observed data, is • Prior elicitation for reviewer trial changed by δN in the no intervention arm and δS in the self-taught arm. • Elicitation of expert opinion • Model • Using this information with MI • One of the MNAR imputed datasets • Comparison of results

Discussion

Missing Data in Clinical Trials – 40 / 45 Using this information with MI

Overview We proceed as follows: Towards a principled approach 1. Fit the imputation model to the observed data. For k =1,...,K, Critique of common methods draw from the posterior distribution of the imputation model k k k k 2,k Missing At Random parameters θ =(β0 ,β1 ,β2 ,σ ). Multiple Imputation under 2. Draw k k from (1) Missing At Random (δN ,δS) k MI: example 3. Using the draws obtained in steps 1 and 2, impute the missing RQIi Multiple Imputation under using (2). Missing Not At Random • MNAR in the reviewer study Steps 1–3 are repeated to create K = 20 imputed data sets. Then we ﬁt • Prior elicitation for the substantive model to each imputed data set and apply Rubin’s rules. reviewer trial • Elicitation of expert opinion • Model • Using this information with MI • One of the MNAR imputed datasets • Comparison of results

Discussion

Missing Data in Clinical Trials – 41 / 45 One of the MNAR imputed datasets

Overview Towards a principled no training self−taught package approach 5 Critique of common methods

Missing At Random

Multiple Imputation under 4 Missing At Random

MI: example

Multiple Imputation under 3 Missing Not At Random • MNAR in the reviewer study • Prior elicitation for reviewer trial 2 • Elicitation of expert Second paper mean RQI opinion • Model • Using this information with MI 1 • One of the MNAR imputed datasets 1 2 3 4 5 1 2 3 4 5 • Comparison of results First (baseline) paper mean RQI Discussion Graphs by Training package

Missing Data in Clinical Trials – 42 / 45 Comparison of results

Overview Towards a principled approach Analysis Est SE p-value 95% CI

Critique of common methods

Missing At Random Missing at random Multiple Imputation under Missing At Random

MI: example Observed data 0.24 0.070 0.001 (0.10, 0.38) Multiple Imputation under MI under MAR, K = 20 0.23 0.071 0.002 (0.09, 0.37) Missing Not At Random • MNAR in the reviewer study Missing not at random • Prior elicitation for reviewer trial • Elicitation of expert opinion Experts’ Prior: 0.21 0.178 0.250 (−0.158, 0.575) • Model • Using this information with MI • One of the MNAR imputed datasets • Comparison of results

Discussion

Missing Data in Clinical Trials – 43 / 45 Summary & Discussion

Overview • Missing data introduce ambiguity into the analysis, beyond the Towards a principled approach familiar sampling imprecision.

Critique of common methods • Extra assumptions about the missingness mechanism are needed;

Missing At Random these assumptions can rarely be veriﬁed from the data at hand. Multiple Imputation under Sensitivity analysis is therefore important. Missing At Random •

MI: example • Single imputation approaches should be avoided. Multiple Imputation under Missing Not At Random • MAR is a natural assumption for the primary analysis.

Discussion • Using MI for inference under MAR allows us to use all the longitudinal • Summary & Discussion follow-up information and all baseline variables that are predictive of • References missing values. • MI can also be used for sensitivity analysis, using the pattern mixture approach as illustrated above. More details and examples in [3]

Missing Data in Clinical Trials – 44 / 45 References

[1] Melanie L Bell, Mallorie Fiero, Nicholas J Horton, and Chiu-Hsieh Hsu. Handling missing data in randomised clinical trials; a review of the top medical journals. BMC Medical Research Methodology, 14:118, 2014 Overview [2] J Carpenter, M Kenward, S Evans, and I White. Letter to the editor: Last observation carry forward and last observation analysis by J. Shao and B. Zhong, Statistics in Medicine, 2003, 22, 2429–2441. Statistics in Medicine, 23:3241–3244, 2004 Towards a principled [3] J R Carpenter. Mulitple imputation based sensitivity analysis, in Wiley Statistics Reference Online, Editors: Ruggeri, F., Piegorsch, approach W., Davidian, M. Kenett, R, Molenberghs, G. and Longford, N. T., 2018 Critique of common [4] James R Carpenter and Michael G Kenward. Missing data in clinical trials — a practical guide. Birmingham: National Health methods Service Co-ordinating Centre for Research Methodology, 2008 [5] A-W Chan and Douglas G Altman. Epidemiology and reporting of randomised trials published in PubMed journals. The Lancet, Missing At Random 365:1159–1162, 2005 [6] R J Cook, L Zeng, and G Y Yi. Marginal analysis of incomplete longitudinal binary data; a cutionary note on locf imputation. Multiple Imputation under Biometrics, pages 820–828, 2004 Missing At Random [7] Alexina J Mason, Manuel Gomes, Richard Grieve, Pinar Ulug, Janet T Powell, and James R Carpenter. Development of a practical MI: example approach to expert elicitation for randomised controlled trials with missing health outcomes: Application to the improve trial. Clinical Trials, 14:357–367, 2017 Multiple Imputation under [8] G Molenberghs, H Thijs, I Jansen, C Beunkens, M G Kenward, C Mallinkrodt, and R J Carroll. Analyzing incomplete longitudinal Missing Not At Random clinical trial data. Biostatistics, 5:445–464, 2004 [9] D B Rubin. Inference and missing data. Biometrika, 63:581–592, 1976 Discussion [10] S Schroter, N Black, S Evans, J Carpenter, F Godlee, and R Smith. Effects of training on quality of peer review: randomised • Summary & Discussion controlled trial. British Medical Journal, 328:673–675, 2004 • References [11] M Smuk, J R Carpenter, and T P Morris. What impact do assumptions about missing data have on conclusions? a practical sensitivity analysis for a cancer survival registry. BMC Med Res Methodol, 17(1):21, 2017 [12] J A C Sterne, I R White, J B Carlin, M Spratt, P Royston, M G Kenward, A M Wood, and J R Carpenter. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. British Medical Journal, 339:157–160, 2009 [13] I White, J Carpenter, Stephen Evans, and Sara Schroter. Eliciting and using expert opinions about non-response bias in randomised controlled trials. Clinical Trials, 4:125–139, 2007 [14] Angela M Wood, Ian R White, and Simon G Thompson. Are missing outcome data adequately handled? a review of published randomized controlled trials in major medical journals. Clinical Trials, 1:368–376, 2004

Missing Data in Clinical Trials – 45 / 45