Meta-Analysis of Odds Ratios With Incomplete Extracted Data Shemra Rizzo1 and Robert E. Weiss2 1;2 Department of Biostatistics, University of California, Los Angeles, CA 90095 Abstract A typical random effects meta-analysis of odds-ratios assumes binomially distributed numbers of events in a treatment and control group and requires the proportion of deaths to be extracted from published papers. This data is often not available in the publications due to loss to follow-up. When the Kaplan Meier survival plot is avail- able, it is common practice to manually measure the needed information from the plot and infer the probability of survival and then to infer a best-guess of the number of deaths. Uncertainty introduced from theses guesses is not accounted for in current models. This naive approach leads to over-certain results and potentially inaccurate conclusions. We propose the Uncertain Reading-Estimated Events model to construct each study's contribution to the meta-analysis separately using the data available for extraction in the publications. We use real and simulated data to illustrate our meth- ods. Meta-analysis based on the observed number of deaths lead to biased estimates while our proposed model does not. Our results show increases in the standard devia- tion of the log-odds as compared to a naive meta-analysis that assumes ideal extracted data, equivalent to a reduction of the overall sample size of 43% in our example. Keywords: Bayesian modeling, missing data, censoring, loss to follow-up, survival data, imputation. 1 INTRODUCTION arXiv:1410.2843v1 [stat.ME] 10 Oct 2014 Meta-analysis is widely used to quantitatively combine results from multiple studies. In the health sciences, meta-analysis can provide stronger and more broadly-based evidence for treatment efficacy. Ideally, meta-analysis would analyze the combined individual patient data from all studies (Stewart and Parmar, 1993; Lambert et al., 2002; Berlin et al., 2002). In practice, this is rarely done (Kovalchik, 2012). Instead, meta-analyses rely on data extracted 1 from journal articles in the published literature or from presentations at major conferences. This is known as meta-analysis of aggregate data. A meta-analysis of odds ratios typically requires four quantities to be extracted per study: number of events and non-events in the treatment and control groups, which can easily be summarized in a 2x2 table. The Cochrane Handbook for Systematic Reviews of Interventions recognizes that data required for the meta-analysis are often not available in published papers (Higgins, J.P.T. and Deeks, J.J. (eds.), 2011), for example, the true number of events. Software typically requires the data from the 2x2 table from each study regardless of the outcome that was computed in each study (odds, risk and hazard ratio) (Melle et al., 2004). The frequent occurrence of incomplete extracted data has led to the common practice of guessing the missing entries of the 2x2 table using other information available from the study. For example, it is a very common occurrence to have all four entries missing, but the row totals are known which correspond to the numbers of people in each group at baseline. Best-guesses for the missing entries are computed from Kaplan-Meier (KM) survival curves which are often available in the published study. Because the survival curves rarely include survival probability values, meta-analysts take manual measurements from the curve to estimate them. They then multiply the survival probability and its complement by the row total of each treatment group to fill in the 2x2 table. Estimates are often rounded to the nearest integer. The guessed data is introduced in the meta-analysis as observed data, leading to unjustified certainty in the results and to potentially inaccurate conclusions. In some cases, the KM survival curve is not available and meta-analysts use the number of observed events reported in the text as the best approximation of the true number of events. However, in the presence of loss to follow-up this is an underestimate of the true number of events and could lead to biased meta-analysis estimates. A typical meta-analysis includes a combination of studies where the KM probabilities are available for extraction and studies where only the observed numbers of events are available. While there have been efforts to promote better reporting practices (Riley et al., 2003), there is no established protocol for addressing the missing information encountered in the data extraction step of meta-analysis. For time-to-event data, some methods have been proposed to handle the missing extracted data for meta-analyses of hazard ratios but not for odds ratios (Parmar et al., 1998; Tierney et al., 2007). However, the proposed methods involve calculations that are only approximate, and do not account for the uncertainty in- troduced by the estimation. Methods to account for uncertainty in meta-analysis of odds ratios have been proposed for data from randomized trials that includes number of missing outcomes (non-observed events) (White et al., 2008b,c). However, in time-to-event studies the number of missing outcomes is typically not reported. We propose the Uncertain Reading-Estimated Events (UR-EE) Bayesian model to ac- count for the uncertainty that arises at the data extraction step of meta-analysis. Our model 2 formally constructs a model to properly account for the contribution of each study to the meta-analysis. Our constructions do not depend on the desired data but depend rather on the actual data available from and extracted from each published study. Data available for extraction includes the number of participants at baseline, and may include one or more of the following: the rounded survival probabilities or measurements taken off the KM plot; confidence intervals for the KM survival probabilities; mean, variance, median, and quartiles of the distributions of follow-up times; the number of observed events, and the number of people at risk at the time of interest. Because the information available for extraction is different for each study, the extracted data from each study must be modeled individually. The UR-EE model improves the validity of results from meta-analysis by accommodating all the uncertainty in the data input to the meta-analysis. The paper is organized as follows. We introduce two datasets in section 2. In section 3 we briefly review the classical and Bayesian random effects models and describe the naive approach to manipulate incomplete extracted data to be able to use the models. In section 4 we define the UR-EE Bayesian model for meta-analysis. Results for the two datasets from UR-EE and naive methods are given in section 5. The paper finishes with discussion. 2 DATASETS 2.1 Unprotected left main coronary artery stenosis data To exemplify the different types of extracted data and proposed methodology for a meta- analysis of odds ratios, we carefully re-evaluate a published meta-analysis that compares two treatments for unprotected left main coronary artery (ULMCA) stenosis (Naik et al., 2009). The current gold standard treatment is coronary artery bypass grafting (CABG), which portends high morbidity. Percutaneous coronary intervention (PCI) has emerged as a plausible alternative. It is desirable to draw a definitive assessment of both treatments. The meta-analysis performed by Naik et al. (2009) included 10 studies with a total of 3,773 patients. The meta-analysis of mortality after 1 year presents multiple challenges in the data extraction step. In all studies the type of extractable data was the same for PCI and CABG. Table 1 is a checklist of the components available for extraction from each study. 3 Table 1: Components of the extracted data from each study in the ULMCA meta-analysis ∗ ∗ ∗ ∗ ∗ 2 Study i nij eij rij xij yij κij aij− aij+ mij vij Q1ij Q2ij Q3ij Brener XXXXXX Palmerini XXXXXXX Seung XXXXXX Wu XXXX Sanmartin XXXXXX Buszman XXXXX Makikallio XXXXX White XXXXXX Serryus XXXXXX Chieffo XX Note: nij is the number of people at baseline, eij is the number of observed deaths, rij is the ∗ ∗ number of people at risk at year 1, xij and yij are the measurements of the KM plot from the x-axis ∗ to the curve at baseline and year 1, κij is the rounded KM survival probability and (aij−; aij+) is 2 its confidence interval, mij and vij are the mean and variance of the follow-up times, Q1ij is the median and [Q1ij;Q3ij] is the interquartile range of the follow-up times for study i and group j. The PCI (j = 1) and CABG (j = 0) groups have the same type of extractable data but different values. The ten studies have different types of extractable data: 1. All ten papers provide the number of people enrolled at baseline by treatment group. 2. Two papers provide the number of deaths observed after one year. The number of observed deaths is less than or equal to the true number of deaths, which is unknown due to loss to follow-up. 3. Four papers provide the number of people at risk after one year. 4. Eight papers provide a measure of central tendency and spread of the follow-up times. Two papers provide pooled follow-up times only. One paper provides the mean of the follow-up times by group but no variance. 5. Seven papers have a KM survival plot. Three of these plots have numerical values for the survival probabilities at year 1. For the remaining four plots, the values must be manually extracted from the plot using a ruler either in the computer screen or in print. An additional paper has a mortality rate plot with rounded mortality rates. 6.