Final Research Report

PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE FINAL RESEARCH REPORT

The Benefits and Challenges of Using Multiple Sources of Information about Clinical Trials

Kay Dickersin, PhD (Principal investigator), Professor; Evan Mayo-Wilson, MPA, DPhil; Tianjing Li, Assistant Professor

AFFILIATION: For the MUDS Investigators, Center for Clinical Trials and Evidence Synthesis, Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland

Original project title: Integrating Multiple Data Sources for Meta-analysis to Improve Patient-Centered Outcomes Research PCORI ID: ME-1303-5785 HSRProj ID: 20143578

______To cite this document, please use: Dickersin K, Mayo-Wilson E, Li T. (2018). The Benefits and Challenges of Using Multiple Sources of Information about Clinical Trials. Patient-Centered Outcomes Research Institute (PCORI). https://doi.org/10.25302/3.2018.ME.13035785

TABLE OF CONTENTS

ABSTRACT ...... 4 BACKGROUND ...... 6 Objectives and Overview ...... 6 Patient-Centered Outcomes ...... 6 Reporting Biases ...... 7 Multiple Data Sources ...... 7 METHODS ...... 9 Selection of Case Studies ...... 9 Methods for Identifying Patient-Centered Outcomes ...... 10 Effect of Multiple Data Sources on Meta-analyses ...... 11 Table 1. Eligibility Criteria for the Case Studies ...... 12 Table 2. The 5 Elements of an Outcome ...... 14 Analysis ...... 15 RESULTS ...... 18 Patient-Centered Outcomes Identified Through Existing Sources and Stakeholder Coinvestigators ...... 18 Effects of Adding and Replacing Data From Multiple Data Sources on the Results of Meta-analyses ...... 20 Table 3. Multiple Sources of Trials of Gabapentin for Neuropathic Pain ...... 22 Table 4. Multiple Sources of Trials of Quetiapine for Bipolar Depression ...... 24 Table 5. Trial Quality (Risk of Bias) for Sources of Gabapentin for Neuropathic Pain Trialsa ...... 26 Table 6. Trial Quality (Risk of Bias) by Source of Quetiapine for Bipolar Depression Trialsa ...... 28 Table 7. Number of Prespecified Outcome Domains in Each Source of Gabapentin for Neuropathic Pain Trials ...... 30 Table 8. Number of Prespecified Outcome Domains in Each Source of Quetiapine for Bipolar Depression Pain Trials ...... 32 Figure 1. Number of Outcomes in a Trial Is a Function of the Number of Definitions of Each of the 5 Elements ...... 33 Figure 2. Number of Outcomes Observed Increased as a Consequence of Multiple Definitions of the Elements ...... 35

Table 9. Series of Meta-analyses and the Effect of Adding or Replacing Data Sources on the Pooled Estimate of Pain Intensity in Trials of Gabapentin for Neuropathic Pain (SMD [95% CI])a ...... 38 Table 10. Series of Meta-analyses and the Effect of Adding or Replacing Data Sources on the Pooled Estimate of Depression in Trials of Quetiapine Bipolar Depression (SMD [95% CI])a ...... 40 Table 11. Main Findings From MUDS and our Recommendations for Application of the Findings...... 42 Guidance for Using Multiple Data Sources in Systematic Reviews ...... 44 Table 12. Recommendations for Systematic Reviews Using Multiple Data Sources ...... 45 DISCUSSION ...... 46 Identifying Patient-Centered Outcomes ...... 46 Effects of Adding and Replacing Data From Multiple Data Sources on the Results of Meta-analyses ...... 46 Implications for Data Sharing ...... 48 Limitations ...... 48 CONCLUSIONS ...... 50 REFERENCES ...... 51 ACKNOWLEDGMENTS ...... 68

ABSTRACT

Background: International standards include recommendations that systematic reviews be comprehensive, but time and resources may render it impractical to search for and extract data from all possible sources of information. For example, many data sources exist for randomized controlled trials (RCTs), a number of which are not publicly available or are difficult or impossible to access. Searching nonpublic sources of RCTs may improve the impact of systematic reviews by identifying information that was recorded but not included in public sources, thus reducing the need for additional RCTs and improving research efficiency.

Objectives: To determine whether multiple data sources about RCTs affected systematic reviews and meta-analyses of patient-centered outcomes (PCOs) research (eg, trial quality assessment, pooled effect estimates) and to produce open access guidance about using multiple data sources, for producers of systematic reviews of PCOs research.

Methods: We conducted a methods study related to the conduct of systematic reviews using 2 case studies: (1) gabapentin for neuropathic pain and (2) quetiapine for bipolar depression. Applying comprehensive searching methods as if we were conducting 2 systematic reviews, we attempted to identify all data sources for eligible RCTs. We extracted data and compared the information about trial design and trial quality (ie, risk of bias) using the multiple data sources. Using these multiple data sources, we extracted information about prespecified outcome domains, including the definition of each outcome (ie, the outcome domain, measure, metric, method of aggregation, and time point) and the associated results (ie, numerical estimates of treatment effectiveness). Then, we compared the results of meta-analyses using multiple data sources by conducting multiple meta-analyses in which we systematically added and replaced data from various sources. We compared the outcomes that matter most to patients with the outcomes that were examined in clinical trials.

Results: Most clinical trials in our case studies were associated with multiple data sources, including public sources (eg, journal articles, conference abstracts, trial registrations, and Food and Drug Administration (FDA) reviews) and nonpublic sources (eg, clinical study reports and individual patient data). We found 21 gabapentin RCTs (74 reports, 6 individual participant data) and 7 quetiapine RCTs (50 reports, 1 individual participant data); we found nonpublic sources for 6 of 21 (29%) gabapentin and 4 of 7 (57%) quetiapine RCTs. We identified literally hundreds of PCOs reported in the sources we found that could be included in our meta- analyses. We surmised that there were many opportunities for selective outcome reporting by trialists and systematic reviewers. The process of identifying all sources of information—and extracting and analyzing data—required considerable time and skill. Nonpublic sources were especially difficult to identify. Clinical study reports were time consuming to extract, and individual participant data required time and skill to prepare the information for analysis.

Data sources differed in completeness. Most RCTs (18/21 [86%] and 6/7 [86%], respectively) were reported in journal articles, which often presented unclear information related to trial quality. When nonpublic sources were available for RCTs, clinical study reports contained

mostly information about trial design and trial quality, and clinical study reports and individual participant data contained the most results. In these case studies, individual participant data were not accompanied by any metadata (eg, codebooks or description of trial methods).

For a single RCT reported in a single source, 1 outcome domain (eg, pain intensity) could be associated with multiple outcome definitions (eg, using multiple measures) and results. Thus, multiple results for an outcome could be presented selectively, even when an outcome domain was prespecified. Multiple data sources associated with a single RCT sometimes contained contradicting information about trial design.

In the series of meta-analyses using results for a single outcome domain, measure, and time point for each case study, the effect of gabapentin on pain intensity ranged from an SMD of – 0.46 (95% CI –0.64 to –0.28) to an SMD of –0.31 (95% CI –0.47 to –0.15), and the effect of quetiapine on depression ranged from SMD = –0.42 (95% CI –0.62 to –0.22) to SMD = –0.44 (95% CI –0.63 to –0.26). The range of results might have been wider had we included other outcome measures and time points in our meta-analyses.

Clinical study reports and individual participant data contained hundreds of harms (adverse events) that were not included in public sources. Further analyses would be required to determine if conclusions about the relative benefits and harms of treatment would be affected by a systematic synthesis of harms included only in nonpublic sources.

Discussion: There is tremendous variation in the information available across multiple data sources from individual trials. Reported estimates from individual trials were vulnerable to selective reporting at both the trial and the systematic review levels.

Although an “open science” environment is a positive move forward, it will not solve all the problems that we identified with how trial information is currently made available. For example, an open trial data set for the purpose of making research reproducible allows for the investigation of new research questions, systematic reviews, and individual participant data meta-analysis; however, the steps required to share a data set and the steps required to utilize an open data set are resource intensive and demand skills that are not taught routinely to trialists or systematic reviewers, such as creating, storing, accessing, synthesizing, and analyzing information from multiple data sources. Furthermore, patients and stakeholders are often unfamiliar with the wide variety of existing data sources and what can be gained from each of them. Thus, current proposals to expand data sharing require careful and community-wide decision making about implementation.

Although guidance for systematic reviewers suggests searching extensively for information sources and including all data sources found, this approach is neither practical nor possible for many systematic reviews. Our research provides practical insights into a complex area that eeds further research and discussion.

BACKGROUND Objectives and Overview Our objectives were to determine whether multiple data sources about randomized controlled trials (RCTs) affected systematic reviews and meta-analyses of patient-centered outcomes (PCOs) research and to produce open access guidance about using multiple data sources for producers of systematic reviews of PCOs research.

To meet these objectives, we conducted a methods study related to the conduct of systematic reviews using 2 case studies: (1) gabapentin for neuropathic pain and (2) quetiapine for bipolar depression. We did not conduct systematic reviews; instead, we examined whether and how the results of systematic reviews might be affected by how trial data are made available, and how meta-analyses might be affected by those who use data from multiple sources for a single trial. Although we investigated methods used in conducting systematic reviews and the effect of those methods on meta-analyses, we did not consider our work to comprise multiple systematic reviews because we did not evaluate the benefits and harms of the interventions per se. Our findings may be of greatest interest to trialists, systematic reviewers, journal editors, and guideline developers.

Patient-Centered Outcomes We aimed to examine whether we found additional PCOs (meaningful to patients) by looking through all reporting sources for each trial, not just a few sources. Systematic reviewers may rely on easy-to-find sources (eg, public sources) for trial outcomes, and they may not list PCOs that could be available from other sources (eg, nonpublic sources). Searching nonpublic sources may improve the impact of systematic reviews by identifying PCOs that were recorded but not included in public sources. Examining nonpublic data sources for PCOs could reduce the need for additional RCTs and improve the efficiency of PCOs research. As far as we are aware, this possibility has never been addressed in research.

Reporting Biases Reporting biases threaten the reliability of RCTs and systematic reviews. Failure to report trials at all and selective reporting of design, methods, or results represents an ethical breach with research participants who believed that by participating in a trial they were contributing to knowledge. Reporting bias means that information from a trial is not transmitted to the public and thus also constitutes wasted information.1 When failure to report trials (or failure to report trials completely) is not random, it biases the trial findings and what is generally believed to be true.2

Multiple Data Sources To decrease the threat of reporting biases, current best practices for the conduct of systematic reviews include a comprehensive search for trial information.3,4 Standards for such a search have focused on public sources of information about trials and include going beyond journal articles and searching the “gray” literature such as conference abstracts, FDA reviews, and trial registries. This is mainly because studies of reporting bias have shown that not all information about trial design and quality can be found in journals, and publication of trial results in a journal is associated with positive trial results.5,6

In addition, studies have shown that additional information about a trial can be found in nonpublic sources.7-9 Although systematic reviewers may wish to search for and include all relevant and reliable data in systematic reviews, comprehensive searching adds to the time and resources required to complete systematic reviews. In addition, information from the various sources may not always agree,8 and systematic reviewers must decide which source is correct.10 The value of searching for nonpublic information is unclear, partly because the best methods for searching are unknown.

Assuming that comprehensive searching for trial information provides additional data, questions remain about which public and nonpublic sources should be searched for and used in systematic reviews. Empirically grounded guidance is needed to help systematic reviewers make choices about the use of data from multiple sources. Increasingly, the move to open

science is making nonpublic sources available.11 Even if all that is currently hidden becomes public, we would still need guidance to address the gains and challenges from accessing multiple data sources for a single trial.

METHODS

We published a protocol describing detailed methods for our study.12 What follows is a summary of our methods.

Selection of Case Studies To evaluate the effect of using data from multiple sources in systematic reviews and meta-analyses, we examined 2 case studies regarding the effectiveness and safety of (1) gabapentin (Neurontin) for neuropathic pain in adults and (2) quetiapine (Seroquel) for the treatment of depression in adults with bipolar disorder following a published protocol.12 We used methods consistent with the National Academy of Medicine (formerly the Institute of Medicine) standards for conducting systematic reviews.4

We selected these cases for several reasons. First, gabapentin and quetiapine are used commonly for these respective conditions, so the included studies are clinically important. Second, pain and depression are associated with several patient-reported outcomes. Methodologically, these cases also allow us to consider different, typical situations that systematic reviewers might face with respect to data from multiple sources.

Gabapentin for Neuropathic Pain Neuropathic pain occurs when there is damage to the peripheral nervous system, which is the array of nerves that transmit information from the central nervous system (brain and spinal cord) to other parts of the body. Between 3% and 10% of the population may be living with neuropathic pain, which affects 30% to 50% of people with diabetes in particular.13 Other consequences include loss of sleep, reduced quality of life, and high health care costs.

The FDA approved gabapentin for the treatment of epilepsy in 1993, and it approved gabapentin for the treatment of postherpetic neuralgia (ie, residual pain in people who have had shingles) in 2002. Gabapentin is used off label for a variety of symptoms, including for neuropathic and other types of pain.

Quetiapine for Bipolar Depression Bipolar disorder is characterized by episodes of depression and at least 1 episode of mania or hypomania14 and has a lifetime prevalence of 1% to 4%.15,16 Work, family life, and social life are impaired significantly by depressive episodes.17,18 People with bipolar disorder are at increased risk of suicide compared with the general population and compared with people who have other mental health problems.19,20

Quetiapine is an antipsychotic medication originally used for the treatment of acute psychotic episodes. Quetiapine is currently recommended as a first-line choice for treatment of acute bipolar depression,21,22 although it is associated with outcomes that are undesirable to patients, including daytime sleepiness, cognitive impairment, loss of libido, and rapid weight gain. Quetiapine and other antipsychotics are also associated with serious adverse events, including cardiac and metabolic effects, and extrapyramidal symptoms.23,24

Methods for Identifying Patient-Centered Outcomes A goal of this project was to examine what matters most to patients who take gabapentin for neuropathic pain and to patients who take quetiapine for bipolar depression. We used existing resources and consulted patient coinvestigators. We compared the list of outcomes that matter most to patients with the outcomes that were examined in clinical trials.

We identified PCOs by reviewing existing information sources, and we worked with our patient coinvestigators and clinical experts to identify those outcomes they consider patient centered. Specifically, we examined the following:

1. Core outcome sets (COSs) using the Core Outcome Measures in Effectiveness Trials database25 (http://www.comet-initiative.org) and the James Lind Alliance (http://www.lindalliance.org) in April and May 2014, respectively. We asked coinvestigators and colleagues to identify COSs related to neuropathic pain and bipolar disorder.

2. A social media website (http://www.patientslikeme.com/) in April 2014. PatientsLikeMe allows patients to enter information about their medical history and interventions they have used.

3. DRUGDEX (http://micromedex.com/) in January 2015. DRUGDEX is a compendium health care providers use to make treatment decisions. The Centers for Medicare and Medicaid may use DRUGDEX ratings to make reimbursement decisions related to medically accepted treatments. DRUGDEX contains a list, by organ system, of each drug’s adverse effects.

4. FDA prescribing information (“package inserts”) accessed through the Drugs@FDA website in April 2014

After examining existing sources, we asked patient and stakeholder partners to identify outcomes they thought should be included in each case study. We discussed the outcomes identified through all sources, and we selected those outcomes that patient and stakeholder partners deemed most important. We then generated a list of outcomes reported in clinical trials for each case study, and we examined whether PCOs were included in sources of clinical trials.

Effect of Multiple Data Sources on Meta-analyses

Eligibility Criteria for a Trial’s Inclusion in Our Study Our published protocol includes detailed inclusion and exclusion criteria for each case study.12 Table 1 describes the eligibility criteria for participants, interventions, outcomes, comparisons, and time points.

Table 1. Eligibility Criteria for the Case Studies

Gabapentin Quetiapine Participants Included: Included: • Adults (≥18 y) • Adults (≥18 y) • Neuropathic pain • Bipolar disorder (type I or II) • Not hospitalized • Acute depressive episode Excluded: Excluded: • Fibromyalgia • Rapid cycling Interventions Included: Included: • Oral gabapentin • Oral quetiapine • ≥300 mg/d • ≥100 mg/d • Regular or extended release • Regular or extended release Excluded: Excluded: • Gabapentin enacarbil (a • Norquetiapine (a metabolite of prodrug of gabapentin) quetiapine) Comparisons Included: Included: • No intervention • No intervention • Pill placebo • Pill placebo • Treatment(s) common to both • Treatment(s) common to both groups (ie, gabapentin was the groups (ie, quetiapine was the only difference between only difference between groups) groups) Excluded: Excluded: • Other doses of quetiapine • Other doses of gabapentin • Other active interventions • Other active interventions • Discontinuation of quetiapine • Discontinuation of gabapentin Outcomes Five prespecified outcome Eight prespecified outcome domains: domains: • Anxiety • Mood • Depression • Pain intensity • Functioning • Pain interference • Psychiatric hospitalization • Quality of life • Quality of life • Sleep disturbance • Sleep • Suicide • Weight Time points • 8 wk (4-13 wk) • 8 wk (4-13 wk) • 18 wk (14-22 wk) • 18 wk (14-22 wk) • 27 wk (23-31 wk) • 27 wk (23-31 wk)

Search Methods for Identification of Trials We conducted electronic and additional searches to identify trials as described in our study protocol.12 We searched for the following types of sources, which are listed by their approximate level of expected detail (ie, less detail to more detail):

1. Trial registrations without results posted on trial registries (eg, www.ClinicalTrials.gov)

2. Short reports (eg, conference abstracts, posters)

3. Trial registrations without with summary data posted on trial registries

4. Peer reviewed journal articles

5. Dissertations (eg, Master’s or doctoral theses)

6. Unpublished manuscripts and reports (eg, reports to funders, clinical study reports)

7. Information from regulators (ie, FDA medical and statistical reviews)

8. Individual patient data

Selection of Trials Two investigators independently screened titles and abstracts to determine which were eligible for inclusion in the review, as described in our study protocol.12 We independently screened full text reports of all potentially relevant trials and resolved disagreements through consensus and by asking a third investigator as needed.

Data Collection We extracted information about trial design as described in our study protocol.12 We recorded all outcomes within the prespecified outcome domains, including the 5 elements described in Table 2.26,27

For sources of aggregate data. For trials associated with multiple sources (eg, journal articles, clinical study reports), we extracted data from each source, including trial

design, trial quality (ie, risk of bias), outcomes, and results. We developed data collection forms and entered data in the Systematic Review Data Repository (SRDR).28,29

From single sources about more than 1 trial, we extracted information about each individual trial; we did not extract results from multiple trials when the data had already been combined (ie, we were unable to separate data for an individual trial from the combined data).

We extracted results from figures and tables that included numerical data (eg, counts and means included in graphs), including statistical significance (eg, P value above or below .05). We did not meta-analyze information that was presented only graphically. We extracted text from sources describing results as “statistically significant” or “not significant.”

Table 2. The 5 Elements of an Outcome

Element Definition Domain Title27 or concept30 to describe 1 or more outcomes Specific measure Instrument used to assess the outcome domain, including (1) the name of the instrument or questionnaire and (2) the total score or the subscales that will be analyzed Specific metric Unit of measurement (eg, change from baseline, value at a time) Method of The primary statistical procedure for estimating the treatment effect aggregation (eg, mean, median), including definition of cutoff or pooling for categorical variables (eg, proportion with 50% decrease, proportion much or very much improved) The procedure for estimating the treatment effect, including (1) if the outcome will be treated as a continuous, categorical, or time-to-event variable, and (2) for continuous variables, a measure of central tendency (eg, mean value); for categorical and time-to-event data variables, proportion with an event and, if relevant, the specific cutoff or categories compared Time point The period of follow-up, including (1) when outcome measurements will be obtained (eg, weeks or months since randomization) and (2) which of the outcome measurements will be analyzed

Two investigators extracted data independently and resolved discrepancies by consensus and through discussion with a third investigator if necessary. We used Stata 1431 to reconcile the records and to produce a final data set that we will share publicly through SRDR at the end of the study.

For individual patient data. We received individual participant data for gabapentin trials from Pfizer as Microsoft Access database files. When codebooks were not available, we compared the individual participant data with case report forms (which often showed how and when data were recorded) and statistical analysis plans (which often showed how data were coded and analyzed) to identify the variables contained in the individual participant data.

We did not receive any data sets about quetiapine from AstraZeneca. In 1 quetiapine clinical study report, we identified an appendix with individual participant data, which we extracted using ABBYY FineReader version 11.32

Analysis

Assessment of Trial Quality To compare the completeness of sources with respect to trial quality, we assessed each source using the Cochrane Collaboration Risk of Bias Tool3 as described in our study protocol.12 Two investigators independently rated each source for risk of bias (low, unclear, or high risk of bias) related to sequence generation, allocation concealment, masking of participants, masking of outcome assessors, masking of providers, and missing data. Investigators resolved discrepancies by consensus and through discussion with a third investigator if necessary.

Series of Meta-analyses For each case study, we conducted a series of meta-analyses to explore the impact of multiple data sources on the overall results. We analyzed these results by adding or replacing data as follows:

1. Including only results from publications in peer reviewed journals

2. Adding results from trial registries to step 1

3. Adding results from short reports to step 2

4. Adding or replacing results (from step 3) using results obtained from regulators (eg, FDA reviews)

5. Adding or replacing results (from step 4) using summary results obtained from the authors or manufacturers (eg, clinical study reports)

6. Replacing the best available aggregate results for all trials (step 5) with individual patient data where available

For example, if there was no source for a trial in steps 1 to 4 (ie, no journal article, trial registration, conference abstract, or FDA review) but there was a clinical study report, we added the results from the clinical study report in step 5. If there was a journal article, a FDA review, and a clinical study report, we replaced the results from the journal article with the results from the FDA review in step 4, and we replaced the results from the FDA review with data from the clinical study report in step 5.

For each outcome in each source, we determined if results included enough information to include them in meta-analyses. We considered results meta-analyzable if they included a point estimate and measure of precision (eg, SE, CI). We meta-analyzed only data for which there were sufficient information in a source to calculate the treatment effect; for example, we did not impute precision if it was not reported.

For each case study, we combined results for the most common measure using the standardized mean difference (SMD) for the difference between groups. In meta-analyses that included both aggregate data and individual participant data, we analyzed individual participant data to obtain an aggregate result for each trial and then we combined the aggregate results across all trials.33-36

We conducted all meta-analyses using random effects, and we report all SMDs with 95% CIs.37-39 We prespecified that we would use random effects because we anticipated that eligible trials might include differences (eg, dosage, participant characteristics) that would violate the

assumptions required for the fixed effect model. We compared information using the Dersimonian-Laird method, a very common method of synthesis, assuming the random effects model. We have no reason to think that differences in meta-analytic effect estimates using multiple data sources would be limited to whether we used 1 statistical model or another to perform the meta-analysis.

Additional analyses—including other outcome measures, metrics, and methods of aggregation—are included in manuscripts submitted for publication.

RESULTS

Manuscripts describing our analyses and findings are currently under review. What follows is a summary of our findings.

Patient-Centered Outcomes Identified Through Existing Sources and Stakeholder Coinvestigators As our protocol12 describes, we selected outcomes that matter most to patients, using existing sources and in collaboration with patient and stakeholder partners.

Core Outcome Sets We identified a core outcome set for pain: the Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials (IMMPACT) consensus recommendations.40 IMMPACT recommends that the following 6 core domains be assessed in clinical trials of chronic pain: pain, physical functioning, emotional functioning, participant ratings of global improvement, symptoms and adverse events, and participant disposition.

We did not identify a core outcome set for bipolar depression. From the James Lind Alliance, we learned that a working group had been formed to identify priorities for bipolar disorder in conjunction with the Oxford Patients Active in Research project; no relevant outputs had been produced at the time of our study

Social Media Information on social media could not be used to identify PCOs. PatientsLikeMe reported the percentage of patients who rated each intervention’s subjective effectiveness and side effects. Although patients could list specific outcomes related to effectiveness (eg, quality of life) or side effects (eg, dizziness), PatientsLikeMe does not include a function that allows patients to rate those specific side effects. PatientsLikeMe reports information about the 6 most commonly rated side effects for each medication.

DRUGDEX DRUGDEX entries for gabapentin and quetiapine included information about the use of the drugs for multiple conditions. We identified information about the outcome domains pain intensity and depression, respectively; however, we found little information about other potential benefits of treatment for either on-label or off-label uses of the drugs. DRUGDEX did not always state whether harms were associated with all conditions for which a drug might be used or with only specific conditions.

Food and Drug Administration Prescribing Information Prescribing information included the outcome domains pain intensity for trials of gabapentin and depression for trials of quetiapine. Prescribing information contained little information about other potential benefits. For gabapentin, prescribing information did not include any information about potential benefits for off-label uses (ie, types of neuropathic pain for which gabapentin is not approved by FDA).

Prescribing information included many potential harms, and it did not always state whether information about harms was applicable to any condition or to only specific conditions.

Patient and Stakeholder Investigators Patient and stakeholder partners identified outcomes they thought should be included in each review. We created a questionnaire with a list of potential outcomes, and patients rated how much the outcomes mattered to them. We discussed the outcomes identified through all sources and selected those outcomes that patient and stakeholder partners thought were most patient centered.

Effects of Adding and Replacing Data From Multiple Data Sources on the Results of Meta-analyses

Results of the Search We identified 21 eligible trials of gabapentin (from 1997 to 2013) and 7 eligible trials of quetiapine (from 2003 to 2014). As Tables 3 and 4 show, most of these trials were associated with more than 1 source (71% and 100%, respectively). We identified 74 (gabapentin) and 50 (quetiapine) reports and extracted information from them. Journal articles and conference abstracts were the most common reports. We identified 1 gabapentin trial only through a conference abstract, and 1 gabapentin trial only on ClinicalTrials.gov (www.clinicaltrials.gov).

FDA reviews included information about 2 trials of gabapentin for postherpetic neuralgia. FDA reviews available online did not include trials of quetiapine for bipolar depression, which was added to the label in a supplemental new drug application.

We identified few registrations for trials of gabapentin on ClinicalTrials.gov and other registries, none of the trials conducted by the manufacturer (Pfizer). In contrast, all eligible trials of quetiapine were registered.

We received clinical study reports and individual participant data for 6 gabapentin trials as a result of the litigation we describe above7,41; we received no additional information from Pfizer as a result of our requests.

By searching online, we identified clinical study reports for 2 quetiapine trials that were released during litigation (http://psychrights.org/); 1 of the clinical study reports included individual participant data in an appendix. AstraZeneca declined our request for more information about quetiapine trials, and our correspondence with AstraZenenca has been published.42 Thus, in our study, all nonpublic sources were previously nonpublic, since they were available to us online from litigation sources (ie, they were disclosed during patient lawsuits), and we received no additional sources from the trials’ sponsors.

In summary, we found multiple sources for most trials, the largest contributors to multiple reports being journal articles and conference abstracts.

Table 3. Multiple Sources of Trials of Gabapentin for Neuropathic Pain

Public sourcesa Nonpublic sources All Journal Journal Conferenc Other Trial FDA sources Trial author/identifier (1 trial) (>1 trial) e abstract source registry review CSR IPD (total) Arai 201043,44 1 0 1 0 0 0 0 0 2 Backonja 199845-58 2 1 5 5 0 0 1 1 15 Caraceni 200459-61 1 0 0 2 0 0 0 0 3 Hahn 200462 1 0 0 0 0 0 0 0 1 Hui 201063,64 1 0 0 0 1 0 0 0 2 Irving 200965,66 2 0 0 0 0 0 0 0 2 Milenkovic 200967 0 0 1 0 0 0 0 0 1 Mishra 201268 1 0 0 0 0 0 0 0 1 NCT0047590469 0 0 0 0 1 0 0 0 1 Rice 200149-53,70-77 2 2 2 4 0 2 1 1 14 Rowbotham 199849- 1 2 2 5 0 2 1 1 14 53,71,73,76-81 Sandercock 201282-84 1 0 0 1 1 0 0 0 3 Sang 201385-101 1 4 10 1 1 0 0 0 17 Serpell 200249-53,71,102-105 1 2 3 3 0 0 1 1 11 Simpson 2001106,107 1 0 1 0 0 0 0 0 2 Study 945-22449-53,108 0 1 2 2 0 0 1 1 7

Public sourcesa Nonpublic sources All Journal Journal Conferenc Other Trial FDA sources Trial author/identifier (1 trial) (>1 trial) e abstract source registry review CSR IPD (total) Study A945-1008109 0 0 0 0 0 0 1 1 2 Tamez Pérez 2000110,111 1 0 0 1 0 0 0 0 2 Wallace 201088-100,112-114 1 4 10 0 1 0 0 0 16 Yildirim 2003115 1 0 0 0 0 0 0 0 1 Zepeda Vazquez 2001116 1 0 0 0 0 0 0 0 1 Trials (total) 17 7 10 9 5 2 6 6 21 Unique sources (total) 20 6 20 15 5 2 6 6 80 Abbreviations: CSR, clinical study report; IPD, individual patient data. aJournal (1 trial) = journal article about a single included trial; journal (>1 trial) = journal article about 2 or more included trials; other source = letters, summaries of journal articles, press releases, brief reports in trade publications (eg, not peer reviewed); trial registry = trial registration on ClinicalTrials.gov or another registry; FDA review = US Food and Drug Administration medical or statistical review. Each cell represents the number of sources of a particular type (eg, journal articles) for a given trial.

Table 4. Multiple Sources of Trials of Quetiapine for Bipolar Depression

Public sourcesa Nonpublic sources All Journal Journal Conferenc Other Trial sources Trial author (1 trial) (>1 trial) e abstract source CSR-S registry CSR IPD (total) Calabrese 2004117-142 4 5 14 1 0 1 1 1 27 Gao 2014143,144 1 0 0 0 0 1 0 0 2 Li 2014145,146 0 0 1 0 0 1 0 0 2 McElroy 2010125- 1 1 7 1 1 1 0 0 12 129,133,141,147-151 Suppes 2010152-154 1 0 1 0 0 1 0 0 3 Thase 2006125-133,138,141,155- 2 5 6 2 0 1 1 0 17 160 Young 2008125- 1 1 8 2 1 1 0 0 14 129,133,141,148,161-166 Trials (total) 6 4 6 4 2 7 2 1 7 Unique sources (total) 10 5 19 5 2 7 2 1 51 Abbreviations: CSR, clinical study report; CSR-S, clinical study report synopsis; IPD, individual patient data. aJournal (1 trial) = journal article about a single included trial; journal (> 1 trial) = journal article about 2 or more included trials; other source = letters, summaries of journal articles, press releases, brief reports in trade publications (eg, not peer reviewed); trial registry = trial registration on ClinicalTrials.gov or another registry. Each cell represents the number of sources of a particular type (eg, journal articles) for a given trial.

Completeness of Information Trial design (eg, participant, intervention characteristics) was best described in clinical study reports and journal articles. For example, conference abstracts were incomplete and sometimes contradicted other sources. Five trial registrations submitted to ClinicalTrials.gov by AstraZeneca described a single intervention, “quetiapine fumarate,” while other sources indicated that 2 groups received quetiapine at different doses (ie, 300 mg and 600 mg).

Trial quality (defined as our assessment of the risk of bias for each trial) was often unclear in the sources we identified (see Tables 5 and 6). When we located enough information to assess risk of bias (usually in a journal article or clinical study report), that information almost always led from an unclear to a low risk of bias rating. The only exception was information about missing data, where identification of additional information led from an unclear to a high risk of bias rating for trials with large amounts of missing data.

Statistical information about many outcomes described in journal articles and short reports did not include sufficient detail for results to be included in meta-analysis. In contrast, data in clinical study reports were usually complete and could almost always be used for analysis.

For these reasons, clinical study reports—when they were available—were the most complete sources of information about methods and results.

Table 5. Trial Quality (Risk of Bias) for Sources of Gabapentin for Neuropathic Pain Trialsa

Masking Sequence Allocation Masking Masking outcome Missing data for generation concealment participants providers assessment primary outcome Trial Total name/author/identifier sources H U L H U L H U L H U L H U L NA H U L NA Arai 201043,44 2 0 1 1 0 2 0 0 2 0 0 2 0 0 2 0 0 0 2 0 0 Backonja 199845-58 14 0 13 1 0 13 1 0 12 2 0 14 0 0 14 0 0 0 12 2 0 Caraceni 200459-61 3 0 3 0 0 2 1 0 3 0 0 3 0 0 3 0 0 0 2 1 0 Hahn 200462 1 0 1 0 0 0 1 0 0 1 0 1 0 0 1 0 0 0 1 0 0 Hui 201063,64 2 0 1 1 0 2 0 0 1 1 0 2 0 0 0 2 0 0 2 0 0 Irving 200965,66 2 0 1 1 0 2 0 0 1 1 0 2 0 0 2 0 0 0 1 1 0 Milenkovic 200967 1 0 1 0 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 Mishra 201268 1 0 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 NCT0047590469 1 0 1 0 0 1 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 Rice 200149-53,70-77 13 0 11 2 0 11 2 0 12 1 0 13 0 0 13 0 0 0 7 5 1 Rowbotham 199849- 13 0 12 1 0 13 0 0 12 1 0 13 0 0 13 0 0 0 11 1 1 53,71,73,76-81 Sandercock 201282-84 3 1 2 0 0 3 0 0 3 0 0 3 0 0 3 0 0 0 1 1 1 Sang 201385-101 17 0 15 2 0 17 0 0 16 1 0 17 0 0 17 0 0 0 13 2 2 Serpell 200249-53,71,102-105 10 0 9 1 0 9 1 0 10 0 0 10 0 0 10 0 0 0 7 2 1 Simpson 2001106,107 2 0 2 0 0 2 0 0 2 0 0 2 0 0 2 0 0 0 2 0 0

Masking Sequence Allocation Masking Masking outcome Missing data for generation concealment participants providers assessment primary outcome Trial Total name/author/identifier sources H U L H U L H U L H U L H U L NA H U L NA Study 945-22449-53,108 6 0 6 0 0 5 1 0 5 1 0 5 1 0 6 0 0 0 5 1 0 Study A945-1008109 1 0 0 1 0 1 0 0 0 1 0 1 0 0 1 0 0 0 0 1 0 Tamez Pérez 2000110,111 2 0 2 0 0 2 0 0 2 0 0 2 0 0 2 0 0 0 2 0 0 Wallace 201088-100,112-114 16 0 15 1 0 15 1 0 15 1 0 16 0 0 16 0 0 0 13 1 2 Yildirim 2003115 1 0 1 0 0 1 0 0 1 0 0 1 0 0 1 0 0 1 0 0 0 Zepeda Vazquez 2001116 1 0 1 0 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 aFor each included trial, this table shows the number of sources (excluding individual participant data) at high (H), unclear (U), and low (L) risk of bias for each item. NA indicates not applicable because the source did not report outcomes.

Table 6. Trial Quality (Risk of Bias) by Source of Quetiapine for Bipolar Depression Trialsa

Masking Sequence Allocation Masking Masking outcome Missing data for generation concealment participants providers assessment primary outcome Total Trial author sources H U L H U L H U L H U L H U L NA H U L NA Calabrese 2004117-142 26 0 22 4 0 22 4 0 25 1 0 25 1 0 22 1 3 5 14 0 7 Gao 2014143,144 2 0 2 0 0 2 0 0 1 1 0 1 1 0 2 0 0 1 1 0 0 Li 2014145,146 2 0 2 0 0 2 0 0 1 1 0 2 0 0 2 0 0 0 2 0 0 McElroy 2010125- 12 0 10 2 0 10 2 0 11 1 0 12 0 0 12 0 0 2 7 0 3 129,133,141,147-151 Suppes 2010152-154 3 0 3 0 0 3 0 0 3 0 0 3 0 0 3 0 0 1 2 0 0 Thase 2006125- 17 0 14 3 0 14 3 0 17 0 0 17 0 0 17 0 0 4 10 0 3 133,138,141,155-160 Young 2008125- 14 0 12 2 0 12 2 0 13 1 0 13 1 0 14 0 0 2 9 0 3 129,133,141,148,161-166 aFor each included trial, this table shows the number of sources (excluding individual participant data) at high (H), unclear (U), and low (L) risk of bias for each item. NA indicates not applicable because the source did not report outcomes.

Locating Patient-Centered Outcomes in Multiple Data Sources Across all sources for all gabapentin trials, we located 4 of 5 prespecified outcome domains as described in section “Patient-centered outcomes identified through existing sources and stakeholder coinvestigators” (Tables 7 and 8). Across all sources for all quetiapine trials, we identified 7 of 8 prespecified domains. Quetiapine sources included about 6 (median) outcome domains, and they were also highly variable (range, 2-6 domains per source). To our knowledge, none of the trials measured the prespecified outcome domains for pain interference (gabapentin) or psychiatric hospitalization (quetiapine).

Conference abstracts and other short reports included little useful information about PCOs. Journal articles contained more PCOs than did other public sources, although clinical study reports were consistently the best source of aggregate information about PCOs. No outcome measures appeared in public sources that did not also appear in corresponding clinical study reports; however, we identified methods of aggregation in public sources (ie, journal articles and an FDA review) for gabapentin trials that were not included in clinical study reports.

In addition to the prespecified outcomes related to treatment benefits, clinical study reports and individual participant data described hundreds of harms that were not included in public sources.167

Table 7. Number of Prespecified Outcome Domains in Each Source of Gabapentin for Neuropathic Pain Trials

Public sourcesa Nonpublic sources All Trial Journal Journal Conference Other Trial FDA sources name/author/identifier (1 trial) (>1 trial) abstract source registry review CSR IPD (total) Arai 201043,44 0 NA 1 NA NA NA NA NA 1 Backonja 199845-58 4 4 4 3 NA NA 4 4 4 Caraceni 200459-61 0 NA NA 0 NA NA NA NA 0 Hahn 200462 2 NA NA NA NA NA NA NA 2 Hui 201063,64 1 NA NA NA 1 NA NA NA 1 Irving 200965,66 2 NA NA NA NA NA NA NA 2 Milenkovic 200967 NA NA 1 NA NA NA NA NA 1 Mishra 201268 1 NA NA NA NA NA NA NA 1 NCT0047590469 NA NA NA NA 1 NA NA NA 1 Rice 200149-53,70-77 3 3 0 1 NA 3 3 3 3 Rowbotham 199849- 4 3 0 4 NA 4 4 4 4 53,71,73,76-81 Sandercock 201282-84 2 NA NA 2 2 NA NA NA 2 Sang 201385-101 2 1 1 1 2 NA NA NA 2 Serpell 200249-53,71,102-105 2 2 2 2 NA NA 2 2 2 Simpson 2001106,107 4 NA 3 NA NA NA NA NA 4

Public sourcesa Nonpublic sources All Trial Journal Journal Conference Other Trial FDA sources name/author/identifier (1 trial) (>1 trial) abstract source registry review CSR IPD (total) Study 945-22449-53,108 NA 3 0 1 NA NA 3 3 3 Study A945-1008109 NA NA NA NA NA NA 4 3 4 Tamez Pérez 2000110,111 1 NA NA 1 NA NA NA NA 1 Wallace 201088-100,112-114 2 1 1 NA 2 NA NA NA 2 Yildirim 2003115 1 NA NA NA NA NA NA NA 1 Zepeda Vazquez 2001116 1 NA NA NA NA NA NA NA 1 Total 4 4 4 4 2 4 4 4 4 Abbreviations: CSR, clinical study report; IPD, individual patient data. aJournal (1 trial) = journal article about a single included trial; journal (> 1 trial) = journal article about 2 or more included trials; other source = letters, summaries of journal articles, press releases, brief reports in trade publications (eg, not peer reviewed); trial registry = trial registration on ClinicalTrials.gov or another registry; FDA review = US Food and Drug Administration medical or statistical review. For each trial, we recorded the number of prespecified domains (pain intensity, pain interference, sleep disturbance, mood, and quality of life) that could be extracted from each type of source and from all sources about that trial. None of the reports described pain interference.

Table 8. Number of Prespecified Outcome Domains in Each Source of Quetiapine for Bipolar Depression Pain Trials

Public sourcesa Nonpublic sources All Journal Journal Conference Other Trial sources Trial author (1 trial) (>1 trial) abstract source CSR-S registry CSR IPD (total) Calabrese 2004117-142 6 3 4 2 NA 0 6 6 6 Gao 2014143,144 3 NA NA NA NA 3 NA NA 3 Li 2014145,146 NA NA 0 NA NA 2 NA NA 2 McElroy 2010125- 6 0 3 4 6 1 NA NA 6 129,133,141,147-151 Suppes 2010152-154 3 NA 1 NA NA 1 NA NA 3 Thase 2006125-133,138,141,155- 6 2 0 3 NA 1 6 NA 6 160 Young 2008125- 6 0 3 4 4 1 NA NA 6 129,133,141,148,161-166 Total 7 4 5 6 6 4 7 6 7 Abbreviations: CSR, clinical study report; CSR-S, clinical study report synopsis; IPD, individual patient data. aJournal (1 Trial) = journal article about a single included trial; journal (> 1 trial) = journal article about 2 or more included trials; other source = letters, summaries of journal articles, press releases, brief reports in trade publications (eg, not peer reviewed); trial registry = trial registration on ClinicalTrials.gov or another registry. For each trial, we recorded the number of prespecified domains (depression, functioning, quality of life, anxiety, sleep, weight, psychiatric hospitalization, and suicide) that could be extracted from each type of source and from all sources about that trial. None of the reports described psychiatric hospitalization.

Multiple Outcome Definitions As noted earlier, it is also possible for a single prespecified outcome domain to be assessed in multiple ways (ie, by varying the measure, metric, and method of aggregation). For this reason, an outcome might appear to be reported as described in the protocol when in fact there were multiple opportunities for changing the associated results (Figure 1).

Figure 1. Number of Outcomes in a Trial Is a Function of the Number of Definitions of Each of the 5 Elements

Note: In this hypothetical example, the number of treatment effects for a single trial is the product of definitions for each element. In a trial with 4 outcome domains, introducing 2 definitions for each of the other elements will result in 64 different outcomes.

For each case study, we identified hundreds of outcomes for the prespecified domains (Figure 2). By varying 4 elements, gabapentin RCTs reported 214 outcomes and quetiapine RCTs reported 81 outcomes. For pain intensity and depression, the most common outcome domains for gabapentin and quetiapine RCTs, we found 119 and 44 outcomes, respectively. Multiple methods of analysis also contributed to multiple results for the same defined outcome. For

these reasons, many meta-analyses were possible for each outcome domain using the same included trials.167

Figure 2. Number of Outcomes Observed Increased as a Consequence of Multiple Definitions of the Elements

Note: The points nearest the center of the circle represent the prespecified outcome domains for which we found any information in eligible trials (ie, 4 outcome domains for gabapentin and 7 outcome domains for quetiapine); these are represented by different colors. The branches represent variations in outcome definitions associated with those domains (ie, different measures, metrics, methods of aggregation, or time points). Each point around the outer edge of the circle thus represents a unique outcome described in 1 or more of the trial sources we identified. (A) Specific measure (name of instrument): BPI = Brief Pain Inventory; CGIC = Clinician Global Impression of Change; ECOG = Eastern Cooperative Oncology Group; GIC = Global Impression of Change; GSS = Global Symptom Score; HADS = Hospital Anxiety and Depression Scale; HRQoLQ = Health Related Quality

of Life Questionnaire; NPS = Neuropathic Pain Scale; Pain (11 pt) = pain score (11-point scale); Pain (4 pt) = pain score (4-point scale); PGIC = Patient Global Impression of Change; POMS = Profile of Mood States; SF-36 = Short Form 36 item; SFMPQ = Short Form McGill Pain Questionnaire; Sleep score = sleep score (11-point scale); TOPS = Treatment Outcomes in Pain Survey. Specific measure (total or subscale): % pain relief = percentage pain relief; Ave. pain = average pain score; Care sat = health care satisfaction; Current pain = current pain score; Fear/Avoid = fear avoidance; Gen health = general health; Globe intensity = global intensity; Globe unpleas = global unpleasantness; Lower body = lower body functional limitations; Ment health = mental health; Obj. social = objective family/social disability; Obj. work = objective work disability; Pain sympt = pain symptom; Passive cope = passive coping; Patient sat = patient satisfaction with outcomes; Phys funct = physical functioning; PPI = present pain intensity; Role emot = role emotional; Role phys = role physical; Social disab = perceived family/social disability; Social funct = social functioning; Solicitous = solicitous responses; Total pain = total pain experience; Total score = total; Upper body = upper body functional limitations; VAS = visual analog scale; Work limit = work limitations. Metric: Change = change from baseline; Value = value at a time point. Method of aggregation: % change mean = percentage change mean; % change median = percentage change median; 2 decrease = at least 2-point decrease; 3 decrease = at least 3-point decrease; 4 decrease = at least 4-point decrease; M or VM improved = much or very much improved; M or VM worse = much worse or very much worse; Min or No change = minimally improved or no change; Min improved = minimally improved; Min worse = minimally worse; Mod improved = moderately improved; Mod or much imp = moderately or much improved; Mod worse = moderately worse; No change = no change (0%); Pain free = totally pain free; Response (ND) = response (not defined); V much improved = very much improved; V much worse = very much worse. (B) Specific measure (name of instrument): BMI = Body Mass Index; CGI-I = Clinical Global Impression - Improvement; CGI-S = Clinical Global Impression - Severity; HAM-A = Hamilton Anxiety Rating Scale; HAM-D = Hamilton Depression Rating Scale; MADRS = Montgomery Asberg Depression Rating Scale; MADRS/YMRS = Montgomery Asberg Depression Rating Scale & Young Mania Rating Scale; MOS Cog = Medical Outcomes Study Cognitive Scale; Not defined = not defined; PSQI = Pittsburgh Sleep Quality Assessment; Q-LES-Q = Quality of Life Enjoyment and Satisfaction Questionnaire; QIDS-SR-16 = Quick Inventory of Depressive Symptomology Self-Report 16; SDS = Sheehan Disability Scale; Ideation = suicidal ideation (self-reported). Metric: Change = change from baseline; Value = value at a time point. Method of aggregation: “= 10” = less than or equal to 10; “= 12” = less than or equal to 12; “= 7” = less than or equal to 7; “= 8” = less than or equal to 8; 15% increase = at least 15% increase; 25% increase = at least 25% increase; 30% decrease = at least 30% decrease; 40% decrease = at least 40% decrease; 50% decrease = at least 50% decrease; 60% decrease = at least 60% decrease; 7% increase = at least 7% increase; 70% decrease = at least 70% decrease; Extremely ill = among the most extremely ill; M or VM improved = much or very much Improved; MADRS = 12, YMRS = 12 = MADRS less than or equal to 12 and YMRS less than or equal to 12; MADRS = 12, YMRS = 8 = MADRS less than or equal to 12 and YMRS less than or equal to 8; Mean % Max = mean percentage of the maximum; Min improved = minimally improved; Min worse = minimally worse; Normal or Borderline = normal, not at all ill or borderline ill; Normal, not ill = normal, not at all ill; Proportion = proportion of participants who experienced the event; Remission (ND) = remission (not defined); Response (ND) = response (not defined); V Much improved = very much improved; V Much worse = very much worse.

Impact of Adding or Replacing Multiple Data Sources on Pooled Effect Sizes and Precision For each case study, we conducted a series of meta-analyses in which we added or replaced data from each of the multiple sources for a single outcome domain, measure, and time point to evaluate the effect of adding or replacing data. In the presence of reporting bias, whereby negative or null results are reported in nonpublic sources but not in public sources, we would expect the addition of nonpublic sources to change the average effects estimated in meta-analyses. This change would lead to a meta-analytic effect estimate closer to the null.

Not all trials could contribute data to a meta-analysis (Tables 9 and 10). Of the 21 gabapentin and 7 quetiapine trials, 9 (43%) and 4 (57%), respectively, could not be included in an analysis of pain intensity or depression using continuous data (eg, the mean) because no source for those trials included meta-analyzable data (see “Series of Meta-analyses”).

When restricted to the most common outcome defined as the same outcome domain, measure, and time point, results in both case studies were largely consistent across sources. For gabapentin, including data from previously nonpublic sources (ie, clinical study reports and individual participant data) reduced the magnitude of the effect estimate; adding additional sources did not affect the statistical significance of the treatment effect. In our series of meta- analyses, the effect of gabapentin on pain intensity ranged from an SMD of –0.46 (95% CI –0.64 to –0.28) to an SMD of –0.31 (95% CI –0.47 to –0.15). For quetiapine, there were no meaningful differences as a result of adding and replacing data from multiple sources. The effect of quetiapine on depression ranged from an SMD of –0.42 (95% CI –0.62 to –0.22) to an SMD of –0.44 (95% CI –0.63 to –0.26).

Table 9. Series of Meta-analyses and the Effect of Adding or Replacing Data Sources on the Pooled Estimate of Pain Intensity in Trials of Gabapentin for Neuropathic Pain (SMD [95% CI])a

FDA reports, CSRs, FDA reports, IPD, CSRs, FDA Journal articles, journal articles, journal articles, reports, journal Including only registries, plus registries, and registries, and articles, registries, Trial peer-reviewed Journal articles conference conference conference and conference name/author/ID journals plus registries abstracts abstracts abstracts abstracts Backonja 199845- –0.57 (–0.88 to – –0.57 (–0.88 to – –0.57 (–0.88 to – –0.57 (–0.88 to – –0.56 (–0.87 to – –0.53 (–0.84 to – 58 0.25) 0.25) 0.25) 0.25) 0.24) 0.22) Irving 200965,66 –0.37 (–0.71 to – –0.37 (–0.71 to – –0.37 (–0.71 to – –0.37 (–0.71 to – –0.37 (–0.71 to – –0.37 (–0.71 to – 0.04) 0.04) 0.04) 0.04) 0.04) 0.04) Mishra 201268 –0.44 (–0.95 to –0.44 (–0.95 to –0.44 (–0.95 to 0.06) –0.44 (–0.95 to –0.44 (–0.95 to –0.44 (–0.95 to 0.06) 0.06) 0.06) 0.06) 0.06) NCT0047590469 No sources 0.32 (0.04-0.60) 0.32 (0.04-0.60) 0.32 (0.04-0.60) 0.32 (0.04-0.60) 0.32 (0.04-0.60) Rice 200149-53,70-77 –0.64 (–0.92 to – –0.64 (–0.92 to – –0.64 (–0.92 to – –0.64 (–0.92 to – –0.63 (–0.86 to – –0.56 (–0.80 to – 0.36) 0.36) 0.36) 0.36) 0.40) 0.33) Rowbotham –0.86 (–1.13 to – –0.86 (–1.13 to – –0.86 (–1.13 to – –0.86 (–1.13 to – –0.86 (–1.13 to – –0.83 (–1.11 to – 199849-53,71,73,76-81 0.59) 0.59) 0.59) 0.59) 0.59) 0.56) Sandercock –0.49 (–0.84 to – –0.49 (–0.84 to – –0.49 (–0.84 to – –0.49 (–0.84 to – –0.49 (–0.84 to – –0.49 (–0.84 to – 201282-84 0.15) 0.15) 0.15) 0.15) 0.15) 0.15) Sang 201385-101 –0.22 (–0.40 to – –0.22 (–0.40 to – –0.22 (–0.40 to – –0.22 (–0.40 to – –0.22 (–0.40 to – –0.22 (–0.40 to – 0.03) 0.03) 0.03) 0.03) 0.03) 0.03) Serpell 200249- No sources No sources No sources No sources –0.22 (–0.45 to –0.20 (–0.43 to 53,71,102-105 0.01) 0.03)

FDA reports, CSRs, FDA reports, IPD, CSRs, FDA Journal articles, journal articles, journal articles, reports, journal Including only registries, plus registries, and registries, and articles, registries, Trial peer-reviewed Journal articles conference conference conference and conference name/author/ID journals plus registries abstracts abstracts abstracts abstracts Study 945-22449- No sources No sources No sources No sources –0.09 (–0.35 to –0.06 (–0.32 to 53,108 0.17) 0.19) Study A945- No sources No sources No sources No sources –0.43 (–0.66 to – –0.26 (–0.46 to – 1008109 0.20) 0.06) Wallace 201088- –0.20 (–0.40 to –0.20 (–0.40 to –0.20 (–0.4 to 0.01) –0.20 (–0.40 to –0.20 (–0.40 to –0.20 (–0.40 to 100,112-114 0.01) 0.01) 0.01) 0.01) 0.01) Total No. of trials 8 9 9 9 12 12 No. of 1805 2019 2019 2019 3074 3144 participants Meta-analytic –0.46 (–0.64 to – –0.38 (–0.60 to – –0.38 (–0.60 to – –0.38 (–0.60 to – –0.34 (–0.51 to – –0.31 (–0.47 to – effect size (SMD) 0.28) 0.15) 0.15) 0.15) 0.18) 0.15) I2 68.75 82.56 82.56 82.56 79.00 76.94 Abbreviations: CSRs, clinical study reports; IPD, individual patient data; SMD, standardized mean difference. aData for individual trials are provided in each row. In the columns, we first estimated the effect sizes using only journal articles; we then estimated the effect by including other sources to the analysis (ie, adding data from new sources or replacing the data from sources appearing in columns the left). When we used IPD, we first obtained aggregated data and then combined data from all sources, as described in the Methods. For each included study, we calculated the SMD and 95% CI for the intervention compared with placebo. We combined the results (“Total”) by conducting a meta-analysis using random effects. Effect sizes are given for all trials that reported pain intensity (measured using the average daily pain score on an 11-point scale) in at least 1 source.

Table 10. Series of Meta-analyses and the Effect of Adding or Replacing Data Sources on the Pooled Estimate of Depression in Trials of Quetiapine Bipolar Depression (SMD [95% CI])a

FDA reports, CSRs, FDA reports, IPD, CSRs, FDA Journal articles, journal articles, journal articles, reports, journal Including only registries, plus registries, and registries, and articles, registries, peer-reviewed Journal articles plus conference conference conference and conference Trial author journals registries abstracts abstracts abstracts abstracts Calabrese –0.63 (–0.84 to – –0.63 (–0.84 to – –0.63 (–0.84 to – –0.63 (–0.84 to – –0.59 (–0.78 to – –0.59 (–0.78 to – 2004117-142 0.41) 0.41) 0.41) 0.41) 0.41) 0.4) Thase 2006125- –0.37 (–0.56 to – –0.37 (–0.56 to – –0.37 (–0.56 to – –0.37 (–0.56 to – –0.46 (–0.66 to – –0.46 (–0.66 to – 133,138,141,155-160 0.17) 0.17) 0.17) 0.17) 0.27) 0.27) Young 2008125- –0.27 (–0.46 to – –0.27 (–0.46 to – –0.27 (–0.46 to – –0.27 (–0.46 to – –0.27 (–0.46 to – –0.27 (–0.46 to – 129,133,141,148,161-166 0.08) 0.08) 0.08) 0.08) 0.08) 0.08) Total No. of trials 3 3 3 3 3 3 No. of 1453 1453 1453 1453 1625 1629 participants Meta-analytic –0.42 (–0.62 to – –0.42 (–0.62 to – –0.42 (–0.62 to – –0.42 (–0.62 to – –0.44 (–0.63 to – –0.44 (–0.62 to – effect size (SMD) 0.22) 0.22) 0.22) 0.22) 0.26) 0.26) I2 66.45 66.45 66.45 66.45 63.70 63.15 Abbreviations: CSRs, clinical study reports; IPD, individual patient data; SMD, standardized mean difference. aData for individual trials are provided in each row. In the columns, we first estimated the effect sizes using only journal articles; we then estimated the effect by including other sources to the analysis (ie, adding data from new sources or replacing the data from sources appearing in columns the left). When we used IPD, we first obtained aggregated data and then combined data from all sources, as described in the Methods. For each included trial, we calculated the SMD and 95% CI for the intervention compared with placebo. We combined the results (“Total”) by conducting a meta-analysis using random effects. Effect sizes are given for all trials that reported depression measured using the Montgomery Åsberg Depression Rating in at least 1 source.

Resources Required to Use Multiple Data Sources in Systematic Reviews and Meta-analyses Available guidance for performing systematic reviews (eg, from the Cochrane Collaboration and the National Academy of Medicine [formerly the Institute of Medicine]) suggests that reviewers should use all data sources for clinical trials.3,4 Identifying and extracting data from multiple data sources was resource intensive and required many skills, and our findings suggest that existing guidance could be improved to help systematic reviewers use multiple data sources efficiently (Table 11).

Table 11. Main Findings From MUDS and our Recommendations for Application of the Findings

Findings Recommendation Audience FINDING 1: Accessing, obtaining, and analyzing multiple sources is resource intensive and tedious, and requires many skills. Most trials lead to multiple • Users of research should be able to identify clinical trials. • Clinicians, patients sources. • PCOs can be found in multiple sources about trials. • Guideline developers, patients, systematic reviewers Multiple sources are needed to • To prevent bias and reduce waste, trials should use common • Funders, trialists understand what was planned PCOs (core outcome sets). and what actually occurred in • All information about a clinical trial should be stored in a single • Funders, trialists clinical trials. public source. • Trialists and manufacturers should make information • Trialists consistent across sources. Some sources (eg, trial protocols) • Protocols submitted to IRBs should be publicly available. • Funders, trialists are rarely available. • IRBs should monitor compliance with registration and • Funders, trialists reporting requirements. Some trials are not reported in • All trials should be registered in advance. • Trialists public sources (eg, journal • Results from all trials should be shared with sufficient • Funders, trialists articles), and nonpublic sources information to understand the variables and to reproduce the are available for many trials that analyses. are reported. • To reduce waste, trials should use common data elements. • Funders, trialists • Individual patient data are useful for some applications but not • Systematic reviewers all.

Findings Recommendation Audience FINDING 2: Information varies tremendously across sources. Sources differ in completeness. • Public sources should be available for all trials with sufficient • Funders, guideline information to understand their design, quality, and results. developers, trialists • Outcomes data should be reported with sufficient information • Trialists to include results in meta-analyses. Sources include different • All sources should include complete descriptions of all • Trialists information. outcomes examined, including all elements of each outcome. • Users should expect variation across sources and prioritize • Guideline developers, variables/sources in advance. systematic reviewers Trials and sources included results • To be usable, databases should employ common fields. • Trialists that could not be compared as • Shared data should be accompanied by codebooks and other well as incomplete information. metadata that allow examination of the data. Nonpublic data sources included • Trialists should anticipate requests for data and plan to share • Funders, trialists additional information. from the time a trial is initiated. • Systems are needed to archive and share new research, and to • Guideline developers, share “legacy” trials related to drugs in current use. funders Abbreviations: MUDS, Integrating multiple data sources; PCOs, patient-centered outcomes.

In general, we were able to extract information from conference abstracts quickly; journal articles took several hours to extract.

Clinical study reports, which included thousands of pages of appendices and hundreds of outcomes and analyses, required dozens of hours. Two extractors each spent at least 5 8- hour days extracting data from each clinical study report.

Because we did not receive codebooks for individual participant data, each data set was a unique puzzle we had to solve even before we could synthesize the data. Members of our team spent months developing codebooks and creating a common data format to combine data across trials. Even with this investment, there were some variables we could not identify with certainty.

Guidance for Using Multiple Data Sources in Systematic Reviews As a final step in our investigation, we developed guidance for systematic reviewers (Table 12). For the open science movement to succeed, systematic reviewers need to make efficient use of evidence from clinical trials, including previously nonpublic information. Although it would have been desirable to identify a best source of information for clinical trials, we did not find a single source or combination of sources that provided complete information about trial design, trial quality, outcomes, and results. Our recommendations incorporate the main findings for each step in the review process.

Table 12. Recommendations for Systematic Reviews Using Multiple Data Sources

Task Guidance for systematic reviewers Selecting • Determine in advance if individual participant data, aggregate data (eg, clinical trials study reports), or both are required to answer the research questions. • Search electronic databases for all publicly available sources. • Use registries (eg, ClinicalTrials.gov), conference abstracts, and regulatory data (eg, FDA, EMA) to identify unpublished trials. • Search repositories (eg, www.clinicalstudydatarequest.com) for clinical study reports and individual participant data. • For unpublished data, allow time to establish data use agreements as required for access to trial data; make alternative plans in case requests are ignored or refused. • Contact trial authors and sponsors to confirm which sources are related to each trial. Assessing • Use clinical study reports to extract information about trial design and trial trial design quality. and trial • In addition to clinical study reports, use journal articles and FDA reviews quality because they sometimes include information not contained in clinical study reports. • Check trial registrations for information about prespecified methods and outcomes. • Consider excluding conference abstracts when other sources are available for data extraction. Extracting • Anticipate multiple definitions of outcomes; prespecify outcomes for results extraction within each outcome domain. • When clinical study reports are available, check journal articles and FDA reviews for additional analyses. • Where clinical study reports are not available, use journal articles to assess effects on primary outcomes. Analyzing • Consider excluding trials reported only in conference abstracts from meta- results analyses. • Be cautious in interpreting trials with only public data sources (eg, conduct sensitivity analyses limited to trials with clinical study reports and individual participant data). • Use individual participant data for investigating research questions not addressed in clinical study reports (eg, subgroup analyses, potential mediators and moderators) and for older trials for which new statistical methods to account for missing data (eg, multiple imputation) might affect the results. • Allow time (12-18 mo) to recode individual participant data for analysis and to merge databases. Abbreviation: EMA, European Medicines Agency.

DISCUSSION Identifying Patient-Centered Outcomes Our efforts to identify PCOs revealed little formal research about what patients with neuropathic pain and bipolar disorder consider important outcomes. Better evidence is needed to determine what should be measured by clinical trials in these areas and to determine how those outcomes should be assessed (eg, choice of specific measures and time points).

Effects of Adding and Replacing Data From Multiple Data Sources on the Results of Meta-analyses In both case studies, most trials were associated with multiple sources of information about their methods and results (mostly reports about the trials and some individual participant data). The most common sources, short reports (eg, conference abstracts), were often the least informative about trial design, trial quality, outcomes, and results. Journal articles contained information that was consistent with nonpublic sources, but they did not include all of the information available from nonpublic sources. FDA reviews contained less information than hoped, largely because the drugs and indications we examined did not meet FDA’s criteria for inclusion in its public information. Trial registries were also less useful than we had hoped because trials supporting current uses of the drugs in our case studies were performed before legislation mandated trial registration and results reporting. The nonpublic sources we obtained were made public through litigation, an approach that has been used in previous studies to compare data sources.168

Because many sources were incomplete and inconsistent regarding trial design (eg, number of groups), sometimes we found it difficult to understand whether sources were about the same trial or different trials.

Most sources for a single trial, viewed on their own, were unclear about whether there was risk of bias in a trial’s design (also referred to as trial quality). An advantage of identifying multiple data sources for each trial was that often information was provided in 1 source but not

another; thus, when all sources were taken together, we were able discern more about the trial quality.

The effects of multiple data sources on meta-analysis for gabapentin were consistent with evidence that using FDA reviews and nonpublic sources may change the results of meta- analyses because additional trials can be included using FDA reviews and nonpublic sources and because results in these sources sometimes differ from results in journal articles and conference abstracts.169-172 Findings for quetiapine were consistent with evidence from other case studies that indicate the results reported in multiple data sources sometimes agree.173-175 We are not aware of any other cases in which researchers have had access to this range and number of sources.

Although they sometimes identified trials not reported in other sources (eg, trials not published in a journal article), short reports such as conference abstracts did not contribute new information about methods and results when other data sources were available.

Most of the outcomes in each trial we examined were associated with a few domains (ie, they measured a few underlying concepts, such as pain intensity and depression), and these domains were assessed using a variety of measures, metrics, methods of aggregation, and time points both within individual trials and across all eligible trials.167 For this reason, we believe it is possible for investigators to prespecify an outcome domain, assess it in multiple ways, and choose the most favorable result for public presentation.

In the 2 case studies we examined, inferences about effectiveness derived from meta- analyses were only slightly different when we added or replaced data following the process described above. However, some results associated with outcome definitions in each outcome domain were statistically significant and some were not; such differences might lead trial authors to report outcomes selectively or lead systematic reviewers to include outcomes selectively in their meta-analyses. Reporting biases related to differences in outcome measures, metrics, and methods of aggregation could lead decision makers to recommend or select 1 treatment over another.

Potential harms were viewed by our patient coinvestigators as among the most important outcomes. There was limited information about harms in publicly available sources. Further investigation of harmful outcomes as reported in public and nonpublic sources is warranted; findings could have important implications for PCOs research.

Implications for Data Sharing Discussions about open access have largely focused on developing systems for archiving and sharing data from new trials. The drugs in our study are widely prescribed even though the trials were completed before trial registration and reporting requirements came into effect.176 Open access to trial data would make what are now nonpublic sources (eg, clinical study reports, individual participant data) more widely available. Our findings highlight the importance of open access to legacy trial data (eg, trials conducted before requirements to share data) because many older drugs enjoy current widespread use. Before data similar to those we examined are lost forever, the research community should develop a system for sharing legacy trials with future generations.

Limitations We were not funded to analyze potential harms, so we do not know whether the relative benefits and harms of these interventions differ between public and nonpublic data sources. Further research is needed to explore the benefits of using clinical study reports and individual patient data meta-analysis for identifying the frequency of specific harms and of groups of harms.

While many studies have shown that selective reporting is a problem, until now only a few studies have explored the potential for systematic reviewers to select outcomes for meta- analysis,177-179 and none have used both public and nonpublic sources. Although our results were consistent with previous studies comparing various sources with trial registrations,180,181 conference abstracts,6,182 FDA reviews,174,183 journal articles,184 protocols,185-187 clinical study reports,8,188 and individual participant data,9,189,190 we do not know the extent of applicability of our results to other CER questions. These case studies included trials of relatively short

duration, and there may be even greater benefits to using nonpublic sources for trials with multiple assessments over long periods of time.

CONCLUSIONS

For systematic reviewers and meta-analysts, using multiple data sources for the same trial is time consuming and comes with both rewards (eg, more complete data) and challenges (eg, the need to resolve differences among contradictory sources). We found no advantage in using conference abstracts except for identifying the existence of unregistered and otherwise unpublished trials. Trial registrations in these case studies were relatively old (eg, we only found information in ClinicalTrials.gov, not other registries), and they provided less information than users might find in more recent registrations. Systematic reviewers should anticipate the vast amount of information that could be extracted from nonpublic sources and plan accordingly. Further research is needed to determine if the results of these case studies are representative of all trials and to determine how multiple data sources might affect meta-analyses of potential harms.

REFERENCES

1. Chan AW, Song F, Vickers A, et al. Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014;383(9913):257-266.

2. Goodman S, Dickersin K. Metabias: a challenge for comparative effectiveness research. Ann Intern Med. 2011;155(1):61-62.

3. Higgins JP, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]: The Cochrane Collaboration; 2011. http://handbook.cochrane.org.

4. Institute of Medicine. Finding What Works in Health Care: Standards for Systematic Reviews. The National Academies Press; 2011.

5. Schmucker C, Schell LK, Portalupi S, et al. Extent of non-publication in cohorts of studies approved by research ethics committees or included in trial registries. PLoS One. 2014;9(12):e114023. doi:10.1371/journal.pone.0114023

6. Scherer RW, Langenberg P, von Elm E. Full publication of results initially presented in abstracts. Cochrane Database Systematic Rev. 2007;(2):MR000005. doi:10.1002/14651858.MR000005.pub3

7. Doshi P, Dickersin K, Healy D, Vedula SS, Jefferson T. Restoring invisible and abandoned trials: a call for people to publish the findings. BMJ. 2013;(346):f2865.

8. Vedula SS, Bero L, Scherer RW, Dickersin K. Outcome reporting in industry-sponsored trials of gabapentin for off-label use. N Engl J Med. 2009;361(20):1963-1971.

9. Stewart LA, Parmar MK. Meta-analysis of the literature or of individual patient data: is there a difference? Lancet. 1993;341(8842):418-422.

10. Page MJ, Forbes A, Chau M, Green SE, McKenzie JE. Investigation of bias in meta- analyses due to selective inclusion of trial effect estimates: empirical study. BMJ Open. 2016;6(4):e011863.

11. Nosek BA, Alter G, Banks GC, et al. Scientific standards: promoting an open research culture. Science. 2015;348(6242):1422-1425.

12. Mayo-Wilson E, Hutfless S, Li T, et al. Integrating multiple data sources (MUDS) for meta-analysis to improve patient-centered outcomes research: a protocol for a systematic review. Syst Rev. 2015;(4):143.

13. Yawn BP, Wollan PC, Weingarten TN, Watson JC, Hooten WM, Melton LJ III. The prevalence of neuropathic pain: clinical evaluation compared with screening tools in a community population. Pain Med. 2009;10(3):586-593.

14. Harris RP, Helfand M, Woolf SH, et al. Current methods of the US Preventive Services Task Force: a review of the process. Am J Prev Med. 2001;20(suppl 3):21-35.

15. Kessler RC, Chiu WT, Demler O, Merikangas KR, Walters EE. Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62(6):617-627.

16. Merikangas KR, Jin R, He JP, et al. Prevalence and correlates of bipolar spectrum disorder in the world mental health survey initiative. Arch Gen Psychiatry. 2011;68(3):241-251.

17. Calabrese JR, Hirschfeld RM, Frye MA, Reed ML. Impact of depressive symptoms compared with manic symptoms in bipolar disorder: results of a U.S. community-based sample. J Clin Psychiatry. 2004;65(11):1499-1504.

18. Hirschfeld RM, Vornik LA. Recognition and diagnosis of bipolar disorder. J Clin Psychiatry. 2004;65(suppl 15):5-9.

19. Ilgen MA, Bohnert AS, Ignacio RV, et al. Psychiatric diagnoses and risk of suicide in veterans. Arch Gen Psychiatry. 2010;67(11):1152-1158.

20. Holma KM, Haukka J, Suominen K, et al. Differences in incidence of suicide attempts between bipolar I and II disorders and major depressive disorder. Bipolar Disord. 2014:16:652-661. doi:10.1111/bdi.12195

21. Nivoli AM, Colom F, Murru A, et al. New treatment guidelines for acute bipolar depression: a systematic review. J Affect Disord. 2011;129(1-3):14-26.

22. NICE. Bipolar Disorder: The Management of Bipolar Disorder in Adults, Children and Adolescents, in Primary and Secondary Care. National Institute for Health and Care Excellence; 2014.

23. Newcomer JW. Second-generation (atypical) antipsychotics and metabolic effects: a comprehensive literature review. CNS Drugs. 2005;19(suppl 1):1-93.

24. Ray WA, Chung CP, Murray KT, Hall K, Stein CM. Atypical antipsychotic drugs and the risk of sudden cardiac death. N Engl J Med. 2009;360(3):225-235.

25. Gargon E, Williamson PR, Altman DG, Blazeby JM, Clarke M. The COMET initiative database: progress and activities update (2014). Trials. 2015;(16):515.

26. Zarin DA, Tse T, Williams RJ, Califf RM, Ide NC. The ClinicalTrials.gov results database— update and key issues. N Engl J Med. 2011;364(9):852-860.

27. Saldanha IJ, Dickersin K, Wang X, Li T. Outcomes in Cochrane systematic reviews addressing four common eye conditions: an evaluation of completeness and comparability. PLoS One. 2014;9(10):e109400. doi:10.1371/journal.pone.0109400

28. Ip S, Hadar N, Keefe S, et al. A web-based archive of systematic review data. Syst Rev. 2012;(1):15.

29. Li T, Vedula SS, Hadar N, Parkin C, Lau J, Dickersin K. Innovations in data collection, management, and archiving for systematic reviews. Ann Intern Med. 2015;162(4):287- 294.

30. Shamseer L, Moher D, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;349:g7647.

31. Statistical Software: Release [computer program]. Version 14.0. StataCorp Corporation; 2001.

32. FineReader [computer program]. Version 11. Milpitas, CA. ABBYY; 2013.

33. Stewart LA, Clarke MJ. Practical methodology of meta-analyses (overviews) using updated individual patient data: Cochrane Working Group. Stat Med. 1995;14(19):2057- 2079.

34. Riley RD, Lambert PC, Abo-Zaid G. Meta-analysis of individual participant data: rationale, conduct, and reporting. BMJ. 2010;(340):c221.

35. Riley RD, Simmonds MC, Look MP. Evidence synthesis combining individual patient data and aggregate data: a systematic review identified current practice and possible methods. J Clin Epidemiol. 2007;60(5):431-439.

36. Riley RD, Lambert PC, Staessen JA, et al. Meta-analysis of continuous outcomes combining individual patient data and aggregate dat. Stat Med. 2008;27(11):1870-1893.

37. Higgins JP, Whitehead A, Turner RM, Omar RZ, Thompson SG. Meta-analysis of continuous outcome data from individual patients. Stat Med. 2001;20(15):2219-2241.

38. Turner RM, Omar RZ, Yang M, Goldstein H, Thompson SG. A multilevel model framework for meta-analysis of clinical trials with binary outcomes. Stat Med. 2000;19(24):3417-3432.

39. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7(3):177- 188.

40. Turk DC, Dworkin RH, Allen RR, et al. Core outcome domains for chronic pain clinical trials: IMMPACT recommendations. Pain. 2003;106(3):337-345.

41. Vedula SS, Bero L, Scherer RW, Dickersin K. Outcome reporting in industry-sponsored trials of gabapentin for off-label use. N Engl J Med. 2009;361(20):1963-1971.

42. Mayo-Wilson E, Doshi P, Dickersin K. Are manufacturers sharing data as promised? BMJ. 2015;(351):h4169.

43. Arai YC, Matsubara T, Shimo K, et al. Low-dose gabapentin as useful adjuvant to opioids for neuropathic cancer pain when combined with low-dose imipramine. J Anesth. 2010;24(3):407-410.

44. Arai YP, Matsubara T, Shimo K, Ushida T, Osuga T, Nishihara M. Low dose gabapenttin as useful adjuvant to opioids for neuropathic cancer pain when combined with low dose imipramine. Anesth Analg. 2010;110(3):S365.

45. Backonja M, Beydoun A, Edwards KR, et al. Gabapentin for the symptomatic treatment of painful neuropathy in patients with diabetes mellitus: a randomized controlled trial. JAMA. 1998;280(21):1831-1836.

46. Backonja MM. Gabapentin monotherapy for the symptomatic treatment of painful neuropathy: a multicenter, double-blind, placebo-controlled trial in patients with diabetes mellitus. Epilepsia. 1999;40(Suppl 6):S57-S59; discussion S73-S74.

47. Backonja M. Gabapentin for painful diabetic neuropathy (in reply). JAMA. 1999;282(2):133-134.

48. Slawson D. How effective is gabapentin (Neurontin) in reducing the pain associated with diabetic peripheral neuropathy? Evidence-Based Practice. 1999;2(3):7.

49. Backonja MM, Mutisya E. Dose response to gabapentin across five multicenter trials for neuropathic pain. Ann Neurol. 2002(suppl 1):S81.

50. Backonja M, Glanzman RI. Gabapentin dosing for neuropathic pain: evidence from randomized, placebo-controlled clinical trials. Clin Ther. 2003;(25):81-104.

51. Backonja MM, Mutisva E. Gabapentin demonstrates a nonlinear dose-reponse across five multicenter trials for neuropathic pain. Poster presented at: EFNS Annual Congress; October 26-29, 2002; Vienna, Austria.

52. Rowbotham MC, Backonja MM, Knapp LE, Yan C, Purcell TJ, Koto E. Gabapentin in the management of neuropathic pain in patients with diabetic peripheral neuropathy, postherpetic neuralgia, and neuropathic pain of varying etiology: Overview of results from 5 placebo-controlled trials. Poster presented at: Annual Meeting of ICMTNP; 2002; San Francisco, CA.

53. Backonja M, Mutisya EM. Review of gabapentin dosing in five placebo-controlled clinical trials for neuropathic pain. Eur J Neurol. 2002;(suppl 2):191.

54. Anonymous. Gabapentin for diabetic neuropathy. Nurses' Drug Alert. 1999;23(2):11.

55. Edwards KR, Bannington VT, Hes MS, LaMoreaux LK, Garofalo EA, Koto EM. Gabapentin (Neurontin) for pain associated with diabetic peripheral neuropathy: a double-blind, placebo-controlled study (945-210) [abstract]. Neurology. 1998;(50):A378-A379.

56. Backonja M, Hes MS, LaMoreaux LK, Garofalo EA, Koto EM, The US Gabapentin Study Group 210. Gabapentin reduces pain in diabetics with painful peripheral neuropathy: results of a double-blind, placebo controlled trial (945-210). Poster presented at: 16th Annual Scientific Meeting of the American Pain Society; 1997. New Orleans, LA.

57. Seidl JJ, Slawson JG. Gabapentin for painful diabetic neuropathy. J Fam Pract. 1999;48(3):173-174.

58. Parke-Davis. Research Report: 720-03908. A double-blind placebo-controlled trial of gabapentin for treatment of painful diabetic peripheral neuropathy. Protocol Number: 945-210 30 December 1998.

59. Caraceni A, Zecca E, Bonezzi C, Bennett MI. Gabapentin significantly improves analgesia in people receiving opioids for neuropathic cancer pain. Cancer Treat Rev. 2005;31(1):58-62.

60. Caraceni A, Zecca E, Bonezzi C, et al. Gabapentin for neuropathic cancer pain: a randomized controlled trial from the Gabapentin Cancer Pain Study Group. J Clin Oncol. 2004;22(14):2909-2917.

61. Caraceni A, Zecca E, Bonezzi C, Spigel DR. Improving neuropathic pain due to cancer using gabapentin. J Clin Outcomes Manag. 2004;11(10):620-621.

62. Hahn K, Arendt G, Braun JS, et al. A placebo-controlled trial of gabapentin for painful HIV-associated sensory neuropathies. J Neurol. 2004;251(10):1260-1266.

63. Hui AC, Wong SM, Leung HW, Man BL, Yu E, Wong LK. Gabapentin for the treatment of carpal tunnel syndrome: a randomized controlled trial. Eur J Neurol. 2011;18(5):726- 730.

64. Chinese University of Hong Kong. Gabapentin for carpal tunnel syndrome: a randomised controlled trial. In: ClinicalTrials.gov [Internet]. Bethesda (MD): National Library of Medicine (US). 2000 [cited 26 May 2015]. https://clinicaltrials.gov/ct2/show/NCT00137735. NLM Identifier: NCT00137735.

65. Jensen MP, Chiang YK, Wu J. Assessment of pain quality in a clinical trial of gabapentin extended release for postherpetic neuralgia. Clin J Pain. 2009;25(4):286-292.

66. Irving G, Jensen M, Cramer M, et al. Efficacy and tolerability of gastric-retentive gabapentin for the treatment of postherpetic neuralgia: results of a double-blind, randomized, placebo-controlled clinical trial. Clin J Pain. 2009;25(3):185-192.

67. Milenkovic T, Percan V, Petrovski G, Ahmeti I, Mishevska Jovanovska S. Management of painful diabetic neuropathy with gabapenthin. J Diabetes Complicat. 2009;(1):A202- A203.

68. Mishra S, Bhatnagar S, Goyal GN, Rana SP, Upadhya SP. A comparative efficacy of amitriptyline, gabapentin, and pregabalin in neuropathic cancer pain: a prospective randomized double-blind placebo-controlled study. Am J Hosp Palliat Care. 2012;29(3):177-182.

69. EpiCept Corporation. A phase 3 multicenter, randomized, double-blind, placebo- controlled study of the safety and efficacy of Gabapentin Extended Release (G-ER) tablets in the treatment of patients with postherpetic neuralgia. In: ClinicalTrials.gov [Internet]. Bethesda (MD): National Library of Medicine (US) 2000 [cited 26 May 2015]. https://clinicaltrials.gov/ct2/show/NCT00475904. NLM Identifier: NCT00475904.

70. Rice ASC, Maton S. Gabapentin in postherpetic neuralgia: a randomised, double blind and placebo controlled study. [Spanish]. Revista de la Sociedad Española del Dolor. 2002;9(2):61-73.

71. Parsons B, Tive L, Huang S. Gabapentin: a pooled analysis of adverse events from three clinical trials in patients with postherpetic neuralgia. Am J Geriatr Pharmacother. 2004;2(3):157-162.

72. Rice AS, Maton S. Gabapentin in postherpetic neuralgia: a randomised, double blind, placebo controlled study. Pain. 2001;94(2):215-224.

73. Anonymous. First approval of oral treatment for postherpetic neuralgia. Formulary. 2002;37(7):335.

74. Rice AS, Maton S. Reply to comment on "Rice ASC, Maton S, the Postherpetic Neuralgia Study Group (UK), gabapentin in postherpetic neuralgia: a randomized, double blind, placebo-controlled study." Pain. 96(3):411-412.

75. Parke-Davis. Research Report:430-00124. A double-blind placebo controlled trial of gabapentin for the treatment of post herpetic neuralgia. Protocol Number: 945-430-295 (NN 025) 3 April 2000.

76. Food and Drug Administration (Center for Drug Evaluation and Research). Statistical Review NDA 21-397. 2002.

77. Food and Drug Administration (Center for Drug Evaluation and Research). Medical Review NDA 21-397. 2002.

78. Rowbotham M, Harden N, Stacey B, Bernstein P, Magnus-Miller L. Gabapentin for the treatment of postherpetic neuralgia: a randomized controlled trial. JAMA. 1998;280(21):1837-1842.

79. Slawson D. How effective and safe is gabapentin (Neurontin) for reducing the pain of postherpetic neuralgia? Evidence-Based Practice. 1999;2(3):7-8.

80. Anonymous. Gabapentin for postherpetic neuralgia. Nurses' Drug Alert. 1999;(23):2.

81. Parke-Davis. Research Report: 995-00070. Double-blind, randomized, placebo- controlled, parallel groups, multi-center trial to determine the efficacy and safety of Neurontin (gabapentin) in subjects with peripheral neuropathy (post-herpetic neuralgia). Protocol Number: 945-211 29 December 1998.

82. Sandercock D, Cramer M, Biton V, Cowles VE. A gastroretentive gabapentin formulation for the treatment of painful diabetic peripheral neuropathy: efficacy and tolerability in a double-blind, randomized, controlled clinical trial. Diabetes Res Clin Pract. 2012;97(3):438-445.

83. Sandercock D, Cramer M, Wu J, Chiang YK, Biton V, Heritier M. Gabapentin extended release for the treatment of painful diabetic peripheral neuropathy: efficacy and tolerability in a double-blind, randomized, controlled clinical trial. Diabetes Care. 2009;32(2):e20.

84. Depomed. A multicenter, randomized, double-blind, placebo-controlled study of the safety and efficacy of gabapentin extended release (G-ER) tablets in the treatment of patients with painful diabetic peripheral neuropathy. In: ClinicalTrials.gov [Internet]. National Library of Medicine (US) 2000 [cited 26 May, 2015]. https://clinicaltrials.gov/ct2/show/NCT00712439. NLM Identifier: NCT00712439.

85. Sang CN, Sathyanarayana R, Sweeney M. Gastroretentive gabapentin (G-GR) formulation reduces intensity of pain associated with postherpetic neuralgia (PHN). Clin J Pain. 2013;29(4):281-288.

86. Once-daily therapy for postherpetic neuralgia. Clinical Advisor. November 21, 2011. https://www.clinicaladvisor.com/home/drug-update/once-daily-therapy-for- postherpetic-neuralgia/

87. Sweeney M, Sathyanarayana R. Efficacy and tolerability of once-daily gabapentin extended-release (G-ER) for the treatment of postherpetic Neuralgia in patients at least 65 years old. J Gen Intern Med. 2011:26(S161). doi:10.1007/s11606-011-1730-9

88. Irving GA, Sweeney M. Tolerability and safety of gastroretentive once-daily gabapentin tablets for the treatment of postherpetic neuralgia. J Pain Res. 2012;(5):203-208.

89. Irving G, Sondag E, Sweeney M. Tolerability and safety of gabapentin extended-release tablets in the treatment of postherpetic neuralgia. Pain Med. 2011;12(3):509.

90. Irving G, Carter F, Sondag E, Sweeney M. Tolerability and safety of gabapentin extended- release tablets in the treatment of postherpetic neuralgia. J Am Pharm Assoc. 2011;51(2):253.

91. Gupta A, Li S. Safety and efficacy of once-daily gastroretentive gabapentin in patients with postherpetic neuralgia aged 75 years and over. Drugs Aging. 2013;30(12):999- 1008.

92. Jensen MP, Hsu PH, Vanhove GF. Early pain reduction can predict treatment response: results of integrated efficacy analyses of a once-daily gastroretentive formulation of gabapentin in patients with postherpetic neuralgia. Pain Med. 2012;13(8):1059-1066.

93. Rauck RL, Irving GA, Wallace MS, Vanhove GF, Sweeney M. Once-daily gastroretentive gabapentin for postherpetic neuralgia: integrated efficacy, time to onset of pain relief and safety analyses of data from two phase 3, multicenter, randomized, double-blind, placebo-controlled studies. J Pain Symptom Manage. 2013;46(2):219-228.

94. Backonja M, Wallace MS, Freeman R, Sweeney M. Effect of gabapentin once daily on Neuropathic Pain Scale (NPS) score in patients with postherpetic neuralgia (PHN). Pain Med. 2012;13(2):341.

95. Sweeney M, Misha B, Wallace M, Freeman R. Effect of a once-daily gastroretentive formulation of gabapentin on the Brief Pain Inventory Scale scores in patients with postherpetic neuralgia. J Pain. 2012;13(4)(suppl 1):S71.

96. Jensen M, Rauck R, Irving G, Sweeney M, Vanhove G. Onset of treatment response to a once-daily gastroretentive formulation of gabapentin in patients with postherpetic neuralgia. J Pain. 2012;13(4)(suppl 1):S70.

97. Rauck R, Irving G, Wallace M, Vanhove G, Sweeney M. Integrated analysis of efficacy and safety of a once-daily gastroretentive formulation of gabapentin in patients with postherpetic neuralgia. Neurology. 2012;78(1).P04.161.

98. Rauck R, Irving G, Wallace MS, Vanhove GF, Sweeney M. Integrated analysis of efficacy and safety of a once-daily gastroretentive formulation of gabapentin in patients with postherpetic neuralgia. Pain Pract. 2012;(12):119.

99. Smith W, Rauck R, Irving G, Wallace M, Vanhove G, Sweeney M. Integrated analysis of efficacy and safety of a once-daily gastroretentive formulation of gabapentin in patients with postherpetic neuralgia. J Am Pharm Assoc. 2012;52(2):263.

100. Kantor D, Mathis JT, Rauck RL, Irving G, Sweeney M, Vanhove GF. Integrated analysis of efficacy of a once-daily gastroretentive formulation of gabapentin in patients with postherpetic neuralgia who are at least 75 years old. J Gen Intern Med. 2012;(27):S231.

101. Depomed. A phase 3 multicenter, randomized, double-blind, placebo-controlled study of the safety and efficacy of Gabapentin Extended Release (G-ER) tablets in the treatment of patients with postherpetic neuralgia. In: ClinicalTrials.gov [Internet]. National Library of Medicine (US) 2000 [cited 6 November 2015]. https://clinicaltrials.gov/ct2/show/NCT00636636. NLM Identifier: NCT00636636.

102. Serpell MP, The Neuropathic Pain Study Group. Gabapentin in neuropathic pain syndromes: a randomised, double‐blind, placebo‐controlled trial. Poster presented at: Fifth International Conference on Mechanisms and Treatment of Neuropathic Pain Annual Meeting; November 21-23, 2002; Hamilton, Bermuda.

103. Serpell MG. Gabapentin in neuropathic pain syndromes: a randomised, double-blind, placebo-controlled trial. Pain. 2002;99(3):557-566.

104. Serpell MG, Neuropathic Pain Study Group. Reply to Comment on: Serpell et al., gabapentin in neuropathic pain syndromes: a randomised double-blind, placebo controlled trial. Pain. 2003;103(1-2):228.

105. Parke-Davis. Research Report: 430-00125. A double blind placebo controlled trial of gabapentin for the treatment of patients exhibiting symptoms of neuropathic pain. Protocol Number: 945-430-306 (NN 026). 5 May 2000.

106. Simpson DA. Gabapentin and venlafaxine for the treatment of painful diabetic neuropathy. J Clin Neuromuscul Dis. 2001;3(2):53-62.

107. Simpson DA. Gabapentin and venlafaxine in the treatment of painful diabetic neuropathy. Muscle Nerve. 2000;23(10):1627.

108. Parke-Davis. Research Report: 720-04130. A double-blind placebo-controlled trial with 3 doses of gabapentin for treatment of painful diabetic neuropathy. Protocol Number: 945-224. 7 February 2000.

109. Pfizer. Final study report: A 15 Week randomized, double-blind, placebo controlled, parallel-group, multicenter study of Neurontin (gabapentin) for eficacy and quality of life in patients with painful diabetic peripheral neuropahty. Protocol A945-1008. 24 March 2005.

110. Pérez HE, Sánchez GF. Gabapentin therapy for diabetic neuropathic pain. Am J Med. 2000;108(8):689.

111. Tamez Pérez HE, Rodríguez Ayala M, Gómez de Ossio MD. Uso de gabapentina en la neuropatía diabética dolorosa [Use of gabapentin on neuropathic diabetic pain]. Med Interna Méx. 1998;14(6):251-253.

112. Wallace MS, Irving G, Cowles VE. Gabapentin extended-release tablets for the treatment of patients with postherpetic neuralgia: a randomized, double-blind, placebo-controlled, multicentre study. Clin Drug Investig. 2010;30(11):765-776.

113. Rowbotham MC. Gabapentin Extended Release (GER) in the treatment of postherpetic neuralgia: a randomized, placebo-controlled clinical trial. Pain Pract. 2009;(9):59-60.

114. Depomed. A phase 3 multicenter, randomized, double-blind, placebo-controlled study of the safety and efficacy of gabapentin extended release (G-ER) tablets in the treatment of patients with postherpetic neuralgia. In: ClinicalTrials.gov [Internet]. National Library of Medicine (US) 2000 [cited 6 November 2015]. https://clinicaltrials.gov/ct2/show/NCT00335933. NLM Identifier: NCT00335933.

115. Yildirim K, Sisecioglu M, Karatay S, et al. The effectiveness of gabapentin in patients with chronic radiculopathy. Pain Clinic. 2003;15(3):213-218.

116. Zepeda-Vazquez TC, Hernandez-Santos JR, Tenopala-Villegas S, et al. Manejo del dolor neuropatico en el paciente diabetico con tramadol via oral comparado con la administracion del mismo asociado a la amitrptilina O Gabapentina. Revista Mexicana de Anestesiología. 2001;1(2):84-87.

117. Calabrese JR, Keck PE, Macfadden W, et al. A randomized, double-blind, placebo- controlled trial of quetiapine in the treatment of bipolar I or II depression. Am J Psychiatry. 2005;162(7):1351-1360.

118. Macfadden W, Calabrese JR, Suppes T, et al. Quetiapine in bipolar I depression: doubleblind, placebo-controlled study. Bipolar Disord. 2005;7(72)(suppl 2).

119. Endicott J, Rajagopalan K, Minkwitz M, Macfadden W. A randomized, double-blind, placebo-controlled study of quetiapine in the treatment of bipolar I and II depression: improvements in quality of life. Int Clin Psychopharm. 2007;22(1):29-37.

120. Cookson J, Keck PE, Ketter TA, Macfadden W, Minkwitz MC, Mullen J. Quetiapine in bipolar depression: NNT and time-to-event analyses. Poster presented at: 158th Annual Meeting of the American Psychiatric Association; May 21-26, 2005; Atlanta, GA.

121. Hirschfeld RM, Weisler RH, Raines SR, Macfadden W. Quetiapine in the treatment of anxiety in patients with bipolar I or II depression: a secondary analysis from a randomized, double-blind, placebo-controlled study. J Clin Psychiatry. 2006;67(3):355- 362.

122. Endicott J, Rajagopalan K, MacFadden W, Minkwitz MC, Gaddy J. Efficacy of quetiapine in improving quality of life in bipolar depression. Poster presented at: 158th Annual Meeting of the American Psychiatric Association; May 21-26, 2005; Atlanta, GA.

123. Cookson J, Keck PE, Ketter TA, Macfadden W. Number needed to treat and time to response/remission for quetiapine monotherapy efficacy in acute bipolar depression: evidence from a large, randomized, placebo-controlled study. Int Clin Psychopharm. 2007;22(2):93-100.

124. Calabrese JR, Macfadden W, McCoy R, Minkwitz M, Wilson E, Mullen J. Double-blind, placebo-controlled study of quetiapine in bipolar depression. Poster presented at: 157th Annual Meeting of the American Psychiatric Association; May 1-6, 2004; New York, NY.

125. Fajutrao L, Gustafsson U, Paulsson B, Eriksson H. Quetiapine in the treatment of bipolar depression: Improvements in quality of life and functioning in four randomized, placebo-controlled trials. Int J Psychiatry Clin Pract. 2009;13(S1):28-29.

126. Eriksson H, Fajutrao L, Gustafsson U, Paulsson B. Quetiapine in the treatment of bipolar depression: improvements in quality of life and functioning in four randomized, placebo- controlled trials. Bipolar Disord. 2009;11(S1):37.

127. Young AH, Calabrese JR, Gustafsson U, et al. The efficacy of quetiapine monotherapy in bipolar II depression: combined data from the BOLDER and EMBOLDEN studies. Bipolar Disord. 2009;11(S1):90.

128. Young AH, Macritchie KAN, Calabrese J, et al. The efficacy of Quetiapine monotherapy in bipolar II depression: combined data from the BOLDER and EMBOLDEN studies. Int J Psychiatry Clin Pract. 2009;13(S1):37.

129. Calabrese J, Macritchie KAN, Young AH, Gustafsson U, Paulsson B. The efficacy of Quetiapine monotherapy in bipolar depression: combined data from the BOLDER and EMBOLDEN studies. Int J Psychiatry Clin Pract. 2009;13(suppl 1):36-37.

130. Weisler RH, Calabrese JR, Thase ME, et al. Efficacy of quetiapine monotherapy for the treatment of depressive episodes in bipolar I disorder: a post hoc analysis of combined results from 2 double-blind, randomized, placebo-controlled studies. J Clin Psychiatry. 2008;69(5):769-782.

131. Endicott J, Paulsson B, Gustafsson U, Schiöler H, Hassan M. Quetiapine monotherapy in the treatment of depressive episodes of bipolar I and II disorder: improvements in quality of life and quality of sleep. J Affect Disord. 2008;111(2-3):306-319.

132. Suppes T, Hirschfeld RM, Vieta E, Raines S, Paulsson B. Quetiapine for the treatment of bipolar II depression: analysis of data from two randomized, double-blind, placebo- controlled studies. World J Biol Psychiatry. 2008;9(3):198-211.

133. Young AH, Calabrese JR, Gustafsson U, et al. Quetiapine monotherapy in bipolar II depression: combined data from four large, randomized studies. Int J Bipolar Disord. 2013;1(10):1-12.

134. AstraZeneca. Safety and efficacy trial of the use of quetiapine fumarate (SEROQUEL) in the treatment of patients with bipolar depression. In: ClinicalTrials.gov [Internet]. National Library of Medicine (US). 2000 [cited 1 June 2015]. https://clinicaltrials.gov/ct2/show/NCT00060489. NLM Identifier: NCT00060489.

135. MacFadden W, Calabrese J, Suppes T, et al. Quetiapine in bipolar I depression: double- blind, placebo-controlled study. Poster presented at: 158th Annual Meeting of the American Psychiatric Association; May 21-26, 2005; Atlanta, GA.

136. Cookson J, Keck PE, Ketter TA, Macfadden W, Minkwitz MC, Mullen J. Quetiapine in bipolar depression: NNT and time-to-event analyses. Bipolar Disord. 2005;7(suppl 2):27- 117.

137. Endicott J, Rajagopalan K, MacFadden W, Minkwitz M, Gaddy J. Efficacy of quetiapine in improving quality of life in patients with bipolar depression. Bipolar Disord. 2005;7(suppl 2):76, 50.

138. Thase ME. Quetiapine monotherapy for bipolar depression. Neuropsychiatr Dis Treat. 2008;4(1):11-21.

139. Anonymous. New data shows the efficacy and good tolerance of Seroquel in bipolar depression. Médecine et Hygiène. 2004;62(2489):1450.

140. Macfadden W, Calabrese JR, McCoy R, Minkwitz M, Wilson E, Mullen J. Antianxiety effects analysis of quetiapine in bipolar depression. Poster presented at: 157th Annual Meeting of the American Psychiatric Association; May 1-6, 2004; New York, NY.

141. Gustafsson U, Fajutrao L. Effect of quetiapine on functioning and quality of life in bipolar depression. Eur Psychiat. 2011; 26(supp 1):1935.

142. AstraZeneca. A multicenter, double-blind, randomized, placebo-controlled, double- dummy trial of the use of quetiapine fumarate (SEROQUEL[Registered]) in the treatment of patients with bipolar depression. Study code: 5077US/0049. Clinical Study Report 28 November 2005.

143. Gao K, Wu R, Kemp DE, et al. Efficacy and safety of quetiapine-XR as monotherapy or adjunctive therapy to a mood stabilizer in acute bipolar depression with generalized anxiety disorder and other comorbidities: a randomized, placebo-controlled trial. J Clin Psychiatry. 2014;75(10):1062-1068.

144. Gao K. Quetiapine XR in the treatment of comorbid generalized anxiety disorder in bipolar depression with or without substance use disorder. In: ClinicalTrials.gov

[Internet]. National Library of Medicine (US) 2000 [cited 6 November 2015]. https://clinicaltrials.gov/ct2/show/NCT00671853. NLM Identifier: NCT00335933.

145. AstraZeneca. A multicenter, double-blind, randomized, placebo-controlled study to evaluate the efficacy and safety of quetiapine fumarate (SEROQUEL) extended release as monotherapy in the treatment of patients with bipolar depression. In: ClinicalTrials.gov [Internet]. National Library of Medicine (US) 2000 [cited 1 June 2015]. https://clinicaltrials.gov/ct2/show/NCT01256177. NLM Identifier: NCT01256177.

146. Li HF, Gu NF, Zhang HY, et al. The efficacy and safety of quetiapine extended release (XR) as mono-therapy in the treatment of Chinese patients with bipolar I or II depression. Bipolar Disord. 2014;(16):65-66.

147. McElroy SL, Weisler RH, Chang W, et al. A double-blind, placebo-controlled study of quetiapine and paroxetine as monotherapy in adults with bipolar depression (EMBOLDEN II). J Clin Psychiatry. 2010;71(2):163-174.

148. Anonymous. EMBOLDEN I and II: quetiapine effective for treating bipolar depression. Brown University Psychopharmacology Update. 2010;21(5):1.

149. AstraZeneca. Multicentre, double-blind, randomised, parallel group, placebo controlled, phase 3 study of the efficacy & safety of quetiapine fumarate & paroxetine as monotherapy in adult patients with bipolar depression for 8 weeks & quetiapine in continuation (Abbreviated). In: ClinicalTrials.gov [Internet]. National Library of Medicine (US) 2000 [cited 1 June 2015]. https://clinicaltrials.gov/ct2/show/NCT00119652. NLM Identifier: NCT00119652.

150. AstraZeneca. An international, multi-centre, double-blind, randomised, parallel-group, placebo-controlled, phase III study of the efficacy and safety of quetiapine fumarate (Seroquel [Registered], single oral 300 mg or 600 mg dose) and paroxetine as monotherapy in adult patients with bipolar depression for 8 weeks and quetiapine in continuation treatment for 26 up to 52 weeks. Study code: D1447C00134. Synopsis 5 October 2007.

151. Young A, McElroy S, Chang W, Olausson B, Paulsson B, Brecher M. Placebo-controlled study with acute and continuation phase of quetiapine in adults with bipolar depression (EMBOLDEN II). Eur Neuropsychopharmacol. 2008;18(S4):S371-S372.

152. Suppes T, Datto C, Minkwitz M, Nordenhem A, Walker C, Darko D. Effectiveness of the extended release formulation of quetiapine as monotherapy for the treatment of acute bipolar depression. J Affect Disord. 2010;121(1-2):106-115.

153. Datto C, Minkwitz M, Nordenhem A, Walker C, Darko D, Suppes T. Effectiveness of the new extended-release formulation of quetiapine as monotherapy for the treatment of acute bipolar depression (trial D144CC00002). Eur Psychiat. 2009;24:S574.

154. AstraZeneca. A multicenter, double-blind, randomized, parallel-group, placebo- controlled, phase III study of the efficacy and safety of quetiapine fumarate (Seroquel SR) sustained-release as monotherapy in adult patients with acute bipolar depression. In: ClinicalTrials.gov [Internet]. National Library of Medicine (US) 2000 [cited 1 June 2015]. https://clinicaltrials.gov/ct2/show/NCT00422214. NLM Identifier: NCT00422214.

155. Thase ME. BOLDER II study of quetiapine therapy for bipolar depression. Future Neurol. 2007;2(4):373-377.

156. Thase ME, Macfadden W, Weisler RH, et al. Efficacy of quetiapine monotherapy in bipolar I and II depression: a double-blind, placebo-controlled study (the BOLDER II study). J Clin Psychopharmacol. 2006;26(6):600-609.

157. AstraZeneca. Controlled study of the use of quetiapine fumarate in the treatment of patients with bipolar depression. In: ClinicalTrials.gov [Internet]. National Library of Medicine (US) 2000 [cited 1 June 2015]. https://clinicaltrials.gov/ct2/show/NCT00083954. NLM Identifier: NCT00083954.

158. BOLDER II study confirms therapeutic potential of SEROQUEL in bipolar depression. EurekAlert! October 21, 2005. http://www.eurekalert.org/pub_releases/2005-10/hak- bis102105.php

159. AstraZeneca. A confirmatory multicenter, double-blind, randomized, placebo-controlled study of the use of quetiapine fumarate (SEROQUEL) in the treatment of patients with bipolar depression. Study code: D1447C00135. Clinical Study Report 1 December 2005.

160. Goodwin GM. Quetiapine more effective than placebo for depression in bipolar I and II disorder. Evid Based Ment Health. 2007;10(3):82.

161. Young AH, McElroy SL, Bauer M, et al. A double-blind, placebo-controlled study of quetiapine and lithium monotherapy in adults in the acute phase of bipolar depression (EMBOLDEN I). J Clin Psychiatry. 2010;71(2):150-162.

162. AstraZeneca. Multi-centre, double-blind, randomised, parallel-group, placebo- controlled, phase 3 study of the efficacy & safety of quetiapine fumarate & lithium as monotherapy in adult patients with bipolar depression for 8 weeks & quetiapine in continuation (Abbreviated). In: ClinicalTrials.gov [Internet]. National Library of Medicine (US) 2000 [cited 1 June, 2015]. https://clinicaltrials.gov/ct2/show/NCT00206141. NLM Identifier: NCT00206141.

163. Young AH. Erratum: A double-blind, placebo-controlled study with acute and continuation phase of quetiapine and lithium in adults with bipolar depression (Embolden I). Bipolar Disord. 2008;10(3):451.

164. AstraZeneca. An international, multi-centre, double-blind, randomised, parallel-group, placebo-controlled, phase III study of the efficacy and safety of quetiapine fumarate

(Seroquel, single oral 300 mg or 600 mg dose) and lithium as monotherapy in adult patients with bipolar depression for 8 weeks and quetiapine in continuation treatment for 26 up to 52 weeks. Study code: D1447C0001. Synopsis.

165. Grunze H. Quetiapine is effective in the treatment of adults in the acute phase of bipolar depression. Evid Based Ment Health. 2010;13(3):88.

166. Young A, McElroy S, Chang W, Olausson B, Paulsson B, Brecher M. Placebo-controlled study with acute and continuation phase of quetiapine in adults with bipolar depression (EMBOLDEN I). Eur Neuropsychopharmacol. 2008;18(S4):S371-S372.

167. Mayo-Wilson E, Fusco N, Li T, et al. Multiple outcomes and analyses in clinical trials create challenges for interpretation and research synthesis. J Clin Epidemiol. 2017;86:39-50. doi:10.1016/j.jclinepi.2017.05.007.

168. Wieland LS, Rutkow L, Vedula SS, et al. Who has used internal company documents for biomedical and public health research and where did they find them? PLoS One. 2014;9(5):e94709. doi:10.1371/journal.pone.0094709

169. Le Noury J, Nardo JM, Healy D, et al. Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence. BMJ. 2015;351:h4320.

170. Eyding D, Lelgemann M, Grouven U, et al. Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ. 2010;341:c4737.

171. Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008;358(3):252-260.

172. Jefferson T, Jones M, Doshi P, Spencer EA, Onakpoya I, Heneghan CJ. Oseltamivir for influenza in adults and children: systematic review of clinical study reports and summary of regulatory comments. BMJ. 2014;(348):g2545.

173. Rodgers MA, Brown JV, Heirs MK, et al. Reporting of industry funded study outcome data: comparison of confidential and published data on the safety and effectiveness of rhBMP-2 for spinal fusion. BMJ. 2013;(346):f3981.

174. Turner EH, Knoepflmacher D, Shapley L. Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration database. PLoS Med. 2012;9(3):e1001189. doi:10.1371/journal.pmed.1001189

175. Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ. 2012;344:d7202.

176. Food and Drug Administration Amendments Act (FDAAA), 42 USC § 8012007. 2007.

177. Tendal B, Nuesch E, Higgins JP, Juni P, Gotzsche PC. Multiplicity of data in trial reports and the reliability of meta-analyses: empirical study. BMJ. 2011;343:d4829.

178. Page MJ, McKenzie JE, Chau M, Green SE, Forbes A. Methods to select results to include in meta-analyses deserve more consideration in systematic reviews. J Clin Epidemiol. 2015;68(11):1282-1291.

179. Page MJ, McKenzie JE, Kirkham J, et al. Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Database Syst Rev. 2014;(10):MR000035.

180. Scherer RW, Huynh L, Ervin AM, Taylor J, Dickersin K. ClinicalTrials.gov registration can supplement information in abstracts for systematic reviews: a comparison study. BMC Med Res Methodol. 2013;(13):79.

181. Baudard M, Yavchitz A, Ravaud P, Perrodeau E, Boutron I. Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses. BMJ. 2017;(356):j448.

182. Saldanha IJ, Scherer RW, Rodriguez-Barraquer I, Jampel HD, Dickersin K. Dependability of results in conference abstracts of randomized controlled trials in ophthalmology and author financial conflicts of interest as a factor associated with full publication. Trials. 2016;17(1):213.

183. MacLean CH, Morton SC, Ofman JJ, Roth EA, Shekelle PG, Southern California Evidence- Based Practice Center: How useful are unpublished data from the Food and Drug Administration in meta-analysis? J Clin Epidemiol. 2003;56(1):44-51.

184. Gotzsche PC. Multiple publication of reports of drug trials. Eur J Clin Pharmacol. 1989;36(5):429-432.

185. Dickersin K, Min YI, Meinert CL. Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. JAMA. 1992;267(3):374-378.

186. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet. 1991;337(8746):867-872.

187. Dwan K, Gamble C, Williamson PR, Kirkham JJ, Reporting Bias Group. Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS One. 2013;8(7):e66844. doi:10.1371/journal.pone.0066844

188. Wieseler B, Kerekes MF, Vervoelgyi V, McGauran N, Kaiser T. Impact of document type on reporting quality of clinical drug trials: a comparison of registry reports, clinical study reports, and journal publications. BMJ. 2012;344:d8141.

189. Fu R, Selph S, McDonagh M, et al. Effectiveness and harms of recombinant human bone morphogenetic protein-2 in spine fusion: a systematic review and meta-analysis. Ann Intern Med. 2013;158(12):890-902.

190. Simmonds MC, Brown JV, Heirs MK, et al. Safety and effectiveness of recombinant human bone morphogenetic protein-2 for spinal fusion: a meta-analysis of individual- participant data. Ann Intern Med. 2013;158(12):877-889.

ACKNOWLEDGMENTS

The MUDS Investigators Lorenzo Bertizzolo, Department of Epidemiology, Johns Hopkins University Bloomberg School of Public Health Joseph K. Canner, Department of Surgery, Johns Hopkins University School of Medicine Terrie Cowley, The TMJ Association, Ltd. Kay Dickersin, PhD (Principal Investigator), Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health Peter Doshi, Pharmaceutical Health Services Research, University of Maryland School of Pharmacy Jeffrey Ehmsen, Department of Neurology-Neuromuscular Medicine, Johns Hopkins University School of Medicine Nicole Fusco, Department of Epidemiology, Johns Hopkins University Bloomberg School of Public Health Gillian Gresham, Department of Epidemiology, Johns Hopkins University Bloomberg School of Public Health Nan Guo, Department of Epidemiology, Johns Hopkins University Bloomberg School of Public Health Jennifer A Haythornthwaite, Department of Psychiatry & Behavioral Sciences, Johns Hopkins University School of Medicine James Heyward, Department of Epidemiology, Johns Hopkins University Bloomberg School of Public Health Hwanhee Hong, Department of Mental Health, Johns Hopkins University Bloomberg School of Public Health Tianjing Li, Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health Diana Lock , Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health Evan Mayo-Wilson, MPA, DPhil, Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health Jennifer L. Payne , Psychiatry and Behavioral Sciences, The Johns Hopkins Hospital Lori Rosman, Welch Medical Library, Johns Hopkins University School of Medicine Elizabeth A. Stuart, Department of Mental Health, Johns Hopkins University Bloomberg School of Public Health

Catalina Suarez-Cuervo, Department of Health Policy & Management, Johns Hopkins University Bloomberg School of Public Health Elizabeth Tolbert, Johns Hopkins University Peabody Institute Claire Twose, Welch Medical Library, Johns Hopkins University School of Medicine Swaroop Vedula, Malone Center for Engineering in Healthcare, Johns Hopkins University Whiting School of Engineering

Copyright© 2018 Johns Hopkins University. All Rights Reserved Disclaimer: The [views, statements, opinions] presented in this report are solely the responsibility of the author(s) and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute® (PCORI®), its Board of Governors or Methodology Committee. Acknowledgment: Research reported in this report was [partially] funded through a Patient-Centered Outcomes Research Institute® (PCORI®) Award (#ME-1303-5785). Further information available at: https://www.pcori.org/research-results/2013/benefits-and-challenges-using-multiple-sources-information-about-clinical