A New Way to Evaluate Randomized Controlled Trials? New Approach Does More Harm Than Good
Total Page:16
File Type:pdf, Size:1020Kb
BCMJ /#47Vol5.final 5/19/05 11:55 AM Page 241 David B. Miller, MD, FRCPC, Karin H. Humphries, DSc A new way to evaluate randomized controlled trials? New approach does more harm than good The Therapeutics Initiative uses a “total serious adverse events” rate to evaluate RCTs, but this flawed approach can lead to invalid conclusions and needless concern on the part of physicians and patients. ABSTRACT: The Therapeutics Initia- recent publication of the Using the TI publication as an tive of British Columbia published Therapeutics Initiative (TI) example, we will show how this ap- “Do statins have a role in primary of British Columbia, “Do proach to evaluating randomized con- prevention?” suggesting that statin A statins have a role in prima- trolled trials (RCTs) can lead to in- medications may be harmful in ry prevention?” suggested that statin valid conclusions and can cause patients without a previous myocar- medications (properly called HMG- needless concern, if not alarm, on the dial infarction. This conclusion was CoA reductase inhibitors) may do part of physicians and patients.4-7 Sev- drawn by evaluating serious adverse more harm than good in patients with- eral aspects of this approach to evalu- events in two randomized, placebo- out a previous myocardial infarction.1 ating RCTs deserve mention. First, controlled studies—AFCAPS/TexCAPS This conclusion was drawn by evalu- there are inherent problems when and PROSPER. The Therapeutics Ini- ating serious adverse events (SAEs) meta-analyses selectively focus on a tiative authors believe that total seri- in two randomized, placebo-controlled small number of trials and secondary ous adverse events, an endpoint that studies—AFCAPS/TexCAPS2 and rather than primary endpoints. Sec- combines safety and efficacy data, PROSPER.3 ond, proposing that total SAE rates (a is the best way to evaluate clinical While the authors agree that statins combination of safety and efficacy trials. We demonstrate in a variety of demonstrated a statistically signifi- endpoints) is the most appropriate ways that this approach to evaluat- cant reduction in the studies’ primary method to evaluate randomized clini- ing randomized controlled trials can endpoints and selected secondary end- cal trials ignores the intent of a clini- lead to invalid conclusions and can points, they argue that these efficacy cal trial and could lead to an under- cause needless concern, if not endpoints are less relevant than the assessment of adverse events. We will alarm, on the part of physicians and “total SAE” rate, or safety endpoint. demonstrate that when this type of patients. The authors believe that a similar total analysis is used, any trial that demon- SAE rate in the placebo and treatment strates superior efficacy against a arms of the studies demonstrates that “statins have not been shown to pro- Dr Miller is an endocrinologist in Victoria, vide an overall health benefit in pri- BC, and head of Endocrinology for the Van- mary prevention trials.”1 Furthermore, couver Island Health Authority, South because the total SAE rates are simi- Islands. Dr Humphries is an epidemiolo- lar in both arms, despite reductions in gist/statistician and assistant professor, the secondary efficacy endpoints, Department of Medicine, Division of Cardi- there is “the possibility that unrecog- ology and Centre for Health Evaluation and nized serious adverse events are Outcome Sciences at St. Paul’s Hospital in increased by statin therapy.”1 Vancouver, BC. VOL. 47 NO. 5, JUNE 2005 BC MEDICAL JOURNAL 241 BCMJ /#47Vol5.final 5/19/05 11:55 AM Page 242 A new way to evaluate randomized controlled trials? New approach does more harm than good background of equivalent safety will How are adverse events defined? clinically significant results from lab- be open to the charge of being inef- The definitions for SAEs are quite uni- oratory tests or other medical proce- fective and having unrecognized seri- form. The International Conference on dures, and clinically significant find- ous adverse events. Harmonisation (ICH) defines an SAE ings uncovered during a physical as, “any untoward medical occurrence examination. Discussion that results in death, is life-threaten- Adverse events should always be Primary versus secondary ing, requires inpatient hospitalization, reported when a trial is presented in endpoints or results in persistent or significant a peer-reviewed journal. The ICH PROSPER was designed and powered disability/incapacity.”8 The United GCP Statistical Guideline11 devotes to evaluate the efficacy of pravastatin States Federal Drug Administration an entire section to the appropriate for the prevention of coronary heart (USFDA) defines an SAE as “any analysis of adverse events, separate disease (CHD) death, non-fatal MI, experience that is fatal or life-threat- from a section on the analysis of pri- and fatal or nonfatal stroke.3 In ening, is permanently disabling, re- mary and secondary endpoints. Sim- AFCAPS/TexCAPS lovastatin was quires hospitalization, or is a congen- ilarly, the Consolidated Standards of used to reduce fatal or nonfatal MI, ital anomaly, cancer, or overdose.”9 Reporting Trials (CONSORT)12 dis- sudden cardiac death, or unstable an- All SAEs must be reported to the cusses the presentation and report- gina.2 The TI evaluation of these two appropriate regulatory agency irre- ing of efficacy endpoints as distinct trials, on the other hand, focuses on spective of whether the SAE—like from the presentation of adverse total mortality, for which neither death or acute myocardial infarc- events. Neither of these internation- study was powered, and MI or stroke, tion—is also an efficacy endpoint for ally recognized groups recommends which were secondary endpoints. The a particular trial. combined reporting of efficacy and absolute risk reduction in PROSPER For the purposes of reporting trial safety events. for the primary endpoint was 2.1% results in a journal, the reader is inter- Why report efficacy and safety (P=.014), while in AFCAPS/Tex- ested in all adverse events, not just endpoints separately? One of the car- CAPS it was 4.1% (P<.001). The sec- those defined as serious. The reader is dinal rules of statistical inference in ondary endpoint chosen in the TI interested in determining whether the clinical trials is the assumption of meta-analysis, MI and stroke, had an efficacy of the intervention was independence of the observations.13 absolute risk reduction of 1.8%. This achieved with acceptable safety by This is relevant in the assessment and is still statistically significant but un- comparing the adverse events in the reporting of both efficacy and safety derestimates the full benefit demon- treatment arm with those in the place- events. Multiple events for an individ- strated in these trials. bo arm. Unfortunately, the definition ual are unlikely to be independent, and of adverse events varies among phar- in a clinical trial the patient is the unit Reporting of serious maceutical companies, research orga- of observation. Data are presented as adverse events nizations, and journals. The ICH the proportion of patients experienc- The main premise of the alternative Good Clinical Practice (GCP) guide- ing a given endpoint. For a composite approach to RCT evaluation is that line defines an adverse event as any efficacy endpoint (like CHD death, because the SAE rates in the statin and unintended and unfavorable sign, MI, or stroke), a patient with both an placebo arms in these two trials were symptom, or disease temporally asso- MI and a CHD death contributes only the same, despite a reduction in MI ciated with the use of a medical prod- one count to the efficacy endpoint. In and stroke in the statin arm, the statin uct, whether or not it is considered addition, if the patient experienced arm must have incurred “unrecog- related to the product.8 Northington two adverse events (like an elevation nized SAEs.”1 This unconventional defines an adverse event as any unfa- in liver enzymes and hospitalization approach of combining efficacy (the vorable change in the structure (signs), for appendicitis), the patient also con- clinical endpoint the trial is designed function (symptoms), or chemistry tributes one count to the SAE total. to assess) with safety outcomes and (laboratory data) of the body tempo- This is different from regulatory then subtracting the efficacy out- rally associated with participation in requirements where both events comes is problematic both with re- the clinical trial, irrespective of the would be counted, because regulatory spect to how SAEs are reported and believed relationship to the study bodies are interested in the actual defined and with respect to method- drug.10 This definition would thus number and nature of adverse events ological and statistical validity. include intercurrent illness or injury, observed. 242 BC MEDICAL JOURNAL VOL. 47 NO. 5, JUNE 2005 BCMJ /#47Vol5.final 5/19/05 11:55 AM Page 243 A new way to evaluate randomized controlled trials? New approach does more harm than good If one adopts the “total SAE” ap- Table 1. Calculation of “other SAE.” proach, the patient who experienced each of these four outcomes would Treatment Placebo Absolute risk count only once. Thus, the total SAE reduction approach can underestimate the true Trial A Primary endpoint 8.0% 9.8% 1.8% magnitude of a drug’s adverse effects. SAE 20% 20% 0% Also in the total SAE analysis, a tran- sient elevation in liver enzymes would Other SAE 12% 10.2% –1.8%* be equivalent to a disabling stroke; Trial B Primary endpoint 5.0% 10.2% 5.2% most physicians and patients would SAE 20% 20% 0% reject this inference. Other SAE 15% 9.8% –5.2%* By assuming that the SAE rate includes all efficacy endpoints, and * (–) suggests harm that subtracting the efficacy endpoints from the total SAE rate can derive Table 2. Adding SAE to the primary outcome.