BCMJ /#47Vol5.final 5/19/05 11:55 AM Page 241

David B. Miller, MD, FRCPC, Karin H. Humphries, DSc

A new way to evaluate randomized controlled trials? New approach does more harm than good

The Therapeutics Initiative uses a “total serious adverse events” rate to evaluate RCTs, but this flawed approach can lead to invalid conclusions and needless concern on the part of physicians and patients.

ABSTRACT: The Therapeutics Initia- recent publication of the Using the TI publication as an tive of British Columbia published Therapeutics Initiative (TI) example, we will show how this ap- “Do statins have a role in primary of British Columbia, “Do proach to evaluating randomized con- prevention?” suggesting that statin A statins have a role in prima- trolled trials (RCTs) can lead to in- medications may be harmful in ry prevention?” suggested that statin valid conclusions and can cause patients without a previous myocar- medications (properly called HMG- needless concern, if not alarm, on the dial infarction. This conclusion was CoA reductase inhibitors) may do part of physicians and patients.4-7 Sev- drawn by evaluating serious adverse more harm than good in patients with- eral aspects of this approach to evalu- events in two randomized, placebo- out a previous .1 ating RCTs deserve mention. First, controlled studies—AFCAPS/TexCAPS This conclusion was drawn by evalu- there are inherent problems when and PROSPER. The Therapeutics Ini- ating serious adverse events (SAEs) meta-analyses selectively focus on a tiative authors believe that total seri- in two randomized, placebo-controlled small number of trials and secondary ous adverse events, an endpoint that studies—AFCAPS/TexCAPS2 and rather than primary endpoints. Sec- combines safety and efficacy data, PROSPER.3 ond, proposing that total SAE rates (a is the best way to evaluate clinical While the authors agree that statins combination of safety and efficacy trials. We demonstrate in a variety of demonstrated a statistically signifi- endpoints) is the most appropriate ways that this approach to evaluat- cant reduction in the studies’ primary method to evaluate randomized clini- ing randomized controlled trials can endpoints and selected secondary end- cal trials ignores the intent of a clini- lead to invalid conclusions and can points, they argue that these efficacy cal trial and could lead to an under- cause needless concern, if not endpoints are less relevant than the assessment of adverse events. We will alarm, on the part of physicians and “total SAE” rate, or safety endpoint. demonstrate that when this type of patients. The authors believe that a similar total analysis is used, any trial that demon- SAE rate in the placebo and treatment strates superior efficacy against a arms of the studies demonstrates that “statins have not been shown to pro- Dr Miller is an endocrinologist in Victoria, vide an overall health benefit in pri- BC, and head of Endocrinology for the Van- mary prevention trials.”1 Furthermore, couver Island Health Authority, South because the total SAE rates are simi- Islands. Dr Humphries is an epidemiolo- lar in both arms, despite reductions in gist/statistician and assistant professor, the secondary efficacy endpoints, Department of Medicine, Division of Cardi- there is “the possibility that unrecog- ology and Centre for Health Evaluation and nized serious adverse events are Outcome Sciences at St. Paul’s Hospital in increased by statin therapy.”1 Vancouver, BC.

VOL. 47 NO. 5, JUNE 2005 BC MEDICAL JOURNAL 241 BCMJ /#47Vol5.final 5/19/05 11:55 AM Page 242

A new way to evaluate randomized controlled trials? New approach does more harm than good

background of equivalent safety will How are adverse events defined? clinically significant results from lab- be open to the charge of being inef- The definitions for SAEs are quite uni- oratory tests or other medical proce- fective and having unrecognized seri- form. The International Conference on dures, and clinically significant find- ous adverse events. Harmonisation (ICH) defines an SAE ings uncovered during a physical as, “any untoward medical occurrence examination. Discussion that results in death, is life-threaten- Adverse events should always be Primary versus secondary ing, requires inpatient hospitalization, reported when a trial is presented in endpoints or results in persistent or significant a peer-reviewed journal. The ICH PROSPER was designed and powered disability/incapacity.”8 The United GCP Statistical Guideline11 devotes to evaluate the efficacy of pravastatin States Federal Drug Administration an entire section to the appropriate for the prevention of coronary heart (USFDA) defines an SAE as “any analysis of adverse events, separate (CHD) death, non-fatal MI, experience that is fatal or life-threat- from a section on the analysis of pri- and fatal or nonfatal stroke.3 In ening, is permanently disabling, re- mary and secondary endpoints. Sim- AFCAPS/TexCAPS lovastatin was quires hospitalization, or is a congen- ilarly, the Consolidated Standards of used to reduce fatal or nonfatal MI, ital anomaly, cancer, or overdose.”9 Reporting Trials (CONSORT)12 dis- sudden cardiac death, or unstable an- All SAEs must be reported to the cusses the presentation and report- gina.2 The TI evaluation of these two appropriate regulatory agency irre- ing of efficacy endpoints as distinct trials, on the other hand, focuses on spective of whether the SAE—like from the presentation of adverse total mortality, for which neither death or acute myocardial infarc- events. Neither of these internation- study was powered, and MI or stroke, tion—is also an efficacy endpoint for ally recognized groups recommends which were secondary endpoints. The a particular trial. combined reporting of efficacy and absolute risk reduction in PROSPER For the purposes of reporting trial safety events. for the primary endpoint was 2.1% results in a journal, the reader is inter- Why report efficacy and safety (P=.014), while in AFCAPS/Tex- ested in all adverse events, not just endpoints separately? One of the car- CAPS it was 4.1% (P<.001). The sec- those defined as serious. The reader is dinal rules of statistical inference in ondary endpoint chosen in the TI interested in determining whether the clinical trials is the assumption of meta-analysis, MI and stroke, had an efficacy of the intervention was independence of the observations.13 absolute risk reduction of 1.8%. This achieved with acceptable safety by This is relevant in the assessment and is still statistically significant but un- comparing the adverse events in the reporting of both efficacy and safety derestimates the full benefit demon- treatment arm with those in the place- events. Multiple events for an individ- strated in these trials. bo arm. Unfortunately, the definition ual are unlikely to be independent, and of adverse events varies among phar- in a the patient is the unit Reporting of serious maceutical companies, research orga- of observation. Data are presented as adverse events nizations, and journals. The ICH the proportion of patients experienc- The main premise of the alternative Good Clinical Practice (GCP) guide- ing a given endpoint. For a composite approach to RCT evaluation is that line defines an adverse event as any efficacy endpoint (like CHD death, because the SAE rates in the statin and unintended and unfavorable sign, MI, or stroke), a patient with both an placebo arms in these two trials were symptom, or disease temporally asso- MI and a CHD death contributes only the same, despite a reduction in MI ciated with the use of a medical prod- one count to the efficacy endpoint. In and stroke in the statin arm, the statin uct, whether or not it is considered addition, if the patient experienced arm must have incurred “unrecog- related to the product.8 Northington two adverse events (like an elevation nized SAEs.”1 This unconventional defines an adverse event as any unfa- in liver enzymes and hospitalization approach of combining efficacy (the vorable change in the structure (signs), for appendicitis), the patient also con- clinical endpoint the trial is designed function (symptoms), or chemistry tributes one count to the SAE total. to assess) with safety outcomes and (laboratory data) of the body tempo- This is different from regulatory then subtracting the efficacy out- rally associated with participation in requirements where both events comes is problematic both with re- the clinical trial, irrespective of the would be counted, because regulatory spect to how SAEs are reported and believed relationship to the study bodies are interested in the actual defined and with respect to method- drug.10 This definition would thus number and nature of adverse events ological and statistical validity. include intercurrent illness or injury, observed.

242 BC MEDICAL JOURNAL VOL. 47 NO. 5, JUNE 2005 BCMJ /#47Vol5.final 5/19/05 11:55 AM Page 243

A new way to evaluate randomized controlled trials? New approach does more harm than good

If one adopts the “total SAE” ap- Table 1. Calculation of “other SAE.” proach, the patient who experienced each of these four outcomes would Treatment Placebo Absolute risk count only once. Thus, the total SAE reduction approach can underestimate the true Trial A Primary endpoint 8.0% 9.8% 1.8% magnitude of a drug’s adverse effects. SAE 20% 20% 0% Also in the total SAE analysis, a tran- sient elevation in liver enzymes would Other SAE 12% 10.2% –1.8%* be equivalent to a disabling stroke; Trial B Primary endpoint 5.0% 10.2% 5.2% most physicians and patients would SAE 20% 20% 0% reject this inference. Other SAE 15% 9.8% –5.2%* By assuming that the SAE rate includes all efficacy endpoints, and * (–) suggests harm that subtracting the efficacy endpoints from the total SAE rate can derive Table 2. Adding SAE to the primary outcome. “other SAE,” any drug that demon- strates statistical efficacy with safety Trial Outcome Treatment Placebo (95% CI) equivalent to placebo will have a high- 1 MI + stroke 8.0% 9.8% 0.82 (0.73-0.92) er other-SAE rate. In fact the greater 2 Total SAE 18.0% 19.8% 0.91 (0.83-0.99) the efficacy, the higher the other-SAE rate will be and the more likely such a 3 Total SAE 28.0% 29.8% 0.94 (0.88-1.02) trial will be found to demonstrate 4 Total SAE 38.0% 39.8% 0.95 (0.89-1.03) “unrecognized serious adverse events.” CI=confidence interval This is illustrated inTable 1 , where a study has a conventionally reported 20% SAE rate in each arm of the study. InTable 2 , Trial 1 duplicates the MI cally significant difference. A useful Of the two studies cited in the TI and stroke outcome of the AFCAPS/ treatment might be rejected. report, Professor James Shepherd, the TexCAPS and PROSPER meta-analysis. Why not just look at mortality? principal investigator of the PROS- In Trial 2, a very modest 10% rate of Many studies are not powered to PER study, has confirmed that the other SAEs is added to the efficacy investigate differences in disease-spe- primary outcomes in his study were outcome in both groups, producing a cific mortality: in the case of statins, not included in the SAE rate quoted in total-SAE rate. Note that the relative cardiovascular mortality. The studies the publication (personal written com- risk reduction has been reduced from are usually powered to find a differ- munication with DBM, 2003). For 18% to 9%, even though the absolute ence in combined endpoints of mor- AFCAPS/TexCAPS, the number of risk reduction has remained at 1.8%. bidity such as nonfatal MI and/or serious adverse events is reported, Also note that the 95% confidence stroke. Nonfatal events are more com- rather than proportions, making assess- interval is now approaching unity and mon than cardiovascular death and are 2 ment on a per patient basis impossible. that has almost valid, clinical endpoints worth pre- Even if both PROSPER and been lost. In Trial 3 and Trial 4, the venting. Finding a difference in cardio- AFCAPS/TexCAPS had reported an other SAE have been set at 20% and vascular death would simply require a SAE rate that comprised both efficacy 30% respectively. In these two analy- larger and/or longer trial. In these and safety endpoints, the statement ses, a non-statistically significant re- studies the lack of a statistically sig- that “unrecognized serious adverse duction in total SAE is seen. In the nificant reduction in mortality could events are increased by statin therapy” AFCAPS/TexCAPS and PROSPER be an example of a type II or β error— cannot be supported. We will demon- meta-analysis, the other SAE rate was failing to find a difference where one strate how adding a relatively large even higher: 34% to 36%. Therefore, truly exists. Examining all-cause number of SAEs, with high variance, where the other SAE rate is high and mortality has the same problem as to a much lower primary event rate, the absolute risk of the primary out- examining total SAE. If cause-specific with lower variance, allows a statisti- come is low, the total SAEs are almost mortality is low and other (non-drug- cally significant difference to be lost. guaranteed to produce a non-statisti- related) mortality is high, and equal in

VOL. 47 NO. 5, JUNE 2005 BC MEDICAL JOURNAL 243 BCMJ /#47Vol5.final 5/19/05 11:55 AM Page 244

A new way to evaluate randomized controlled trials? New approach does more harm than good

the treatment and placebo arms of a Statin treatment is useful in many 7. Munro M. Cholesterol pill’s side effects study, small differences in cause-spe- settings, both in primary and sec- worry UBC drug specialists. Victoria cific mortality will be diluted and no ondary prevention; though where the Times Colonist. 16 September 2003;A4. statistically significant difference in risk of an event is low, as in the pri- 8. ICH Steering Committee. International total mortality will be found. This has mary prevention of cardiovascular Conference on Harmonisation Tripartite occurred in many statin studies in sec- disease, even an efficacious treatment Guideline for Good Clinical Practice. ondary prevention—statistically sig- will have a small absolute risk reduc- 1997. Minister of Public Works and Gov- nificant reductions in cardiovascular tion and a high number needed to ernment Services Canada. Ottawa, ON. morbidity and/or mortality have been treat.16 The safety of statins has been Cat #H42-2/67-11-1997IN. seen without reductions in all-cause established in dozens of major clinical 9. Chow S-C, Liu J. Safety Assessment. In: mortality. trials, which have included tens of Design and Analysis of Clinical Trials Con- thousands of patients studied for up to cepts and Methodologies. New York: Conclusion 8 years.17 The unconventional evalua- John Wiley & Sons Inc.; 1998. Authors affiliated with the Therapeu- tion of statin safety and efficacy pro- 10. Northington B. A review of issues in the tics Initiative have proposed using the posed by the TI is methodologically collection and reporting of adverse total SAE rate as a better way to eval- and statistically unsound and will ulti- events. Biopharmaceut Rep 1996;4:1-5. uate clinical trials.14 Through the Ther- mately do more harm than good. 11. ICH Steering Committee. International apeutics Letter,1,15 this approach has Conference on Harmonization Tripartite been presented to physicians in British Competing interests Guideline for Good Clinical Practice Columbia and through newspapers4-7 None declared. Process. Statistical Principles for Clinical to patients throughout Canada. The Trials. 2003. Minister of Public Works and aim of providing more information on References Government Services Canada. Ottawa, adverse events is clearly laudable, 1. Therapeutics Initiative. Do statins have a ON. Cat #H49-171/2003E. especially in light of recent findings role in primary prevention? Therapeutics 12. Moher D, Schulz KF, Altman DG. The related to COX-2 inhibitors. But the Letter 2003;48. www.ti.ubc.ca/pages/ CONSORT statement: Revised recom- proposed approach will likely under- letter48.htm (accessed 27 January 2005). mendations for improving the quality of estimate the adverse event rates and 2. Downs JR, Clearfield M, Weis S, et al. reports of parallel-group randomised tri- thus have the opposite effect. Only by Primary prevention of acute coronary als. Lancet 2001;357:1191-1194. rigorously adopting the reporting for- events with lovastatin in men and wo- 13. Bolton S. Independence and statistical mat of the CONSORT group, in which men with average cholesterol levels: inference in clinical trial designs: A tutor- efficacy and safety endpoints are re- Results of AFCAPS/TexCAPS. Air Force/ ial review. J Clin Pharmacol 1998;38:408- ported separately, can one critically Texas Coronary Atherosclerosis Preven- 412. evaluate the results of an RCT. tion Study. JAMA 1998;279:1615-1622. 14. Wright JM, Puil L, Bassett CL. Analysis There are many ways to evaluate 3. Shepherd J, Blauw GJ, Murphy MB, et of serious adverse events. Lipid-lower- and interpret clinical trials of pharma- al. Pravastatin in elderly individuals at risk ing therapy revisited. Can Fam Physician ceutical agents. One can look at sta- of vascular disease (PROSPER): A ran- 2002;48:486-499, 92-95. tistical and clinical significance. One domised controlled trial. Lancet 2002; 15. Therapeutics Initiative. Serious adverse can look at primary outcomes, second- 360:1623-1630. event analysis: Lipid-lowering therapy ary outcomes, and adverse events. 4. Taylor P. Unclogging the heart debate: Do revisited. Therapeutics Letter 2001;42. One can look at relative risk reduc- cholesterol drugs help or harm? Globe www.ti.ubc.ca/pages/letter42.htm tions, absolute risk reductions, and and Mail. 24 January 2004;F4. (accessed 27 January 2005). numbers needed to treat.16 One should 5. Munro M. Cholesterol pills carry risk, 16. Miller D. Secondary prevention for look at adverse events but one shouldn’t UBC group says: “Fine print” raised ischemic heart disease. Relative num- look at the total SAE rate, as many doubts: Finding applies only to preven- bers needed to treat with different ther- effective treatments will be discarded tive use. National Post. 16 September apies. Arch Intern Med 1997;157:2045- as useless, or even harmful, and per- 2003;A1. 2052. versely, treatments with high adverse 6. Munro M. Friend or foe?: First we were 17. Gotto AM Jr, Pownall H. Chapter 9. In: event rates will likely be underesti- told statins could save lives. Now some Manual of Lipid Disorders. Lippincott, mated using this approach. say they could be putting lives at risk. Williams & Wilkins; 2003:169-198. National Post. 16 September 2003;A15.

244 BC MEDICAL JOURNAL VOL. 47 NO. 5, JUNE 2005