Users' Guides to the Medical Literature

THE MEDICAL LITERATURE

Users’ Guides to the Medical Literature XIX. Applying Clinical Trial Results B. Guidelines for Determining Whether a Drug Is Exerting (More Than) a Class Effect

Finlay A. McAlister, MD, FRCPC your patients with elevated choles- Herein, we define a drug class as those Andreas Laupacis, MD, MSc, FRCPC terol levels, but are uncertain which of drugs that share a similar structure and the statins on the market is best. You mechanism of action. Most classes of George A. Wells, MSc, PhD ask a general internist, cardiologist, and drugs include multiple compounds, and David L. Sackett, FRSC, MD, FRCP endocrinologist for their opinions, and because of their similar mechanisms of for the Evidence-Based Medicine each suggests a different statin, citing action, they are generally thought to con- Working Group different reasons. You contact pharma- fer similar pharmacologic effects and ceutical representatives to provide you clinical outcomes (class effects). This 3 OST CLASSES OF DRUGS IN- with evidence that their statins are bet- assumption is a key medical heuristic clude multiple com- ter than those of their competitors. Al- and underlies clinical practice guide- pounds. The opinions of though you use the JAMA series on Us- lines in which evidence from studies clinicians, manufacturers’ Guides to the Medical Literature to involving 1 or more drugs within a class Mers, and purchasers may differ as to assess the validity of published stud- is extrapolated to other drugs of the same whether a particular drug is more effi- ies, faced with a variety of competing class. For example, it is recommended cacious, safer, or more cost-effective claims, you realize that you need a that ␤-blockers be prescribed for survi- than others in its class.1 In this article, framework for grading the strength of vors of myocardial infarction or angio- we review the types of evidence com- these studies. tensin-converting enzyme inhibitors to monly cited to support the prescrib- patients with heart failure. In this cir- ing of a particular drug rather than an- The Policymaker cumstance, clinicians are likely to be other of the same class and provide a Your colleague, a purchaser for a large interested in the drug within each class hierarchy for grading studies that com- health maintenance organization (HMO), is faced with a similar di- Author Affiliations: National Health Service Research pare a drug with another of the same and Development Centre for Evidence-Based Medi- class, expanding on our discussion in lemma when she is asked to consider re- cine, John Radcliffe Hospital, Oxford, England (Drs part A of this Users’ Guide.2 placing the statin on her HMO’s formu- McAlister and Sackett); Division of General Internal Medicine, Ottawa Hospital (Dr Laupacis), and Clinical lary with a newer one. She wonders Epidemiology Unit, Loeb Health Research Institute (Drs CLINICAL SCENARIOS whether there is enough evidence to sup- Laupacis and Wells), Ottawa, Ontario. The Clinician port the contention that the new statin Dr McAlister is currently at the Division of General Internal Medicine, University of Alberta Hospital, As a clinician, you care for many pa- is as good as, or better than, the one cur- Edmonton. rently on formulary. While the new The original list of members (with affiliations) ap- tients with elevated serum cholesterol pears in the first article of this series (JAMA. 1993; levels. A speaker at a recent continu- statin is cheaper, it has been evaluated 270:2093-2095). A list of new members appears in only in short-term trials, with choles- the 10th article of the series (JAMA. 1996;275:1435- ing medical education event reviewed 1439). The following members contributed to this ar- the benefits of cholesterol-lowering terol lowering as the solitary end point. ticle: Gordon Guyatt, MD, MSc, Virginia Moyer, MD, therapy, particularly with 3-hydroxy- MPH, and David Naylor, MD, DPhil. DRUG CLASSES Corresponding Author: Finlay A. McAlister, MD, 3-methylglutaryl coenzyme A reduc- FRCPC, Division of General Internal Medicine, 2E3.24 tase inhibitors (statins), in the pri- Although there is no uniformly accepted WMC, University of Alberta, Canada Hospital, 8440 112 St, Edmonton, Alberta, Canada T5G 2R7. mary and secondary prevention of definition of a drug class—and some Reprints: Gordon Guyatt, MD, MSc, McMaster Uni- ischemic heart disease but did not rec- argue that it cannot be defined at all— versity Health Sciences Centre, 1200 Main St W, Room 2C12, Hamilton, Ontario, Canada L8N 3Z5. ommend a particular statin. You de- drugs are generally said to belong to the Users’ Guides to the Medical Literature Section Editor: cide to consider statin therapy for all same class for 1 of 3 reasons (TABLE 1). Drummond Rennie, MD, Deputy Editor (West), JAMA.

this class effect, because the overall OR effects other than the mechanism of Table 1. Definitions of Drug Classes* in 32 trials involving 7105 patients was action that defined them as being from Definition Example 0.77 (95% CI, 0.67-0.88), the CIs for each the same class. It therefore may be inac- Drugs with similar Dihydropyridine chemical CCBs have of the angiotensin-converting enzyme curate to extrapolate the clinical out- structure dihydropyridine rings inhibitors overlapped, and there was no comes shown in randomized trials of 1 Drugs with similar CCBs block the mechanism of voltage-dependent statistical heterogeneity between trials of drug in a class to another member of action calcium channels on different agents. that class that has not been subjected the surfaces of cell membranes to similar outcome-centered trials. For Drugs with similar Antihypertensives Risks of Assuming a Class Effect example, some authors have argued pharmacologic (eg, CCBs, ACE Although drugs of the same class typi- that, although all of the statins act on effects inhibitors, ␤-blockers, thiazides, ␣-blockers) cally exhibit similar pharmacological ef- the 3-hydroxy-3-methylglutaryl coen- lower blood pressure fects and clinical outcomes, this may not zyme A reductase enzyme, they may *CCB indicates calcium channel blocker; ACE, angiotensin- always be the case. Note the current con- have different nonlipid effects on the converting enzyme. troversy regarding the safety of sotalol atherothrombotic process that may hydrochloride in myocardial infarction influence their clinical efficacy.13 with the most attractive efficacy-to- survivors with congestive heart failure To reduce the risk of faulty extrapo- safety ratio; purchasers, in the most after the publication of the SWORD lation and to maximize the optimal se- cost-effective drug from a class; and Trial,8 which suggested an increase in lection of treatments within a class of manufacturers, in the most frequent mortality with sotalol, compared with drugs, it may be useful to develop and prescribing of their drugs. the decrease in mortality with other apply a hierarchy of evidence when mak- The absolute treatment effects seen ␤-blockers. It is useful to recall a previ- ing decisions about the comparative with a drug (defined by the absolute risk ous controversy regarding the efficacy clinical efficacy and safety of drugs reduction or number needed to treat) are of ␤-blockers with intrinsic sympathetic within a class. As pointed out in part A influenced by the baseline risk or con- activity (ISA) in patients with myocar- of this Users’ Guide, no matter how trol event rate of those patients in whom dialinfarction.Althoughameta-analysis9 strong the pathophysiologic rationale or it is used. Thus, the absolute risk reduc- suggested that the treatment effect was indirect evidence, the efficacy and safety tion varies considerably among differ- greater with non-ISA ␤-blockers, sub- of a new drug must be established in ent groups of patients. On the other hand, sequent trials10 failed to confirm this, and clinical outcome studies that test more the relative treatment effect of a drug (de- the evidence11 suggests there is little dif- than just biological plausibility. fined by the relative risk reduction ference between ␤-blocker subgroups. [RRR]) is often (but not always4) simi- It would seem reasonable to accept a Levels of Evidence lar, irrespective of the baseline risk of trial priori that drugs within the same class Levels of evidence are increasingly used participants.5,6 If 2 drugs are tested in exert similar effects, unless there is clear by groups that make recommenda- separate placebo-controlled trials, only evidence of important differences. tions about patient care,14-16 and we have proportional effects such as the RRR However, this assumption can lead used some of them to develop guide- resulting from each drug can be com- to 2 important errors of extrapolation lines for comparing 1 drug with other pared (and then only under the assump- with major clinical consequences. First, drugs in the same class (TABLE 2). This tion of constant RRR over different con- when agents in a class of drugs (such comparison should occur as part of a sys- trol event rates). Although the point as the thiazide diuretics) all produce tematic review of all the relevant evi- estimates of effect size vary, a class effect similar pharmacological effects (blood dence on the effects of a treatment, iden- is considered to be present when drugs pressure lowering) and similar clini- tified and assessed by thorough and clear with similar mechanisms of action gen- cal effects (stroke reduction), a sec- methods such as those used in the erate RRRs (or odds ratios [ORs]) that ond class of drugs (for example, the cal- Cochrane Collaboration [Update Soft- are similar in direction and magnitude. cium channel blockers) that produce the ware, Oxford, England; 1998]. We will For example, the Collaborative Group same pharmacological effects might be describe each level in turn, using the on ACE Inhibitor Trials7 suggested that assumed to produce the same clinical choice of statin drugs as an example to there is a class effect for angiotensin- benefits. In the absence of randomized illustrate their use (TABLE 3). converting enzyme inhibitors in patients trials verifying that final assumption, Level 1. Level 1 includes random- with symptomatic heart failure, despite this type of extrapolation may be erro- ized clinical trials providing head-to- the fact that the OR point estimates for neous. For example, consider the issue head comparisons of the drug of inter- effects on total mortality ranged from 0.14 raised in part A of this Guide—some est with other drugs of the same class (95% confidence interval [CI], 0-7.6) for calcium channel blockers have unfa- for their effects on clinically impor- perindopril (1 trial, 125 patients) to 0.78 vorable effects on total mortality.12 Sec- tant outcomes. This would generate the (95% CI, 0.67-0.91) for enalapril (7 tri- ond, even within the same class, indi- strongest evidence for the decision als, 3381 patients). We are confident in vidual drugs may have physiologic maker; however, there are potential

1372 JAMA, October 13, 1999—Vol 282, No. 14 ©1999 American Medical Association. All rights reserved. USERS’ GUIDES TO THE MEDICAL LITERATURE threats to validity (Table 2) and sev- comes could include symptom scores mortality that we agreed to accept the eral methodologic issues unique to and other quality-of-life measures. surrogate end point of LDL cholesterol these trials. First, at least 1 of the drugs Although there are examples of level lowering as a proxy for clinically impor- should have been shown to have a clini- 1 evidence in other branches of medi- tant outcomes. Thus, to accept head-to- cally important impact vs placebo in cine,18,19 they are rare in the cardiovas- head comparisons for surrogate out- previous trials carried out in a popula- cularliterature.Ourliteraturesearchfailed comes as level 2 evidence, at least 1 of tion similar to that of the current trial. to find any level 1 evidence for statins. the comparators must have demon- Second, the choice of appropriate dos- Level 2. Level 2 includes random- strated efficacy in long-term trials with age for each drug is a complicated is- ized clinical trials providing head-to- clinically important outcomes. sue, as this will affect the outcomes and head comparisons of the drug of inter- Whereas a randomized trial26 com- safety profiles for both drugs. Finally, est with other drugs of the same class paring 4 statins for their effects on LDL one must carefully consider the trial size for their effects on validated surrogate cholesterol, high-density lipoprotein and methods before concluding equiva- outcomes or comparisons across 2 or cholesterol, and triglycerides during an lence of 2 drugs—equivalence trials re- more placebo-controlled trials for effects 8-week period would be an example of quire much larger sample sizes than on clinically important outcomes or vali- level 2 evidence, it also is important to standard trials,17 and any laxity in trial dated surrogate outcomes. Part A of this incorporate considerations of the size conduct or patient compliance will tend Users’ Guide discussed criteria for decid- and duration of trials in the decision- to mask any real differences between ing whether to accept results of trials making process. drugs. based on surrogate outcomes. Ecologic On the other hand, a number of level The choice of clinically important out- studies, cohort studies, and random- 2 comparisons can be made between comes for level 1 studies depends on the ized clinical trials with prestatin lipid- various statins—for example, one can target intervention. In the case of thera- lowering agents were supportive of the compare the treatment effects seen with pies designed to prevent or arrest ath- lipid-lowering hypothesis20 (that low- simvastatin vs pravastatin in secondary erosclerosis (such as statins), this implies ering low-density lipoprotein [LDL] cho- prevention trials (such as the 4S21 and long-term efficacy data on events such lesterol levels lowers the risk of athero- LIPID25 studies [Table 3]). Although con- as myocardial infarction, stroke, and all- sclerotic heart disease); however, it was sistency of effects in such comparisons cause mortality. On the other hand, for not until the publication of the large- would be strong evidence for the pres- interventionsdesignedtotreatsymptom- scale statin trials21-25 (Table 3) consis- ence of a class effect, these comparisons atic diseases (such as gastroesophageal tently linking reductions in LDL cho- are less useful in determining whether reflux disease), clinically important out- lesterol to reductions in morbidity and a drug is more efficacious than another,

Table 2. Levels of Evidence for Comparing the Efficacy of Drugs Within the Same Class* Level Comparison Study Patients Outcomes Threats to Validity 1 Within a head-to-head RCT Identical (by definition) Clinically important Failure to conceal randomization scheme Failure to achieve complete follow-up Failure to achieve double-blinding Soundness of outcome assessment 2 Within a head-to-head RCT Identical (by definition) Validated surrogate Those of level 1 plus validity of surrogate outcome for clinically important outcomes 2 Across RCTs of different Similar or different Clinically important Those of level 1 plus differences between trials in: drugs vs placebo (in disease and or validated Methodologic quality (adequacy of blinding, risk factor status) surrogate allocation concealment, etc) End point definitions Compliance rates Baseline risk of outcomes 3 Across subgroup analyses Similar or different Clinically important Those of level 1 (plus or minus those of level 2) plus: from RCTs of different or surrogate Multiple comparisons, posthoc data dredging drugs vs placebo Underpowered subgroups Misclassification into subgroups 3 Across RCTs of different Similar or different Unvalidated Surrogate outcomes may not capture all of the drugs vs placebo surrogate effects (beneficial or hazardous) of a therapeutic agent 4 Between nonrandomized Similar or different Clinically important Confounding by indication, compliance, studies (observational and/or calendar time studies and administrative Unknown/unmeasured confounders database research) Measurement error For outcomes research: limited databases, coding systems not suitable for research *Clinically important outcomes refer to long-term efficacy data, and the particular end points depend on the condition being treated. For statins used to prevent or treat athero- sclerotic disease, clinically important outcomes would include all-cause mortality, myocardial infarction, and stroke. Surrogate outcomes are considered validated only when the relationship between the surrogate outcome and clinically important outcomes has been established in long-term randomized clinical trials (RCTs).

©1999 American Medical Association. All rights reserved. JAMA, October 13, 1999—Vol 282, No. 14 1373 USERS’ GUIDES TO THE MEDICAL LITERATURE because the advantages of randomiza- prevention trial (such as simvastatin in the OR (or relative risk) from studies of tion are lost, and the comparison is es- 4S21), such a comparison would only be drug A vs placebo (p) and the OR from sentially that between 2 or more co- valid if the drug efficacy is known to be studies comparing drug B vs placebo: horts. In addition to the potential biases independent of baseline risk, an assump- ORAvsB=ORAvsp/ORBvsp. However, this outlined in Table 2, there is also the pos- tion that appears valid to make in some assumes that none of the potential bi- sibility of confounding a subject’s risk or situations (such as antiplatelet6 or anti- ases outlined in Table 2 are operative and responsiveness with exposure to a par- hypertensive5 therapy) but has been that an intervention’s treatment effect is ticular treatment in those situations in questioned for the statins.27-32 consistent across different patient sub- which subjects from different trials have It is theoretically possible to com- groups. Furthermore, these indirect es- different risk statuses. For example, if one pare the efficacy of 2 drugs tested in timates may provide substantially dif- were to compare the statin used in a pri- separate placebo-controlled trials. As ferent effect-size estimates than direct mary prevention trial (such as lovas- outlined by Bucher et al,33 an indirect es- comparisons of drug A against drug B. tatin in AFCAPS/TexCAPS24) with an- timate of the association between drugs For example, a systematic overview of other statin tested in a secondary A and B can be obtained by comparing strategies to prevent Pneumocystis cari-

Table 3. Features of Randomized, Placebo-Controlled Statin Trials Designed to Detect Differences in Clinically Important End Points* Trial

4S21 WOSCOPS22 CARE23 AFCAPS/TexCAPS24 LIPID25 Study design Secondary prevention, Primary prevention, Secondary prevention, Primary prevention, Secondary prevention, multicenter single center multicenter multicenter multicenter Treatment (dose once daily) Simvastatin (20 mg) Pravastatin (40 mg) Pravastatin (40 mg) Lovastatin (40 mg) Pravastatin (40 mg) Patient inclusion criteria† Age 35-70 y, prior Age 45-64 y, no prior Age 21-75 y, prior Age 45-73 y (males) or Age 31-75 y, prior AMI angina or AMI, AMI, fasting LDL AMI, fasting LDL 55-73 y (females), or unstable angina, fasting total cholesterol cholesterol no prior AMI, fasting total cholesterol 4.0-6.0 mmol/L 3.0-4.5 mmol/L fasting LDL cholesterol 5.5-8.0 mmol/L cholesterol 4.0-7.0 mmol/L 3.4-4.9 mmol/L Cointerventions, % Aspirin 37 None 83 None 82 ␤-Blockers 57 None 40 None 47 Duration of follow-up, y 5.4 (Median) 4.9 (Mean) 5.0 (Median) 5.2 (Mean) 6.1 (Mean) Patients No. 4444 6595 4159 6605 9014 Mean age, y 58.6 55.2 59 58 62 Males, % 81 100 86 85 83 Smokers, % 26 44 21 12 10 Diabetes mellitus, % 5 1 15 2 9 Baseline cholesterol, mean mmol/L† Total 6.8 7.0 5.4 5.7 5.6 LDL 4.9 5.0 3.6 3.9 3.9 Control event rates, % Death 11.5 4.1 9.4 0.44 14.1 AMI 22.6 7.9 10 0.56 10.3 Treatment effects Change in lipids −25 (Total cholesterol) −20 (Total cholesterol) −20 (Total cholesterol) −18 (Total cholesterol) −18 (Total cholesterol) (active treatment −35 (LDL cholesterol) −26 (LDL cholesterol) −28 (LDL cholesterol) −25 (LDL cholesterol) −25 (LDL cholesterol) vs placebo), % +8 (HDL cholesterol) +5 (HDL cholesterol) +5 (HDL cholesterol) +6 (HDL cholesterol) +5 (HDL cholesterol) −10 (Triglycerides) −12 (Triglycerides) −14 (Triglycerides) −15 (Triglycerides) −11 (Triglycerides) Relative risk reductions, % (95% CI) Death 30 (15 to 42) 22 (0 to 40) 9 (−12 to 26) −4 (Not given) 22 (13 to 31) AMI 27 (20 to 34) 31 (17 to 43) 25 (8 to 39) 40 (17 to 57) 29 (18 to 38) Number needed to treat‡ To prevent 1 death 27 (5 y) 111 (5 y) 125 (5 y) 5000 to harm§ 32 (6 y) To prevent 1 AMI 10 (5 y) 42 (5 y) 40 (5 y) 435 (5 y) 34 (6 y) *4S indicates Scandinavian Simvastatin Survival Study; WOSCOPS, West of Scotland Coronary Prevention Study; CARE, Cholesterol and Recurrent Events Trial; AFCAPS/ TexCAPS, Air Force/Texas Coronary Atherosclerosis Prevention Study; LIPID, Long-term Intervention with Pravastatin in Ischaemic Disease Study; AMI, acute myocardial infarction; LDL, low-density lipoprotein; HDL, high-density lipoprotein; and CI, confidence interval. †To convert cholesterol levels to milligrams per deciliter, divide by 0.02586. ‡Point estimates only. Years in parentheses indicate number of years needed to treat that number of patients to prevent 1 event. §Since all-cause mortality was nonsignificantly increased in the active treatment arm, results are presented as number needed to treat to cause 1 death.

1374 JAMA, October 13, 1999—Vol 282, No. 14 ©1999 American Medical Association. All rights reserved. USERS’ GUIDES TO THE MEDICAL LITERATURE nii pneumonia in human immunodefi- monlyusedbypractitioners.Nonrandom- row confidence limits around the differ- ciency virus–positive patients docu- ized evidence can include cohort or case- ence between drugs), we recognize that mented that the indirect comparison of control studies, modeling studies (using it is rarely available and is unlikely to ever trimethoprim-sulfamethoxazole vs a risk-prediction equations such as those be available for many classes of drugs be- combination of dapsone and pyrimeth- derived from the Framingham data36), cause of difficulties in funding and con- amine suggested a much larger effect size and/or outcomes research using admin- ducting trials so large that they are un- from trimethoprim-sulfamethoxazole istrative databases. Although these types likely to appeal to researchers, manufac- (OR, 0.37; 95% CI, 0.21-0.65) than was of analyses can provide useful insights turers, or funders. In this situation, the seen in the direct comparisons (overall (particularlywithrespecttodose-response amount of level 2 evidence becomes im- OR, 0.64 in the 9 trials of trimethoprim- relationships),37 they are best viewed as portant. For instance, one would feel sulfamethoxazole vs dapsone and pyri- exercisesinhypothesis-generation.Inpar- more comfortable in concluding that a methamine; 95% CI, 0.45-0.90).33 Thus, ticular, outcomes research studies, origi- drug produced a class effect if there were the strength of inference from indirect nally developed to determine whether the a number of placebo-controlled trials comparisons is limited. efficacy of interventions proven in ran- demonstrating that various drugs from Level 3. Level 3 includes compari- domized trials have their anticipated im- the same class had similar treatment ef- sons across subgroups from different pla- pactsatapopulationlevel,havesometimes fects.However,ourgoalisnottosetalevel cebo-controlled trials or comparisons been used to pursue the primary deter- that must be achieved before a drug can across placebo-controlled trials in which minationofefficacy—apurposeforwhich be claimed to be superior to others in its outcomes are restricted to unvalidated they were not intended. When used to es- class or before a class effect can be estab- surrogate markers. In addition to the tablish efficacy, they present, in addition lished.Thosearedecisionsthatindividual biases that affect higher-level studies, to other limitations (Table 2), unique clinicians or policymakers must make, comparisons based on subgroup analy- problemsininterpretationthatrestrictthe taking into account their local circum- sis are potentially flawed (Table 2). Both validity of inferences drawn from them stances and individual comfort levels. simple statistics and experience have about the relative efficacy of medications Safety. In the past decade, there have taught us that many initial subgroup con- from the same class.38 been numerous examples of drugs clusions (especially those that result from An example of level 4 evidence is a re- within the same class that have been data-dredging) are subsequently dis- cent reanalysis of the WOSCOPS data- shown to have different safety pro- proven.34,35 An example of such a com- base, designed to infer whether pravas- files. Although not our primary focus, parison would be looking at the efficacy tatin’s efficacy exceeds that expected of considerations of drug safety are part of simvastatin in the 4S subgroup with other statins.29 Using the constellation of of any treatment or purchasing deci- the lowest lipid levels (241 patients with risk factors and mean on-treatment cho- sion, so we offer a set of levels of evi- total cholesterol levels of 5.5-6.24 mmol/L lesterol levels seen in the trial, the ob- dence for determining drug safety in [213-241 mg/dL])28 vs the efficacy of servedcoronaryeventratesinpravastatin- TABLE 4. Phase 1 drug studies in pravastatin in the CARE subgroup with treatedpatientswerecomparedwiththose humans are designed to determine the comparable lipid profiles (2087 patients predictedfromtheFraminghamcoronary maximally tolerated dose, and clinical with total cholesterol levels of 5.4-6.21 risk equation to determine whether the trials are generally designed to deter- mmol/L [209-240 mg/dL]).23 treatment benefit with pravastatin ex- mine the efficacy of the drug. As such, Level 3 evidence may also include ceeded that expected from the degree of the sample sizes of neither are adequate the use of surrogate markers that, al- cholesterol lowering achieved. to detect uncommon adverse effects. though they may lie along a recognized Level 3 and 4 studies have numerous The inverse rule of 3 states that to be pathogenetic pathway from mecha- flawsasoutlinedaboveandarebestviewed 95% sure of seeing at least 1 adverse nisms of action to important clinical out- as exercises in hypothesis generation. drug reaction that occurs once in every comes, have not been validated in long- given number of patients, you need to term randomized clinical trials. To Other Considerations follow up 3 times that many patients.39 return to an example cited in part A of Amount of Efficacy Evidence. While we Given the size and duration of most this Users’ Guide, this would involve have thus far focused on the validity of clinical trials, adverse effects that occur making inferences about reductions in the evidence, the number, size, and du- in fewer than 1 in 1000 participants or fractures from the effects on bone den- ration of studies are essential factors to that take more than 6 months to appear sity of 2 different bisphosphonates in 2 be considered in the decision-making will generally remain undetected.3 How- independent randomized trials. process. Certainly, the superiority of 1 ever, randomized clinical trials are still Level 4. Level 4 includes comparisons drug within a class can only be defini- the strongest design for detecting real involving or confined to nonrandomized tively established with level 1 evidence. differences in adverse effects (such evidence.Thistypeofevidenceisonlypos- However, while level 1 evidence would as the different rates of intracranial sible for conditions in which there are a be ideal for establishing that a group of bleeding with different thrombolytic largenumberofpotentialtreatmentscom- drugsexertaclasseffect(byshowingnar- agents40,41), and meta-analyses of such

©1999 American Medical Association. All rights reserved. JAMA, October 13, 1999—Vol 282, No. 14 1375 USERS’ GUIDES TO THE MEDICAL LITERATURE trials can give unbiased estimates of mined by formal economic analyses, and ates most often in a cost-utility gray excess hazards. In the absence of clini- the Users’ Guides series offers criteria for zone between these 2 extremes.45 cal trials, premarketing safety data must evaluating methodological quality.44 Al- be considered preliminary, and large, though cost-minimization analysis is the RESOLUTION OF SCENARIOS phase 4 studies or systematic postmar- simplest and least controversial of the The Clinician keting surveillance data are necessary economic analysis techniques, it re- Given the qualitative consistency of the to confirm the safety of new drugs. quires proof that the outcomes result- RRR for acute myocardial infarction in Convenience/Compliance. While ing from both alternatives are the same. patients treated with 3 of the statins in once-a-day medications are more con- As this rarely exists, the policymaker large trials with clinically important out- venient and usually have higher com- must rely on other types of analyses comes (Table 3) and the convincing na- pliance rates, evidence about drug (cost-effectiveness, cost-benefit, or cost- ture of LDL cholesterol lowering as a sur- compliance derived from trials may utility analyses) that involve varying rogateoutcome,20,30,47-49 ourcliniciancon- translate poorly in clinical practice. For degrees of assumption and guesswork. cludes that there is a class effect of statin instance, while compliance with the vari- As pointed out by Naylor and col- drugsontheoccurrenceofischemicheart ous statins described in Table 3 ranged leagues,45 economic analyses should be disease. In the apparent absence of dif- from 90% to 94% during the course of viewed as “promising, clearly helpful, ferences in safety or compliance profile the trials, analyses of administrative da- still in need of refinement and open, like between the various statins, he decides tabases in Canada and the United States42 any new technology, to both wise use to pursue a cost-minimization strategy. revealed that only half of statin-treated and well-intentioned abuse.” While the newer statin has been evalu- patients were still taking their medica- The decision as to whether a new atedonlyforcholesterol-loweringefficacy tion 1 year after it was prescribed. drug is efficient enough to warrant its in a short-term trial (Ͻ6 months), he de- Cost. Faced with a decision as to adoption depends critically on the so- cidestoprescribeitbecauseitisthecheap- whether a new drug from a class should cial, political, and economic realities of est statin in his local setting. be offered to eligible patients within the the particular health care setting, com- population, clinicians and policymak- plicating the policymaker’s task. Thus, The Policymaker ers have different perspectives. For cli- attempts to establish universal cut- The policymaker agrees with the clini- nicians, this decision usually hinges on points (using cost or quality-adjusted cian that the statins appear to exert a class the efficacy, safety, convenience or com- life-year ratios) have been largely un- effect in terms of efficacy. However, she pliance, cost of the new drug vs the old, successful.46 Although there are occa- is concerned that the efficacy of the and the applicability of the trial evi- sions for which there is compelling evi- newer statin has not been evaluated in dence to their patients.43 However, for dence for a new drug’s adoption (the long-term trials with clinically impor- policymakers, these issues form only 1 new drug is as effective or more effec- tant outcomes or validated surrogate out- piece of the puzzle. They also must tive than others of its class and is less comes. Thus, she decides to keep the evaluate the efficiency, affordability, and costly) or rejection (the new drug is less older (and more expensive) statin on her opportunity costs of any new drugs. The effective than others of its class and is formulary until level 1 or long-term level efficiency of any intervention is deter- more costly), the policymaker oper- 2 evidence is available that proves that the newer statin is as good as or better than the currently provided statin. Table 4. Levels of Evidence for Comparing the Safety of Drugs Within the Same Class Level Type of Study Advantages Threats to Validity CONCLUSION 1 Randomized Only design that permits the Underpowered for detecting clinical trial(s) detection of adverse effects adverse effects While it would be preferable that every when the adverse effect is drug in each class (and indeed every dose similar to the event that and every formulation) be evaluated in treatment is trying to prevent randomizedclinicaltrialswithactivecom- 2 Cohort Prospective data collection, Critically depends on follow-up, defined cohort classification, and parators from the same class for its effects measurement accuracy onclinicallyimportantoutcomes,thishas 3 Case-control Cheap and fast to perform Selection and recall bias; not been accomplished for several impor- temporal relationship may not be clear tant classes of drugs. We believe that ad- 4 Phase 4 If sufficiently large, can detect No, or unmatched, control vocatesofnewerdrugswithinaclassmust rare but important adverse group; critically depends on provide evidence of equivalence (or effects follow-up, classification, and measurement accuracy superiority) to the older agents and “ran- 5 Case series Cheap and fast to perform Small sample size; selection bias; domized comparative trials...remain no control group the preferred evidentiary standard.”50 6 Case report(s) Cheap and fast to perform Small sample size; selection bias; Recognizing that this criterion standard no control group is not always attainable (in the case of

1376 JAMA, October 13, 1999—Vol 282, No. 14 ©1999 American Medical Association. All rights reserved. USERS’ GUIDES TO THE MEDICAL LITERATURE the statins, such randomized clinical tri- 13. Rosenson RS, Tangney CC. Antiatherothrom- Arterioscler Thromb Vasc Biol. 1997;17:3527-3533. botic properties of statins. JAMA. 1998;279:1643- 31. Smith GD, Song F, Sheldon TA. Cholesterol low- als would require very large sample sizes 1650. ering and mortality [published correction appears in and long follow-up to detect significant 14. Guyatt GH, Sackett DL, Sinclair JC, Hayward RC, BMJ. 1993;306:1648]. BMJ. 1993;306:1367-1373. Cook DJ, Cook RJ, for the Evidence-Based Medicine 32. Sacks FM, Gibson CM, Rosner B, Pasternak RC, differences in myocardial infarction or Working Group. Users’ guides to the medical litera- Stone PH, for the Harvard Atherosclerosis Reversibil- deathbetween2differentstatins),wesug- ture, IX: a method for grading health care recommen- ity Project Research Group. The influence of pretreat- gest that discussions about class effects dations [published correction appears in JAMA. 1995; ment low density lipoprotein cholesterol concentra- 275:1232]. JAMA. 1995;274:1800-1804. tions on the effect of hypocholesterolemic therapy on will benefit from citing the levels of evi- 15. Canadian Task Force on the Periodic Health Ex- coronary atherosclerosis in angiographic trials. Am J dence behind the arguments and recog- amination. The Periodic Health Examination, 2: 1987 Cardiol. 1995;76:78C-85C. update. CMAJ. 1988;138:618-626. 33. Bucher HC, Guyatt GH, Griffith LE, Walter SD. nizing the strengths and weaknesses in- 16. Cook DJ, Guyatt GH, Laupacis A, Sackett DL, Gold- The results of direct and indirect treatment compari- herent in each study design. berg RJ. Clinical recommendations using levels of evi- sons in meta-analysis of randomized controlled trials. dence for antithrombotic agents. Chest. 1995;108(4 J Clin Epidemiol. 1997;50:683-691. Funding/Support: Dr McAlister is supported by the suppl):227S-230S. 34. Oxman AD, Guyatt GH. A consumer’s guide to sub- Medical Research Council of Canada and the Alberta 17. Donner A. Approaches to sample size estimation group analyses. Ann Intern Med. 1992;116:78-84. Heritage Foundation for Medical Research, Dr Laupa- in the design of clinical trials. Stat Med. 1984;3:199- 35. Yusuf S, Wittes J, Probstfield J, Tyroler HA. Analy- cis is supported by the Medical Research Council of 214. sis and interpretation of treatment effects in sub- Canada, and Dr Sackett is supported by the Research 18. Naguib M, el Bakry AK, Khoshim MH, et al. Pro- groups of patients in randomized clinical trials. JAMA. and Development Programme of the National Health phylactic antiemetic therapy with ondansetron, tro- 1991;266:93-98. Service, United Kingdom. pisetron, granisetron and metoclopramide in pa- 36. Anderson KM, Wilson PW, Odell PM, Kannel WB. Acknowledgment: We gratefully acknowledge the in- tients undergoing laparoscopic cholecystectomy. Can An updated coronary risk profile: a statement for health put of various members of the University of Ottawa J Anaesth. 1996;43:226-231. professionals. Circulation. 1991;83:356-362. Clinical Epidemiology Unit and attendees of the 19. Korttila K, Clergue F, Leeser J, et al. Intravenous 37. Psaty BM, Siscovick DS, Weiss NS, et al. Hyperten- EQUINOX symposium for earlier discussions of this dolasetron and ondansetron in prevention of postop- sion and outcomes research: from clinical trials to clini- topic, J. Glennie, PharmD, for providing information erative nausea and vomiting. Acta Anaesthesiol Scand. cal epidemiology. Am J Hypertens. 1996;9:178-183. on the approaches of regulators to these issues, and 1997;41:914-922. 38. Marshall WJ. Administrative databases: fact or fic- Ian Chalmers, MB, ChB, and F. Lawson, MB, ChB, for 20. Law MR, Wald NJ, Thompson SG. By how much tion? CMAJ. 1998;158:489-490. reviewing earlier versions of this article. and how quickly does reduction in serum cholesterol 39. Sackett DL, Haynes RB, Gent M, et al. Compli- concentration lower risk of ischaemic heart disease? ance. In: Inman WH, ed. Monitoring for Drug Safety. BMJ. 1994;308:367-373. Philadelphia, Pa: Lippincott; 1980:427-438. REFERENCES 21. Scandinavian Simvastatin Survival Study Group. 40. Gruppo Italiano per lo Studio della Sopravvivenza Randomised trial of cholesterol lowering in 4444 pa- nell’ Infarto Miocardico. GISSI-2: a factorial ran- 1. Skolnick AA. Drug firm suit fails to halt publica- tients with coronary heart disease. Lancet. 1994;344: domised trial of alteplase versus streptokinase and hep- tion of Canadian Health Technology Report. JAMA. 1998;280:683-684. 1383-1389. arin versus no heparin among 12,490 patients with acute 2. Bucher HC, Guyatt GH, Cook DJ, Holbrook A, 22. Shepherd J, Cobbe SM, Ford I, et al, for the West myocardial infarction. Lancet. 1990;336:65-71. McAlister FA, for the Evidence-Based Medicine Work- of Scotland Coronary Prevention Study Group. Pre- 41. Third International Study of Infarct Survival Col- ing Group. Users’ guides to the medical literature, XIX: vention of coronary heart disease with pravastatin in laborative Group. ISIS-3: a randomised comparison of applying clinical trial results, A: how to use an article men with hypercholesterolemia. N Engl J Med. 1995; streptokinase vs tissue plasminogen activator vs an- measuring the effect of an intervention on surrogate 333:1301-1307. istreplase and of aspirin plus heparin vs aspirin alone end points. JAMA. 1999;282:771-778. 23. Sacks FM, Pfeffer MA, Moye LA, et al, for the Cho- among 41,299 cases of suspected acute myocardial 3. McDonald CJ. Medical heuristics. Ann Intern Med. lesterol and Recurrent Events Trial Investigators. The infarction. Lancet. 1992;339:753-770. 1996;124:56-62. effect of pravastatin on coronary events after myo- 42. Avorn J, Monette J, Lacour A, et al. Persistence 4. Rothwell PM. Can overall results of clinical trials be cardial infarction in patients with average cholesterol of use of lipid-lowering medications: a cross-national applied to all patients? Lancet. 1995;345:1616-1619. levels. N Engl J Med. 1996;335:1001-1009. study. JAMA. 1998;279:1458-1462. 5. Collins R, Peto R, MacMahon S, et al. Blood pres- 24. Downs JR, Clearfield M, Weis S, et al, for the 43. Dans AL, Dans LF, Guyatt GH, Richardson S, for sure, stroke, and coronary heart disease, part 2: short- AFCAPS/TexCAPS Research Group. Primary preven- the Evidence-Based Medicine Working Group. Us- term reductions in blood pressure. Lancet. 1990;335: tion of acute coronary events with lovastatin in men ers’ guides to the medical literature, XIV: how to de- 827-838. and women with average cholesterol levels: results of cide on the applicability of clinical trial results to your 6. Antiplatelet Trialists’ Collaboration. Collaborative AFCAPS/TexCAPS. JAMA. 1998;279:1615-1622. patient. JAMA. 1998;279:545-549. overview of randomised trials of antiplatelet therapy—I 25. The Long-term Intervention with Pravastatin in 44. Drummond MF, Richardson WS, O’Brien BJ, [published correction appears in BMJ. 1994;308: Ischaemic Disease (LIPID) Study Group. Prevention of Levine M, Heyland D, for the Evidence-Based Medi- 1540]. BMJ. 1994;308:81-106. cardiovascular events and death with pravastatin in cine Working Group. Users’ guides to the medical lit- 7. Garg R, Yusuf S, for the Collaborative Group on patients with coronary heart disease and a broad range erature, XIII: how to use an article on economic analy- ACE Inhibitor Trials. Overview of randomized trials of of initial cholesterol levels. N Engl J Med. 1998;339: sis of clinical practice, A: are the results of the study angiotensin-converting enzyme inhibitors on mortal- 1349-1357. valid? JAMA. 1997;277:1552-1557. ity and morbidity in patients with heart failure [pub- 26. Jones P, Kafonek S, Laurora I, Hunninghake D. 45. Naylor CD, Williams JI, Basinski A, Goel V. Tech- lished correction appears in JAMA. 1995;274:462]. Comparative dose efficacy study of atorvastatin ver- nology assessment and cost-effectiveness analysis: mis- JAMA. 1995;273:1450-1456. sus simvastatin, pravastatin, lovastatin, and fluvas- guided guidelines? CMAJ. 1993;148:921-924. 8. Waldo AL, Camm AJ, de Ruyter H, et al, for the tatin in patients with hypercholesterolemia [pub- 46. Laupacis A, Feeny D, Detsky AS, Tugwell PX. How SWORD Investigators. Effect of d-sotalol on mortal- lished correction appears in Am J Cardiol. 1998;82: attractive does a new technology have to be to warrant ity in patients with left ventricular dysfunction after 128]. Am J Cardiol. 1998;81:582-587. adoption and utilization? CMAJ. 1992;146:473-481. recent and remote myocardial infarction [published cor- 27. Sacks FM, Moye LA, Davis BR, et al. Relation- 47. Gotto AM Jr. Cholesterol management in theory rection appears in Lancet. 1996;348:416]. Lancet. ship between plasma LDL concentrations during treat- and practice. Circulation. 1997;96:4424-4430. 1996;348:7-12. ment with pravastatin and recurrent coronary events 48. Grover SA, Paquet S, Levinton C, Coupal L, Zow- 9. Yusuf S, Peto R, Lewis J, Collins R, Sleight P. Beta in the Cholesterol and Recurrent Events Trial. Circu- all H. Estimating the benefits of modifying risk fac- blockade during and after myocardial infarction. Prog lation. 1998;97:1446-1452. tors of cardiovascular disease [published correction ap- Cardiovasc Dis. 1985;27:335-371. 28. Scandinavian Simvastatin Survival Study Group. pears in Arch Intern Med. 1998;158:1228]. Arch Intern 10. Boissel JP, Leizorovicz A, Picolet H, Peyrieux JC. Baseline serum cholesterol and treatment effect in the Med. 1998;158:655-662. Secondary prevention after high-risk acute myocar- Scandinavian Simvastatin Survival Study (4S). Lan- 49. Lacour A, Derderian F, LeLorier J. Comparison of dial infarction with low-dose acebutolol. Am J Car- cet. 1995;345:1274-1275. efficacy and cost among lipid-lowering agents in pa- diol. 1990;66:251-260. 29. West of Scotland Coronary Prevention Study Group. tients with primary hypercholesterolemia. Can J Car- 11. McAlister FA, Teo KK. Antiarrhythmic therapies Influence of pravastatin and plasma lipids on clinical diol. 1998;14:355-361. for the prevention of sudden cardiac death. Drugs. events in the West of Scotland Coronary Prevention 50. Tu JV, Naylor CD. Choosing among drugs of dif- 1997;54:235-252. Study (WOSCOPS). Circulation. 1998;97:1440-1445. ferent price for similar indications [published correc- 12. Furberg CD, Psaty BM. Calcium antagonists. Am 30. Fager G, Wiklund O. Cholesterol reduction and tion appears in Can J Cardiol. 1998;14:662]. Can J J Hypertens. 1996;9:122-125. clinical benefit: are there limits to our expectations? Cardiol. 1998;14:349-351.