10 Confirmatory clinical trials: Analysis of categorical efficacy data 10.1 Introduction: Regulatory views of advance and evaluated. As a rule, confirmatory substantial evidence trials are necessary to provide firm evidence of efficacy or safety. In such trials the key hypoth- esis of interest follows directly from the trial’s When thinking about the use of statistics in clin- primary objective, is always pre-defined, and is ical trials, the first thing that comes to mind for the hypothesis that is subsequently tested when many people is the process of hypothesis testing the trial is complete. In a confirmatory trial it is and the associated use of p values. This is very equally important to estimate with due precision reasonable, because the role of a chance outcome the size of the effects attributable to the treat- is of utmost importance in study design and the ment of interest and to relate these effects to interpretation of results from a study. A sponsor’s their clinical significance. objective is to develop an effective therapy that It is common practice to use earlier phase can be marketed to patients with a certain disease studies such as therapeutic exploratory studies or condition. From a public health perspective, to characterize the size of the treatment effect, the benefits of a new treatment cannot be sepa- while acknowledging that the effect size found rated from the risks that are tied to it. Regulatory in these studies is associated with a certain agencies must protect public health by ensuring amount of error. As noted earlier, confidence that a new treatment has “definitively” been intervals can be helpful for planning confirma- demonstrated to have a beneficial effect. The tory studies. The knowledge and experience meaning of the word “definitively” as used here gained in these earlier studies can lead to is rather broad, but we discuss what it means in hypotheses that we wish to test (and hopefully this context – that is, we operationally define the confirm) in a therapeutic confirmatory trial, for term “definitively” as it applies to study design, example, the mean reduction in systolic blood data analysis, and interpretation in new drug pressure (SBP) for the test treatment is 20 mmHg development. greater than the mean reduction in SBP for Most of this chapter is devoted to describing placebo. As we have seen, a positive result from various types of data and the corresponding a single earlier trial could be a type I error, so a analytical strategies that can be used to demon- second study is useful in substantiating that strate that an investigational drug, or test result. treatment, is efficacious. First, however, it is The description of a confirmatory study in informative to discuss the international stand- ICH Guidance E9 (1998) also illustrates the ards for demonstrating efficacy of a new product, importance of the study design employed. The and examine how regulatory agencies have study should be designed with several important interpreted these guidelines. ICH Guidance E9 characteristics: (1998, p 4) addresses therapeutic confirmatory studies and provides the following definition: • It should test a specific hypothesis. • It should be appropriately sized. A confirmatory trial is an adequately controlled • It should be able to differentiate treatment trial in which the hypotheses are stated in effects from other sources of variation (for 128 Chapter 10 • Confirmatory clinical trials: Analysis of categorical efficacy data example, time trends, regression to the mean, The terms “firm evidence” and “robust” do bias). not have explicit definitions. However, as clin- • The size of the treatment effect that is being ical trials have been conducted and reported in confirmed should be clinically relevant. recent years, some practical (operational) defini- tions have emerged, and these are discussed The clinical relevance, or clinical significance, of shortly. a treatment effect is an extremely important In its guidance document Providing Clinical consideration. The size of a treatment effect that Evidence of Effectiveness for Human Drug and is deemed clinically relevant is best defined by Biological Products, the US Food and Drug medical, clinical, and regulatory specialists. Administration (US Department of Health and Precise description of the study design and Human Services, FDA, 1998) describes the adherence to the study procedures detailed in introduction of an effectiveness requirement the study protocol are particularly important for according to a standard of “substantial evidence” confirmatory studies. Quoting again from ICH in the Federal Food, Drug, and Cosmetic Act Guidance E9 (1998, p 4): (the FDC Act) of 1962: Confirmatory trials are intended to provide firm Substantial evidence was defined in section 505(d) evidence in support of claims and hence adher- of the Act as “evidence consisting of adequate ence to protocols and standard operating proce- and well-controlled investigations, including dures is particularly important; unavoidable clinical investigations, by experts qualified by changes should be explained and documented, scientific training and experience to evaluate the and their effect examined. A justification of the effectiveness of the drug involved, on the basis design of each such trial, and of other important of which it could fairly and responsibly be statistical aspects such as the principal features concluded by such experts that the drug will of the planned analysis, should be set out in have the effect it purports or is represented to the protocol. Each trial should address only a have under the conditions of use prescribed, limited number of questions. recommended, or suggested in the labeling or proposed labeling thereof.” Confirmatory studies should also provide quantitative evidence that substantiates claims US Department of Health and in the product label (for example, the package Human Services, FDA (1998, p 3) insert) as they relate to an appropriate popula- The phrase “adequate and well-controlled inves- tion of patients. In the following quote from tigations” has typically been interpreted as at ICH Guidance E9 (1998, p 4), the elements of least two studies that clearly demonstrated that statistical and clinical inference can be seen: the drug has the effect claimed by the sponsor Firm evidence in support of claims requires that submitting a marketing approval. Furthermore, a the results of the confirmatory trials demonstrate type I error of 0.05 has typically been adopted as that the investigational product under test has a reasonable standard upon which data from clinical benefits. The confirmatory trials should clinical studies are judged. That is, it was widely therefore be sufficient to answer each key clinical believed that the intent of the FDC Act of 1962 question relevant to the efficacy or safety claim was to state that a drug could be concluded to be clearly and definitively. In addition, it is impor- effective if the treatment effect was clinically tant that the basis for generalisation . to the relevant and statistically significant at the a ϭ intended patient population is understood and 0.05 level in two independent studies. explained; this may also influence the number The ICH Guidance E8 (1998, p 4) clarified this and type (e.g. specialist or general practitioner) of issue: centres and/or trials needed. The results of the confirmatory trial(s) should be robust. In some The usual requirement for more than one circumstances the weight of evidence from a adequate and well-controlled investigation single confirmatory trial may be sufficient. reflects the need for independent substantiation Introduction: Regulatory views of substantial evidence 129 of experimental results. A single clinical experi- substantiation of experimental results addresses mental finding of efficacy, unsupported by other such problems by providing consistency across independent evidence, has not usually been more than one study, thus greatly reducing the considered adequate scientific support for a possibility that a biased, chance, site-specific, or conclusion of effectiveness. The reasons for this fraudulent result will lead to an erroneous include the following: conclusion that a drug is effective. • Any clinical trial may be subject to unantici- This guidance further clarified that the need for pated, undetected, systematic biases. These substantiation does not necessarily require two biases may operate despite the best intentions or more identically designed trials: of sponsors and investigators, and may lead Precise replication of a trial is only one of a to flawed conclusions. In addition, some number of possible means of obtaining indepen- investigators may bring conscious biases to dent substantiation of a clinical finding and, at evaluations. times, can be less than optimal as it could leave • The inherent variability in biological systems the conclusions vulnerable to any systematic may produce a positive trial result by chance biases inherent to the particular study design. alone. This possibility is acknowledged, and Results that are obtained from studies that are quantified to some extent, in the statistical of different design and independent in execu- evaluation of the result of a single efficacy tion, perhaps evaluating different populations, trial. It should be noted, however, that endpoints, or dosage forms, may provide hundreds of randomized clinical efficacy trials support for a conclusion of effectiveness that is are conducted each year with the intent of as convincing as, or more convincing than, a submitting favorable results to the FDA. Even repetition of the same study. if all drugs tested in such trials were ineffec- tive, one would expect one in forty of those ICH Guidance E8 (1998, p 5) trials to “demonstrate” efficacy by chance Regulatory agencies have traditionally accepted alone at conventional levels of statistical significance. It is probable, therefore, that only two-sided hypotheses because, theoreti- false positive findings (that is, the chance cally, one could not rule out harm (as opposed to appearance of efficacy with an ineffective simply no effect) associated with the test treat- drug) will occur and be submitted to FDA as ment.