Psychological Methods Copyright 1998 by the American Psychological Association, Inc. 1998, Vol. 3, No. 3, 339-353 1082-989X/98/S3.00 Using Odds Ratios as Effect Sizes for Meta-Analysis of Dichotomous Data: A Primer on Methods and Issues C. Keith Haddock David Rindskopf University of Missouri—Kansas City Graduate Center of the City University of New York William R. Shadish University of Memphis Many meta-analysts incorrectly use correlations or standardized mean difference statistics to compute effect sizes on dichotomous data. Odds ratios and their loga- rithms should almost always be preferred for such data. This article reviews the issues and shows how to use odds ratios in meta-analytic data, both alone and in combination with other effect size estimators. Examples illustrate procedures for estimating the weighted average of such effect sizes and methods for computing variance estimates, confidence intervals, and homogeneity tests. Descriptions of fixed- and random-effects models help determine whether effect sizes are functions of study characteristics, and a random-effects regression model, previously unused for odds ratio data, is described. Although all but the latter of these procedures are already widely known in areas such as medicine and epidemiology, the absence of their use in psychology suggests a need for this description. Psychological researchers often use dichotomous programs; then the proportion of each group that be- outcome variables in which participants are classified came pregnant after treatment would be determined. into two distinct groups, such as pregnant and not Here, both the antecedent factor (treatment condition) pregnant, recidivist and not recidivist, improved and and the outcome variable (pregnant vs. not pregnant) not improved, or passed and failed. When a dichoto- are dichotomous. Jacobson, Follette, and Revenstorf mous outcome variable is a function of a comparison (1984) and Blanchard and Schwarz (1988) suggested between two treatment conditions, a common way of methods of statistically defining a clinically signifi- presenting such data is by means of a fourfold table. cant change in psychotherapy treatment that poten- In a fourfold table, two dichotomous variables are tially yields a dichotomous outcome (clinically sig- crossed with one another to form four possible cat- nificant change or not). Of course, the two variables egories. For example, a psychologist could study the need not be treatment and outcome; for example, one relative effectiveness of two programs for lowering might be interested in the relationship between sex the teenage pregnancy rate in a community. Partici- (male vs. female) and hiring decisions (hired vs. not pants might be randomly assigned to one of the two hired). The issues we outline in the present article apply equally well to all fourfold designs, even though we focus on experiments. C. Keith Haddock, Department of Psychology, Univer- In these cases, effect sizes are often inappropriately sity of Missouri—Kansas City; David Rindskopf, Depart- computed with methods developed for continuous ment of Psychology, Graduate Center of the City University data, such as standardized mean difference statistics of New York; William R. Shadish, Department of Psychol- or phi correlation coefficients. The present article re- ogy, University of Memphis. The authors are listed in alphabetical order. views better effect size estimates for these situations Correspondence concerning this article should be ad- and appropriate statistical models for meta-analysis of dressed to C. Keith Haddock, Department of Psychology, such effect size data. The effect size measures pre- University of Missouri, 5319 Holmes, Kansas City, Missouri sented in this article are widely used in epidemiology, 64110-2499. chemistry, genetics, and medicine (Feinberg, 1980; 339 340 HADDOCK, RINDSKOPF, AND SHADISH Sandercock, 1989). Indeed, in meta-analysis, these is- used. However, x2 is not a measure of the magnitude sues and the proper analyses are well known by sta- of the effect; rather, x2 is a function of both the mag- tistical experts (e.g., Fleiss, 1994), and some of the nitude of effect and the number of participants in the material we present has been described previously study. Theoretically, the number of participants stud- (e.g., Shadish & Haddock, 1994). However, these ied should not affect a measure of the magnitude of methods almost never have been used in psychology the association between two variables (Fisher, 1954; and education. Rather, most researchers in psychol- Fleiss, 1981). ogy and education continue to use incorrect effect size Table 2 presents data from 24 studies examining estimates such as the standardized mean difference the effectiveness of psychosocial treatments for indi- statistic or the phi correlation. Hence, a primer is viduals who were addicted to illegal drugs, to alcohol, needed to sensitize the psychological and educational or to smoking cigarettes. In all these studies, the out- communities to the issues and appropriate methods come was categorized as successful (e.g., drug free) for such meta-analytic work.' In addition, we present or not successful. Therefore, the principle outcome a hierarchical linear model for the fixed- and random- measure in these studies is a dichotomous variable. effects regression analysis of such data in meta- We use these data to illustrate various computations analysis. Although this extension of standard hierar- with fourfold tables throughout this article. chical models to such data is straightforward, such an extension has not been presented in the literature pre- Odds Ratio Measures of Effect Size viously for odds ratio data. There is widespread consensus among statisticians that, with a few exceptions that we discuss shortly, the Analyzing Fourfold Tables most appropriate measure of effect size from a four- fold table is the odds ratio (Agresti, 1990; Fleiss, The structure and notation of a fourfold table are 1981; Kleinbaum, Kupper, & Morgenstern, 1982; presented in Table 1. ThepyS indicate the proportion Mantel & Hankey, 1975; Sandercock, 1989; Somes & of participants or values falling in the cell in row i and O'Brien, 1985) or a transformation such as the loga- column ;', and the n^s represent the raw frequency of rithm of the odds ratio. These previous works have participants falling into the respective cells. The + presented the formulas and various examples of how symbols in the row and column totals represent sum- to apply them. We summarize such work here, but the mation across rows or columns. Thus, for example, interested reader is referred to these works for more P1+ represents the proportion of individuals in row 1, detail. We can express ftg, the odds of improving summed across columns. Many readers will be more given that one is treated with a psychosocial treatment familiar with the sample sizes (i.e., the n^s) of the (E), as the following ratio of conditional probabilities: cells of fourfold tables labeled as A through D; there- fore, this notation is also included in Table 1. nE = P(l\E)/P(l'\E), In order to determine whether a statistically signifi- cant relationship exists between the variables repre- where P(IIE) is the probability of improving given 2 sented in a fourfold table, the familiar x test can be psychosocial treatment and P(I'IE) = 1 - P(IIE) is the probability of not improving given the same treat- Table 1 ment. Similarly, fl^ can be defined for those patients Fourfold Table not receiving the psychosocial treatment (E' denotes receiving the alternative or control treatment) with the Characteristic B probability of improving given the comparison treat- Characteristic A Present Absent Row total ment, P(IIE'), and the probability of not improving Present nu = A «,, = B «,+ PM Pl2 PI + Absent «2I = C n22 = D «2+ 1 P21 P22 P2+ Although this article focuses on only fourfold tables as Column total "+i "+2 " + * the exemplar case in meta-analysis, the arguments extend in P+i P+2 1.0 principle to polytomous categorical variables as well. How- ever, the issues become more complicated given the diffi- Note. PIJ — proportion of participants or values in row /. column j; n^ = raw frequency of participants in row i, column j; + = culty identifying a single correct summary estimate for summation across row or column. more than two categories of outcome. META-ANALYSIS OF DICHOTOMOUS DATA 341 Table 2 Data From 24 Studies of Treatment for Addiction Treatment Comparison Study Success Failure Success Failure Alcohol studies Bowers & Al-Redha (1990) 4 3 1 4 Drummond et al. (1990) 5 14 6 12 Ferrell&Galassi(1981) 5 3 3 6 Madill et al. (1966) 3 9 3 12 Olson etal. (1981) 16 10 19 3 Orford & Edwards (1977) 18 28 17 29 Penick et al. (1969) 9 12 15 8 Primo et al. (1972) 5 2 1 3 Sannibale (1988) 15 15 4 4 Shaffer etal. (1964) 21 7 20 8 Stein et al. (1975) 11 17 8 20 Walker etal. (1983) 42 52 49 63 Substance abuse studies Henggeler et al. (1991) 89 3 71 13 Joanning et al. (1992) 19 16 4 21 Lewis et al. (1990) 24 20 15 25 McClellan et al. (1993) 21 10 3 7 Ziegler-Driscoll (1977) 18 19 3 9 Smoking cessation studies Areechon & Punnotok (1988) 56 22 37 27 Hall et al. (1985) 71 49 36 81 Harackiewicz et al. (1988) 16 74 8 30 Hjalraarson (1984) 38 65 19 77 Jamrozik et al. (1984) 10 90 8 89 Puska et al. (1979) 29 55 21 55 RCBTS (1983) 68 212 76 189 Nnte. RCBTS = Research Committee of the British Thoracic Society. given the comparison treatment, P(I'\E'). The popu- (2) lation parameter flE is estimated by the sample value: Pii/P+i Pii Notice that Equation 2 is equivalent to multiplying the P12/P+1 Pl2 main diagonal entries (raw frequencies) of the cells of the fourfold table and then dividing by the product of An overall measure of effect size for a study is the the off-diagonal entries.