Quick viewing(Text Mode)

Observational Study

Observational Study

Observational Study

PAUL R. ROSENBAUM Volume 3, pp. 1451–1462

in

Encyclopedia of in Behavioral

ISBN-13: 978-0-470-86080-9 ISBN-10: 0-470-86080-4

Editors

Brian S. Everitt & David C. Howell

 John Wiley & Sons, Ltd, Chichester, 2005 it is desired to discover, or to assign subjects at ran- Observational Study dom to different procedures.

When subjects are not assigned to treatment or con- Observational Studies Defined trol at random, when subjects select their own treat- ments or their environments inflict treatments upon In the ideal, the effects caused by treatments are them, differing outcomes may reflect these initial investigated in that randomly assign sub- differences rather than effects of the treatments [12, jects to treatment or control, thereby ensuring that 59]. Pretreatment differences or selection are comparable groups are compared under competing of two kinds: those that have been accurately mea- treatments [1, 5, 15, 23]. In such an , sured, called overt biases, and those that have not comparable groups prior to treatment ensure that dif- be measured but are suspected to exist, called hid- ferences in outcomes after treatment reflect effects of den biases. Removing overt biases and addressing the treatment (see Clinical Trials and Intervention uncertainty about hidden biases are central issues Studies). uses chance to form in observational studies. Overt biases are removed comparable groups; it does not use measured charac- by adjustments, such as , stratification or teristics describing the subjects before treatment. As adjustment (see ), a consequence, random assignment tends to make the which are discussed in the Section titled ‘Adjusting groups comparable both in terms of measured char- for Biases Visible in Observed Covariates’. Hidden acteristics and characteristics that were not or could biases are addressed partly in the design of an obser- not be measured. It is the unmeasured characteristics vational study, discussed in the Section titled ‘Design that present the largest difficulties when randomiza- of Observational Studies’ and ‘Elaborate ’, tion is not used. More precisely, random assignment and partly in the analysis of the study, discussed in the ensures that the only differences between treated and Section titled ‘Appraising Sensitivity to Hidden ’ control groups prior to treatment are due to chance – and ‘Elaborate Theories’ (see Quasi-experimental the flip of a coin in assigning one subject to treat- Designs; Internal Validity;andExternal Validity). ment, another to control – so if a common statistical test rejects the hypothesis that the difference is due to chance, then a treatment effect is demonstrated Examples of Observational Studies [18, 22]. Experiments with human subjects are often ethical Several examples of observational studies are and feasible when (a) all of the competing treat- described. Later sections refer to these examples. ments under study are either harmless or intended and expected to benefit the recipients, (b) the best treatment is not known, and in light of this, subjects Long-term Psychological Effects of the Death of a consent to be randomized, and (c) the investigator Close Relative can control the assignment and delivery of treatments. Experiments cannot ethically be used to study treat- In an attempt to estimate the long-term psychologi- ments that are harmful or unwanted, and experiments cal effects of bereavement, Lehman, Wortman, and are not practical when subjects refuse to cede control Williams [30] collected following the sudden of treatment assignment to the experimenter. When death of a spouse or a child in a car crash. They experiments are not ethical or not feasible, the effects matched 80 bereaved spouses and parents to 80 con- of treatments are examined in an observational study. trols drawn from 7581 individuals who came to renew Cochran [12] defined an observational study as an a drivers license. Specifically, they matched for gen- empiric comparison of treated and control groups in der, age, family income before the crash, education which: level, number, and ages of children. Contrasting their findings with the views of Bowlby and Freud, they the objective is to elucidate cause-and-effect rela- concluded: tionships [... in which it] is not feasible to use controlled experimentation, in the sense of being able Contrary to what some early writers have sug- to impose the procedures or treatments whose effects gested about the duration of the major symptoms 2 Observational Study

of bereavement ... both spouses and parents in our who were exposed to lead at work. They matched 33 study showed clear evidence of depression and lack children whose parents worked in a battery factory to of resolution at the time of the interview, which was 33 unexposed control children of the same age and 5 to 7 years after the loss occurred. neighborhood, and used Wilcoxon’s signed rank test to compare the level of lead found in the children’s Effects on Criminal Violence of Laws Limiting blood, finding elevated levels of lead in exposed Access to Handguns children. In addition, they compared exposed children whose parents had varied levels of exposure to lead Do laws that ban purchases of handguns by convicted at the factory, finding that parents who had higher felons reduce criminal violence? It would be difficult, exposures on the job in turn had children with more perhaps impossible, to study this in a randomized lead in their blood. Finally, they compared exposed experiment, and yet an observational study faces sub- children whose parents had varied hygiene upon leav- stantial difficulties as well. One could not reasonably ing the factory at the end of the day, finding that poor estimate the effects of such a law by comparing hygiene of the parent predicted higher levels of lead the rate of criminal violence among convicted felons in the blood of the child. barred from handgun purchases to the rate among all other individuals permitted to purchase handguns. After all, convicted felons may be more prone to Design of Observational Studies criminal violence and may have greater access to illegally purchased guns than typical purchasers of Observational studies are sometimes referred to handguns without felony convictions. For instance, in as natural experiments [36, 56] or as quasi- his ethnographic account of violent criminals, Athens experiments [61] (see Quasi-experimental Designs). (1997, p. 68) depicts their sporadic violent behav- These differences in terminology reflect certain ior as consistent with stable patterns of thought and differences in emphasis, but a shared theme is that the , and writes: “... the self-images of vio- early stages of planning or designing an observational lent criminals are always congruent with their violent study attempt to reproduce, as nearly as possible, criminal actions.” some of the strengths of an experiment [47]. Wright, Wintemute, and Rivara [68] compared A treatment is a program, policy, or intervention two groups of individuals in California: (a) indi- which, in principle, may be applied to or withheld viduals who attempted to purchase a handgun but from any subject under study. A variable measured whose purchase was denied because of a prior felony prior to treatment is not affected by the treatment and conviction, and (b) individuals whose purchase was is called a covariate. A variable measured after treat- approved because their prior felony arrest had not ment may have been affected by the treatment and is resulted in a conviction. The comparison looked for- called an outcome. An analysis that does not carefully ward in time from the attempt to purchase a hand- distinguish covariates and outcomes can introduce gun, recording arrest charges for new offenses in the biases into the analysis where none existed previ- subsequent three years. Presumably, group (b) is a ously [43]. The effect caused by a treatment is a mixture of some individuals who did not commit comparison of the outcome a subject exhibited under the felony for which they were arrested and others the treatment the subject actually received with the who did. If this presumption were correct, group (a) potential but unobserved outcome the subject would would be more similar to group (b) than to typical have exhibited under the alternative treatment [40, purchasers of handguns, but substantial biases may 59]. Causal effects so defined are sometimes said remain. to be counterfactual (see Counterfactual Reason- ing), in the specific sense that they contrast what Effects on Children of Occupational Exposures to did happen to a subject under one treatment with Lead what would have happened under the other treatment. Causal effects cannot be calculated for individuals, Morton, Saah, Silberg, Owens, Roberts, and because each individual is observed under treatment Saah [39] asked whether children were harmed by or under control, but not both. However, in a random- lead brought home in the clothes and hair of parents ized experiment, the treated-minus-control difference Observational Study 3 in outcomes is an unbiased and consistent esti- deliberate choice, assigned subjects to treatment mate of the average effect of the treatment on the or control. For instance, in the United States, subjects in the experiment. class sizes in government run schools are largely In planning an observational study, one attempts determined by the degree of wealth in the local to identify circumstances in which some or all of the region, but in Israel, a rule proposed by Mai- following elements are available [47]. monides in the 12th century still requires that a class of 41 must be divided into two sepa- • Key covariates and outcomes are available for rate classes. In Israel, what separates a class of treated and control groups. The most basic ele- size 40 from classes half as large is the enroll- ments of an observational study are treated and ment of one more student. Angrist and Lavy [2] control groups, with important covariates mea- exploited Maimonides rule in their study of the sured before treatment, and outcomes measured effects of class size on academic achievement after treatment. If data are carefully collected in Israel. Similarly, Oreopoulos [41] studies the longitudi- over time as events occur, as in a economic effects of living in a poor neigh- nal study (see Longitudinal Data Analysis), borhood by exploiting the policy of Toronto’s then the temporal order of events is typically public housing program of assigning people to clear, and the distinction between covariates and housing in quite different neighborhoods simply outcomes is clear as well. In contrast, if data based on their position in a waiting list. Lehman are collected from subjects at a single time, as et al. [30], in their study of bereavement in the in a cross-sectional study (see Cross-sectional Section titled ‘Long-term Psychological Effects Design) based on a single survey interview, of the Death of a Close Relative’, limited the then the distinction between covariates and out- study to car crashes for which the driver was comes depends critically on the subjects’ recall, not responsible, on the grounds that car crashes and may not be sharp for some variables; this is a weakness of cross-sectional studies. Age for which the driver was responsible were rela- and sex are covariates whenever they are mea- tively less haphazard events, perhaps reflecting sured, but current recall of past , expe- forms of addiction or psychopathology. Random riences, moods, habits, and so forth can easily assignment is a fact, but haphazard assignment be affected by subsequent events. is a judgment, perhaps a mistaken one; however, • Haphazard treatment assignment rather than haphazard assignments are preferred to assign- self-selection. When is not used, ments known to be severely biased. • treated and control groups are often formed Special populations offering reduced self- by deliberate choices reflecting either the per- selection. Restriction to certain subpopulations sonal preferences of the subjects themselves or may diminish, albeit not eliminate, biases due else the view of some provider of treatment to self-selection. In their study of the effects of that certain subjects would benefit from treat- adolescent abortion, Zabin, Hirsch, and Emer- ment. Deliberate selection of this sort can lead son [69] used as controls young women who to substantial biases in observational studies. visited a clinic for a pregnancy test, but whose For instance, Campbell and Boruch [10] dis- test result came back negative, thereby ensuring cuss the substantial systematic biases in many that the controls were also sexually active. The observational studies of compensatory programs use in the Section titled ‘Effects on Criminal intended to offset some disadvantage, such as Violence of Laws Limiting Access to Handguns’ the US Head Start Program for preschool chil- of controls who had felony arrests without con- dren. Campbell and Boruch note that the typical victions may also reduce hidden bias. study compares disadvantaged subjects eligi- • Biases of known direction. In some settings, the ble for the program to controls who were not direction of unobserved biases is quite clear eligible because they were not sufficiently dis- even if their magnitude is not, and in cer- advantaged. When randomization is not possi- tain special circumstances, a treatment effect ble, one should try to identify circumstances in that overcomes a bias working against it may which an ostensibly irrelevant event, rather than yield a relatively unambiguous conclusion. For 4 Observational Study

instance, in the Section titled ‘Effects on Crimi- be assumed to have occurred randomly with nal Violence of Laws Limiting Access to Hand- respect to other risk factors of depression and guns’, one expects that the group of convicted to be inescapable, in which case matched com- parison can be used to make causal inferences felons denied handguns contains fewer inno- about long-term stress effects. A good exam- cent individuals than does the arrested-but-not- ple is the matched comparison of the parents convicted group who were permitted to purchase of children having cancer, diabetes, or some handguns. Nonetheless, Wright et al. [68] found other serious childhood physical disorder with fewer subsequent arrests for gun and violent the parents of healthy children. Disorders of offenses among the convicted felons, suggest- this sort are quite common and occur, in most ing that the denial of handguns may have had an cases, for reasons that are unrelated to other risk factors for parental psychiatric disorder. effect large enough to overcome a bias working The small amount of research shows that these in the opposite direction. Similarly, it is often childhood physical disorders have significant claimed that payments from disability insurance psychiatric effects on the family.” (p. 197) provided by US Social Security deter recipients • from returning to work by providing a finan- Additional structural features in quasi-experi- cial disincentive. Bound [6] examined this claim ments intended to provide information about by comparing disability recipients to rejected hidden biases. The term quasi-experiment is applicants, where the rejection was based on an often used to suggest a design in which certain administrative judgment that the injury or dis- structural features are added in an effort to ability was not sufficiently severe. Here, too, the provide information about hidden biases; see direction of bias seems clear: rejected applicants Section titled ‘Elaborate Theories’ for detailed should be healthier. However, Bound found that discussion. In the Section titled ‘Effects on Chil- even among the rejected applicants, relatively dren of Occupational Exposures to Lead’, for few returned to work, suggesting that even instance, data were collected for control chil- fewer of the recipients would return to work dren whose parents were not exposed to lead, without insurance. Some general about together with data about the level of lead expo- studies that exploit biases of known direction is sure and the hygiene of parents exposed to lead. given in Section 6.5 of [49]. An actual effect of lead should produce a quite • An abrupt start to intense treatments.Inan specific pattern of associations: more lead in the experiment, the treated and control conditions blood of exposed children, more lead when the are markedly distinct, and these conditions level of exposure is higher, more lead when become active at a specific known time. Lehman the hygiene is poor. In general terms, Cook et al.’s [30] study of the psychological effects et al. [14] write that: of bereavement in the Section titled ‘Long-term “... the warrant for causal inferences from Psychological Effects of the Death of a Close quasi-experiments rests [on] structural ele- Relative’ resembles an experiment in this sense. ments of design other than random assign- The study concerned the effects of the sudden ment–pretests, comparison groups, the way loss of a spouse or a child in a car crash. In treatments are scheduled across groups ... – contrast, the loss of a distant relative or the grad- [which] provide the best way of ruling out ual loss of a parent to chronic might threats to internal validity ... [C]onclusions are more plausible if they are based on evi- possibly have effects that are smaller, more dence that corroborates numerous, complex, or ambiguous, more difficult to discern. In a gen- numerically precise predictions drawn from a eral discussion of studies of stress and depres- descriptive causal hypothesis.” (pp. 570-1) sion, Kessler [28] makes this point clearly: Randomization will produce treated and control “... a major problem in interpret[ation] ... is groups that were comparable prior to treatment, and that both chronic role-related stresses and the chronic depression by definition have occurred it will do this mechanically, with no understanding for so long that deciding unambiguously which of the context in which the study is being conducted. came first is difficult ... The researcher, When randomization is not used, an understand- however, may focus on stresses that can ing of the context becomes much more important. Observational Study 5

Context is important whether one is trying to iden- Relative’ and Morton et al.’s [39] study of lead expo- tify what covariates to measure, or to locate set- sure in the Section titled ‘Effects on Children of tings that afford haphazard treatment assignments Occupational Exposures to Lead’, each treated sub- or subpopulations with reduced selection biases, or ject is matched to exactly one control, but other to determine the direction of hidden biases. Ethno- matching structures may yield either greater bias graphic and other qualitative studies (e.g., [3, 21]) reduction or estimates with smaller standard errors may provide familiarity with context needed in plan- or both. In particular, if the reservoir of potential ning an observational study, and moreover quali- controls is large, and if obtaining data from con- tative methods may be integrated with quantitative trols is not prohibitively expensive, then the standard studies [55]. errors of estimated treatment effects can be substan- Because even the most carefully designed obser- tially reduced by matching each treated subject to vational study will have weaknesses and ambiguities, several controls [62]. When several controls are used, a single observational study is often not decisive, substantially greater bias reduction is possible if the and is often necessary. In replicating an number of controls is not constant, instead varying observational study, one should seek to replicate from one treated subject to another [37]. the actual treatment effects, if any, without replicat- ing any biases that may have affected the original Multivariate Matching Using Propensity Scores. study. Some strategies for doing this are discussed In matching, the first impulse is to try to match each in [48]. treated subject to a control who appears nearly the same in terms of observed covariates; however, this Adjusting for Biases Visible in Observed is quickly seen to be impractical when there are many covariates. For instance, with 20 binary covariates, Covariates there are 220 or about a million types of individuals, Matched so even with thousands of potential controls, it will often be difficult to find a control who matches a Selecting from a Reservoir of Potential Controls. treated subject on all 20 covariates. Among methods of adjustment for overt biases, the Randomization produces covariate balance, not most direct and intuitive is matching, which compares perfect matches. Perfect matches are not needed to each treated individual to one or more controls who balance observed covariates. Multivariate matching appear comparable in terms of observed covariates. methods attempt to produce matched pairs or sets that Matched sampling is most common when a small balance observed covariates, so that, in aggregate, treated group is available together with a large the distributions of observed covariates are similar reservoir of potential controls [57]. in treated and control groups. Of course, unlike ran- The structure of the study of bereavement by domization, matching cannot be expected to balance Lehman et al. [30] in the Section titled ‘Long-term unobserved covariates. Psychological Effects of the Death of a Close Rel- The propensity score is the conditional probabil- ative’ is typical. There were 80 bereaved spouses ity (see Probability: An Introduction) of receiv- and parents and 7581 potential controls, from whom ing the treatment rather than the control given the 80 matched controls were selected. Routine admin- observed covariates [52]. Typically, the propensity istrative records were used to identify and match score is unknown and must be estimated, for instance, bereaved and control subjects, but additional informa- using [19] of the binary category, tion was needed from matched subjects for research treatment/control on the observed covariates. The purposes, namely, psychiatric outcomes. It is neither propensity score is defined in terms of the observed practical nor important to obtain psychiatric outcomes covariates, even when there are concerns about hid- for all 7581 potential controls, and instead, match- den biases due to unobserved covariates, so estimat- ing selected 80 controls who appear comparable to ing the propensity score is straightforward because treated subjects. the needed data are available. For nontechnical sur- Most commonly, as in both Lehman et al.’s [30] veys of methods using propensity scores, see [7, 27], study of bereavement in the Section titled ‘Long- and see [33] for discussion of propensity scores for term Psychological Effects of the Death of a Close doses of treatment. 6 Observational Study

Matching on one variable, the propensity score, Model-based Adjustments tends to balance all of the observed covariates, even though matched individuals will typically differ on Unlike matched sampling and stratification, which many observed covariates. As an alternative, match- compare treated subjects directly to actual controls ing on the propensity score and one or two other who appear comparable in terms of observed covari- key covariates will also tend to balance all of the ates, model-based adjustments, such as covariance observed covariates. If it suffices to adjust for the adjustment, use data on treated and control subjects observed covariates – that is, if there is no hidden bias without regard to their comparability, relying on a due to unobserved covariates – then it also suffices model, such as a model, to predict to adjust for the propensity score alone. These results how subjects would have responded under treatments are Theorems 1 through 4 of [52]. A study of the they did not receive. In a from labor psychological effects of prenatal exposures to barbi- , Dehejia and Wahba [20] compared the turates balanced 20 observed covariates by matching performance of model-based adjustments and match- on an estimated propensity score and sex [54]. ing, and Rubin [58, 60] compared performance using One can and should check to confirm that the simulation. Rubin found that model-based adjust- propensity score has done its job. That is, one ments yielded smaller standard errors than matching should check that, after matching, the distributions of when the model is precisely correct, but model-based observed covariates are similar in treated and control adjustments were less robust than matching when the groups; see [53, 54] for examples of this simple pro- model is wrong. Indeed, he found that if the model is cess. Because theory says that a correctly estimated substantially incorrect, model-based adjustments may propensity score should balance observed covariates, not only fail to remove overt biases, they may even this check on covariate balance is also a check on increase them, whereas matching and stratification are the model used to estimate the propensity score. If fairly consistent at reducing overt biases. Rubin found some covariates are not balanced, consider adding to that the combined use of matching and model-based the logit model interactions or quadratics involving adjustments was both robust and efficient, and he rec- these covariates; then check covariate balance with ommended this strategy in practice. the new propensity score. Bergstralh, Kosanke, and Jacobsen [4] provide SAS software for an optimal matching algorithm. Appraising Sensitivity to Hidden Bias

Stratification With care, matching, stratification, model-based adjustments and combinations of these techniques Stratification is an alternative to matching in which may often be used to remove overt biases accurately subjects are grouped rather than paired. Cochran [13] recorded in the data at hand, that is, biases visible showed that five strata formed from a single contin- in imbalances in observed covariates. However, uous covariate can remove about 90% of the bias in when observational studies are subjected to critical that covariate. Strata that balance many covariates at evaluation, a common concern is that the adjustments once can often be formed by forming five strata at the failed to control for some covariate that was not quintiles of an estimated propensity score. A study measured. In other words, the concern is that of coronary bypass surgery balanced 74 covariates treated and control subjects were not comparable using five strata formed from an estimated propensity prior to treatment with respect to this unobserved score [53]. covariate, and had this covariate been measured The optimal stratification – that is, the stratifica- and controlled by adjustments, then the conclusions tion that makes treated and control subjects as similar about treatment effects would have been different. as possible within strata – is a type of matching This is not a concern in randomized experiments, called full matching in which a treated subject can because randomization balances both observed and be matched to several controls or a control can be unobserved covariates. In an observational study, matched to several treated subjects [45]. An optimal a sensitivity analysis (see Sensitivity Analysis in full matching, hence also an optimal stratification, can Observational Studies) asks how such hidden biases be determined using network optimization. of various magnitudes might alter the conclusions of Observational Study 7 the study. Observational studies vary greatly in their 0.0001 and 0.041. Analogous bounds may be com- sensitivity to hidden bias. puted for point estimates and confidence intervals. Cornfield et al. [17] conducted the first formal How large must be before the conclusions of the sensitivity analysis in a discussion of the effects of study are qualitatively altered? If for = 9, the P cigarette smoking on health. The objection had been value for testing no effect is between 0.00001 and raised that smoking might not cause lung cancer, but 0.02, then the results are highly insensitive to bias – rather that there might be a genetic predisposition only an enormous departure from random assignment both to smoke and to develop lung cancer, and that of treatments could explain away the observed associ- this, not an effect of smoking, was responsible for ation between treatment and outcome. However, if for the association between smoking and lung cancer. = 1.1, the P value for testing no effect is between Cornfield et al. [17] wrote: 0.01 and 0.3, then the study is extremely sensitive to hidden bias – a tiny bias could explain away the ... if cigarette smokers have 9 times the risk of observed association. This method of sensitivity anal- nonsmokers for developing lung cancer, and this is ysis is discussed in detail with many examples in not because cigarette smoke is a causal agent, but Section 4 of [49] and the references given there. only because cigarette smokers produce hormone X, then the proportion of hormone X-producers among For instance, Morton et al.’s [39] study in the cigarette smokers must be at least 9 times greater Section titled ‘Effects on Children of Occupational than among nonsmokers. (p. 40) Exposures to Lead’ used Wilcoxon’s signed-ranks test to compare blood lead levels of 33 exposed Though straightforward to compute, their sensitivity and 33 matched control children. The pattern of analysis is an important step beyond the familiar matched pair differences they observed would yield fact that association does not imply causation. A a P value less than 10−5 in a randomized exper- sensitivity analysis is a specific statement about the iment. For = 3, the of possible P values magnitude of hidden bias that would need to be is from about 10−15 to 0.014, so a bias of this present to explain the associations actually observed. magnitude could not explain the higher lead lev- Weak associations in small studies can be explained els among exposed children. In words, if Morton away by very small biases, but only a very large bias etal.[39]hadfailedtocontrolbymatchingavari- can explain a strong association in a large study. able strongly related to blood lead levels and three A simple, general method of sensitivity analysis times more common among exposed children, this introduces a single sensitivity parameter that mea- would not have been likely to produce a difference sures the degree of departure from random assign- in lead levels as large as the one they observed. ment of treatments. Two subjects with the same The upper bound on the P value is just about 0.05 observed covariates may differ in their odds of receiv- when = 4.35, so the study is quite insensitive to ing the treatment by at most a factor of .Inan hidden bias, but not as insensitive as the studies experiment, random assignment of treatments ensures of heavy smoking and lung cancer. For = 5and that = 1, so no sensitivity analysis is needed. In = 6, the upper bounds on the P value are 0.07 and an observational study with = 2, if two subjects 0.12, respectively, so biases of this magnitude could were matched exactly for observed covariates, then explain the observed association. Sensitivity analy- one might be twice as likely as the other to receive the ses for point estimates and confidence intervals for treatment because they differ in terms of a covariate this example are in Section 4.3.4 and Section 4.3.5 not observed. Of course, in an observational study, of [49]. is unknown. A sensitivity analysis tries out sev- Several other methods of sensitivity analysis are eral values of to see how the conclusions might discussed in [16, 24, 26], and [31]. change. Would small departures from random assign- ment alter the conclusions? Or, as in the studies Elaborate Theories of smoking and lung cancer, would only very large departures from random assignment alter the conclu- Elaborate Theories and Pattern Specificity sions? For each value of , it is possible to place bounds on a – perhaps for = 3, What can be observed to provide evidence about the true P value is unknown, but must be between hidden biases, that is, biases due to covariates that 8 Observational Study were not observed? Cochran [12] summarizes the or if higher exposure predicted lower lead levels, or view of Sir , the inventor of the if poor hygiene predicted lower lead levels, then this : would be difficult to explain as an effect caused by lead exposure, and would likely be understood as a About 20 years ago, when asked in a meeting consequence of some unmeasured bias, some way what can be done in observational studies to clarify children who appeared comparable were in fact not the step from association to causation, Sir Ronald Fisher replied: “Make your theories elaborate.” The comparable. Indeed, the pattern specific comparison reply puzzled me at first, since by Occam’s razor, is less sensitive to hidden bias. In detail, suppose that the advice usually given is to make theories as the exposure levels are assigned numerical scores, 1 simple as is consistent with known data. What Sir for a child whose father had either low exposure or Ronald meant, as subsequent discussion showed, good hygiene, 2 for a father with high exposure and was that when constructing a causal hypothesis one poor hygiene, and 1.5 for the several intermediate should envisage as many different consequences of situations. The sensitivity analysis discussed in the its truth as possible, and plan observational studies to discover whether each of these consequences is Section titled ‘Appraising Sensitivity to Hidden Bias’ found to hold. used the signed rank test to compare lead levels of the 33 exposed children and their 33 matched controls, Similarly, Cook & Shadish [15] (1994, p. 565): “Suc- and it became sensitive to hidden bias at = 4.35, cessful prediction of a complex pattern of multi- because the upper bound on the P value for testing variate results often leaves few plausible alternative no effect had just reached 0.05. Instead, using the explanations. (p. 95)” Some patterns of response are dose-signed-rank [46, 50] to incorporate the scientifically plausible as treatment effects, but others predicted pattern, the comparison becomes sensitive are not [25], [65]. “[W]ith more pattern specificity,” at = 4.75; that is, again, the upper bound on writes Trochim [63], “it is generally less likely that the P value for testing no effect is 0.05. In other plausible alternative explanations for the observed words, some biases that would explain away the effect pattern will be forthcoming. (p. 580)” higher lead levels of exposure children are not large enough to explain away the pattern of associations Example of Reduced Sensitivity to Hidden Bias predicted by the elaborate theory. To explain the Due to Pattern Specificity entire pattern, noticeably larger biases would need to be present. Morton et al.’s [39] study of lead exposures in the A reduction in sensitivity to hidden bias can occur Section titled ‘Effects on Children of Occupational when a correct elaborate theory is strongly confirmed Exposures to Lead’ provides an illustration. Their by the data, but an increase in sensitivity can occur elaborate theory predicted: (a) higher lead levels if the pattern is contradicted [50]. It is possible to in the blood of exposed children than in matched contrast competing design strategies in terms of their control children, (b) higher lead levels in exposed ‘design sensitivity;’ that is, their ability to reduce children whose parents had higher lead exposure sensitivity to hidden bias [51]. on the job, and (c) higher lead levels in exposed children whose parents practiced poorer hygiene upon Common Forms of Pattern Specificity leaving the factory. Since each of these predictions was consistent with observed data, to attribute the There are several common forms of pattern specificity observed associations to hidden bias rather than or elaborate theories [44, 49, 61]. an actual effect of lead exposure, one would need to postulate biases that could produce all three • Two control groups.Campbell [8] advocated associations. selecting two control groups to systematically In a formal sense, elaborate theories play two vary an unobserved covariate, that is, to roles: (a) they can aid in detecting hidden biases [49], select two different groups not exposed to and (b) they can make a study less sensitive to the treatment, but known to differ in terms hidden bias [50, 51]. In Section titled ‘Effects on of certain unobserved covariates. For instance, Children of Occupational Exposures to Lead’, if Card and Krueger [11] examined the common exposed children had lower lead levels than controls, claim among economists that increases in the Observational Study 9

minimum wage cause many minimum wage • Unaffected outcomes; ineffective treatments.An earners to loose their jobs. They did this by elaborate theory may predict that certain out- looking at changes in employment at fast food comes should not be affected by the treat- restaurants – Burger Kings, Wendy’s, KFCs, ment or certain treatments should not affect and Roy Rogers’ – when New Jersey increased the outcome; see Section 6 of [49] and [67]. its minimum wage by about 20%, from $4.25 For instance, in a case- [34], to $5.05 per hour, on 1 April 1992, comparing Mittleman et al. [38] asked whether bouts of employment before and after the increase. In anger might cause myocardial infarctions or certain analyses, they compared New Jersey heart attacks, finding a moderately strong and restaurants initially paying $4.25 to two control highly significant association. Although there groups: (a) restaurants in the same chains are reasons to think that bouts of anger might across the Delaware River in Pennsylvania cause heart attacks, there are also reasons to where the minimum wage had not increased, doubt that bouts of curiosity cause heart attacks. and (b) restaurants in the same chains in Mittleman et al. found curiosity was not asso- affluent sections of New Jersey where the ciated with myocardial infarction, writing: “the starting wage was at least $5.00 before 1 April specificity observed for anger ... as opposed to 1992. An actual effect of raising the minimum curiosity ... argue against recall bias.” McKil- wage should have negligible effects on both lip [35] suggests that an unaffected or ‘control’ control groups. In contrast, one anticipates outcome might sometimes serve in place of a differences between the two control groups control group, and Legorreta et al. [29] illus- if, say, Pennsylvania Burger Kings were poor trate this possibility in a study of changes in the demand for a type of surgery following a tech- controls for New Jersey Burger Kings, or nological change that reduced cost and increased if employment changes in relatively affluent safety. sections of New Jersey are very different from those in less affluent sections. Card and Krueger found similar changes in employment in the two control groups, and similar results Summary in their comparisons of the treated group In the design of an observational study, an attempt with either control group. An algorithm for is made to reconstruct some of the structure and optimal pair matching with two control groups strengths of an experiment. Analytical adjustments, is illustrated with Card and Krueger’s study such as matching, are used to control for overt biases, in [32]. that is, pretreatment differences between treated and • Coherence among several outcomes and/or sev- control groups that are visible in observed covariates. eral doses. Hill [25] emphasized the importance Analytical adjustments may fail because of hidden of a coherent pattern of associations and of dose- biases, that is, important covariates that were not response relationships, and Weiss [66] further measured and therefore not controlled by adjust- developed these ideas. Campbell [9] wrote: “... ments. Sensitivity analysis indicates the magnitude great inferential strength is added when each of hidden bias that would need to be present to theoretical parameter is exemplified in two or alter the qualitative conclusions of the study. Obser- more ways, each being as independent vational studies vary markedly in their sensitivity as possible of the other, as far as the theoret- to hidden bias; therefore, it is important to know ically irrelevant components are concerned (p. whether a particular study is sensitive to small biases 33).” Webb [64] speaks of triangulation. The or insensitive to quite large biases. Hidden biases lead example in the Section titled ‘Example may leave visible traces in observed data, and a of Reduced Sensitivity to Hidden Bias Due variety of tactics involving pattern specificity are to Pattern Specificity’ provides one illustration aimed at distinguishing actual treatment effects from and Reynolds and West [42] provide another. hidden biases. Pattern specificity may aid in detect- Related is in [46, 51] and the ing hidden bias or in reducing sensitivity to hidden references given there. bias. 10 Observational Study

Acknowledgment [15] Cook, T.D. & Shadish, W.R. (1994). Social experiments: Some developments over the past fifteen years, Annual This work was supported by grant SES-0345113 from Review of 45, 545–580. the US National Science Foundation. [16] Copas, J.B. & Li, H.G. (1997). Inference for non- random samples (with discussion), Journal of the Royal Statistical Society. Series B 59, 55–96. References [17] Cornfield, J., Haenszel, W., Hammond, E., Lilien- feld, A., Shimkin, M. & Wynder, E. (1959). Smoking [1] Angrist, J.D. (2003). Randomized trials and quasi- and lung cancer: recent evidence and a discussion of experiments in education research. NBER Reporter, some questions, Journal of the National Cancer Institute Summer, 11–14. 22, 173–203. [2] Angrist, J.D. & Lavy, V. (1999). Using Maimonides’ [18] Cox, D.R. & Read, N. (2000). The Theory of the Design rule to estimate the effect of class size on scholas- of Experiments, Chapman & Hall/CRC, New York. tic achievement, Quarterly Journal of Economics 114, [19] Cox, D.R. & Snell, E.J. (1989). Analysis of Binary Data, 533–575. Chapmann & Hall, London. [3] Athens, L. (1997). Violent Criminal Acts and Actors [20] Dehejia, R.H. & Wahba, S. (1999). Causal effects in Revisited, University of Illinois Press, Urbana. nonexperimental studies: Reevaluating the evaluation of [4] Bergstralh, E.J., Kosanke, J.L. & Jacobsen, S.L. (1996). training programs, Journal of the American Statistical Software for optimal matching in observational stud- Association 94, 1053–1062. ies, 7, 331–332. http://www.mayo. [21] Estroff, S.E. (1985). Making it Crazy: An Ethnography edu/hsr/sasmac.html of Psychiatric Clients in an American Community,Uni- [5] Boruch, R. (1997). Randomized Experiments for Plan- versity of California Press, Berkeley. ning and Evaluation, Sage Publications, Thousand Oaks. [22] Fisher, R.A. (1935). The , Oliver [6] Bound, J. (1989). The health and earnings of rejected & Boyd, Edinburgh. disability insurance applicants, American Economic [23] Friedman, L.M., DeMets, D.L. & Furberg, C.D. (1998). Review 79, 482–503. Fundamentals of Clinical Trials, Springer-Verlag, New [7] Braitman, L.E. & Rosenbaum, P.R. (2002). Rare out- York. comes, common treatments: analytic strategies using [24] Gastwirth, J.L. (1992). Methods for assessing the sensi- propensity scores, Annals of Internal Medicine 137, tivity of statistical comparisons used in title VII cases to 693–695. omitted variables, 33, 19–34. [8] Campbell, D.T. (1969). Prospective: artifact and Control, [25] Hill, A.B. (1965). The environment and disease: associ- in Artifact in Behavioral Research,R.Rosenthal& ation or causation? Proceedings of the Royal Society of R. Rosnow, eds, Academic Press, New York. Medicine 58, 295–300. [9] Campbell, D.T. (1988). Methodology and Epistemol- [26] Imbens, G.W. (2003). Sensitivity to exogeneity assump- ogy for : Selected Papers, University of tions in program evaluation, American Economic Review Chicago Press, Chicago, pp. 315–333. 93, 126–132. [10] Campbell, D.T. & Boruch, R.F. (1975). Making the case [27] Joffe, M.M. & Rosenbaum, P.R. (1999). Propen- for randomized assignment to treatments by considering sity scores, American Journal of Epidemiology 150, the alternatives: six ways in which quasi-experimental 327–333. evaluations in compensatory education tend to underes- [28] Kessler, R.C. (1997). The effects of stressful life timate effects, in Evaluation and Experiment,C.A.Ben- events on depression, Annual Review of Psychology 48, nett & A.A. Lumsdaine, eds, Academic Press, New York, 191–214. pp. 195–296. [29] Legorreta, A.P., Silber, J.H., Costantino, G.N., Kobylin- [11] Card, D. & Krueger, A. (1994). Minimum wages and ski, R.W. & Zatz, S.L. (1993). Increased cholecystec- employment: a case study of the fast-food industry tomy rate after the introduction of laparoscopic cholecys- in New Jersey and Pennsylvania, American Economic tectomy, Journal of the American Medical Association Review 84, 772–793. 270, 1429–1432. [12] Cochran, W.G. (1965). The planning of observational [30] Lehman, D., Wortman, C. & Williams, A. (1987). studies of human populations (with Discussion), Journal Long-term effects of losing a spouse or a child in a of the Royal Statistical Society. Series A 128, 134–155. motor vehicle crash, Journal of Personality and Social [13] Cochran, W.G. (1968). The effectiveness of adjustment Psychology 52, 218–231. by subclassification in removing bias in observational [31] Lin, D.Y., Psaty, B.M. & Kronmal, R.A. (1998). Assess- studies, Biometrics 24, 295–313. ing the sensitivity of regression results to unmeasured [14] Cook, T.D., Campbell, D.T. & Peracchio, L. (1990). confounders in observational studies, Biometrics 54, Quasi-experimentation, in Handbook of Industrial and 948–963. Organizational Psychology, M. Dunnette & L. Hough, [32] Lu, B. & Rosenbaum, P.R. (2004). Optimal pair match- eds, Consulting Psychologists Press, Palo Alto, ing with two control groups, Journal of Computational pp. 491–576. and Graphical Statistics 13, 422–434. Observational Study 11

[33] Lu, B., Zanutto, E., Hornik, R. & Rosenbaum, P.R. [50] Rosenbaum, P.R. (2003). Does a dose-response relation- (2001). Matching with doses in an observational study ship reduce sensitivity to hidden bias? 4, of a media campaign against drug abuse, Journal of the 1–10. American Statistical Association 96, 1245–1253. [51] Rosenbaum, P.R. (2004). Design sensitivity in observa- [34] Maclure, M. & Mittleman, M.A. (2000). Should we use tional studies, Biometrika 91, 153–164. a case-crossover design? Annual Review of Public Health [52] Rosenbaum, P.R. & Rubin, D.B. (1983). The central role 21, 193–221. of the propensity score in observational studies for causal [35] McKillip, J. (1992). Research without control groups: effects, Biometrika 70, 41–55. a control construct design, in Methodological Issues [53] Rosenbaum, P. & Rubin, D. (1984). Reducing bias in Applied Social Psychology,F.B.Bryant,J.Edwards in observational studies using subclassification on the & R.S. Tindale, eds, Plenum Press, New York, propensity score, Journal of the American Statistical pp. 159–175. Association 79, 516–524. [36] Meyer, B.D. (1995). Natural and quasi-experiments in [54] Rosenbaum, P. & Rubin, D. (1985). Constructing a con- economics, Journal of Business and Economic Statistics trol group using multivariate matched sampling methods 13, 151–161. that incorporate the propensity score, American Statisti- [37] Ming, K. & Rosenbaum, P.R. (2000). Substantial gains cian 39, 33–38. in bias reduction from matching with a variable number [55] Rosenbaum, P.R. & Silber, J.H. (2001). Matching and of controls, Biometrics 56, 118–124. thick description in an observational study of mortality [38] Mittleman, M.A., Maclure, M., Sherwood, J.B., after surgery, Biostatistics 2, 217–232. Mulry, R.P., Tofler, G.H., Jacobs, S.C., Friedman, R., [56] Rosenzweig, M.R. & Wolpin, K.I. (2000). Natural “nat- Benson, H. & Muller, J.E. (1995). Triggering of acute ural experiments” in economics, Journal of Economic myocardial infarction onset by episodes of anger, Literature 38, 827–874. Circulation 92, 1720–1725. [57] Rubin, D. (1973a). Matching to remove bias in obser- [39] Morton, D., Saah, A., Silberg, S., Owens, W., vational studies, Biometrics 29, 159–183.Correction: Roberts, M. & Saah, M. (1982). Lead absorption 1974,. 30, 728. in children of employees in a lead-related industry, [58] Rubin, D.B. (1973b). The use of matched sampling and American Journal of Epidemiology 115, 549–555. regression adjustment to remove bias in observational [40] Neyman, J. (1923). On the application of probability studies, Biometrics 29, 185–203. theory to agricultural experiments: essay on principles, [59] Rubin, D.B. (1974). Estimating causal effects of treat- Section 9 (In Polish), Roczniki Nauk Roiniczych, Tom X, ments in randomized and nonrandomized studies, Jour- 1–51, Reprinted in English with Discussion in Statistical nal of Educational Psychology 66, 688–701. Science 1990,. 5, 463–480. [60] Rubin, D.B. (1979). Using multivariate matched sam- [41] Oreopoulos, P. (2003). The long-run consequences of pling and regression adjustment to control bias in obser- living in a poor neighborhood, Quarterly Journal of vational studies, Journal of the American Statistical Economics 118, 1533–1575. Association 74, 318–328. [42] Reynolds, K.D. & West, S.G. (1987). A multiplist [61] Shadish, W.R., Cook, T.D. & Campbell, D.T. (2002). strategy for strengthening nonequivalent control group Experimental and Quasi-Experimental Designs for Gen- designs, Evaluation Review 11, 691–714. eralized Causal Inference, Houghton-Mifflin, Boston. [43] Rosenbaum, P.R. (1984a). The consequences of adjust- [62] Smith, H.L. (1997). Matching with multiple controls ment for a concomitant variable that has been affected to estimate treatment effects in observational studies, by the treatment, Journal of the Royal Statistical Society. Sociological Methodology 27, 325–353. Series A 147, 656–666. [63] Trochim, W.M.K. (1985). Pattern matching, validity [44] Rosenbaum, P.R. (1984b). From association to causa- and conceptualization in program evaluation, Evaluation tion in observational studies, Journal of the American Review 9, 575–604. Statistical Association 79, 41–48. [64] Webb, E.J. (1966). Unconventionality, triangulation and [45] Rosenbaum, P.R. (1991). A characterization of optimal inference, Proceedings of the Invitational Conference on designs for observational studies, Journal of the Royal Testing Problems, Educational Testing Service, Prince- Statistical Society. Series B 53, 597–610. ton, pp. 34–43. [46] Rosenbaum, P.R. (1997). Signed rank statistics for [65] Weed, D.L. & Hursting, S.D. (1998). Biologic plausi- coherent predictions, Biometrics 53, 556–566. bility in causal inference: current method and practice, [47] Rosenbaum, P.R. (1999). Choice as an alternative to con- American Journal of Epidemiology 147, 415–425. trol in observational studies (with Discussion), Statistical [66] Weiss, N. (1981). Inferring causal relationships: elab- Science 14, 259–304. oration of the criterion of ‘dose-response’, American [48] Rosenbaum, P.R. (2001). Replicating effects and biases, Journal of Epidemiology 113, 487–90. American 55, 223–227. [67] Weiss, N.S. (2002). Can the “specificity” of an associ- [49] Rosenbaum, P.R. (2002). Observational Studies, 2nd ation be rehabilitated as a basis for supporting a causal Edition, Springer-Verlag, New York. hypothesis? Epidemiology 13, 6–8. 12 Observational Study

[68] Wright, M.A., Wintemute, G.J. & Rivara, F.P. (1999). education, psychological status, and subsequent preg- Effectiveness of denial of handgun purchase to persons nancy, Family Planning Perspectives 21, 248–255. believed to be at high risk for firearm violence, American Journal of Public Health 89, 88–90. PAUL R. ROSENBAUM [69] Zabin, L.S., Hirsch, M.B. & Emerson, M.R. (1989). When urban adolescents choose abortion: effects on

Top View