EXPERIMENTALAND QUASI-EXPERIMENTAL DESIGNSFOR GENERALIZED ii:. CAUSALINFERENCE William R. Shadish Trru UNIvERSITYop MEvPrrts ** .jr-*"- Thomas D. Cook NonrrrwpsrERN UNrvPnslrY iLli" '"+.'-,, fr Donald T.Campbell HOUGHTONMIFFLIN COMPANY Boston New York Experimentsand GeneralizedCausal lnference Ex.per'i'ment (ik-spEr'e-mant):[Middle English from Old French from Latin experimentum, from experiri, to try; seeper- in Indo-European Roots.] n. Abbr. exp., expt, 1. a. A test under controlled conditions that is made to demonstratea known truth, examine the validity of a hypothe- sis, or determine the efficacyof something previously untried' b. The processof conducting such a test; experimentation. 2' An innovative "Democracy act or procedure: is only an experiment in gouernment" (.V{illiam Ralph lnge). Cause (k6z): [Middle English from Old French from Latin causa' teason, purpose.] n. 1. a. The producer of an effect, result, or consequence. b. The one, such as a person, an event' or a condition, that is responsi- ble for an action or a result. v. 1. To be the causeof or reason for; re- sult in. 2. To bring about or compel by authority or force. o MANv historiansand philosophers,the increasedemphasis on experimenta- tion in the 15th and L7th centuriesmarked the emergenceof modern science from its roots in natural philosophy (Hacking, 1983). Drake (1981) cites '1.6'!.2 'Water, Galileo's treatrseBodies Tbat Stay Atop or Moue in It as usheringin modern experimental science,but earlier claims can be made favoring \Tilliam Gilbert's1,600 study Onthe Loadstoneand MagneticBodies,Leonardo da Vinci's (1,452-1.51.9)many investigations,and perhapseven the Sth-centuryB.C.philoso- pher Empedocles,who used various empirical demonstrationsto argue against '1.969a, Parmenides(Jones, 1'969b).In the everyday senseof the term, humans have beenexperimenting with different ways of doing things from the earliestmo- ments of their history. Suchexperimenting is as natural a part of our life as trying a new recipe or a different way of starting campfires. z | 1. EXeERTMENTsANDGENERALTzED cAUsAL INFERENcE I However, the scientific revolution of the 1.7thcentury departed in three ways from the common use of observation in natural philosophy atthat time. First, it in- creasingly used observation to correct errors in theory. Throughout historg natu- ral philosophers often used observation in their theories, usually to win philo- sophical arguments by finding observations that supported their theories. However, they still subordinated the use of observation to the practice of deriving "first theoriesfrom principles," starting points that humans know to be true by our nature or by divine revelation (e.g., the assumedproperties of the four basic ele- ments of fire, water, earth, and air in Aristotelian natural philosophy). According to some accounts,this subordination of evidenceto theory degeneratedin the 17th "The century: Aristotelian principle of appealing to experiencehad degenerated among philosophers into dependenceon reasoning supported by casual examples and the refutation of opponents by pointing to apparent exceptionsnot carefully '1,98"1., examined" (Drake, p. xxi).'Sfhen some 17th-century scholarsthen beganto use observation to correct apparent errors in theoretical and religious first princi- ples, they came into conflict with religious or philosophical authorities, as in the case of the Inquisition's demands that Galileo recant his account of the earth re- volving around the sun. Given such hazards,the fact that the new experimental sci- ence tipped the balance toward observation and ^way from dogma is remarkable. By the time Galileo died, the role of systematicobservation was firmly entrenched as a central feature of science,and it has remained so ever since (Harr6,1981). Second,before the 17th century, appealsto experiencewere usually basedon passiveobservation of ongoing systemsrather than on observation of what hap- pens after a system is deliberately changed. After the scientific revolution in the L7th centurS the word experiment (terms in boldface in this book are defined in the Glossary) came to connote taking a deliberate action followed by systematic observationof what occurredafterward. As Hacking (1983) noted of FrancisBa- "He con: taught that not only must we observenature in the raw, but that we must 'twist also the lion's tale', that is, manipulate our world in order to learn its se- crets" (p. U9). Although passiveobservation reveals much about the world, ac- tive manipulation is required to discover some of the world's regularities and pos- sibilities (Greenwood,, 1989). As a mundane example, stainless steel does not occur naturally; humans must manipulate it into existence.Experimental science came to be concerned with observing the effects of such manipulations. Third, early experimenters realized the desirability of controlling extraneous influences that might limit or bias observation. So telescopeswere carried to higher points at which the air was clearer, the glass for microscopeswas ground ever more accuratelg and scientistsconstructed laboratories in which it was pos- sible to use walls to keep out potentially biasing ether waves and to use (eventu- ally sterilized) test tubes to keep out dust or bacteria. At first, thesecontrols were developedfor astronomg chemistrg and physics, the natural sciencesin which in- terest in sciencefirst bloomed. But when scientists started to use experiments in areas such as public health or education, in which extraneous influences are harder to control (e.g., Lind , 1,753lr, they found that the controls used in natural EXPERTMENTSAND CAUSATTONI I sciencein the laboratoryworked poorly in thesenew applications.So they devel- oped new methodsof dealingwith extraneousinfluence, such as random assign- ment (Fisher,1,925) or addinga nonrandomizedcontrol group (Coover& Angell, 1.907).As theoreticaland observationalexperience accumulated across these set- tings and topics,more sourcesof bias wereidentified and more methodswere de- velopedto copewith them (Dehue,2000). TodaSthe key featurecommon to all experimentsis still to deliberatelyvary somethingso asto discoverwhat happensto somethingelse later-to discoverthe effectsof presumedcauses. As laypersonswe do this, for example,to assesswhat happensto our blood pressureif we exercisemore, to our weight if we diet less, or ro our behaviorif we read a self-helpbook. However,scientific experimenta- tion has developedincreasingly specialized substance, language, and tools, in- cluding the practiceof field experimentationin the socialsciences that is the pri- mary focus of this book. This chapter begins to explore these matters by (1) discussingthe natureof causationthat experimentstest, (2) explainingthe spe- cializedterminology (e.g., randomized experiments, quasi-experiments) that de- scribessocial experiments,(3) introducing the problem of how to generalize causalconnections from individual experiments,and (4) briefly situatingthe ex- perimentwithin a largerliterature on the nature of science. EXPERIMENTSAND CAUSATION A sensiblediscussion of experimentsrequires both a vocabularyfor talking about causationand an understandingof key conceptsthat underliethat vocabulary. DefiningCause, Effect, and CausalRelationships Most peopleintuitively recognizecausal relationships in their daily lives.For in- stance,you may say that another automobile'shitting yours was a causeof the damageto your car; that the number of hours you spentstudying was a causeof your testgrades; or that the amountof food a friend eatswas a causeof his weight. You may evenpoint to more complicatedcausal relationships, noting that a low test gradewas demoralizing,which reducedsubsequent studying, which caused evenlower grades.Here the samevariable (low grade)can be both a causeand an effect,and there can be a reciprocal relationship betweentwo variables(low gradesand not studying)that causeeach other. Despitethis intuitive familiarity with causalrelationsbips, a precisedefinition of causeand effecthas eluded philosophers for centuries.lIndeed, the definitions 1. Our analysisrefldcts the useof the word causationin ordinary language,not the more detaileddiscussions of causeby philosophers.Readers interested in suchdetail may consulta host of works that we referencein this chapter,including Cook and Campbell(1979). 4 | 1. EXPERTMENTSAND GENERALTZEDCAUSAL INFERENCE of termssuch as cause and,effectdepend partly on eachother and on the causal relationshipin which both are embedded.So the 17th-centuryphilosopher "That John Locke said: which producesany simpleor complexidea, we denoteby the generalname caLtse, and that which is produced, effect" (1,97s, p. 32fl and also: " A cAtrseis that which makesany other thing, either simpleidea, substance,or mode,begin to be; and an effectis that, which had its beginningfrom someother thing" (p. 325).Since then, other philosophers and scientistshave given us useful definitionsof the threekey ideas--cause,effect, and causalrelationship-that are more specificand that betterilluminate how experimentswork. Wewould not de- fend any of theseas the true or correctdefinition, given that the latter haseluded philosophersfor millennia;but we do claignthat theseideas help to clarify the sci- entific practiceof probing causes. Cause 'We Considerthe causeof a forest fire. know that fires start in differentways-a match tossedfrom a ca\ a lightning strike, or a smolderingcampfire, for exam- ple. None of thesecauses is necessarybecause a forest fire can start evenwhen,