sign in sign up
<<
Home , Social experiment

PSYCHOLOGICAL BULLETIN VOL. 54, No. 4, 1957

FACTORS RELEVANT TO THE OF IN SOCIAL SETTINGS1 DONALD T. CAMPBELL Northwestern University

What do we seek to control in ex- the process of analyzing three pre- perimental designs? What extraneous experimental designs. In the subse- variables which would otherwise con- quent evaluation of the applicability found our interpretation of the of three true experimental designs, do we wish to rule out? factors leading to external invalidity The present paper attempts a specifi- will be introduced. The effects of cation of the major categories of such these extraneous variables will be extraneous variables and employs considered at two levels: as simple or these categories in evaluating the main effects, they occur independ- validity of standard designs for ex- ently of or in addition to the effects perimentation in the social sciences. of the experimental variable; as Validity will be evaluated in terms interactions, the effects appear in of two major criteria. First, and as a conjunction with the experimental basic minimum, is what can be called variable. The main effects typically : did in fact the ex- turn out to be relevant to internal perimental stimulus make some sig- validity, the effects to ex- nificant difference in this specific in- ternal validity or representativeness. stance? The second criterion is that The following designation for ex- of external validity, representativeness, perimental designs will be used: X or generalizability: to what popula- will represent the exposure of a group tions, settings, and variables can this to the experimental variable or event, effect be generalized? Both criteria the effects of which are to be meas- are obviously important although it ured; 0 will refer to the process of turns out that they are to some ex- observation or measurement, which tent incompatible, in that the con- can include watching what people do, trols required for internal validity listening, recording, interviewing, ad- often tend to jeopardize representa- ministering tests, counting lever de- tiveness. pressions, etc. The Xs and Os in a The extraneous variables affecting given row are applied to the same internal validity will be introduced in specific persons. The left to right di-

1 mension indicates temporal order. A dittoed version of this paper was pri- Parallel rows represent equivalent vately distributed in 1953 under the title "Designs for Social Science Experiments." samples of persons unless otherwise The author has had the opportunity to benefit specified. The designs will be num- from the careful reading and suggestions of bered and named for cross-reference L. S. Burwen, J. W. Cotton, C. P. Duncan, purposes. D. W. Fiske, C. I. Hovland, L. V. Jones, E. S. Marks, D. C. Pelz, and B. J. Underwood, THREE PRE-EXPERIMENTAL DESIGNS among others, and wishes to express his ap- preciation. They have not had the opportu- AND THEIR CONFOUNDED EXTRANE- nity of seeing the paper in its present form, OUS VARIABLES and bear no responsibility for it. The author also wishes to thank S. A. Stouffer (33) and The One-Shot Case Study. As B. J. Underwood (36) for their public en- Stouffer (32) has pointed out, much couragement. social science still uses De- 297 298 DONALD T. CAMPBELL sign 1, in which a single individual or in which all extraneous stimuli are group is studied in detail only once, eliminated. The approximation of and in which the observations are such control in much physical and attributed to exposure to some prior biological research has permitted the situation. satisfactory employment of Design 2. X 0 1. One-Shot Case Study But in social and the other social sciences, if history is con- This design does not merit the title of founded with X the results are gen- experiment, and is introduced only to erally uninterpretable. provide a reference point. The very The second class of variables con- minimum of useful scientific infor- founded with X in Design 2 is here mation involves at least one formal designated as maturation. This covers comparison and therefore at least two those effects which are systematic careful observations (2). with the passage of time, and not, The One-Group Pretest-Posttest De- like history, a function of the specific sign. This design does provide for one events involved. Thus between Oi formal comparison of two observa- and 02 the respondents may have tions, and is still widely used. grown older, hungrier, tireder, etc.,

0i X 02 2. One-Group Pretest-Posttest and these may have produced the Design difference between 0\ and 02, inde- However, in it there are four or five pendently of X. While in the typical categories of extraneous variables left brief experiment in the psychology uncontrolled which thus become rival , maturation is unlikely to explanations of any difference be- be a source of change, it has been a tween 0i and 02, confounded with problem in research in develop- the possible effect of X. ment and can be so in extended ex- The first of these is the main effect periments in and of history. During the time span education. In the form of "spontane- ous remission" and the general between 0i and 02 many events have occurred in addition to X, and the processes of healing it becomes an results might be attributed to these. important variable to control in Thus in Collier's (8) experiment, medical research, psychotherapy, and 2 social remediation. while his respondents were reading Nazi propaganda materials, France There is a third source of fell, and the obtained attitude that could explain the difference be- changes seemed more likely a result of tween Oi and 02 without a recourse to this event than of the propaganda.3 the effect of X. This is the effect of By history is meant the specific event testing itself. It is often true that series other than X, i.e., the extra- persons taking a test for the second experimental uncontrolled stimuli. time make scores systematically dif- Relevant to this variable is the con- ferent from those taking the test for cept of experimental isolation, the the first time. This is indeed the case employment of experimental settings for intelligence tests, where a second may be expected to run as s In line with the central focus on social much as five IQ points higher than psychology and the social sciences, the term the first one. This possibility makes respondent is employed in place of the terms subject, patient, or client. important a distinction between re- 3 Collier actually used a more adequate de- active measures and nonreactive meas- sign than this, an approximation to Design 4. ures. A reactive measure is one VALIDITY OF EXPERIMENTS IN SOCIAL SETTINGS 299 which modifies the phenomenon un- niques must be rated as reactive, as der study, which changes the very shown, for example, by Crespi's (9) thing that one is trying to measure. evidence. In general, any measurement proce- Even within the personality and dure which makes the subject self- attitude test domain, it may be conscious or aware of the fact of the found that tests differ in the degree to experiment can be suspected of being which they are reactive. For some a reactive measurement. Whenever purposes, tests involving voluntary the measurement process is not a part self-description may turn out to be of the normal environment it is prob- more reactive (especially at the inter- ably reactive. Whenever measure- action level to be discussed below) ment exercises the process under than are devices which focus the study, it is almost certainly reactive. respondent upon describing the ex- Measurement of a person's height is ternal world, or give him less latitude relatively nonreactive. However, in describing himself (e.g., 5). It measurement of weight, introduced seems likely that, apart from consid- into an experimental design involving erations of validity, the Rorschach adult American women, would turn test is less reactive than the TAT or out to be reactive in that the process MMPI. Where the reactive nature of of measuring would stimulate weight the testing process results from the reduction. A photograph of a crowd focusing of attention on the experi- taken in secret from a second story mental variable, it may be reduced window would be nonreactive, but a by imbedding the relevant content in news photograph of the same scene a comprehensive array of topics, as might very well be reactive, in that has regularly been done in Hovland's the presence of the photographer attitude change studies (14). It would modify the behavior of people seems likely that with attention to seeing themselves being photo- the problem, observational and meas- graphed. In a factory, production urement techniques can be devel- records introduced for the purpose of oped which are much less reactive an experiment would be reactive, but than those now in use. if such records were a regular part of Instrument decay provides a fourth the operating environment they uncontrolled source of variance which would be nonreactive. An English could produce an Oi-Os difference anthropologist may be nonreactive as that might be mistaken for the effect a participant-observer at an English of X. This variable can be exem- wedding, but might be a highly reac- plified by the fatiguing of a spring tive measuring instrument at a Dobu scales, or the condensation of water nuptials. Some measures are so ex- vapor in a cloud chamber. For psy- tremely reactive that their use in a chology and the social sciences it pretest-posttest design is not usually becomes a particularly acute problem considered. In this class would be when human beings are used as a part tests involving surprise, deception, of the measuring apparatus, as rapid adaptation, or stress. Evidence judges, observers, raters, coders, etc. is amply present that tests of learn- Thus 0i and 02 may differ because ing and memory are highly reactive the raters have become more experi- (35, 36). In the field of opinion and enced, more fatigued, have acquired attitude research our well-developed a different adaptation level, or have interview and attitude test tech- learned about the purpose of the ex- 300 DONALD T. CAMPBELL periment, etc. However infelicitously, at some prior time. (The absence of this term will be used to typify those equivalence of groups is problems introduced when shifts in symbolized by the row of dashes.) measurement conditions are con- This design has its most typical oc- founded with the effect of X, includ- currence in the social sciences, and ing such crudities as having a differ- both its prevalence and its weakness ent observer at Oi and 02, or using a have been well indicated by Stouffer different interviewer or coder. Where (32). It will be recognized as one the use of different interviewers, form of the correlational study. It is observers, or experimenters is un- introduced here to complete the list of avoidable, but where they are used factors. If the Os differ, in large numbers, a sampling equiva- this difference could have come about lence of interviewers is required, with through biased selection or recruit- the relevant N being the N of inter- ment of the persons making up the viewers, not interviewees, except as groups; i.e., they might have differed refined through con- anyway without the effect of X. siderations (18). Frequently, exposure to X (e.g., some A possible fifth extraneous factor mass communication) has been vol- deserves mention. This is statistical untary and the two groups have an regression. When, in Design 2, the inevitable systematic difference on group under investigation has been the factors determining the choice selected for its extremity on 0\, involved, a difference which no Oi-Oj shifts toward the mean will amount of matching can remove. occur which are due to random im- A second variable confounded with perfections of the measuring instru- the effect of X in this design can be ment or random instability within called experimental mortality. Even the population, as reflected in the if the groups were equivalent at some test-retest reliability. In general, re- prior time, Oi and Oa may differ now gression operates like maturation in not because individual members have that the effects increase systemati- changed, but because a biased subset cally with the Oi-Ot time interval. of members have dropped out. This McNemar (22) has demonstrated the is a typical problem in making infer- profound mistakes in interpretation ences from comparisons of the atti- which failure to control this factor tudes of college freshmen and college can introduce in remedial research. seniors, for example. The Static Group Comparison. The third pre-experimental design is the TRUE EXPERIMENTAL DESIGNS Static Group Comparison. The Pretest-Posttest Control Group Design. One or another of the above X Oi ----- 3 considerations led be- ยง Static Group Comparison tween 1900 and 1925 (2, 30) to ex- pand Design 2 by the addition of a In this design, there is a comparison control group, resulting in Design 4. of a group which has experienced X 0i X Oi 4. Pretest-Posttest Control Group with a group which has not, for the Oi 0< Design purpose of establishing the effect of X. In with Design 6, there Because this design so neatly con- is in this design no of certify- trols for the main effects of history, ing that the groups were equivalent maturation, testing, instrument de- VALIDITY OF EXPERIMENTS IN SOCIAL SETTINGS 301 cay, regression, selection, and mor- the same individual or small-ses- tality, these separate sources of vari- sion approximation to simultaneity ance are not usually made explicit. needed for history is required. The It seems well to state briefly the rela- occasional practice of running the tionship of the design to each of these experimental group and control group confounding factors, with particular at different times is thus ruled out on attention to the application of the this ground as well as that of history. design in social settings. Otherwise the observers may have If the differences between Oi and become more experienced, more hur- 0^ were due to intervening historical ried, more careless, the maze more events, then they should also show redolent with irrelevant cues, the up in the 03-04 comparison. Note, lever-tension and friction diminished, however, several complications in etc. Only when groups are effectively achieving this control. If respondents simultaneous do these factors affect are run in groups, and if there is only experimental and control groups one experimental session and one alike. Where more than one experi- control session, then there is no con- menter or observer is used, counter- trol over the unique internal histories balancing experimenter, time, and of the groups. The Oi-Os difference, group is recommended. The balanced even if not appearing in 03-0^, may is frequently useful for be due to a chance distracting factor this purpose (4). appearing in one or the other group. While regression is controlled in Such a design, while controlling for the design as a whole, frequently the shared history or event series, secondary analyses of effects are still confounds X with the unique made for extreme pretest scorers in session history. Second, the design the experimental group. To provide implies a simultaneity of Oi with 03 a control for effects of regression, a and 02 with On which is usually im- parallel analysis of extremes should possible. If one were to try to achieve also be made for the control group. simultaneity by using two experimen- Selection is of course handled by ters, one working with the experi- the sampling equivalence ensured mental respondents, the other with through the employed the controls, this would confound in assigning persons to groups, per- experimenter differences with X (in- haps supplemented by, but not sup- troducing one type of instrument planted by, matching procedures. decay). These considerations make it Where the experimental and control usually imperative that, for a true groups do not have this sort of equiv- experiment, the experimental and alence, one has a compromise design control groups be tested and exposed rather than a true experiment. Fur- individually or in small subgroups, thermore, the 0i~0a comparison pro- and that sessions of both types be vides a check on possible sampling temporally and spatially intermixed. differences. As to the other factors: if matura- The design also makes possible the tion or testing contributed an 0i-02 examination of experimental mor- difference, this should appear equally tality, which becomes a real problem in the 03-04 comparison, and these for experiments extended over weeks variables are thus controlled for their or months. If the experimental and main effects. To make sure the de- control groups do not differ in the sign controls for instrument decay, number of lost cases nor in their 302 DONALD T. CAMPBELL pretest scores, the experiment can be from which these samples were taken judged internally valid on this point, to which one wants to generalize. although mortality reduces the gen- A concrete example will help make eralizability of effects to the original this clearer. In the NORC study of population from which the groups a United Nations cam- were selected. paign (31), two equivalent samples, For these reasons, the Pretest- of a thousand each, were drawn from Posttest Control Group Design has the city's population. One of these been the ideal in the social sciences samples was interviewed, following for some thirty years. Recently, which the city of Cincinnati was however, a serious and avoidable subjected to an intensive publicity imperfection in it has been noted, campaign using all the mass media perhaps first by Schanck and Good- of communication. This included man (2Q). Solomon (30) has expressed special features in the newspapers the point as an interaction effect of and on the radio, bus cards, public testing. In the terminology of anal- lectures, etc. At the end of two ysis of variance, the effects of his- months, the second sample of 1,000 tory, maturation, and testing, as was interviewed and the results com- described so far, are all main effects, pared with the first 1,000. There manifesting themselves in mean dif- were no differences between the two ferences independently of the pres- groups except that the second group ence of other variables. They are was somewhat more pessimistic about effects that could be added on to the likelihood of Russia's cooperating other effects, including the effect of for world peace, a result which was the experimental variable. In con- attributed to history rather than to trast, interaction effects represent a the publicity campaign. The second joint effect, specific to the concomi- sample was no better informed about tance of two or more conditions, and the United Nations nor had it no- may occur even when no main effects ticed in particular the publicity are present. Applied to the testing campaign which had been going on. variable, the interaction effect might In connection with a program of re- involve not a shift due solely or search on panels and the reinterview directly to the measurement process, problem, Paul Lazarsfeld and the but rather a sensitization of respon- Bureau of Applied Social Research dents to the experimental variable so arranged to have the initial sample that when X was preceded by 0 there reinterviewed at the same time as would be a change, whereas both the second sample was interviewed, X and 0 would be without effect if after the publicity campaign. This occurring alone. In terms of the reinterviewed group showed signifi- two types of validity, Design 4 is cant attitude changes, a high degree internally valid, offering an adequate of awareness of the campaign and basis for generalization to other important increases in information. sampling-equivalent pretested groups. The inference in this case is unmis- But it has a serious and systematic takably that the initial interview weakness in representativeness in had sensitized the persons interviewed that it offers, strictly speaking, no to the topic of the United Nations, basis for generalization to the un- had raised in them a focus of aware- pretested population. And it is us- ness which made the subsequent ually the unpretested larger universe publicity campaign effective for them VALIDITY OF EXPERIMENTS IN SOCIAL SETTINGS 303 but for them only. This study and ing by t test the experimental and other studies clearly document the control groups. Extension of this possibility of interaction effects which of analysis to the Solomon seriously limit our capacity to gen- Four-Group Design introduces an eralize from the pretested experi- inelegant awkwardness to the other- mental group to the unpretested wise elegant procedure. It involves general population. Hovland (15) assuming as a pretest score for the reports a general finding which is of unpretested groups the mean value of the opposite nature but is, nonethe- the pretest from the first two groups. less, an indication of an interactive This restricts the effective degrees of effect. In his Army studies the initial freedom, violates assumptions of in- pretest served to reduce the effects of dependence, and leaves one without the experimental variable, presum- a legitimate base for testing the ably by creating a commitment to a significance of main effects and inter- given position. Crespi's (9) findings action. An alternative analysis is support this expectation. Solomon available which avoids the assumed (30) reports two studies with school pretest scores. Note that the four children in which a spelling pretest posttests form a simple two-by-two reduced the effects of a training design: period. But whatever the direction of the effect, this flaw in the Pretest- NoX X Posttest Control Group Design is Pretested Ot 0, serious for the purposes of the social Unpretested Oe scientist. The Solomon Four-Group Design. It The column means represent the is Solomon's (30) suggestion to con- main effect of X, the row means the trol this problem by adding to the main effect of pretesting, and the in- traditional two-group experiment two teraction term the interaction of pre- unpretested groups as indicated in testing and X. (By use of a t test the Design 5. combined main effects of maturation and history can be tested through 0, X 02 comparing Oe with 0\ and 03.) Oi Oi 5. Solomon Four-Group Design The Posttest-Only Control Group X Ot 0, Design. While the statistical proce- dures of analysis of variance intro- This Solomon Four-Group Design duced by Fisher (10) are dominant in enables one both to control and meas- psychology and the other social ure both the main and interaction sciences today, it is little noted in our effects of testing and the main effects discussions of experimental arrange- of a composite of maturation and ments that Fisher's typical agricul- history. It has become the new ideal tural experiment involves no pretest: design for social scientists. A word equivalent plots of ground receive needs to be said about the appro- different experimental treatments priate statistical analysis. In Design and the subsequent yields are meas- 4, an efficient single test embodying ured.4 Applied to a social experiment the four measurements is achieved 4 This is not to imply that the pretest is through computing for each individ- totally absent from Fisher's designs. He sug- ual a pretest-posttest difference gests the use of previous year's yields, etc., in score which is then used for compar- covariance analysis. He notes, however, "with 304 DONALD T. CAMPBELL as in testing the influence of a motion applicants for psychotherapy, time picture upon attitudes, two randomly since A introduces maturation (spon- assigned audiences would be selected, taneous remission) and regression one exposed to the movie, and the factors. In Design 6 these effects attitudes of each measured subse- would be confounded with the effect quently for the first time. of X if the As as well as the Os were A X Oi 6. Posttest-Only Control Group not contemporaneous for experimen- A Oa Design tal and control groups. Like Design 4, this design controls In this design the symbol A had been for the effects of maturation and his- added, to indicate that at a specific tory through the practical simul- time prior to X the groups were made taneity of both the ^4s and the Os. In equivalent by a random sampling superiority over Design 4, no main or assignment. A is the point of selec- interaction effects of pretesting are tion, the point of allocation of in- involved. It is this feature that rec- dividuals to groups. It is the exist- ommends it in particular. While it ence of this process that distinguishes controls for the main and interaction Design 6 from Design 3, the Static effects of pretesting as well as does Group Comparison. Design 6 is not Design 5, the Solomon Four-Group a static cross-sectional comparison, Design, it does not measure these but instead truly involves control and effects, nor the main effect of history- observation extended in time. The maturation. It can be noted that sampling procedures employed assure Design 6 can be considered as the two us that at time A the groups were unpretested "control" groups from equal, even if not measured. A pro- the Solomon Design, and that Solo- vides a point of prior equality just as mon's two traditional pretested does the pretest. A point A is, of groups have in this sense the sole pur- course, involved in all true experi- pose of measuring the effects of pre- ments, and should perhaps be in- testing and history-maturation, a dicated in Designs 4 and 5. It is es- purpose irrelevant to the main aim of sential that A be regarded as a studying the effect of X (25). How- specific point in time, for groups ever, under normal conditions of not change as a function of time since A, quite perfect sampling control, the through experimental mortality. four-group design provides in addi- Thus in a public opinion survey situ- tion greater assurance against mis- ation employing probability sampling takenly attributing to X effects which from lists of residents, the longer the are not due it, inasmuch as the effect time since A, the more the sample of X is documented in three different underrepresents the transient seg- fashions (0\ vs. Oa, 02 vs. On, and Os ments of society, the newer dwelling vs. Oe). But, short of the four-group units, etc. When experimental design, Design 6 is often to be pre- groups are being drawn from a self- ferred to Design 4, and is a fully valid selected extreme population, such as experimental design. annual agricultural crops, knowledge of yields Design 6 has indeed been used in of the experimental area in a previous year the social sciences, perhaps first of all under uniform treatment has not been found in the classic experiment by Gosnell, sufficiently to increase the precision to war- rant the adoption of such uniformity trials as Getting Out the Vote (11). Schanckand a preliminary to projected experiments" (10, Goodman (2Q), Hovland (IS) and p. 176). others (1, 12, 23, 24, 27) have also VALIDITY OF EXPERIMENTS IN SOCIAL SETTINGS 305 employed it. But, in spite of its more than dissipated by the costs and manifest advantages of simplicity loss in experimental result- and control, it is far from being a ing from the requirement of two test- popular design in social research and ing sessions, over and above the indeed is usually relegated to an in- considerations of representativeness. ferior position in discussions of exper- Design 4 has a particular advan- imental designs if mentioned at all tage over Design 6 if experimental (e.g., 15, 16, 32). Why is this the mortality is high. In Design 4, one case? can examine the pretest scores of lost In the first place, it is often con- cases in both experimental and con- fused with Design 3. Even where 5s trol groups and check on their com- have been carefully assigned to ex- parability. In the absence of this in perimental and control groups, one is Design 6, the possibility is opened for apt to have an uneasiness about the a mean difference resulting from dif- design because one "doesn't know ferential mortality rather than from what the subjects were like before." individual change, if there is a sub- This objection must be rejected, as stantial loss of cases. our standard tests of significance are A final objection comes from those designed precisely to evaluate the who wish to study the relationship of likelihood of differences occurring by pretest attitudes to kind and amount chance in such sample selection. It is of change. This is a valid objection, true, however, that this design is and where this is the interest, Design particularly vulnerable to selection 4 or 5 should be used, with parallel bias and where is analysis of experimental and control not possible it remains suspect. groups. Another common type of Where naturally aggregated units, individual difference study involves such as classes, are employed intact, classifying persons in terms of these should be used in large numbers amount of change and finding asso- and assigned at random to the experi- ciated characteristics such as sex, age, mental and control conditions; clus- education, etc. While unavailable in ter sampling statistics (18) should be this form in Design 6, essentially the used to determine the error term. If same correlational information can be but one or two intact classrooms are obtained by subdividing both experi- available for each experimental mental and control groups in terms treatment, Design 4 should certainly of the associated characteristics, and be used in preference. examining the experimental-control A second objection to Design 6, in difference for such subtypes. comparison with Design 4, is that it For Design 6, the Posttest-Only often has less precision. The differ- Control Group Design, there is a ence scores of Design 4 are less vari- class of social settings in which it is able than the posttest scores of optimally feasible, settings which Design 6 if there is a pretest-posttest should be more used than they now correlation above .50 (15, p. 323), are. Whenever the social contact and hence for test-retest correlations represented by X is made to single above that level a smaller mean dif- individuals or to small groups, and ference would be statistically signifi- where the response to that stimulus cant for Design 4 than for Design 6, can be identified in terms of indivi- for a constant number of cases. This duals or type of X, Design 6 can be advantage to Design 4 may often be applied. Direct mail and door-to- 306 DONALD T. CAMPBELL door contacts represent such settings. siderations lead into designs in which The alternation of several appeals multiple groups are used, each with a from door-to-door in a fund-raising different X\, X%, Xs, Xn, or in multi- campaign can be organized as a true ple factorial design, as X\a, Xu, Xza, experiment without increasing the Xu, etc. Applied to Designs 4 and 6, cost of the solicitation. Experimental this introduces one additional group variation of persuasive materials in a for each additional X. Applied to 5, direct-mail sales campaign can pro- The Solomon Four-Group Design, vide a better experimental laboratory two additional groups (one pretested, for the study of mass communication one not, both receiving Xn) would be and persuasion than is available in added for each variant on X. any university. The well-established, In many experiments, X\, X2, Xs, if little-used, split-run technique in and Xn are all given to the same comparing alternative magazine ads group, differing groups receiving the is a true experiment of this type, Xs in different orders. Where the usually limited to coupon returns problem under study centers around rather than sales because of the prob- the effects of order or combination, lem of identifying response with stim- such counterbalanced multiple X ar- ulus type (20). The split-ballot tech- rangements are, of course, essential. nique (7) long used in public opinion Studies of transfer in learning are a polls to compare different question case in point (34). But where one wordings or question sequences pro- wishes to generalize to the effect of vides an excellent example which can each X as occurring in isolation, such obviously be extended to other topics designs are not recommended be- (e.g., 12). By and large these labora- cause of the sizable interactions tories have not yet been used to study among Xs, as repeatedly demon- social science theories, but they are strated in learning studies under such directly relevant to hypotheses about labels as proactive inhibition and social persuasion. learning sets. The use of counter- Multiple X designs. In presenting balanced sets of multiple Xs to the above designs, X has been op- achieve experimental equation, where posed to No-X, as is traditional in natural groups not randomly assem- discussions of experimental design in bled have to be used, will be dis- psychology. But while this may be a cussed in a subsequent paper on legitimate description of the stimulus- compromise designs. isolated physical science laboratory, Testing for effects extended in time. it can only be a convenient shorthand The of Hovland and his in the social sciences, for any No-.X' associates (14, IS) have indicated period will not be empty of po- repeatedly that the longer ef- tentially change-inducing stimuli. fects of persuasive Xs may be quali- The experience of the control group tatively as well as quantitatively might better be categorized as an- different from immediate effects. other type of X, a control experience, These results emphasize the im- an Xc instead of No-X. It is also portance of designing experiments to typical of advance in science that we measure the effect of X at extended are soon no longer interested in the periods of time. As the misleading qualitative fact of effect or no-effect, early research on reminiscence and on but want to specify degree of effect the consolidation of the memory for varying degrees of X. These con- trace indicate (36), repeated measure- VALIDITY OF EXPERIMENTS IN SOCIAL SETTINGS 307 ment of the same persons cannot be studies can be made on the content trusted to do this if a reactive meas- of the voting (6). Essential to this urement process is involved. Thus, design is the ability to create specified for Designs 4 and 6, two separate randomly equated groups, the ability groups must be added for each post- to expose one of these groups to X test period. The additional control while withholding it (or providing group cannot be omitted, or the ef- X%) from the other group, and the fects of intervening history, matura- ability to identify the performance of tion, instrument decay, regression, each individual or unit in the subse- and mortality are confounded with quent 0. Since such measures are the delayed effects of X. To follow natural parts of the environment to fully the logic of Design S, four addi- which one wishes to generalize, they tional groups are required for each are not reactive, and Design 4, the posttest period. Pretest-Posttest Control Group De- True experiments in which O is not sign, is feasible if 0 has a predictable not under E's control. It seems well to periodicity to it. With the precinct call the attention of the social scien- as a unit, this was the design of Hart- tist to one class of true experiments mann's classic study of emotional vs. which are possible without the full rational appeals in a public election experimental control over both the (13). Note that S, the Solomon Four- "when" and "to whom" of both X Group Design, is not available, as it and 0. As far as this analysis has requires the ability to withhold 0 been able to go, no such true experi- experimentally, as well as X. ments are possible without the ability to control X, to withhold it from FURTHER PROBLEMS OF REPRESEN- carefully randomly selected respon- TATIVENESS dents while presenting it to others. The interaction effect of testing, af- But control over 0 does not seem so fecting the external validity or repre- indispensable. Consider the follow- sentativeness of the experiment, was ing design. treated extensively in the previous A X 0, section, inasmuch as it was involved A 02 6. Posttest Only Design, where 0 in the comparison of alternative de- (O) cannot be withheld from any signs. The present section deals with (O) respondent the effects upon representativeness (0) of other variables which, while The parenthetical Os are inserted to equally serious, can apply to any of indicate that the studied groups, ex- the experimental designs. perimental and control, have been The interaction effects of selection. selected from a larger universe all of Even though the true experiments which will get 0 anyway. An election control selection and mortality for provides such an 0, and using internal validity purposes, these fac- "whether voted" rather than "how tors have, in addition, an important voted," this was Gosnell's design bearing on representativeness. There (11). Equated groups were selected is always the possibility that the ob- at time A, and the experimental tained effects are specific to the ex- group subjected to persuasive ma- perimental population and do not terials designed to get out the vote. hold true for the populations to which Using precincts rather than persons one wants to generalize. Defining the as the basic sampling unit, similar universe of reference in advance and 308 DONALD T. CAMPBELL selecting the experimental and con- Design 6 might be expected to require trol groups from this at random less cooperation than Design 4 or S, would guarantee representativeness if especially in the natural individual it were ever achieved in practice. contact setting. The interactive ef- But inevitably not all those so des- fects of experimental mortality are of ignated are actually eligible for selec- similar nature. Note that, on these tion by any contact procedure. Our grounds, the longer the experiment is best survey sampling techniques, for extended in time the more respond- example, can designate for potential ents are lost and the less representa- contact only those available through tive are the groups of the original residences. And, even of those so universe. designated, up to 19 per cent are not Reactive arrangements. In any of contactable for an interview in their the experimental designs, the re- own homes even with five callbacks spondents can become aware that (37). It seems legitimate to assume they are participating in an experi- that the more effort and time required ment, and this awareness can have an of the respondent, the larger the loss interactive effect, in creating reac- through nonavailability and nonco- tions to X which would not occur had operation. If one were to try to as- X been encountered without this semble experimental groups away "I'm a guinea pig" attitude. Lazars- from their own homes it seems rea- feld (19), Kerr (17), and Rosenthal sonable to estimate a 50 per cent se- and Frank (28), all have provided lection loss. If, still trying to extra- valuable discussions of this problem. polate to the general public, one fur- Such effects limit generalizations to ther limits oneself to docile preassem- respondents having this awareness, bled groups, as in schools, military and preclude generalization to the units, studio audiences, etc., the pro- population encountering X with non- portion of the universe systematically experimental attitudes. The direc- excluded through the sampling proc- tion of the effect may be one of ess must approach 90 per cent or negativism, such as an unwillingness more. Many of the selection factors to admit to any persuasion or change. involved are indubitably highly sys- This would be comparable to the tematic. Under these extreme selec- absence of any immediate effect from tion losses, it seems reasonable to discredited communicators, as found suspect that the experimental groups by Hovland (14). The result is prob- might show reactions not characteris- ably more often a cooperative re- tic of the general population. This sponsiveness, in which the respondent point seems worth stressing lest we accepts the experimenter's expecta- unwarrantedly assume that the selec- tions and provides psueudoconfirma- tion loss for experiments is compar- tion. Particularly is this positive able to that found for survey inter- response likely when the respondents views in the home at the respondent's are self-selected seekers after the cure convenience. Furthermore, it seems that X may offer. The Hawthorne plausible that the greater the cooper- studies (21), illustrate such sym- ation required, the more the respon- pathetic changes due to awareness of dent has to deviate from the normal experimentation rather than to the course of daily events, the greater specific nature of X. In some settings will be the possibility of nonrepre- it is possible to disguise the experi- sentative reactions. By and large, mental purpose by providing plausi- VALIDITY OF EXPERIMENTS IN SOCIAL SETTINGS 309 ble facades in which X appears as an sonnel involved, using different fa- incidental part of the background fades, separating the settings and (e.g., 26, 27, 29). We can also make times, and embedding the Jf-relevant more extensive use of experiments content of 0 among a disguising taking place in the intact social variety of other topics.6 situation, in which the respondent is Generalizing to other Xs. After the not aware of the experimentation at internal validity of an experiment has all. been established, after a dependable The discussion of the effects of effect of X upon 0 has been found, selection on representativeness has the next step is to establish the limits argued against employing intact nat- and relevant dimensions of general- ural preassembled groups, but the ization not only in terms of popula- issue of conspicuousness of arrange- tions and settings but also in terms ments argues for such use. The of categories and aspects of X. The machinery of breaking up natural actual X in any one experiment is a groups such as departments, squads, specific combination of stimuli, all and classrooms into randomly as- confounded for interpretative pur- signed experimental and control poses, and only some relevant to the groups is a source of reaction which experimenter's intent and theory. can often be avoided by the use of Subsequent experimentation should preassembled groups, particularly in be designed to purify X, to discover educational settings. Of course, as that aspect of the original conglom- has been indicated, this requires the erate X which is responsible for the use of large numbers of such groups effect. As Brunswik (3) has empha- under both experimental and control sized, the representative sampling of conditions. Xs is as relevant a problem in linking The problem of reactive arrange- experiment to theory as is the sam- ments is distributed over all features pling of respondents. To define a of the experiment which can draw the category of Xs along some dimension, attention of the respondent to the and then to sample Xs for experi- fact of experimentation and its pur- mental purposes from the full range poses. The conspicuous or reactive of stimuli meeting the specification pretest is particularly vulnerable, in- while other aspects of each specific asmuch as it signals the topics and stimulus complex are varied, serves purposes of the experimenter. For to untie or unconfound the defined communications of obviously per- dimension from specific others, lend- suasive aim, the experimenter's topi- ing assurance of theoretical relevance. cal intent is signaled by the X itself, In a sense, the placebo problem can if the communication does not seem be understood in these terms. The a part of the natural environment. Even for the posttest-only groups, ' For purposes of completeness, the inter- the occurrence of the posttest may action of X with history and maturation should be mentioned. Both affect the gen- create a reactive effect. The respon- eralizability of results. The interaction effect dent may say to himself, "Aha, now of history represents the possible specificity of I see why we got that movie." This results to a given historical , a possi- consideration justifies the practice of bility which increases as problems are more societal, less biological. The interaction of disguising the connection between 0 maturation and X would be represented in the and X even for Design 6, as through specificity of effects to certain maturational having different experimental per- levels, fatigue states, etc. 310 DONALD T. CAMPBELL experiment without the placebo has continues the confounding of theory- clearly demonstrated that some as- relevant aspects of X and 0 with pect of the total X stimulus complex specific artifacts of unknown influ- has had an effect; the placebo experi- ence. On the other hand, the con- ment serves to break up the complex fusion in our literature generated by X into the suggestive connotation of the heterogeneity of results from stud- pill-taking and the specific pharma- ies all on what is nominally the cological properties of the drug- "same" problem but varying in im- separating two aspects of the X pre- plementation, is leading some to call viously confounded. Subsequent for exact of initial proce- studies may discover with similar dures in subsequent research on a logic which chemical fragment of the topic. Certainly no science can complex natural herb is most essen- emerge without dependably repeat- tial. Still more clearly, the sham able experiments. A suggested res- operation illustrates the process of X olution is the transition experiment, purification, ruling out general effects in which the need for varying the of surgical shock so that the specific theory-independent aspects of X and effects of loss of glandular or neural 0 is met in the form of a multiple X, tissue may be isolated. As these multiple 0 design, one segment of parallels suggest, once recurrent un- which is an "exact" replication of the wanted aspects of complex Xs have original experiment, exact at least in been discovered for a given field, con- those major features which are nor- trol groups especially designed to mally reported in experimental writ- eliminate these effects can be regu- ings. larly employed. Internal vs. external validity. If one Generalizing to other Qs. In parallel is in a situation where either internal form, the scientist in practice uses a validity or representativeness must complex measurement procedure be sacrificed, which should it be? The which needs to be refined in subse- answer is clear. Internal validity is quent experimentation. Again, this the prior and indispensable considera- is best done by employing multiple tion. The is, of course, Os all having in common the theoret- one having both internal and external ically relevant attribute but varying validity. Insofar as such settings are widely in their irrelevant specificities. available, they should be exploited, For Os this process can be introduced without embarrassment from the ap- into the initial experiment by em- parent opportunistic warping of the ploying multiple measures. A major content of studies by the availability practical reason for not doing so is of laboratory techniques. In this that it is so frequently a frustrating sense, a science is as opportunistic as experience, lending hesitancy, in- a bacteria culture and grows only decision, and a feeling of failure to where growth is possible. One basic studies that would have been inter- necessity for such growth is the preted with confidence had but a machinery for selecting among al- single response measure been em- ternative hypotheses, no matter how ployed. limited those hypotheses may have Transition experiments. The two to be. previous paragraphs have argued against the exact replication of experi- SUMMARY mental apparatus and measurement In analyzing the extraneous vari- procedures on the grounds that this ables which experimental designs for VALIDITY OF EXPERIMENTS IN SOCIAL SETTINGS 311 social settings seek to control, seven ity or generalizability of experimental categories have been distinguished: results. Standard experimental de- history, maturation, testing, instru- signs vary in their susceptibility to ment decay, regression, selection, and these interactive effects. Stress is mortality. In general, the simple or also placed upon the differences main effects of these variables jeopar- among measuring instruments and dize the internal validity of the ex- arrangements in the extent to which periment and are adequately con- they create unwanted interactions. trolled in standard experimental de- The value for social science purposes signs. The interactive effects of these of the Posttest-Only Control Group variables and of experimental ar- Design is emphasized. rangements affect the external valid-

REFERENCES 1. ANNIS, A. D., & MEIER, N. C. The induc- in determining election results. J. tion of opinion through suggestion by abnorm. soc. Psychol., 1936, 31, 99-114. means of planted content. /. soc. 14. HOVLAND, C. E., JANIS, I. L., & KELLEY, Psychol., 1934, 5, 65-81. H. H. Communication and persuasion. 2. BORING, E. G. The nature and history of New Haven: Yale Univer. Press, 1953. experimental control. Amer. J. Psy- 15. HOVLAND, C. I., LUMSDAINE, A. A., & chol., 1954, 67, 573-589. SHEFFIELD, F. D. Experiments on mass 3. BRUNSWIK, E. Perception and the repre- communication. Princeton: Princeton sentative design of psychological experi- Univer. Press, 1949. ments. Berkeley: Univer. of California 16. JAHODA, M., DEUTSCH, M., & COOK, Press, 1956. S. W. Research methods in social rela- 4. BUGELSKI, B. R. A note on Grant's dis- tions. New York: Dryden Press, 1951. cussion of the Latin square principle in 17. KEKR, W. A. Experiments on the effect of the design and analysis of psychological music on factory production. Appl. experiments. Psychol. Bull., 1949, 46, Psychol. Monogr., 1945, No. 5. 49-50. 18. KlSH, L. Selection of the sample. In 5. CAMPBELL, D. T. The indirect assessment L. Festinger and D. Katz (Eds.), Re- of social attitudes. Psychol. Bull., 1950, search methods in the behavioral sciences. 47, 15-38. New York: Dryden Press, 1953, 175- 6. CAMPBELL, D. T. On the possibility of 239. experimenting with the "Bandwagon" 19. LAZARSFELD, P. F. Training guide on the effect. Int. J. Opin. Attitude Res., 1951, controlled experiment in social re- 5, 251-260. search. Dittoed. Columbia Univer., 7. CANTRIL, H. Gauging public opinion. Bureau of Applied Social Research, Princeton: Princeton Univer. Press, 1948. 1944. 20. LUCAS, D. B., & BRITT, S. H. Advertising 8. COLLIER, R. M. The effect of propaganda psychology and research. New York upon attitude following a critical exami- McGraw-Hill, 1950. nation of the propaganda itself. /. soc. 21. MAYO, E. The human problems of an in- Psychol., 1944, 20, 3-17. dustrial civilization. New York: 9. CRESPI, L. P. The interview effect in Macmillan, 1933. polling. Publ. Opin. Quart., 1948, 12, 22. McNEMAR, Q. A critical examination of 99-111. the University of Iowa studies of en- 10. FISHER, R. A. The . vironmental influences upon the IQ. Edinburgh: Oliver & Boyd, 1935. Psychol. Bull, 1940, 37, 63-92. 11. GOSNELL, H. F. Getting out the vote: an 23. MBNEFEE, S. C. An experimental study experiment in the stimulation of voting. of strike propaganda. Soc. Forces, 1938, Chicago: Univer. of Chicago Press, 1927. 16, 574-582. 12. GREENBERG, A. Matched samples. 24. PARRISH, J. A., & CAMPBELL, D. T. J. Marketing, 1953-54, 18, 241-245. Measuring propaganda effects with 13. HARTMANN, G. W. A field experiment on direct and indirect attitude tests. the comparative effectiveness of "emo- /. abnorm. soc. Psychol., 1953, 48, 3-9. tional" and "rational" political leaflets 25. PAYNE, S. L. The ideal model for con- 312 DONALD T. CAMPBELL

trolled experiments. Publ. Opin. Quart., nati plan for the United Nations. 1951, IS, 557-562. Amer. J. Social., 1949-50, 55, 389. 26. POSTMAN, L., &BRUNER,]. S. Perception 32. STOUFFER, S. A. Some observations on under stress. Psychol. Rev., 1948, S5, study design. Amer. J. Social., 1949-50, 314-322. 55, 355-361. 27. RANKIN, R. E., & CAMPBELL, D. T. 33. STOUFFER, S. A. Measurement in sociol- Galvanic skin response to Negro and ogy. Amer. social. Rev., 1953, 18, 591- white experimenters. J. abnorm. soc, 597. Psychol, 1955, 51, 30-33. 34. UNDERWOOD, B. J. Experimental psychol- 28. ROSENTHAL, D., & FRANK, J. O. Psycho- ogy. Appleton-Century-Crofts, 1949. therapy and the placebo effect. Psy- 35. UNDERWOOD, B. J. Interference and for- chol. Bull., 1956, S3, 294-302. getting. Psychol. Rev., 1957, 64, 49-60. 29. SCIIANCK, R. L., & GOODMAN, C. Reac- 36. UNDERWOOD, B. J. Psychological re- tions to propaganda on both sides of a controversial issue. Publ. Opin. Quart., search. New York: Appleton-Century- 1939, 3, 107-112. Crofts, 1957. 30. SOLOMON, R. W. An extension of control 37. WILLIAMS, R. Probability sampling in the group design. Psychol. Bull., 1949, 46, field: a case history. Publ. Opin. Quart., 137-150. 1950, 14, 316-330. 31. STAR, S. A., & HUGHES, H. M. Report on an educational campaign: the Cincin- Received October 7, 1956