sign in sign up
<<
Home , Blinded experiment

Experimental Methods in the Social Sciences

Introduction to

August 5, 2013 Experiments Increasingly Important in the Social Sciences

· Field experiments in political science as early as 1920s

· Growing use of experiments in development economics, transportation, education, criminology,

· Global experiments in economic development, alternative energy sources, education

2 Increase in Articles Using Experiments in American Political Science Review

3 Why More Experiments Now?

· Better technology

· Maturation of social sciences

· Increased understanding of the complexity and interdependence of research in social sciences

· Social sciences catching up to physical sciences and medicine

4 Key Elements of Experiments

· Standardization

·

· Treatment versus Control Groups

· Between-Subjects versus Within-Subjects Design

· Internal versus External

· Experimental

5 Standardization

· The Dependent Variable (DV) – the thing to be explained – is measured the same way in the same context across subjects

· Independent Variables (IVs) – the explanatory variables – also controlled and standardized

· Compared to field studies, experiments eliminate or control unmeasured influences on the DV and IVs 6 Treatment versus Control

· Treatment is the Independent Variable that the experimenter manipulates.

· The Treatment Group receives the treatment.

· The Control Group that does not receive the treatment

· An may contain multiple treatments and a control group.

7

· In a blind experiment, subjects do not know whether they received the treatment or control.

· A is a false treatment, meant to make the control group believe they received the treatment.

· In a double-blind experiment, the experiment administrators do not know who received the treatment or control

8 Randomization

· The key feature of an experiment is randomization

· In laboratory and survey experiments, subjects assigned randomly to treatment and control groups

· If subjects volunteer for either treatment or control, we cannot be sure that different outcomes are due to or to the treatment 9 Randomization

· A simple randomizing device such as a coin flip can assign subjects to treatment or control.

· An n-factor design includes n simultaneous treatments.

· Each subject randomized across the n treatments.

10 Factorial Design

· Factorial design indicates the number of different treatments and conditions per treatment

· Each subject receives a combination of treatments (or controls)

· For example, a researcher may want to test whether a combination of: vitamins (vitamin or no vitamin) exercise (exercise or no exercise drugs (Drug A, Drug B, no drug) affect a subject’s health 11 · A 2x2x3 factorial design indicates three separate treatments.

· The first and second have two conditions (such as a treatment and control)

· The third has three conditions (such as a control and two treatments)

· The 2x2x3 design breaks subjects into 12 different groups for analysis.

12 Between-Subjects versus Within Subjects

· Between Subjects breaks subjects into treatment and control groups to measure differences between the two groups after treatment.

· Within Subjects design measures subjects before treatment and after treatment. Each subject is its own control.

· Within Subjects also a Pre-post design.

13 Randomized Controlled Trial (RCT)

· The most powerful experimental technique in scientific research

· RCT’s are usually necessary in a for medical treatments

· Randomized Trial similar to RCT but does not contain a control group

14 RCT’s Classified by Study Design

1. Parallel Group: Each subject randomly assigned to a group. All subjects in a group receive the same treatment (or control)

2. Crossover: Each subject receives or does not receive the treatment in a random sequence

15 3. Cluster: Pre-existing groups of subjects are selected to receive the treatment (or control). Groups may include schools, classes, villages, city blocks, provinces, countries.

4. Factorial: Each subject randomly assigned to a group that receives combinations of treatments (or controls).

16 Standard Reporting Flowchart for Parallel RCT

17 18 Example: Dartmouth SPORT Study (Weinstein, et al.)

· Spine Patients Outcome Research Trial (SPORT)

· RCT to assess whether surgery or conservative care (rest, physical therapy, drugs) provided better outcomes for herniated disc, spondyliosis, and other ailments of the spine

· Published in the Journal of the American Medical Association, 2007, 2008

19 Surgical vs. Nonoperative Treatment for Lumbar Disc Herniation • Weinstein et al 2791

2,720 Patients were screened

729 Patients were ineligible

426 Were not surgical candidates 19 Had fracture, infection, or deformity 129 Had inadequate non-operative treatment 20 Had cancer 135 Had other reasons

1,991 Patients were eligible

747 Patients declined to participate

501 enrolled in the randomized cohort 743 enrolled in the observational cohort

256 were assigned to 222 chose non-surgical 245 were assigned to surgery 521 chose surgery non-surgical treatment treatment

203 Were available at 6 wk 219 Were available at 6 wk 464 Were available at 6 wk 197 Were available at 6 wk 40 Missed the follow-up visit 37 Missed the follow-up visit 57 Missed the follow-up visit 24 Missed the follow-up visit 2 Withdrew 0 Withdrew 0 Withdrew 1 Withdrew 0 Died 0 Died 0 Died 0 Died

75 (31%) Had undergone surgery 46 (18%) Had undergone surgery 471 (90%) Had undergone surgery 4 (2%) Had undergone surgery

198 Were available at 3 mo 211 Were available at 3 mo 434 Were available at 3 mo 187 Were available at 3 mo 45 Missed the follow-up visit 44 Missed the follow-up visit 84 Missed the follow-up visit 34 Missed the follow-up visit 2 Withdrew 1 Withdrew 2 Withdrew 1 Withdrew 0 Died 0 Died 1 Died 0 Died

116 (47%) Had undergone surgery 73 (29%) Had undergone surgery 489 (94%) Had undergone surgery 19 (9%) Had undergone surgery

200 Were available at 6 mo 210 Were available at 6 mo 443 Were available at 6 mo 187 Were available at 6 mo 37 Missed the follow-up visit 41 Missed the follow-up visit 70 Missed the follow-up visit 33 Missed the follow-up visit 8 Withdrew 5 Withdrew 7 Withdrew 2 Withdrew 0 Died 0 Died 1 Died 0 Died

133 (54%) Had undergone surgery 96 (38%) Had undergone surgery 492 (94%) Had undergone surgery 35 (16%) Had undergone surgery

202 Were available at 1 yr 213 Were available at 1 yr 448 Were available at 1 yr 189 Were available at 1 yr 29 Missed the follow-up visit 27 Missed the follow-up visit 56 Missed the follow-up visit 28 Missed the follow-up visit 14 Withdrew 15 Withdrew 16 Withdrew 5 Withdrew 0 Died 1 Died 1 Died 0 Died

139 (57%) Had undergone surgery 106 (41%) Had undergone surgery 493 (95%) Had undergone surgery 44 (20%) Had undergone surgery

187 Were available at 2 yr 191 Were available at 2 yr 429 Were available at 2 yr 192 Were available at 2 yr 35 Missed the follow-up visit 36 Missed the follow-up visit 48 Missed the follow-up visit 14 Missed the follow-up visit 23 Withdrew 27 Withdrew 43 Withdrew 15 Withdrew 0 Died 2 Died 1 Died 1 Died

141 (58%) Had undergone surgery 110 (43%) Had undergone surgery 493 (95%) Had undergone surgery 48 (22%) Had undergone surgery

382 Were available at 3 yr 175 Were available at 3 yr 180 Were available at 3 yr 170 Were available at 3 yr 76 Missed the follow-up visit 24 Missed the follow-up visit 35 Missed the follow-up visit 47 Missed the follow-up visit 60 Withdrew 22 Withdrew 29 Withdrew 37 Withdrew 3 Died 1 Died 1 Died 2 Died 493 (95%) Had undergone surgery 52 (23%) Had undergone surgery 142 (58%) Had undergone surgery 111 (43%) Had undergone surgery

149 Were available at 4 yr 150 Were available at 4 yr 342 Were available at 4 yr 166 Were available at 4 yr 47 Missed the follow-up visit 46 Missed the follow-up visit 100 Missed the follow-up visit 28 Missed the follow-up visit 33 Withdrew 43 Withdrew 76 Withdrew 27 Withdrew 1 Died 2 Died 3 Died 1 Died 15 Had visits pending 15 Had visits pending 493 (95%) Had undergone surgery 53 (24%) Had undergone surgery 144 (59%) Had undergone surgery 115 (45%) Had undergone surgery Figure 1. Exclusion, enrollment, randomization, and follow-up of trial participants. The values for surgery, withdrawal, and death are cumulative over 4 years. For example, a total of 1 patient in the group assigned to surgery died during the 4-year follow-up period. [Data set 04/10/2008]. SPORT Study Has Many of the Same Problems Social Scientists Encounter

· Many subjects did not comply. Some assigned to surgery opted for conservative care. Some assigned to conservative care opted for surgery

· Many subjects could not re-contacted

· Potential placebo effect from surgery

· Some outcomes measured using surveys: People were asked how they felt 21 Internal versus

· Internally validity asks, “Did the experimental treatment make a difference in this specific experiment.”

· External validity asks, “To what populations, settings, treatment variables, and measurement variables can this effect be generalized?”

22 Threatened by...

· History

· Intra-experiment events (intrasession history)

· Selection bias in subjects

· Maturation of subjects

· Performance effects (conditioning)

· Regression toward the

· Mortality or attrition of subjects 23 External Validity Threatened by...

· Uncontrolled interactions, omitted variables, spurious correlations

· Testing may make subjects more sensitive than rest of population to variables under investigation

· Unrepresentative samples (college students in particular)

24 External Validity Threatened by...

· Hawthorne effects: subjects change their behavior when observed

· Professionalized subjects (Panel conditioning: experiment may change subjects)

· Mortality or attrition of subjects

25 Experimental Bias

· Expectancy Effects: the experimenter influences subjects

· Demand Effects: Subjects anticipate the purpose of the experiment

· Experimenter Bias. in recruiting subjects, selecting experiment time and place, coding and analyzing data.

26 Ethical Issues

· No physical, financial, emotional harm

· Subjects must give informed consent

· Subjects should not reveal information that is illegal or threatening to them

· Deceiving subjects acceptable as long as they are debriefed

27 Common Types of Experiments

· Laboratory experiments

· Field experiments

· Survey experiments

· Natural experiments

· Quasi-experiments: usually non-random

28 Other Thoughts

· “You can solve only one hard problem at a time.”

· Think about causation and mechanisms

· Start with a theory. Experiments test theories.

· Proof is theoretical. Confirmation is empirical.

· Read across academic disciplines

· To paraphrase Marvin Minsky: for any problem (research question), the best solution (experiment) is the one that uses the least time, energy, resources 29