Using Randomized Control Trials in Economic Research

HU-GESA - Technical Seminar Series Howard University

Prof. Jevay Grooms April 8, 2020

1 / 36 Randomized Controlled Trials (RCTs)

RCT Defined An experiment which randomly assigns an intervention to a target population in an effort to circumvent any bias caused by unobserved characteristics.

it is seen as close to a scientific study as economists can get

JPAL - RCT quickly explained (video)

2 / 36 Impossible to answer without controlling for other covariates (school resources, textbook, teacher quality...).

Example from JPAL

Does intensive math tutoring boost test scores?

3 / 36 Example from JPAL

Does intensive math tutoring boost test scores?

Impossible to answer without controlling for other covariates (school resources, textbook, teacher quality...).

4 / 36 randomized controlled trial.

Example cont

How can we test the impact of intensive math tutoring on tests scores in a scientific way?

5 / 36 Example cont

How can we test the impact of intensive math tutoring on tests scores in a scientific way? randomized controlled trial.

6 / 36 Example cont

How can we test the impact of intensive math tutoring on tests scores in a scientific way? randomized controlled trial.

7 / 36 Example Explained

Why use an RCT here?

not knowingly harming any participants (Belmont principle of justice) allows policymakers to test a widespread policy on a smaller scale mitigate the costs of a widespread policy if the intervention/policy is found to be insignificant

8 / 36 9 / 36 Types of Experiments

1. Controlled (Randomization - RCTs) - randomly assign some treatment to a subset and then observe the outcomes of an intervention (or policy). Researchers are directly involved in introducing the intervention/policy.

Examples of Randomized Controlled Trials

RAND Health Insurance Experiment - what if we provided people with free health insurance? Turning a Shove into a Nudge? A “Labeled Cash Transfer” for Education, Benhassine et al (2014)

10 / 36 Types of Experiments cont.

2. Observational - nothing is purposely introduced but the outcomes of an intervention (or policy) are observed. There may or may not be any randomization introduced in the study. Researchers are not directly involved in introducing the intervention/policy only in observing the outcomes. Although researchers don’t implement the study sometimes an observational experiment can closely resemble that of an RCT, these are called natural experiments. Some level of randomization was introduced naturally.

11 / 36 12 / 36 Examples of Observational Experiments

Natural Experiments Lifetime Earnings and the Vietnam Era Draft Lottery: Evidence from Social Security Administrative Records, Angrist (1990) Oregon Health Insurance Experiment - while treatment is randomly assigned, the researchers did not design the randomization

Less Random Observational Studies/Topics: Ban the Box, Convictions, and Public Employment, Craigie (2019) Examining Medicaid Expansion and the Treatment of Substance Use Disorders, Grooms & Ortega (2019) free busing programs

13 / 36 Concluding - types of experiments

Ideally both studies have clearly defined treatment and control groups with easily identifiable distinctions between covariates and outcomes. And treatment & control groups are comparable at baseline.

Find an intro to the differences in experimental designs here.

14 / 36 How exactly are RCTs implemented?

Now lets take a moment to think about how an RCT is implemented?

15 / 36 But first, how is it accomplished?

1 define two groups: treatment & control 2 observe outcomes of interest prior to intervention (observe at baseline) 3 apply an intervention (or policy) 4 observe outcomes after intervention 5 use control to estimate effect of the intervention on the treatment group

RCTs

Researchers use RCTs as a clean and clear way to get at causal inference.

16 / 36 RCTs

Researchers use RCTs as a clean and clear way to get at causal inference.

But first, how is it accomplished?

1 define two groups: treatment & control 2 observe outcomes of interest prior to intervention (observe at baseline) 3 apply an intervention (or policy) 4 observe outcomes after intervention 5 use control to estimate effect of the intervention on the treatment group

17 / 36 RCTs - visually

Example: how to model a RCT for a fictitious “back to work” program

cite: Guide for using RCTs to test public policies in the UK.

18 / 36 Main elements of RCT

Researchers who use RCTS are interested in: effect of causes, not causes of the effect the causal effect of a policy or intervention the counterfactual

19 / 36 Average causal effect

Average treatment effect is estimated using the Rubin Causal Model

E[Yi (1)|T = 1] − E[Yi (0)|T = 0]

More on ‘treatment effect’ can be found here

Effect of Causes

Not interested in the entire model. Simply interested in the effect of the intervention, not understanding why it is this intervention had this effect.

20 / 36 Effect of Causes

Not interested in the entire model. Simply interested in the effect of the intervention, not understanding why it is this intervention had this effect.

Average causal effect

Average treatment effect is estimated using the Rubin Causal Model

E[Yi (1)|T = 1] − E[Yi (0)|T = 0]

More on ‘treatment effect’ can be found here

21 / 36 Counterfactual

Counterfactual defined... In simplest terms - “What would have happened?” More specifically it is the unobserved outcome which would have occurred in the absence of an intervention (policy).

Counterfactual is used in various ways in economic research. The next 2 slides will use the difference-in-difference (DID) econometric technique to visually explain.

22 / 36 Counterfactual DID - visually

23 / 36 Calculating the Average Treatment Effect in DID

24 / 36 RCT Summarized

two clear groups are identified (treatment & control) intervention/policy is applied to the treatment group outcomes of interests are measured to estimate the average treatment effect the study is set up such all that covariates which may impact the outcome variable are controlled for.

25 / 36 Critics of the widespread use of RCTs

RCTs are great, but a few economists have pointed out, most notably Angus Deaton, they are not perfect.

This came after some began to call RCTs the “gold standard” for causal inference. Two of Deaton’s papers illustrating his points are below: Understanding and Misunderstanding Randomized Controlled Trials Randomization in the tropics revisited: a theme and eleven variations

26 / 36 That aside let us take a look at some drawbacks of RCTs which are essential to take into account how practical an RCT might be and also to ensure the validity of a study.

27 / 36 $$$ It could be very costly to implement!

Second, consider Controlled vs Clinical Trials Some of the drawback of RCTs are common with clinical trials adequately choosing the targeted population selection bias is the intervention translatable (RCT results might be too narrow to be applied on a large scale)

Context is ESSENTIAL!

Drawbacks

First...

28 / 36 Second, consider Controlled vs Clinical Trials Some of the drawback of RCTs are common with clinical trials adequately choosing the targeted population selection bias is the intervention translatable (RCT results might be too narrow to be applied on a large scale)

Context is ESSENTIAL!

Drawbacks

First... $$$ It could be very costly to implement!

29 / 36 Drawbacks

First... $$$ It could be very costly to implement!

Second, consider Controlled vs Clinical Trials Some of the drawback of RCTs are common with clinical trials adequately choosing the targeted population selection bias is the intervention translatable (RCT results might be too narrow to be applied on a large scale)

Context is ESSENTIAL!

30 / 36 esp in poorer/less developed countries. researchers have historically used people they deemed more disposable to run experiments on minorities, prisoners, populations of developing countries

Drawbacks cont

Third, and most important... ethical dilemmas,

31 / 36 Drawbacks cont

Third, and most important... ethical dilemmas, esp in poorer/less developed countries. researchers have historically used people they deemed more disposable to run experiments on minorities, prisoners, populations of developing countries

32 / 36 Ethical Dilemmas

3 main ethical principles in studies: 1 respect for persons 2 beneficence 3 justice

Unethical examples 1 Stanford Prison Experiment 1971 2 Tuskegee Study 1932 -1974 3 Several examples of unethical trials in the developing world

33 / 36 Conclusion

While RCTs are extremely useful, they have a time and place. Before employing an RCT it is essential researchers consider all the pros and cons to ensure the experiment isn’t unethical or design in a way which provide bias or results which are too narrow.

It is also important to note observational studies also have a place and time.

34 / 36 Resources

Please note there are a lot of live links embedded in the slide deck. Below are additional resources I thought may be useful.

An Introduction to the “Handbook of Field Experiments” Esther Duflo, MIT Lecture on Randomized Controlled Trials and Policy Making in Developing Countries Using Administrative Data for Randomized Evaluations Difference-in-Difference Estimation in Population Health The Belmont Report - principles of ethical studies

35 / 36 Questions

Thank You

If you have any questions or comments please feel free to email me. [email protected]

36 / 36