Field Experiments: Design and Policy
Total Page:16
File Type:pdf, Size:1020Kb
Field Experiments: Design and Policy Lecture 1 HSE, 6.10.2014 Dagmara Celik Katreniak Course Overview • Part I: Field Experiments – Introduction to Randomized Control Trials (RCTs) • Why to randomize? • Field versus Lab Experiments – Experimental Design • Types of designs, designing stages • Implementation of an RCT • Implementation issues (attrition, spillover, selection, …) – Data Analysis • Treatment effects Course Overview • Part II: Topics in Development Economics – Field experiments in health – Field experiments in education – Field experiments in consumer choice – Field experiments in labor economics – Field experiments in microfinance – Field experiments in credit and savings Outcome • Proposal – Topic closest to your interest – Literature review – Arguments for the proposed project – Methodology – Sample size and power calculation – Budget – Logistics and Implementation, suggested solutions for possible implementation issues Expectations and Evaluation • Attendance (5% of final score) • Reading list • Quizzes (15% of final score) • Midterm examination (20% of final score) • Final examination (30% of final score) • Proposal presentation (first draft) – Individual or in pairs • Proposal (final draft, 30% of final score) Contact • Office hours: on request • Email: [email protected] • Course website: hopefully soon Today • Experimental Economics • Types of Experiments – Laboratory Experiments – Natural Experiments – Field Experiments – Laboratory Experiments in the Field • Program Evaluations • Randomized Control Trials • Randomization “Split or Steal” • British Game • Jackpot £100.150 SPLIT STEAL SPLIT £50.075, £50.075 £ 0, £100.150 STEAL £100.150, £ 0 £ 0, £ 0 • What would be the equilibrium? • What behavior do you expect? Experimental Economics Application of experimental methods … – i.e. observing an event in controlled circumstances: • Control over a variable manipulated by the experimenter • Control over background variables set by the experimenter … to studying economic questions • Do people behave as if they maximize utility? • Is there any space for other regarding preferences? • What is the most appropriate mechanism design? • Do poor behave differently compared to rich? Falk, Econometrica (2007) Economics as Experimental Discipline • Natural sciences rely heavily on controlled experiments – e.g. Large Hadron Collider in particle physics, clinical trials in biology and medicine, etc. Economics as Experimental Discipline • Economics – Has relied mostly on naturally occurring processes – First (?) experiment in economics: • Bernoulli (1738) on the St. Petersburg paradox? – Initial pot of $2 doubles if head appears, game ends if tail, what is the fair entrance fee to the game? – Experiments in the lab: • Market vs. game vs. individual decision-making experiments • Chamberlin (late 1940s) and Smith (late 1950s) Experimental Economics • To test theory – Existence of Giffen good – Individual choice • To establish causality – The effect of deworming pill – The effect of mosquito net distribution • Fact-finding – Investigation of gender differences in attitudes toward risk and competition • Whispering in the Ears of Princess – Metaphor by Roth, A.: the intention of the experimental economics to provide advice to policy makers Jensen & Miller, AER (2010) BIGGEST TASK TO FIND A SUITABLE COUNTERFACTUAL BIG QUESTION - VALIDITY • Internal Validity – Our confidence in the causal relationship we measure – Randomization (!!!) • External Validity – Our confidence in extrapolation of our results to real settings / generalizability of results • It is a question of a design and data collection Experiments (1) • Division by choice of counterfactual • Randomized Experiments – Random assignment used – Laboratory, field experiments • Quasi-experiments – Non-random assignment but counterfactual exists – Natural experiments, pre-post, etc. • Non-experiments – Non-random assignment and no counterfactual Experiments (2) • Division by implementation • Laboratory experiments – employs a standard subject pool of students, an abstract framing, and an imposed set of rules – directly construct control groups via randomization • Field experiments – subjects are in their natural environment – directly construct control groups via randomization • Natural experiments – subjects are in their natural environment – find naturally occurring comparison group to mimic control group and assume that selection into treatment is random • Laboratory experiments in the field Laboratory Experiments • Advantages – Controlled environment – Ability to observe counterfactual scenarios – Ability to test novel designs • Disadvantages – External validity – Non-representativeness of subject pools – Heterogeneity in individual preferences and norms – Potential unobserved loss of control What makes a good experiment? • Should an experiment replicate reality? – No! We have happenstance data for that. – a good experiment captures most relevant features of reality in a simple, carefully controlled environment • Should experiment replicate a formal model? – No! We have theory for that. – a good experiment is designed to test specific hypotheses, derived from theory or previous empirical observations Take control over background variables • Experimenter should control all elements of the environment in which the experiment takes place: – determine the rules: what choices are available to subjects, when decisions are made, and what the consequences of these decisions are – control subjects’ payoffs as a function of actions they take – control the information available to subjects Manipulate key variables • Between treatments, experimenter only changes variables which are directly relevant to the hypothesis being tested, otherwise holding the environment fixed: – control vs. treatment groups – no confounds (Don’t change more than one thing at a time) – variables that cannot be directly controlled are typically controlled via randomization Steps in Designing an Experiment • Identify an interesting question – issues that are better addressed in a controlled experiment than by gathering happenstance data • Formulate research hypotheses – Ex1: Increased payoffs → fewer mistakes in logic problems – Ex2: Face-to-face interaction → greater role of fairness • Design a simple environment for testing hypotheses – the more complicated the environment is, the more likely you are to lose control Issues in Designing a Lab Experiment • Test hypotheses by varying a small number of variables within the experiment – if possible, vary one variable at a time • Within- vs. between-subject designs – within-subject: concerns about order effects – between-subject: concerns about subjects’ heterogeneity • Reasons for order effects: – subjects get bored – subjects get confused – subjects gain experience Rules widely accepted by experimental economists – lab experiments • No deception! – if you deceive subjects, what is going to happen the next time you try to run an experiment? • Anonymity – in most experiments, subjects are guaranteed that nobody other than researcher will be able to ever identify their actions or payoffs – in some experiments, subjects are guaranteed that even researchers will be unable to match their actions and payoffs with their real names – in some experiments, abandoning anonymity is an important part of the design, but subjects have the option of withdrawing if they don’t want information to be revealed Where and how to get subjects • What population to use as subjects – undergraduate university students are the easiest to get – subjects with relevant experience are more interesting • How to get subjects – place advertisements on posters, webpages, newspapers – spamming is generally not good • What should be in advertisement – description, expected payoffs and duration, contact info – stress monetary payoffs A Typical Advertisement Earn Money in Economics Experiments The Center for Institutional Studies is looking for participants to take part in experiments. Experiments are an important tool of academic research in economics. They help us to better understand the way the economy works. A typical experiment takes about 2 hours and is held at our computerized laboratory in the center at Myastnitskaya. Participants earn money by making decisions. This can involve, for example, buying and selling a good in a market. How much you earn in an experiment depends, for example, on the prices at which you buy and sell. No special knowledge of economics, mathematics, or computers is required to participate. However, a good command of the language of the experiment is necessary to understand instructions. You may register to participate in experiments carried out in English and/or Russian - the only requirement is a sufficient understanding of the language. The Center organizes experiments on a regular basis. To participate in an experiment you must first register. Registering means that you are in principle interested in participating. We will then send you invitations for specific experiments which you may accept or decline. Select Subjects Carefully • Avoid unintentional selection of subject population • Avoid clustering (assign to treatments randomly) • Anticipate that only about 80% of subjects will show up • Never use a subject twice for the same set of experiments Information Provided in Experimental Instructions • Detailed description of the rules – The main goal is to make certain that subjects understand the rules. If they do not, you have immediately lost control. – Instructions