Field Experiments: Design and Policy

Lecture 1 HSE, 6.10.2014 Dagmara Celik Katreniak Course Overview

• Part I: Field Experiments – Introduction to Randomized Control Trials (RCTs) • Why to randomize? • Field versus Lab Experiments – Experimental Design • Types of designs, designing stages • Implementation of an RCT • Implementation issues (attrition, spillover, selection, …) – Data Analysis • Treatment effects Course Overview

• Part II: Topics in Development Economics – Field experiments in health – Field experiments in education – Field experiments in consumer choice – Field experiments in labor economics – Field experiments in microfinance – Field experiments in credit and savings Outcome • Proposal – Topic closest to your interest – Literature review – Arguments for the proposed project – Methodology – Sample size and power calculation – Budget – Logistics and Implementation, suggested solutions for possible implementation issues Expectations and Evaluation

• Attendance (5% of final score) • Reading list • Quizzes (15% of final score) • Midterm examination (20% of final score) • Final examination (30% of final score) • Proposal presentation (first draft) – Individual or in pairs • Proposal (final draft, 30% of final score) Contact

• Office hours: on request • Email: [email protected] • Course website: hopefully soon Today

• Experimental Economics

• Types of Experiments – Laboratory Experiments – Natural Experiments – Field Experiments – Laboratory Experiments in the Field

• Program Evaluations

• Randomized Control Trials

• Randomization “Split or Steal”

• British Game • Jackpot £100.150

SPLIT STEAL SPLIT £50.075, £50.075 £ 0, £100.150 STEAL £100.150, £ 0 £ 0, £ 0

• What would be the equilibrium? • What behavior do you expect? Experimental Economics

Application of experimental methods … – i.e. observing an event in controlled circumstances: • Control over a variable manipulated by the experimenter • Control over background variables set by the experimenter … to studying economic questions • Do people behave as if they maximize utility? • Is there any space for other regarding preferences? • What is the most appropriate mechanism design? • Do poor behave differently compared to rich? Falk, Econometrica (2007) Economics as Experimental Discipline

• Natural sciences rely heavily on controlled experiments – e.g. Large Hadron Collider in particle physics, clinical trials in biology and medicine, etc.

Economics as Experimental Discipline

• Economics – Has relied mostly on naturally occurring processes – First (?) experiment in economics: • Bernoulli (1738) on the St. Petersburg paradox? – Initial pot of $2 doubles if head appears, game ends if tail, what is the fair entrance fee to the game? – Experiments in the lab: • Market vs. game vs. individual decision-making experiments • Chamberlin (late 1940s) and Smith (late 1950s)

Experimental Economics

• To test theory – Existence of Giffen good – Individual choice • To establish causality – The effect of deworming pill – The effect of mosquito net distribution • Fact-finding – Investigation of gender differences in attitudes toward risk and competition • Whispering in the Ears of Princess – Metaphor by Roth, A.: the intention of the experimental economics to provide advice to policy makers Jensen & Miller, AER (2010) BIGGEST TASK

TO FIND A SUITABLE COUNTERFACTUAL BIG QUESTION - VALIDITY

• Internal Validity – Our confidence in the causal relationship we measure – Randomization (!!!) • External Validity – Our confidence in extrapolation of our results to real settings / generalizability of results

• It is a question of a design and data collection Experiments (1)

• Division by choice of counterfactual • Randomized Experiments – Random assignment used – Laboratory, field experiments • Quasi-experiments – Non-random assignment but counterfactual exists – Natural experiments, pre-post, etc. • Non-experiments – Non-random assignment and no counterfactual Experiments (2) • Division by implementation • Laboratory experiments – employs a standard subject pool of students, an abstract framing, and an imposed set of rules – directly construct control groups via randomization • Field experiments – subjects are in their natural environment – directly construct control groups via randomization • Natural experiments – subjects are in their natural environment – find naturally occurring comparison group to mimic control group and assume that selection into treatment is random • Laboratory experiments in the field Laboratory Experiments

• Advantages – Controlled environment – Ability to observe counterfactual scenarios – Ability to test novel designs • Disadvantages – External validity – Non-representativeness of subject pools – Heterogeneity in individual preferences and norms – Potential unobserved loss of control What makes a good experiment?

• Should an experiment replicate reality? – No! We have happenstance data for that. – a good experiment captures most relevant features of reality in a simple, carefully controlled environment • Should experiment replicate a formal model? – No! We have theory for that. – a good experiment is designed to test specific hypotheses, derived from theory or previous empirical observations Take control over background variables

• Experimenter should control all elements of the environment in which the experiment takes place: – determine the rules: what choices are available to subjects, when decisions are made, and what the consequences of these decisions are – control subjects’ payoffs as a function of actions they take – control the information available to subjects Manipulate key variables

• Between treatments, experimenter only changes variables which are directly relevant to the hypothesis being tested, otherwise holding the environment fixed: – control vs. treatment groups – no confounds (Don’t change more than one thing at a time) – variables that cannot be directly controlled are typically controlled via randomization Steps in Designing an Experiment

• Identify an interesting question – issues that are better addressed in a controlled experiment than by gathering happenstance data • Formulate research hypotheses – Ex1: Increased payoffs → fewer mistakes in logic problems – Ex2: Face-to-face interaction → greater role of fairness • Design a simple environment for testing hypotheses – the more complicated the environment is, the more likely you are to lose control Issues in Designing a Lab Experiment

• Test hypotheses by varying a small number of variables within the experiment – if possible, vary one variable at a time • Within- vs. between-subject designs – within-subject: concerns about order effects – between-subject: concerns about subjects’ heterogeneity • Reasons for order effects: – subjects get bored – subjects get confused – subjects gain experience Rules widely accepted by experimental economists – lab experiments • No deception! – if you deceive subjects, what is going to happen the next time you try to run an experiment? • Anonymity – in most experiments, subjects are guaranteed that nobody other than researcher will be able to ever identify their actions or payoffs – in some experiments, subjects are guaranteed that even researchers will be unable to match their actions and payoffs with their real names – in some experiments, abandoning anonymity is an important part of the design, but subjects have the option of withdrawing if they don’t want information to be revealed Where and how to get subjects

• What population to use as subjects – undergraduate university students are the easiest to get – subjects with relevant experience are more interesting • How to get subjects – place advertisements on posters, webpages, newspapers – spamming is generally not good • What should be in advertisement – description, expected payoffs and duration, contact info – stress monetary payoffs A Typical Advertisement

Earn Money in Economics Experiments The Center for Institutional Studies is looking for participants to take part in experiments. Experiments are an important tool of academic research in economics. They help us to better understand the way the economy works. A typical experiment takes about 2 hours and is held at our computerized laboratory in the center at Myastnitskaya. Participants earn money by making decisions. This can involve, for example, buying and selling a good in a market. How much you earn in an experiment depends, for example, on the prices at which you buy and sell. No special knowledge of economics, mathematics, or computers is required to participate. However, a good command of the language of the experiment is necessary to understand instructions. You may register to participate in experiments carried out in English and/or Russian - the only requirement is a sufficient understanding of the language. The Center organizes experiments on a regular basis. To participate in an experiment you must first register. Registering means that you are in principle interested in participating. We will then send you invitations for specific experiments which you may accept or decline. Select Subjects Carefully

• Avoid unintentional selection of subject population • Avoid clustering (assign to treatments randomly) • Anticipate that only about 80% of subjects will show up • Never use a subject twice for the same set of experiments Information Provided in Experimental Instructions

• Detailed description of the rules – The main goal is to make certain that subjects understand the rules. If they do not, you have immediately lost control. – Instructions should be as clear and complete as possible. – Any critical points should be repeated at least once. – If in doubt, make the instructions too detailed. • Language of the instructions – It is common to frame instructions in a generic context that does not have any real world connotations. – Sometimes, it is desirable to specify the context. – Any loaded terms should be avoided. Additional Issues in Writing Instructions • Avoid demand effects – Never suggest how the subjects ought to behave in a way that compromises the experimental design • Include Payoff Quizzes – ensure that subjects understand the relationship between their choices, the choices of others, and their payoffs – reinforce important points from the instructions – avoid demand effects in writing your payoff quiz Include consent form if you plan on publishing results How to Run an Experiment Smoothly

• Decide well in advance between – computerized experiment (program needs to be written) – pen & paper experiment (logistics needs to be planned) • Before the first session starts: – do some practice (pilot) sessions to test the design – set the room up in advance – keep careful records of subjects’ arrivals to the lab – separate people who arrive at the same time – make it as hard as possible for subjects to see the materials being filled out by other subjects Using Assistance in Running an Experiment • Experimental team – one person who runs the experiment (a “conductor”) • reads the instructions • oversees the experiment • does not have a particular authority – a group of people who help out (“assistants”) • all the other tasks involved in running an experiment • Behavioral code for the experimental team – avoid any demand effects – do not lurk over subjects while they are making decisions – do not rush subjects if they are thinking about a choice – don’t talk among themselves during the session Completing an Experiment

• Paying subjects – pay subjects in cash privately immediately after the session – calculate payoffs yourself – collect receipts: name, ID, date, amount paid, signature • Collecting data – in designing experiment, make it as easy as possible to figure out afterwards what happened – design the action sheets carefully and store them neatly as the experiment progresses – do not rely on subjects’ record sheets – these are often inaccurate Additional Issues in Experimental Techniques (1) • As much as possible, random number generation should be public and credible. For pencil and paper experiments, use devices people will understand – rolling dice and flipping coins. Occasionally pre-generating the random numbers is desirable. • Many experiments involve rematching subjects. If they are not supposed to be matched with the same person, you need to think out the matchings in advance to make certain that they won’t be rematched. This is easy for 1 x 1 matchings, harder for group matchings. • For many experiments, we don’t want subjects to know whom they are playing. For most experiments, using ID numbers rather than names will be good enough. For experiments with rematching, you may want to conceal the ID numbers. Additional Issues in Experimental Techniques (2) • Risk Aversion: To limit the effects of risk aversion for experiments where it is a nuisance variable, many experiments use a “lottery technique.” The value of this technique is dubious and it confuses subjects. You should be sensitive to the possible effects of risk aversion on your results. • Feedback: Experimental results are quite sensitive to what feedback subjects are given. Any sort of feedback is acceptable, just think about its effects in advance and be certain that you aren’t accidentally leaking additional information. Often times the feedback is an important component of the experimental design. • Asymmetric Information: Often times you want some information to only go to some subjects. Be certain that this doesn’t slip over into deception. The other common problem is accidentally revealing information to people who aren’t supposed to have it. Al Roth – experimental lecture (2005)

“While the history of experimental economics can be traced back much further, only since the 1980’s have we begun to see many series of experiments, in which groups of experimenters with different theoretical predispositions looked at the same phenomena in the laboratory.” Experimental Economics Nowadays

• Significant growth in publications – Above 100 a year by the end of 1980s – Above 200 a year by the end of 1990s

• Growing recognition in the profession – Vernon Smith and Daniel Kahneman (Nobel prize, 2002)

• Close connection to Behavioral Economics From 01/2001 until 12/2010

• Only laboratory experiments – No field experiments, no neuroeconomic studies, no comments, replies and meta-analyses • Nine prominent journals – American Economic Review, Econometrica, The Economic Journal, Journal of Political Economy, The Quarterly Journal of Economics, Review of Economic Studies, Games and Economic Behavior, Journal of Economic Behavior and Organization and Experimental Economics • 716 experimental papers identified Source From 01/2001 until 12/2010

• 75% increase of published papers in experimental economics in 2006-2010 over 2001-2005

• The fraction of experimental papers published also increases

Source From 01/2001 until 12/2010, by topics

Source The most favorite topics

• Social Preferences – Esp. Social dilemmas • Individual decision making – Risk – Consumer Behavior/Willingness-to-pay • Markets – Auctions – Asset Markets – Industrial Organization • Games – Coordination • Equilibrium selection – Beauty contest Early Lab Experiments Concerning Individual Choice (1) • Experiments testing ordinal utility theory • Thurstone (1931): asked each subject to make a large number of hypothetical choices between commodity bundles consisting of hats and coats, hats and shoes, or shoes and coats • conclusion: choice data could be adequately represented by indifference curves • Wallis & Friedman (1942): criticized hypothetical choices • Rousseas & Hart (1951): asked each subject to make a single choice between breakfast menus consisting of specified number of eggs and strips of bacon; combined data from groups of subjects • May (1954): elicited intransitive preferences Early Lab Experiments Concerning Individual Choice (2) • Experiments testing expected utility theory proposed in von Neumann & Morgenstern (1944) • Mosteller & Nogee (1951) proposed a general plan to test assumption about individual choice behavior: – 1. observe how subjects take or refuse certain gambles or risks entailing use of real money; – 2. construct a utility curve for each subject; – 3. using utility curves, make predictions toward other risks; – 4. test predictions by examining further behavior. • Allais (1953) reported a violation of expected utility theory • based on two hypothetical choices (“Allais paradox”) Allais Paradox

Choose between A and B • A: • Certainty of receiving 100 million (dollars) • B: • Probability .1 of receiving 500 million • Probability .89 of receiving 100 million • Probability .01 of receiving zero

Choose between C and D • C: • Probability .11 of receiving 100 million • Probability .89 of receiving zero • D: • Probability .1 of receiving 500 million • Probability .9 of receiving 0 million. Early Lab Experiments Concerning Interactive Behavior • Prisoner’s Dilemma – introduced by Dresher & Flood (1950) as hundred-fold repetition of the matrix game between a fixed pair of subjects who communicated their choices of row or column

(-1, 2) (0.5, 1) (0, 0.5) (1, -1)

– Nash equilibrium ..., cooperative play outcome ... – experimental result: subjects do not play Nash – Nash’s response: equilibrium concept cannot be applied to a repeated game between a fixed pair of players Early Lab Experiments Concerning Industrial Organization • Chamberlin (1948): an experimental market with human buyers and sellers – hypothesis: when prices cannot be renegotiated, market outcomes differ from competitive equilibrium – reservation price for buyers: losses if buy above this price – reservation price for sellers: losses if sell below this price – market: all buyers and sellers meet and negotiate prices of individual transactions to form pairs – unambiguous prediction for price and volume under competitive equilibrium – results: in 42 out of 46 markets, traded volumes above competitive prediction; in 39 out of 46 markets, average price below competitive prediction Theory and Empirical Work

• Alternation of theory and empirical work – theory organizes knowledge and helps us predict behavior in new situations – theory suggests ways to analyze new data – data collection and analysis turn up regularities not explained by existing theory – discovered empirical regularities spur refinement of theory

• Role of experimental methods – access to new sources of data – more theoretical propositions can be empirically tested Testing a Theory?

• Formally, a theory consists of: – set of axioms and assumptions – conclusions that logically follow from them – it is valid if conclusions are provable from assumptions

• What can be learned about theory by conducting experiments: – (more of interest to psychologists) descriptive validity of behavioral assumptions – (more of interest to economists) validity of conclusions even when behavioral assumptions are not satisfied Field Experiments

• Oxford English Dictionary definition of the word “field”: – “Used attributively to denote an investigation, study, etc., carried out in the natural environment of a given material, language, animal, etc., and not in the laboratory or office” • Criteria that define field experiments: – nature of subject pool – nature of information that subjects bring to task – nature of commodity – nature of task or trading rules applied – nature of stakes – nature of environment Field Experiments • Advantage • Causation (!!!) • Natural environment • Own design • Cheaper than huge data collections

• Disadvantage • Still quite expensive (hard to get funding for replications) • Prone to unexpected shocks • Higher risk of attrition • Not suitable for every research

Issues in Designing a Field Experiment

• Attrition • Externalities / Spillover Effects – Information spillover – Peer effects • Budget versus Sample size versus Power of the Experiment versus Time – Statistical versus Scientific significance

Sample Size and Power Calculations

Sample Size ↑ES ↓ SS ↓SS ↓$

↑ES ↓ $ Effect Size Budget ↑SS ↑ T (or ↑↑ $)

Design Specific ↓$ ↓T Time

• Statistical versus Scientific significance • Enough Budget? Enough Sample size? High enough Effect Size? Rules widely accepted by experimental economists – field experiments • Think twice, do once – Statistical versus Scientific significance

• No deception

• Anonymity

• Protocol Natural and Quasi Experiments

• Field experiments without control of random assignment • Treatment is an unplanned event exogenous to the outcome – “Teacher responses to wage changes”(Falch 2011, AER: P&P) – “Peer Effects in Academic Outcomes”(Zimmerman 2003, The Review of Economics and Statistics) – Cherokee Casino in reservations, North Dakota Natural and Quasi Experiments

• Advantage – Observed in undisturbed form – Usually bigger scale

• Disadvantage – Hard to establish causal relationship – Outcome variables given – Researcher is not the designer – Question of validity Laboratory Experiments in the Field

• Combination of laboratory and field experiments • Jakiela et al. (2012): “You've Earned It: Combining Field and Lab Experiments to Estimate the Impact of Human Capital on Social Preferences” 

• Can a laboratory experiment use randomization? • What are the disadvantages of lab experiments? • What is internal and external validity? • What is randomization?

Miguel & Kremer, Econometrica, 2003