8 Using Natural Experiments to Provide “Arguably Exogenous” Treatment Variability

8 Using Natural Experiments to Provide “Arguably Exogenous” Treatment Variability The cost to families of investing in their children’s education is of concern the world over. In some developing countries, the question is whether charging fees for enrollment in secondary school reduces access to education markedly for low-income families. Recently, some countries, beginning with Brazil and Mexico, have turned this question on its head and examined whether charging negative prices (that is, making cash pay- ments to low-income families that enroll their children in secondary school) will increase enrollment (Fiszbein, Schady, & Ferreira, 2009 ). In the United States and other industrialized countries, the focus is on the impact of college cost on decisions to enroll in post-secondary education. In every country, knowledge of the sensitivity of families’ educational decisions to the cost of education is critical to sound educational policy- making. A challenge, then, is to provide compelling evidence about the causal impact of a change in cost on families’ educational decisions. As we have argued in Chapter 4, a randomized experiment provides the most persuasive strategy for answering educational policy questions about the causal impact of school fees or scholarships on families’ educational enrollment decisions. Indeed, as we explain in Chapter 14, a number of recent randomized experiments have shed new light on the impact of costs on school-enrollment decisions. Researcher-designed randomized experiments on this topic are often diffi cult to carry out, however, due to cost and diffi culty in obtaining the requisite cooperation of participants and educational institutions. As a result, researchers often try to learn from experiments that occur naturally. These “natural” experiments are situations in which some external agency, perhaps a natural disaster, or an idiosyncrasy of geography or 135 136 Methods Matter birth date, or a sudden unexpected change in a longstanding educational policy “assigns” participants randomly to potential “treatment” and “control” groups. The challenge for the researcher is to recognize such natural experiments when they occur and to be prepared to deal with the opportunities and challenges they present. In the next section of this chapter, we explain the respects in which investigator-designed experiments and natural experiments are similar and the respects in which they are different. We then use data from two excellent studies to illustrate how researchers have taken advantage of natural experiments to address important policy questions about cause and effect. In the subsequent section, we point out sources of natural experiments that have proven productive in the past, and we sum- marize their common features. Then, we describe two important threats to internal validity that are an integral part of working with data from a special kind of natural experiment that has what we will refer to as a discontinuity design . We then explain how a novel analytic approach known as difference-in-differences estimation responds sensibly to one of these validity threats. Natural- and Investigator-Designed Experiments: Similarities and Differences Central to the internal validity of an experiment is the assignment of participants to the experimental conditions. When we say that experimental assignment is exogenous, we mean that it is beyond any possible manipu- lation by the participants themselves, so that membership in either a treatment or a control group is totally independent of the participants’ own motivations and decisions. So, when investigators assign participants in an experiment randomly to experimental conditions, the assignment is exogenous because, by defi nition, participants in a fair lottery cannot infl uence the outcome. Randomized assignment, in turn, renders mem- bers of the treatment and control groups equal in expectation prior to the intervention. Consequently, any between-group difference detected in the average value of the outcome, post-treatment, must be a causal consequence of the intervention itself. Participants are sometimes randomized to different program options, to innovative practices, or to different incentives by exogenous mecha- nisms that are not under the direct control of an investigator, but still provide the equality in expectation prior to treatment that supports causal inference. Provided that we can argue persuasively that those participants who are then subject to the contrasting and naturally occurring Natural Experiments 137 “experimental” conditions are indeed equal in expectation prior to treatment, we have a logical basis for making unbiased inferences about the causal impact of the treatment. However, although data from natural experiments can sometimes be analyzed in the same way as data from investigator-designed experiments, you may need to modify your analytic strategy to respond to additional threats to internal validity that may occur when an experiment arises naturally. Two Examples of Natural Experiments We begin by describing examples of two prototypical natural experiments. The fi rst occurred when the U.S . Department of Defense introduced military draft lotteries during the Vietnam War era. Each lottery created two experimental groups of young males who were arguably equal in expectation and differed only in that one group was offered— that is, could be drafted into — military service, whereas the second could not. Our second example occurred when the federal government ended, in 1982, a program that had previously provided college fi nancial aid to children who were the survivors of Social Security benefi ciaries. This abrupt policy shift meant that high-school seniors in and before 1981, whose fathers were deceased Social Security recipients, were “assigned” effectively to a treatment group that received an offer of college fi nancial aid. High- school seniors immediately after 1981, whose fathers were deceased Social Security recipients, were not made this aid offer. So long as these groups of high-school seniors were otherwise equal in expectation, we are pre- sented with naturally formed experimental groups that differed only in the offer of aid. The Vietnam-Era Draft Lottery The question of whether military service affects long-term labor-market outcomes for participants is a question many governments ask in the pro- cess of designing manpower policies. As Joshua Angrist ( 1990 ) pointed out, this question cannot be answered by using census data to compare the long-term earnings of men who served in the military and those who did not serve. The reason is that men are not usually assigned randomly to military service. Instead, men who have relatively unattractive employ- ment opportunities in the civilian sphere may tend to enter the military. Unobserved differences between those who volunteer to serve and those who do not may therefore create bias in the estimation of the long-term labor market consequences of military service. 138 Methods Matter A natural experiment that took place during the Vietnam War era provided Angrist with an opportunity to obtain unbiased estimates of the impact of military draft eligibility on long-term labor market outcomes. Between 1970 and 1975, the U.S. Department of Defense conducted fi ve draft lotteries that determined which American males in a particular age group were eligible to be drafted into military service. The 1970 lottery included men aged 19 through 26, and the lotteries in the four subsequent years included men aged 19 to 20. In each lottery, a random-sequence number (RSN), ranging from 1 through 365, was assigned to each birth date. Then, only men in the relevant age cohorts whose birthdays had RSNs less than an exogenously determined ceiling, which was specifi ed by the Department of Defense each year, were subject to induction. Angrist called such men “draft-eligible.” A simple comparison of the annual earnings in the early 1980s for the group of men in a particular cohort who were draft-eligible with those who were not provides an unbiased estimate of the impact on earnings of being draft-eligible. Note that the treatment being administered in this natural experiment is “eligibility for the draft,” not the actual experience of military service. This is because it is only the assignment of young men to draft eligibility that was randomized, not military service itself. Indeed, some of those in the draft-eligible “treatment group” avoided military service by enrolling in college, by being declared unfi t for military service due to physical lim- itations, or by having been arrested prior to the draft. Among white males born in 1950, 35 % of those declared draft-eligible as a result of having a low draft number actually served in the military, compared to 19% of those declared draft-ineligible (Angrist, 1990 ; Table 2, p. 321). Of course, the fact that many draft-eligible men did not serve in the military, and some men who were not draft-eligible did serve, does not threaten the unbiased estimation of the impact of being “draft-eligible” on later labor- market outcomes. In fact, this natural experiment resembles the New York Scholarship Program (NYSP), the investigator-designed experiment in which the treatment was the randomized offer of a scholarship to help pay private-school tuition. As explained in Chapter 4, not all families that received the scholarship offer sent their child to a private school. As we will see in Chapter 11, it is possible to use a more sophisticated technique, called instrumental-variables estimation , to tease out the causal impact of actual military service (or actual private-school attendance), using the original randomly assigned offer as an “instrument.” In this chapter, however, we focus only on estimating the impact of draft eligibility on later labor-market earnings. Combining information from the draft lotteries with information on subsequent earnings from the Social Security Administration, Angrist Natural Experiments 139 estimated the impact of draft-eligibility status on future annual earnings, separately for white and non-white men, in the several birth cohorts in which the draft lotteries were conducted.

Load more