Basic Concepts and Principles of Social Experimentation SOCIAL EXPERIMENTATION: EVALUATING PUBLIC PROGRAMS WITH EXPERIMENTAL METHODS PART 2 This is the second in a series of papers on the design, imple- What does society stand to gain from mentation, and analysis of social experiments. In the the experiment? previous paper, we defined a social experiment as a com- parison of the outcomes of two or more groups randomly A social experiment benefits society by providing better assigned to different policy regimes. In this paper, we ex- information on which to base public policy. Such informa- plore in more detail the considerations involved in tion can improve policy in one of two ways: It can lead constructing this comparison and in interpreting the result- policymakers to adopt a program or policy that is found to ant differences in outcomes. We discuss the basic concepts have net social benefits (i.e., benefits to society that out- and principles involved in: weigh its costs) or it can lead to the termination of an existing program that is found to have net social costs (i.e., costs n Deciding whether to experiment; that outweigh its benefits to society).1,2 n Specifying the experimental treatment; Of course, one cannot know before the fact whether any particular experiment will lead to a change in policythat n Specifying the outcomes of interest; depends on the experimental findings and whether n Interpreting the treatment-control service differential; policymakers act on those findings. In deciding whether to and, conduct the experiment, then, one must act on the expected value of the experiment. This can be expressed as: n Interpreting treatment-control differences in outcomes. Expected value of experiment = (Value of change in policy Deciding Whether to Experiment × Probability of change in policy) Cost of experiment A social experiment generally begins either with interest in For a new program, the value of a change in policy due to some new program or a desire to determine whether an ex- the experiment is the net social benefit that will accrue if isting program is achieving its objectives. Unfortunately, the experimental program is adopted. For an existing pro- given the limited resources available for program evalua- gram, the value of a change in policy is its net social cost; tion, not all new ideas and existing programs can be this is a measure of the resource savings that will accrue if evaluated experimentally; sponsoring agencies must choose the program is terminated. These values can be quite large, among competing uses of their evaluation budgets. The as witness the policy impacts of the National JTPA Study. choice of policies or programs to be evaluatedand within That evaluation found that the out-of-school youth compo- that set the ones to be evaluated experimentallyrequires nent of JTPA had essentially no impact on the earnings of a careful assessment of the likely value of the information to be obtained through rigorous evaluation and the cost and feasibility of obtaining it. In this section, we discuss the 1 The following line of argument is stated more formally in Burtless and questions that should be addressed in deciding which pro- Orr (1986). We will discuss in a subsequent paper how net social ben- grams or policy issues to investigate experimentally. efits and costs are estimated. 2 In principle, experiments could also prevent the termination of an effective existing program or prevent the adoption of an ineffective new program. The former is analytically identical to the case in which the experiment leads to the adoption of an effective program and the latter is identical to the case in which the experiment leads to the termination of an ineffective program. 2 PART 1: WHY EXPERIMENT? THE RATIONALE AND HISTORY OF SOCIAL EXPERIMENTS youths who participated in it. As a direct result of this find- programs are based almost entirely on the premise that the ing, funding for the out-of-school youth component of JTPA services they provide will increase the employment and was reduced by over $500 million per year. In just a few earnings of participants, a behavioral premise that can years, this savings of resources that would otherwise have readily be tested experimentally, whereas Social Security been wasted on ineffective training services easily surpassed benefits for the aged are justified primarily on equity the cost of all the social experiments that have been con- grounds, without regard to any behavioral effects they may ducted in the last 30 years.3,4 have. It is obviously difficult to predict either the value of a change The likelihood that evaluation results will be acted upon in policy or the probability it will occur, but one can make will also be influenced by their timing, relative to the life- some statements about them that are useful in discriminat- span of the policy issue they address. Social experiments ing among potential experiments. First, other things equal, take time to plan, implement, and analyze. Often the treat- the larger the program the larger its social benefit or cost is ment itself lasts a year or more and several more years may likely to be. Thus, social experiments focused on larger pro- be required to observe the outcomes of interest and analyze grams are likely to have higher social value. Second, in some the programs impact on them. The interval between the cases previous research may allow one to make at least quali- decision to mount an experiment and the availability of re- tative statements about the probability that a new program sults is often five to ten years. Only if an experiment will be found to be effective or an existing program to be addresses a relatively fundamental policy issue will its re- ineffective. The more credible nonexperimental evidence sults still be relevant to the policy process after such a lag. there is that a new program may be effective or that an ex- isting program may be ineffectivei.e., that an experiment The income maintenance experiments and the health in- would indicate that a change in policy is warrantedthe surance experiment are examples of social experiments that higher the expected value of the experiment is likely to be.5 focused on fundamental policy issues that were still rel- evant many years after the experiments were completed. The social value of an experiment depends not only on the Rather than estimating the impact of a specific policy, these inherent importance and validity of the information it pro- experiments were designed to estimate underlying behav- vides, but also on whether it is used to improve policy. An ioral parametersthe elasticity of supply of labor and the experiment is of no value to society if its results never influ- price elasticity of demand for medical carethat would be ence policy. It is, of course, extremely difficult to predict a relevant to a wide range of policies. And while these ex- priori whether a particular set of evaluation results will be periments never resulted in the adoption of any specific acted upon. Evaluation is only one of a number of forces income maintenance or health insurance programs, their impinging on the political process and in many cases not results have been used extensively in the analysis of a num- the most important one. Still, one can identify certain fac- ber of proposed programs and policies in these areas. tors that make it more or less likely that evaluation results Evaluations of ongoing programs are also highly likely still will play a key role in policy deliberations. For example, the to be relevant when their results become available, because results are more likely to influence policy if the behavioral the program is likely still to be in place. questions that evaluation can address are central to the policy debate than if policy decisions turn on philosophical or ideo- In contrast, novel program or policy proposals may have logical issues. To cite two extreme examples, job training such a short life-span that they are irrelevant to policy by the time an experimental test can be conducted. This is particularly true if the proposal has only limited support to 3 See Greenberg and Shroder (1997) for a catalog of the social experi- begin withe.g., a proposal developed by a single govern- ments that have been conducted and their cost. ment official without significant support in the rest of the 4 This result benefited not only the taxpayers, but also the disadvantaged executive branch or the legislature. Neither that official nor youths who were the intended beneficiaries of the program. Rather than the proposal are likely to be around five years later. perpetuating a program that wasted their time and raised false expecta- tions, the government initiated a search for more effective ways to improve youths earnings. Whether that search will be successful depends on the Proposals that involve a complex package of programmatic outcome of several experimental tests of youth training programs that are components are particularly susceptible to shifts in policy underway as this is written. interest away from the specific combination of program el- 5 This presumes, of course, that the nonexperimental evidence is not ements evaluated before the results become available, even sufficiently compelling to convince policymakers to make the change in though there may still be substantial interest in its indi- policy without the benefit of an experiment. This may often be the case, however, because of the inherent risk that nonexperimental evidence may vidual components. The experimental evaluations of state be contaminated by selection bias or the other threats to validity dis- welfare reform demonstrations conducted in recent years cussed in the first paper in this series.