Use of Experimental Methods in Workforce Evaluations Gary Burtless, Brookings Institution David H. Greenberg, University of Maryland This report has been funded, either wholly or in part, with Federal funds from the U.S. Department of Labor, Employment and Training Administration under Contract Number AF-12985-000-03-30. The contents of this publication do not necessarily reflect the views or policies of the Department of Labor, nor does mention of trade names, commercial products, or organizations imply endorsement of same by the U.S. Government. U.S. Department of Labor Employment and Training Administration Occasional Paper 2005-08 Table of Contents 1.0 Introduction—When Is an Experiment Worthwhile? ............................................. 1 1.1 Advantages of Randomization .......................................................................... 2 1.2 Disadvantages and Threats to Experimental Validity ........................................ 7 1.3 Implications for Experimentation ..................................................................... 14 2.0 Past and Ongoing Experiments .......................................................................... 15 2.1 Target Group Typology ................................................................................... 17 2.2 Program Area Typology .................................................................................. 20 3.0 Lessons from Past and Current Experiments ..................................................... 30 3.1 Feasibility, Cost-Effectiveness, and Time Costs ............................................. 30 3.2 Have Experimental Results Influenced Workforce Policy? ............................. 33 3.3 Welfare-To-Work Experiments ........................................................................ 33 4.0 Some Suggestions for Social Experiments ......................................................... 36 4.1 Higher Priority Experiments ............................................................................ 36 4.2 Lower Priority Experiments ............................................................................. 44 4.3 Postscript ........................................................................................................ 47 Endnotes ....................................................................................................................... 48 Appendix A: Topologies and List of Experiments ......................................................... 52 Appendix B: Summaries of Selected Social Experiments ............................................ 72 U.S. Department of Labor Employment and Training Administration 1.0 Introduction—When Is an Experiment Worthwhile? Randomized trials are an increasingly popular method for evaluating policy alternatives in a variety of fields. The growing popularity of experiments is due to a simple fact. Use of random assignment increases the likelihood that an evaluation will produce a valid estimate of the impact of a policy intervention. This report considers the pros and cons of using randomized trials to improve the effectiveness of workforce policies. Experimentation can be used both to assess the effectiveness of current policies and to develop reliable information about the likely effects of new policy innovations. Of course, randomized trials are not always feasible, and even where feasible they do not necessarily provide reliable evidence about policy impacts. But a wide range of public policies are aimed at improving labor market outcomes at the individual level, and for this kind of program a randomized experiment has important advantages over other methods of policy evaluation. In particular, a randomized trial offers researchers a convincing benchmark for measuring program effectiveness. An experiment allows us to answer a question that is usually unanswerable in nonexperimental studies: How would people enrolled in the tested program have fared if they had not been offered services under the program? The report is divided into four main sections. In the remainder of this section 1.0, we define randomized trials and describe their advantages and disadvantages in comparison with other techniques for learning about program effectiveness. We describe the circumstances in which experiments should be used and those in which they should be avoided. In section 2.0, we review previous use of social experiments in workforce evaluation. Our review provides guidance in thinking about what kinds of experiments should be conducted in the future. By examining the target groups and the types of programs that have previously been the subject of social experiments, we can also identify areas where major gaps remain. What target groups have been missed in past and current experiments assessing workforce interventions? What kinds of treatments have not been subject to experimental evaluation? Section 3.0 considers lessons from past workforce experiments. We are particularly interested in learning about policy questions and the kinds of workforce interventions where experimentation has proved most fruitful. Have past experiments produced knowledge that is useful for evaluating and improving policy? Under what conditions has knowledge obtained in experiments been used to change policy? Did the results of experiments offer a reliable guide to impacts when tested policies were adopted? In the last section 4.0, future social experiments that the Department of Labor (DOL) might sponsor are considered. We develop two lists of potential experiments. The experiments in the first list should have high priority for the future improvement of workforce policy. The proposed experiments in the second list, while worthwhile, have lower priority either because they face greater technical challenges or because they deal with less important research questions. Recommendations are based on our judgment of DOL’s research priorities as well as on our assessment of the successes and failures of previous experiments. Both technical issues and political feasibility when developing recommendations have been taken into account, and U.S. Department of Labor Employment and Training Administration 1 suggested experiments that are likely to be particularly difficult to implement are indicated. In discussion of high-priority experiments, how a randomized trial might be designed and the possible limitations of the research findings are described. 1.1 Advantages of Randomization The critical element that distinguishes controlled experiments from all other research methods is the random assignment of meaningfully different treatments to the observational units of study. A randomized field trial (or social experiment) is simply a controlled experiment that takes place outside a laboratory setting, in the usual environment where social and economic interactions occur. In the simplest kind of experiment, a single treatment is assigned to a ran- domly selected group (the treatment group) and withheld from the remainder of the enrolled sample (the control or null-treatment group). Many social experiments have tested a variety of different treatments rather than only one. Some have not enrolled a pure control group at all. Instead, investigators have attempted to measure the differences in effect of two or more distinctive new treatments. The definition of an experiment can include tests of innovative new policies as well as studies that are intended to measure the impact of a current policy relative to a null treatment. In the absence of information from social experiments, analysts rely on four main kinds of information to learn about the effectiveness of particular programs or interventions. One source of information is data on the relationship between nationwide or communitywide aggregates, such as average educational attainment, on the one hand, and median wage earnings on the other. The effect of treatment differences is inferred by measuring the outcome differences over time or across communities. However, aggregate statistics are inappropriate for analyzing many kinds of detailed relationships, such as the effectiveness of different instructional methods in workforce training programs. A second source of information is management data collected in the administration of existing programs. Information from an existing program almost never provides any data about what the participants’ experiences would have been if they had been enrolled in a different program or in no program at all, however. This greatly limits the usefulness of such information for inferring the impact of program services on participants’ behavior or job market success. A third source of information is new survey data. Such data are more costly to obtain than existing programmatic data, but they provide information about the experiences of nonparticipants as well as participants in a program, and thus offer some evidence about behavior or outcomes in the absence of treatment. A fourth source of information is data generated by special demonstration programs. Like experiments, demonstrations involve the special provision of a possibly novel treatment, data collection about behavioral outcomes, and analysis of treatment effects. Unlike experiments, demonstrations do not necessarily involve random assignment. (When research projects are called “demonstrations” but involve random assignment, we classify them as “experiments.”) What all experiments have in common is that the tested treatments are randomly assigned to observational units—that is, to