LEARNING from the TEST: RAISING SELECTIVE COLLEGE ENROLLMENT by PROVIDING INFORMATION Sarena Goodman*

LEARNING FROM THE TEST: RAISING SELECTIVE COLLEGE ENROLLMENT BY PROVIDING INFORMATION Sarena Goodman*

Abstract—Between 2000 and 2010, five U.S. states adopted mandates and this remains the situation in most states.1 Using exam requiring high school juniors to take a college entrance exam. In the two earliest-adopting states, nearly half of all students were induced into test- data, I show that in each of the two early-adopting states ing, and 40% to 45% of them earned scores high enough to qualify for (Colorado and Illinois), between one-third and one-half of selective colleges. Selective enrollment rose by 20% following imple- students are induced to take the ACT by the mandates. mentation of the mandates, reflecting substitution away from noncompetitive schools. I conclude that a large number of high-ability students Large shares of the new test takers—40% to 45% of appear to dramatically underestimate their candidacy for selective col- the total—earn scores qualifying them for competitive- leges. Policies aimed at reducing this information shortage are likely to admission schools. Disproportionately many—of both the increase human capital investment for a substantial number of students. new test takers and the new high scorers—are from disadvantaged backgrounds. I. Introduction Yet in a simple model of test taking under plausible para- HIS paper examines recent reforms in several states meter values, students who prefer to attend a selective col- T that required high school students to take exams neces- lege, which I define as any school that rejects applicants on sary for admission to selective colleges. A side effect of the basis of quality, and think they stand a nontrivial chance compulsory testing is the new availability of a salient test of admission should take the test. This makes the large score—a direct measure of candidacy for selective share of new test takers who score highly a puzzle unless schools—for students who would not otherwise have taken nearly all are uninterested in attending selective schools.2 the exam. I link the mandates to large increases in test tak- Since I cannot directly observe preferences, I examine ing and show that many of the new test takers score compe- realized enrollment outcomes. In the primary empirical ana- titively. By comparing college enrollment patterns with and lysis, I use difference-in-differences to show that mandates without compulsory testing, I show that, with a mandate, cause substantial increases in enrollment at selective col- more students attend selective colleges. I interpret these leges—10% to 20% (depending on the selectivity mea- results within a model of test participation and conclude sure)—with no effect on college attendance overall (which students are systematically biased in their decision making, is dominated by unselective schools; see Kane, 1998).3 My which I attribute to students underestimating their admissi- results imply that about 20% of the new high scorers (or bility to selective programs. 10% of students induced into testing) attend selective col- Between 2000 and 2010, five U.S. states adopted manda- leges when they otherwise would not have.4 tory ACT testing for their public high school students. The My results cannot be consistent with rational decision ACT is a nationally standardized test, designed to measure making about test taking unless prospective test takers have preparedness for college, that is widely used in selective downward-biased forecasts of their probabilities of being college admissions in the United States. It has traditionally admitted to selective colleges or of their probabilities of been taken only by students applying to selective colleges, attending if admitted. There are two potential channels for this sort of bias. First, students may misperceive their own Received for publication April 3, 2014. Revision accepted for publica- ability, and thus understate their chances of achieving a tion August 10, 2015. Editor: Amitabh Chandra. high score on the exam. Second, students may not have * Board of Governors of the Federal Reserve System. The analysis and conclusions set forth are my own and do not indicate much information about selective schools and may assume concurrence by other members of the research staff or the Board of Gov- that they would choose not to attend even if they were ernors of the Federal Reserve System. I am especially grateful to Eliza- admitted, only to learn more about them and update that beth Ananat and Jesse Rothstein for their thoughtful comments and advice. I also thank Miguel Urquiola, Brendan O’Flaherty, Daniel Gross, Lesley Turner, Todd Kumler, Bernard Salanie´, Bentley MacLeod, David 1 Traditionally, selective college–bound students in some states take the Card, Danny Yagan, Bruce Sacerdote, John Mondragon, Gabrielle Elul, ACT, while in others the SAT is dominant. Most selective colleges Jeffrey Clemens, Joshua Goodman, Joshua Hyman, and seminar partici- require one test, but most schools that do will accept either test. At non- pants at the Columbia applied microeconomics colloquium, the UC Ber- selective colleges, which Kane (1998) finds account for the majority of keley combined public finance and labor seminar, the UT Dallas EPPS enrollment, test scores are generally not required or are used only for departmental seminar, the National Center for Analysis of Longitudinal placement purposes. Data in Education Research, the Federal Reserve Board of Governors, 2 For the purposes of my analysis, students who, absent a mandate, the RAND Corporation, Tufts University, NERA Economic Consulting, would have taken the SAT exam to satisfy their college admissions and the Upjohn Institute, and the All-California Labor Economics Confer- requirements are included in the ‘‘uninterested’’ group, as their college- ence for useful discussions and feedback. I am grateful to ACT, Inc. for going behavior is unexpected to change under the mandate. the data used in this paper, and to Jesse Rothstein, Amanda Pallais, 3 States with later mandates experience relative enrollment increases of Julie Noble, and Ken Bozer for assistance with ACT microdata use and similar magnitude. acquisition. 4 If some of the new high scorers (or students induced into testing by A supplemental appendix is available online at http://www.mitpress the mandates) would have taken the SAT to satisfy their admissions journals.org/doi/suppl/10.1162/REST_a_00600. requirements, these estimates represent a lower bound.

The Review of Economics and Statistics, October 2016, 98(4): 671–684 No rights reserved. This work was authored as part of the Contributor’s ofﬁcial duties as an Employee of the United States Government and is therefore a work of the United States Government. In accordance with 17 U.S.C. 105, no copyright protection is available for such works under U.S. law. doi:10.1162/REST_a_00600

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/REST_a_00600 by guest on 25 September 2021 672 THE REVIEW OF ECONOMICS AND STATISTICS

preference after taking the exam. In the discussion at the mechanism generating this effect: introduction of the end of the paper, I argue that the former explanation is more requirement coincided with test preparation, meals on the plausible: the evidence on college enrollment decisions sug- exam day, and rewards for test taking, which could all gen- gests that the kind of nonspecific information made avail- erate score gains.6 In sum, students exhibit rather large able to test takers does not have large enough effects on postsecondary schooling responses to small shocks to their their preferences to generate the enrollment impacts I find, information sets when they are making their postsecondary whereas it is quite possible that a student who knows little enrollment decisions.7 about the ACT will be a poor judge of her or his likely This paper addresses two policy-relevant questions. First, performance. do exam mandates affect enrollment? The answer to this is Disparities in college attendance and earnings between clearly yes: a very low-cost intervention at late stages of ado- children from low- and high-income families underscore lescence can have a large impact on educational choices.8 the importance of policies that raise access to higher educa- Second, what explains this effect? My results indicate that tion, and especially selective schools, among disadvantaged many students, who are likely high-need, dramatically students.5 Financial aid programs alone have not been able underestimate their postsecondary opportunities, which to close the gap that persists between socioeconomic groups negatively affects their attendance outcomes. This finding is (Kane, 1995). Yet it appears that small nudges, like increas- important, given evidence that disadvantaged students gain ing the allotment of free score sends that accompanies the most from selective colleges and growing recognition admissions tests, improve outcomes for low-income groups that such students are particularly unlikely to make optimal (Pallais, 2015). Identifying other factors amenable to in- college choices when left to their own devices. tervention may reduce disparities in college access and enrollment. Several such factors have been identified in previous II. ACT Mandates work. For instance, the complexity of and lack of knowl- edge about available aid programs appear to stymie their The ACT is a standardized national test for high school potential usefulness (Bettinger et al., 2012). Related experi- achievement and college admissions. First administered in ments have simplified the college application process: 1959, it contains four main sections—English, Math, Read- assisting disadvantaged students in selecting a portfolio of ing, and Science—and (since 2005) an optional Writing colleges led to subsequent enrollment increases (Avery, section. Students receive scores between 1 and 36 on each 2010); and mailing semicustomized information about the section and a composite score that is the average of their college application process to students generated increased scores across the four main sections. The ACT competes with another exam, the SAT, in a fairly stable geographi- applications and admissions, predominantly among low- 9 income high achievers (Hoxby & Turner, 2013). Finally, in cally differentiated duopoly. Traditionally, the ACT has related work, Hurwitz et al. (2014) examine Maine’s deci- been more popular in the South and Midwest, and the SAT sion to make the SAT mandatory beginning in 2007, as part on the coasts. Every four-year college and university in the of comprehensive reform to the state’s accountability sys- United States that requires such a test will now accept tem, and estimate a 4% to 6% increase in four-year college either. going. In their setting, they are unable to pinpoint the The ACT is generally taken in the eleventh and twelfth grades and is offered several times throughout the year. The testing fee is about $50, which includes the option to send 10 5 The disparity in college attendance between children from low- and score reports to up to four colleges. The scores supple- high-income families has been increasing over time (Bailey & Dynarski, ment the student’s secondary school record in college 2011), while earnings have been essentially steady among the college educated and have dropped substantially for everyone else (Baum & Ma, admissions, helping to benchmark locally normed perfor- 2007; Deming & Dynarski, 2010). Earnings are correlated with not just mance measures like GPA. According to a recent ACT the level of an individual’s education, but the caliber of college she or he Annual Institutional Data Questionnaire, 81% of colleges attends (Hoekstra, 2009; Card & Krueger, 1992; Black & Smith, 2006). Educational attainment has been linked to institutional quality (Cohodes & Goodman, 2012), even though students appear to discount potential 6 They also find that mandate compliers are rarely high scoring, so mar- future earnings as much as 94 cents on the dollar. Most studies show sub- ginal students may differ from those in ACT mandate states. stantial effects of college selectivity (Hoxby, 2009). One exception is 7 This phenomenon is not unique to the U.S. educational system. In Dale and Krueger (2002, 2011), which finds effects near zero, albeit in a developing countries, experiments that inform students of the benefits of specialized sample. Even in their setting, there are positive effects of higher education have raised human capital investment along several selectivity for disadvantaged students. While disadvantaged students dimensions (Jensen, 2010; Dinkelman & Martıńez, 2011). A Canadian appear to gain the most from selective schools (Zimmerman, 2014; experiment found college information sessions beneficial (Oreopoulos & McPherson, 2006; Dale & Krueger, 2002, 2011; Saavedra, 2008), they Dunn, 2012). are vastly underrepresented at top institutions (Bowen, Kurzweil, & 8 For example, $1,000 in grants raises college going by 3.6 percentage Tobin, 2005; Hill & Winston, 2005; Pallais & Turner, 2006; Hoxby & points (Dynarski, 2003). Avery, 2012). These studies generally rely on admissions test data in 9 Still, the SAT covers a smaller range of topics and, in particular, which disadvantaged students are vastly underrepresented (Bowen et al., excludes a section on science. The tests also differ in format and treat- 2005; Clotfelter & Vigdor, 2003). Therefore, the shortage of these stu- ment of incorrect responses. dents at and applying to top schools is probably even larger than estimates 10 The cost is $35 if the Writing section is omitted. Additional score suggest. reports are around $10 per school for either test.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/REST_a_00600 by guest on 25 September 2021 LEARNING FROM THE TEST 673

require or use the ACT or the SAT in admissions. Even so, FIGURE 1.—AVERAGE ACT PARTICIPATION RATES many students attend noncompetitive schools with open admissions policies. According to the Carnegie Foundation, 55% of students enrolled in either two-year or full-time four- year institutions attend noncompetitive schools and likely need not have taken the ACT or the SAT for admission. Between 2000 and 2010, five states (Colorado, Illinois, Kentucky, Michigan, and Tennessee) began requiring public high school students to take the ACT (see the online appendix, table 1).11 Colorado and Illinois were the earliest adopters: both states have administered the ACT to all public school students in the eleventh grade since 2001 and thereby first required the exam for the 2002 graduating cohort.12 Kentucky, Michigan, and Tennessee each adopted mandates more than five years later. Participation rates plotted for even years only. Each mandate state’s first treated graduating class There are two primary motivations for these policies. appears in brackets. Data are published by ACT, Inc. but corroborate with participation rates in the sam- The first relates to the 2001 amendment of the Federal Ele- ple used for analysis (appendix figure 1). mentary and Secondary Education Act (ESEA) of 1965, popularly referred to as No Child Left Behind (NCLB). had similar levels and trends in ACT taking. The slow With NCLB, there has been national pressure on states to upward trend in participation continued through 2010 in the adopt statewide accountability measures. The act requires states that never adopted mandates, with average test taking states to assess basic skills among students in particular among graduates rising gradually from 65% to just over grades if those states are to receive federal funding for 70% over the last sixteen years. By contrast, in the early- schools. Specific provisions mandate several rounds of test- adopting states, participation jumped enormously, from ing in math, reading, and science, one of which must occur 68% to approximately 100%, in 2002, immediately after in grade 10, 11, or 12. Since the ACT is a nationally recog- the mandates were introduced. The later-adopting states nized assessment tool, includes all the requisite material had a slow upward trend in participation through 2006, then (unlike the SAT), and tests proficiency at the high school saw their participation rates jump by over 20 percentage level, states can elect to outsource their NCLB accountabil- points over the next four years as their mandates were in- ity testing to the ACT, thereby avoiding a large cost of troduced. Altogether, this picture demonstrates that the developing their own metric.13 The second motivation mandate programs had large effects on ACT participation, relates to the increasingly popular belief that all high school that compliance with the mandates is nearly universal, and graduates should be ‘‘college ready.’’ In an environment that in the absence of mandates, participation rates are fairly where this view dominates, a college entrance exam serves stable and have been comparable in level and trend between as a natural graduation requirement. mandate and nonmandate states. Figure 1 presents average ACT participation rates by gra- Due to data availability, the empirical analysis in this duation year for mandate states, divided into two groups by paper focuses on the two early adopters. However, I also the timing of their adoption, and for the twenty other ‘‘ACT estimate short-term enrollment effects within the other states’’ from 1994 to 2010.14 State-level participation rates ACT mandate states and contextualize them using the reflect the fraction of projected (public and private) high longer-term findings from Colorado and Illinois. school graduates who take the ACT test within the three academic years prior to graduation and are published by III. The Effect of the Mandates on Test Taking ACT, Inc. Prior to the mandate, the three groups of states and the Score Distribution The test-taker data come from microdata samples of 11 In addition, one state, Maine, mandates the SAT. ACT test takers who graduated in 1994, 1996, 1998, 2000, 12 In practice, states can adapt a testing format and process separate and 2004, matched to the public high schools they attended. from the national administration, but the content and use of the ACT test The data set includes a 50% sample of nonwhite students remain true to the national test. For instance, in Colorado, the ‘‘CO ACT’’ is administered once in April and once in May to eleventh graders. The and a 25% sample of white students who took the ACT state website notes that the CO ACT is equivalent to ACT assessments exam each year. Each student observation includes the administered on national test dates throughout the country and can be sub- ACT composite score, the metric most relied on in the college mitted for college entry. 13 ACT, Inc. administers other tests that can be used with the ACT to admissions process, and responses to a survey completed track progress toward college readiness among its test takers (and satisfy before the exam concerning enrollment, socioeconomic additional NCLB criteria). Recently, the College Board developed an status, and demographics.15 The data contain high school analogous battery of assessments to be used with the SAT. 14 These are the states in which the ACT (rather than the SAT) is the dominant test. See Figures 1a and 1b in Clark, Rothstein, and Schanzen- 15 My analysis omits tests takers missing scores or indicating they are bach (2009) for the full list. already enrolled in college.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/REST_a_00600 by guest on 25 September 2021 674 THE REVIEW OF ECONOMICS AND STATISTICS

identifiers that can be linked to records from the students who opt into testing on their own.20 This is also Common Core of Data (CCD), an annual census of unsurprising, since more female and white students tend to public schools that captures the size and minority share of pursue postsecondary education, especially at selective a school’s student body. I use year-earlier CCD data schools. Finally, the posttreatment test takers more often describing eleventh graders, the population at risk of test tend to be enrolled in high-minority high schools.21 taking.16 Figure 2 plots score frequencies for the two treated states A limitation of a data set constructed from ACT test and their neighbors in the years before and after treatment, takers, as opposed to state administrative records, is that I adjusting for the graduating class size.22 The plots include a do not observe students who do not take the ACT exam. vertical line at 18, reflecting a conventionally used thresh- Unobserved students either take no test or only the SAT to old by colleges with the most liberal, but still competitive, satisfy admissions requirements. (Appendix A examines the admissions processes.23 In both treatment states, the influx validity of the assumptions underlying my identification of new test takers shifted the score distribution to the left. strategy.) Moreover, the distributions, particularly in Colorado, My analyses rely on neighboring ACT states to generate widened. Thus, accompanying a mandate, there is both a a counterfactual for the experiences of test takers from the decline in mean scores and a considerable increase in the two early-adopting states.17 A counterfactual test taker con- number of students scoring above 18: test takers scoring structed from surrounding states, rather than all ACT states, between 18 and 20 (inclusive) grew by 60% in Colorado is likely to be more demographically and environmentally and 55% in Illinois; at scores above 23, the growth rates similar to the marginal test taker in a mandate state. These were 40% and 25%, respectively. In the neighboring states, characteristics may be linked to experiences such as the there were no such shifts. likelihood she attends public school (and thus is exposed to Conceptually, students can be separated into two groups: the mandate) or her ambitiousness and cannot otherwise be those who will take the ACT whether or not they are fully accounted for in the data.18 required and those who will take it only if subject to a Appendix table 4 presents average test-taker characteris- mandate. Following the program evaluation literature, I tics in my sample. Test taking in each treatment state jumped refer to these students as ‘‘always takers’’ and ‘‘compliers,’’ substantially around the timing of the mandates. The number respectively (Angrist, Imbens, & Rubin, 1996).24 Because of test takers more than doubled from the pretreatment average in Colorado and increased about 80% in Illinois. In 20 Minority equals 1 if a student selects a race or ethnicity other than each case, the neighboring states saw growth of less than ‘‘white/non-Hispanic.’’ 10%. Predictably, requiring all students to take the test low- 21 High-minority schools are those in which minorities represent more ers the average score: both treatment states experienced than 25% of enrollment. The differences between the pretreatment averages in the treatment states and the averages in neighbor states suggest drops of about 1.5 points in their mean scores after their differences in test participation rates by state, differences in the underlying mandates took effect. Similarly, the mean parental income is distribution of graduates by state, or differences brought on by a combina- lower among the posttreatment test takers than among those tion of the two. Both Colorado and Illinois have higher minority shares 19 and slightly higher relative income among voluntary test takers than do who voluntarily test. Posttreatment, test takers exhibit a their neighbors. The striking stability in test-taker characteristics in more equal gender and minority balance than the group of untreated states (other than slight increases in share minority and share from a high-minority high school) over the period in which the mandates were enacted lend confidence that the abrupt changes observed in treatment states do in fact result from the mandates. Plotting the biennial demo- 16 My sample comprises test takers who can be matched to a school in graphic and test score data in mandate states reveals patterns that mimic the CCD sample. A fraction of students in the ACT data are missing high each of their respective composite states, with a sharp divergence between school codes so cannot be matched. Missing codes are more common in the mandate and nonmandate states in 2004 for both samples. Figures not premandate than in postmandate data, particularly for low-achieving stu- included but available on request. dents. This may lead me to understate the test score distribution for 22 To abstract from changes in cohort size over time, pretreatment score mandate compliers. I also exclude school years for which there are no stu- cells are rescaled by the ratio of public enrollment in the earlier period to dents taking the ACT. that in the later period. 17 The neighboring states include all states that share a border with either 23 Although U.S. college admissions decisions are multidimensional of the treatment states, excluding Indiana, which is an SAT state: Wiscon- and typically not governed by strict test score cutoffs, ACT Inc. publishes sin, Kentucky, Missouri, and Iowa for Illinois; Kansas, Nebraska, Wyoming, benchmarks to help test takers understand the competitiveness of their Utah, and New Mexico for Colorado. scores. According to definitions from their website, a ‘‘liberal’’ admis- 18 Appendix figure 1 reproduces figure 1 for the matched ACT-CCD sions school accepts freshmen in the lower half of a high school’s gradu- sample. Matched-sample participation rates resemble those reported for ating class (ACT: 18–21); a ‘‘traditional’’ admissions school accepts the full population by ACT. Also, participation rates in the neighboring freshmen in the top 50% of a high school’s graduating class (ACT: 20– states track those in the mandate states as closely as a composite formed 23); and a ‘‘selective’’ admissions school tends to accept freshmen in the from the other ACT states. Public school students apparently have a lower top 25% of a high school’s graduating class (ACT: 22–27). (See http: participation rate than private school students (who are included in the //www.act.org/newsroom/releases/view.php?year=2010&p=734&lang published data), but the trends are similar. =english.) According to a recent concordance (College Board, 2009), an 19 The ACT survey asks students to estimate their parents’ pretax 18 on the ACT corresponds roughly to an 870 on the SAT (out of 1600); income according to up to nine broad income categories, which vary a 20 ACT to a 950 SAT, and a 22 ACT to a 1030 SAT. across years. To make a comparable measure over time, each student’s 24 In theory, there may be students who do not take the exam despite selection is recoded to the midpoint of the provided ranges (or the ceiling the mandate (never takers), but figure 1 shows this group is negligible. and floor of the categories noted above), and income quantiles are calcu- The analysis holds in the presence of never-takers, so long as there are no lated within each year. defiers who take the test without mandates but not with a mandate.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/REST_a_00600 by guest on 25 September 2021 LEARNING FROM THE TEST 675

FIGURE 2.—ACT SCORE DISTRIBUTION

Graphs plot the number of test takers with score s (scaled by class size). ‘‘1994–2000 Average’’ refers to the average calculated over even years only. The vertical line at 18 reﬂects an admissions cutoff commonly used by selective colleges. This cutoff is approximately equal to a score of 870 on the SAT.

my data pertain to test takers, compliers are not present in share of total enrollment.25 The g’s represent state and time nonmandate state-year cells. In mandate cells, they present effects that absorb any permanent differences between but not directly identifiable. Therefore, characterizing the states and any time series variation that is common across complier group requires explicit assumptions about the evo- states. lution of characteristics of the at-risk population that I can- I estimate equation (1) separately for the two early-adopt- not observe (eleventh-grade students planning to graduate ing states and their respective neighbors and decompose the from high school). number of test takers into compliers and always-takers.26 Since there were (small) changes in the number of test About 45% of eleventh graders in Colorado and 39% in Illi- takers and the score distributions in nonmandate states nois, are compliers (table 1). as well as in mandate states, I employ a difference-in- It is more complex to identify the score distribution of differences strategy to identify the number of compliers compliers as it requires an assumption about the evolution scoring at each level, net of any common trends (which will of score distributions. My estimator relies on the fact that be attributed to the always-taker group). The first step is to the fraction of all test takers scoring at any value, r,ina estimate the number of new test takers. My key assumption mandate state can be written as a weighted average of com- is that absent the policy, the (voluntary) test-taking rate pliers and always-takers scoring at that value, where the would have evolved in a mandate state the same way it did weights are the share of students each group represents. I in its neighbors. In other words, the likelihood that a randomly selected student elects to take the exam would have 25 Note that an alternative specification of equation (1) is available: increased by the same amount over the sample period ln(Ast) ¼ b0 þ b1 mandatest þ b2 ln(Nst) þ gt þ gs þ est, where Ast is regardless of state lines. The mandates’ effects on test the number of test takers in a state-year, which implies a more flexible but still proportional relationship between the number of test takers and the taking—the share of students induced to take the exam by number of students. The two approaches yield similar results. the mandates—can be estimated with 26 I investigate the sensitivity of my results to specifications that address the small number of states in these regressions. Heteroskedastic-robust Pst ¼ b þ b mandatest þ c þ c þ est; (1) and wild cluster bootstrap-t standard error calculations do not affect infer- 0 1 t s ences (Cameron, Gelbach, & Miller, 2008). Results are not statistically different if estimated using synthetic control composite states derived where Pst is test participation in a state-year and b1 from the set of ACT states and demographic controls from equation (2) represents the estimated size of the complier group as a (Abadie & Gardeazabal, 2003).

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/REST_a_00600 by guest on 25 September 2021 676 THE REVIEW OF ECONOMICS AND STATISTICS

TABLE 1.—ESTIMATED MANDATE EFFECT ON TEST TAKING minority high schools and are more often young men and Participation Rate (0 to 1) minorities than those who opt into testing voluntarily. This Dependent Variable Treatment State Colorado Illinois is true even holding ability constant. These patterns are consistent with previous literature that finds these groups are mandatest 0.455*** 0.393*** (0.007) (0.022) vastly underrepresented at selective colleges, suggesting N 30 25 that early in their educational careers, such students do not Implied number of test takers in each group consider attending selective colleges at the same rate as Compliers 22,803 52,718 Always takers 28,497 82,750 other students. Total ACT test takers 51,300 135,468 Each column in the top panel reports coefficients from an OLS regression. Standard errors, clustered on the state, are in parentheses. The participation rate is the share of potential public high school graduates who take the ACT exam in each state-year cell. The estimation sample in each column is one of two IV. The Effects of the Mandates on College Enrollment early-adopting states, treated beginning in 2002, and its neighboring states in 1994, 1996, 1998, 2000, and 2004. Regressions include state and year effects. See the text for an explanation of the bottom panel. In this section, I explore how and whether the ACT mandates affect college attendance, under the assumption that the shifts in enrollment I identify arise among the group of recovered estimates for these weights in the last exercise; students induced into testing by the mandates described in thus, estimating the share of compliers at any given score, the previous section. r, requires the share of always-takers at r. The score distri- A key by-product of required testing is the availability bution in nonmandate neighboring states, which by defini- of an exam score for students who would not otherwise tion reflects the universe of always-takers in those states, have taken the ACT.30 The score may contain new informa- provides a convenient counterfactual for always-takers in 27 tion for the mandate compliers, revealing either their true mandate states; Thus, I assume that absent the mandate, ability or simply their admissibility to selective colleges. the likelihood a randomly selected always-taker earns a Two distinct decisions may be affected by this information: score of r increases (or decreases) by the same amount in the decision to attend college and to attend a selective both groups.28 Repeating this exercise for each score cell, I 29 college. recover the full complier score distribution. Whether to attend college is a complex function of indi- Table 2 summarizes the results for test takers scoring vidual-specific returns to college attendance and the oppor- below 17, between 18 and 20, between 21 and 24, and tunity cost of college each student faces. Further, it is above 25. The estimates are broadly consistent with figure 2 unclear whether the return to college is an increasing, and suggest that while a majority of compliers earned low decreasing, or nonmonotonic function of test scores. The scores (more than twice as often as their always-taker coun- ability to attend college does not depend on test scores, as terparts from earlier years), many earned high scores the majority of American college students attend open- (column 3). As a consequence, a substantial portion of the enrollment colleges that do not require test scores. Still, it is high scorers in mandate states came from the induced group possible that students use the score as information about (column 5). I estimate that around 40% to 45% of com- whether they can succeed in college (Stange, 2012), in pliers, amounting to about 20% of all postmandate test which case the effect on their choice to attend college at all takers, earned scores surpassing conventional thresholds for is theoretically ambiguous. Altogether, it appears that the admission to competitive colleges. information contained in a student’s ACT score would Appendix B shows how I can link the above methodol- likely have little influence on her decision to enroll in col- ogy to test taker characteristics to estimate complier lege, and the direction of influence is not obvious. shares in subgroups of interest (e.g., high-scoring minority By contrast, conditional on attending college, returns are students). Compliers come from poorer families and high- higher to attending a selective college, and acceptance at a selective college is a direct function of test scores. Thus, the new availability of a score, on net, could positively 27 I derive a composite counterfactual distribution from neighboring states and pre-2004 years. influence a student’s decision to attend a selective school. 28 Were I able to observe the population distribution of scores in non- To identify mandate-induced changes in college atten- mandate state-years, a counterfactual formed from test-taking rates would dance, I use information on college matriculation available be analogous to the method used to estimate the weights. Alternatively, I could use levels instead of rates, but it is less plausible. in the Integrated Postsecondary Education Data System 31 29 An advantage of my approach is that it does not require knowing the (IPEDS). IPEDS surveys are completed annually by more underlying scoring potential of the at-risk population. A disadvantage is than 7,500 colleges, universities, and technical and voca- that it poses stringent requirements on the relationship between the at-risk populations and their corresponding test-taking rates. Without these, at tional institutions that participate in the federal student least some of the differential changes in the score distribution among test financial aid programs. I use data on first-time, first-year takers might have been driven by shifts in the student population. These enrollment of degree- or certificate-seeking students, disag- additional constraints underscore the importance of a comparison population that exhibits similar traits (both demographically and educationally). In appendix A, I examine the plausibility of this assumption by comparing 30 The discussion that follows sets aside sticker prices of test taking and the test-taker composition to observable characteristics of eleventh gra- college applications. ders in the matched CCD schools. 31 Data were most recently accessed June 11, 2012.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/REST_a_00600 by guest on 25 September 2021 LEARNING FROM THE TEST 677

TABLE 2.—SUMMARY TABLE OF COMPLIER TESTING STATISTICS (1) (2) (3) (4) (5) Share of Complier Share of Pre (1994–2000 Post Compliers Compliers Test Takers (4)/ Score Range Average)a (2004) (2004) (2004) (2) (2004) N%N%% N % Colorado 0–17 4,880 21% 17,884 35% 53% 12,016 67% 18–20 5,507 24% 10,992 21% 19% 4,417 40% 21–24 6,973 30% 11,750 23% 13% 3,021 26% 25 and over 6,060 26% 10,674 21% 15% 3,349 31% All 23,421 100% 51,300 100% 100% 22,803 44% Illinois 0–17 17,984 24% 51,656 38% 60% 31,394 61% 18–20 16,256 22% 27,826 21% 19% 9,797 35% 21–24 19,817 27% 29,130 22% 13% 6,975 24% 25þ 19,741 27% 26,856 20% 9% 4,554 17% All 73,799 100% 135,468 100% 100% 52,718 39% Compliers are students who take the ACT only if subject to a mandate. aRefers to the average calculated over even years only.

gregated by state of residence, which are reported by each selective (‘‘competitive’’ institutions and above), very institution in even years.32 selective (‘‘very competitive’’ institutions and above), I merge the IPEDS data to classifications of schools highly selective (‘‘highly competitive’’ institutions and into nine selectivity categories from the Barron’s College above), and most selective (‘‘most competitive’’ institu- Admissions Selector (appendix table 7).33 Designations tions, only).34 In the appendix, I explore analyses cross- range from noncompetitive, where nearly 100% of an insti- classifying institutions by selectivity and institutional char- tution’s applicants are granted admission and ACT scores acteristics, constructed from IPEDS. are often not required, to most competitive, where less than Figure 3 presents suggestive evidence linking ACT man- one-third of applicants are accepted. Matriculates at ‘‘com- dates to enrollment patterns. The top panel plots overall petitive’’ institutions tend to have ACT scores around 24, enrollment over time for freshmen from the ACT states. while those at ‘‘less competitive’’ schools (the category just The solid line represents students from Illinois and Color- above ‘‘noncompetitive’’) tend to score below 21. I create ado, and the dashed line is students from the remaining 23 six summary enrollment selectivity measures: overall (any states. For comparability, the series are indexed to 1994 institution, including those not ranked by Barron’s), selec- levels. The bottom panel presents the same construction for tive (‘‘less competitive’’ institutions and above), more selective and more selective enrollment. There is a series break between 2000 and 2002, corresponding to the introduction of the mandates. 32 IPEDS also releases counts for the number of first-time, first-year enrollees who have graduated from high school or obtained an equivalent The graphs highlight several important phenomena. First, degree in the last twelve months (i.e., immediate enrollment), but these there are important time trends in all three series: overall are less complete. Specifically, there are 203 selective schools (out of enrollment rose by about 30% between 1994 and 2000 1,262) that report nonzero immediate enrollment in 1998 and 2006 but have 0 immediate enrollment reported for either 2002 or 2004. This is not (reflecting, in part, increased coverage of the IPEDS sur- normal year-to-year variation: only ten schools have nonzero immediate vey) among freshmen from the nonmandate states and by enrollment in 1998 and 2006 but are missing it in either 1996 or 2008. 15% among freshmen from the mandate states, while selec- When I correct for this underreporting, either by restricting attention to a balanced panel of schools (schools that report nonzero immediate enroll- tive and more selective enrollment rose by around 15% over ment in each year) or by using first-time, first-year enrollment to fill in the this period from both groups. Second, after 2002, the rate of missing values, I obtain results quite similar to those that I report in the increase of each series slowed somewhat among freshmen main analysis (available on request). The number of reporting institutions varies over time. To obtain the from the nonmandate states. The mandate states experi- broadest snapshot of enrollment at any given time, I compile enrollment enced a similar slowing in overall enrollment growth for statistics for the full sample of institutions reporting in any covered year. much of that period, but the growth of selective and more The number of reporting institutions grows from 3,166 in 1994 to 6,597 in 2010. My analysis primarily focuses on the 1,262 competitive institutions in my sample, of which 99% or more report every year, so the increase in coverage should not affect my main results. The 3,735 institu- 34 Year-to-year changes in the Barron’s designations are uncommon. I tions in the 2010 data that do not report in 1994 represent around 15% of follow Pallais (2015) and rely on Barron’s data from a single base year total 2010 enrollment and 3% of 2010 selective enrollment. (2001 edition). 33 Barron’s selectivity rankings are constructed from admissions statis- Appendix figure 2 depicts the distribution of enrollment by institutional tics describing the year-earlier first-year class. About 80% of schools in selectivity in 2000. Together, the solid colors (35% of the pie) represent my sample are not ranked by Barron’s. Most of these schools are for- ‘‘more selective’’ enrollment, and the solid colors plus the hatched slice profit and two-year institutions that generally offer open admissions to (45%) represent ‘‘selective.’’ The majority of students attend noncompeti- interested students. I classify all unranked schools as noncompetitive. The tive institutions. These shares are consistent with the Carnegie Foundation Barron’s data were generously provided to me by Lesley Turner. statistics.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/REST_a_00600 by guest on 25 September 2021 678 THE REVIEW OF ECONOMICS AND STATISTICS

FIGURE 3.—COLLEGE ATTENDANCE IN THE UNITED STATES BY STATE OF RESIDENCE population of 18-year-olds, and population characteristics from the Current Population Survey. College going increased around 5 percentage points within both groups, whereas attendance at selective and more selective colleges grew around 2 percentage points among students from mandate states and was essentially flat among those from nonmandate states.37 The population in mandate states dif- fers somewhat from nonmandate states, but the differences are stable over time and thus unlikely to confound identification in my estimation strategy. I will present specifications controlling for these observables. I next turn to a regression version of the estimator, using data from 1994 to 2008:

Est ¼ b0 þ b1 mandatest þ Xsth þ ct þ cs þ est: (2)

Here, Est is the log of enrollment in year t among students residing in s, aggregated across institutions (in all states) in a particular selectivity category. The g’s are state and time effects that absorb any stable differences between states and any time series variation that is common across states. The variable mandatest is an indicator for a treatment state after the mandate is introduced; thus, b1 represents the mandate effect, the differential change in the mandate states following implementation of the mandate. Standard errors are clustered at the state level.38

Data for even years only. The discontinuity in the series reflects timing of ACT mandates in early- Xst represents a vector of controls that vary over time adopting states. within states. For my primary analyses, I consider three specifications of X that vary in how I measure students at selective enrollment from these states accelerated. By 2010, risk of enrolling. In the first set of analyses, I do not include selective enrollment from the mandate states grew almost an explicit measure of cohort size. In the second, I include 30% from its 2000 level, compared with 9% growth in the the size of the potential enrolling class (measured as the log other states. Finally, there is a discrete jump in selective of the state population of 16-year-olds in year t2). And in college going unique to students from mandate states, con- 35 the third, for only selective and more selective enrollment, I sistent with a shock to trend around the mandate. use total postsecondary enrollment as a summary statistic Comparing average selective enrollment in 1994–2000 to for factors influencing the demand for higher education. 2002–2010, the top five institutions absorbing the growth in Because (as I show below and as figure 3 makes clear), students from Colorado, from most to least, were Colorado there is little sign of a relationship between mandates and State University; University of Colorado, Boulder; overall enrollment, this control makes little difference to Metropolitan State College of Denver; University of Color-

ado, Denver; and University of Colorado at Colorado 37 36 Appendix table 9 shows that the same general pattern—relatively lar- Springs. Each contributed at least 6% to selective enroll- ger growth among the students from mandate states—holds for an alterna- ment growth. The top five institutions absorbing Illinois tive measure of institutional selectivity, schools that primarily offer four- growth were: DePaul University; Southern Illinois Univer- year degrees, as well as across a wide variety of subgroups of selective institutions, including those both public and private and both in-state and sity Edwardsville; University of Illinois at Urbana-Cham- out-of-state. paign; Indiana University-Bloomington; and University of 38 Conley and Taber (2011) argue that clustered standard errors may be inconsistent in difference-in-differences regressions with a small number Iowa. Each contributed at least 3.5%. All ten require an of treated clusters and propose an alternative estimator for the confidence admissions test, reject at least 20% of applicants, and report interval. Conley-Taber confidence intervals are slightly larger than those that at least 75% of their students score above 18 on the implied by the standard errors in table 3, but the differences are small. For instance, in panel B, specification 5, the Conley-Taber confidence ACT. interval is (0.059, 0.219), while the clustered confidence interval is Appendix table 8 summarizes enrollment by mandate sta- (0.104, 0.180). Conley-Taber confidence intervals exclude 0 in each of tus in 2000 and 2002, denominated as a share of the at-risk the specifications marked as significant in table 3. Robust standard errors are smaller than clustered, except for some instances in table 3—specifications 5 and 6—where they are slightly lar- 35 The jump in selective enrollment among students from the three ger but do not affect inferences. A small-sample correction for critical later-adopting states (relative to those from states without a mandate) was values using a t-distribution with 23 degrees of freedom (G – 1) does not almost exactly the same magnitude. affect inferences (Hansen, 2007). Wild cluster bootstrap-t standard errors, 36 Institutions in bold are state flagships. Italicized institutions are as recommended by Cameron et al. (2008), produce qualitatively similar located out of state. results.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/REST_a_00600 by guest on 25 September 2021 LEARNING FROM THE TEST 679

TABLE 3.—EFFECT OF MANDATES ON LOG FIRST-TIME FRESHMEN ENROLLMENT, 1994–2008 (1) (2) (3) (4) (5) (6) A. Overall mandatest 0.054 0.032 0.003 0.006 (0.133) (0.129) (0.109) (0.108) B. Selective mandatest 0.159*** 0.138** 0.111*** 0.102*** 0.142*** 0.128*** (0.053) (0.050) (0.032) (0.031) (0.017) (0.020) C. More selective mandatest 0.163** 0.140** 0.110** 0.100** 0.145*** 0.130*** (0.067) (0.064) (0.042) (0.043) (0.024) (0.027) Controls ln(population) X X ln(overall enrollment) XX Demographic controls X X X Each column in each panel represents a separate OLS regression. The dependent variable in each regression is the log of first-time freshmen enrollment of students from state s in year t at a subset of schools, with the definition of selectivity increasing in stringency from the top to the bottom of the table. All regressions include state and year effects. Demographic controls include poverty and minority share, the June–May unemployment rate, and the share of residents over 25 years old with a B.A. The estimation sample is all ACT states (excluding Michigan) in even years between 1994 and 2008 (inclusive). Two states are treated beginning in 2002. Standard errors, clustered on the state, are in parentheses. Significant at *10%, **5%, ***1%.

the results. I estimate each specification with and without the mandate year to 0, omits the year just before the demographic controls. mandate is introduced (2000), and includes lead and lag Table 3 presents the regression results for the period terms for each year of data between 1994 and 2008 (inclu- between 1994 and 2008, where the estimation sample sive) (appendix table 10). The coefficients on the mandate includes all ACT states39 and the treatment states are Color- lead terms are not significantly different from 0; thus, there ado and Illinois. Each panel reflects a different enrollment is no evidence of anticipatory effects or important heteroge- outcome, with the definition of selectivity increasing in neity in enrollment pretrends. When either of the selective stringency from the top to the bottom of the table. Within categories is the outcome of interest, the lag terms are each panel, I present up to six variations of my main equa- highly significant. (They are 0 for overall enrollment.) Esti- tion: Specification 1 includes no additional controls beyond mated effects increase sharply over the first four years of the state and time effects, specification 2 adds demographic the policy and gradually after. I examine this further in controls, specification 3 controls only for the size of the table 4 and provide a discussion in the appendix. high school cohort, specification 4 adds the demographic Table 4 explores specifications that examine the robust- controls, and specifications 5 and 6 replace the size of the ness of my findings. (More extensions are presented in high school cohort in columns 3 and 4 with total college appendix C.) I report results for selective enrollment, con- enrollment.40 trolling for overall enrollment (panel B, specification 5 in Results are quite stable across specifications. There is no table 3).41 Column 1 reviews results from table 3. sign that the mandates affect overall enrollment probabil- In column 2, I add state-specific time trends. The man- ities. However, they do influence selective enrollment: date effect vanishes in this specification.42 Figure 3 and the when mandates are introduced, selective and more selective analysis in appendix table 10 suggest that the mandate college enrollment each increase between 10% and 20%. effects grow over time, a pattern that may be absorbed with The regression results coincide with the descriptive evi- a linear trend and a single step-up mandate effect. There- dence: a mandate induces students who would otherwise fore, I explore another specification that allows the treat- enroll in nonselective schools to instead enroll in selective ment effect to phase in:43 institutions. ÀÁ Still, in order for my strategy to identify the effect of an Est ¼ a0 þ a1 treatments ðÞ< 4years of policy t ACT mandate on enrollment of freshmen, college-going þ a2 ðtreatments ðÞ4 years of policy tÞ trends among students from mandate states would, absent the þ a3 overallst þ ct þ cs þ ws t þ est: ð2 Þ new requirement, resemble those observed in nonmandate states. To examine this, I follow Autor (2003) and estimate The results are presented without state-specific trends in an event-study version of equation (2) that normalizes column 3 and with them in column 4. Column 3 indicates

39 The main analysis omits Michigan due to dramatic decreases in Michigan’s state aid around its mandate (see www.michigan.gov/mistu 41 Results using the other speciﬁcations from table 3 are similar (avail- dentaid/0,4636,7-128-60969_61002_61357-279168--,00.html, accessed able on request). June 6, 2013). Headline results are not very sensitive to Michigan’s inclu- 42 In columns 2, 4, and 6, I present robust standard errors, as they are sion as a control through 2007 and treatment state thereafter. Michigan is more conservative here than the clustered standard errors of 0.016, 0.017, included in some robustness checks. 0.033, and 0.031, respectively. 40 I have also estimated the speciﬁcations presented in table 3 weighting 43 Note that the complier analysis in section III includes test-taker data the regressions by population and total enrollment (where applicable). that extend only through 2004, corresponding to the period covered by Results are mostly unchanged. the short-term effect in equation (20).

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/REST_a_00600 by guest on 25 September 2021 680 THE REVIEW OF ECONOMICS AND STATISTICS

TABLE 4.—EFFECT OF MANDATES ON LOG FIRST-TIME FRESHMEN ENROLLMENT AT SELECTIVE COLLEGES,ALTERNATIVE SPECIFICATIONS (1) (2) (3) (4) (5) (6) Table 3, State State Time Late-Adopting Panel B, Time Phased-In Trends with Late-Adopting States (excluding Speciﬁcation Column 5 Trends Treatment Phased-in Treatment States Michigan)

mandatest 0.142*** 0.043 (0.017) (0.044) mandates (<4 years of policy)t 0.095*** 0.065** 0.070** 0.103*** (0.014) (0.029) (0.030) (0.034) mandates (4 years of policy)t 0.190*** 0.140*** (0.029) (0.034) Joint significance test F-statistic 30.962*** 9.426*** P-value 0.000 0.000 Control states 22 22 22 22 20 20 N 192 192 192 192 207 198 Each column represents a separate OLS regression. Sample and specification for columns 1–4 are as in table 3, panel B, specification 5 except as noted. Samples for columns 5 and 6 exclude Colorado and Illinois; column 5 includes Michigan. In column 5, one state is treated beginning in 2008, and two states are treated in 2010. In column 6, two states are treated in 2010. Standard errors, clustered on the state (except in columns 2, 4, and 6, which use more conservative heteroskedastic-robust standard errors), are in parentheses. Significant at the *10%, **5%, ***1%.

that the treatment effect is 10% in the first years after effect of the mandates on selective college enrollment—are mandate implementation and then grows to 20%. This spec- incompatible with a view in which students make rational ification is much more robust to the inclusion of state- test-taking decisions based on unbiased forecasts of their specific trends (column 4) than was the version with a sin- probabilities of earning high scores. The pass rate among gle treatment effect. The hypothesis that both treatment compliers who would attend a selective college if admitted coefficients are 0 is rejected at the 1% level. There are a (LE compliers) is roughly 0.16. Because a rate well below number of possible explanations for a growing treatment that threshold would be sufficient to justify the cost of tak- effect, including changes in student aspirations and better ing the exam, there must be a divergence between the actual preparation for testing, as well as expanded slots at state rate at which such students pass the exam (P*) and their schools to accommodate admissible mandate compliers. I subjective judgments of their likelihood of passing (P). discuss these explanations in appendix D and provide sug- Many students have systematically downwardly biased gestive evidence that slots at state schools are not exclu- forecasts of their scores. sively driving this pattern. Figure 4 graphs the probability that a student will earn a Finally, I estimate the effects of more recent mandates in competitive score against the return to selective colleges. Kentucky, Michigan, and Tennessee over the full sample The solid curve represents the locus of points (return to period, omitting Colorado and Illinois from the sample selective college, probability of admission) at which a stu- (column 5).44 Because I observe only one postmandate year dent will be indifferent between taking the test or not. For a in Kentucky and Tennessee and two in Michigan, based on given anticipated return, students who perceive their prob- column 3, we should expect a smaller treatment effect than ability of earning a high score as above this curve will find was seen for Illinois and Colorado. This is indeed what we it rational to take the ACT even if it is not mandated, while see.45 Due to dramatic decreases in Michigan’s state college students who perceive their probability of earning a high aid around the timing of its mandate, I reestimate the enroll- score as below the line will not. The dashed horizontal line ment effect in the late-adopting states, excluding Michigan represents my lower-bound estimate for the passing rate entirely (column 6). The coefficient is almost 50% larger among LE compliers, 16%.46 This crosses the decision and much closer in magnitude to the comparable column 3 threshold at $963, indicating that the observed decisions estimate for the early-adopting states. can be consistent with full information only if students perceive the return to attending a selective college as under $1,000. The vertical lines on figure 4 are derived returns V. Assessing the Test-Taking Decision estimates from the literature, ranging from $5,000 (Black & In appendix E, I present a simple model of test taking to Smith, 2006, adjusting for Cohodes & Goodman, 2012; see show that the combination of results above—the high share appendix E for the derivation of this estimate) to $507,000 of mandate compliers who earn high scores and the large (Dale and Krueger, 2011, for their minority subsample; in

44 There is no detectable overall enrollment effect among the later- adopting states (not shown). 46 Heterogeneity in P* among LE compliers would imply a higher 45 Accounting for two-year-earlier eleventh-grade public enrollment break-even point. In appendix F, an alternative calculation using the full shrinks the early-adopter estimates closer to those in column 5; however, score distribution and external estimates of the ACT’s reliability implies it does not reduce the estimated effect in Kentucky and Tennessee. With- that students with P* as high as 40% to 45% are choosing not to take the out theoretical motivation for its inclusion, I omit this measure. test.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/REST_a_00600 by guest on 25 September 2021 LEARNING FROM THE TEST 681

FIGURE 4.—IMPLIED TEST-TAKING THRESHOLDS FOR VARIOUS ESTIMATES In the paper, I interpret the results as suggesting students OF THE RETURN TO COLLEGE QUALITY would like to attend competitive colleges but choose not to take the test out of an incorrect belief that they cannot score high enough to gain admission. Much research, mostly by psychologists and sociologists, has examined the profound effect students’ experiences and the expectations of those around them have on the goals they set themselves for (see figure 1 in Jacob & Wilder, 2010).48 These goals, whether aligned with ability or not, are strongly predictive of later enrollment decisions, but they are also malleable:49 high school students’ future educational plans appear to fluctuate with the limited new information available in their GPAs (Jacob & Wilder, 2010). In an environment of compulsory testing, students may upwardly revise their college plans because they had otherwise lacked information about their admissibility to particular schools. Students who perceive their probability of earning a high score to be above the solid line will find it Another compatible explanation is that students underes- rational to take the ACT even if it is not mandated, while those below will not. The horizontal line plots the probability of admission among the ‘‘low expectations’’ (LE) compliers that I derive in the text, and timate the utility of going to a selective school (before tak- the vertical lines depict estimates of the returns to college quality from the literature. The Dale and Krue- ger (2011) estimate over their full sample implies a 0 return (not shown). However, they examine col- ing the test). Agencies that administer admissions tests may leges with Barron’s designations ranging only from ‘‘competitive’’ to ‘‘most competitive,’’ all of which cooperate with college campuses by providing test taker are considered ‘‘selective’’ in my categorization. contact information for marketing purposes, and students’ perceptions of colleges may update with information they the full sample, Dale and Krueger estimate a return of receive after taking the ACT. Because I do not observe zero).47 All are well above the value consistent with the beliefs, I cannot distinguish a student underestimating her estimated passing rate, implying much lower decision or his ability from an upward revision to her or his per- thresholds for student test taking, ranging from 3% down to ceived returns to a selective college after testing. I can only 0.03%. truly conclude that students systematically underestimate These calculations do not account for psychic costs of their abilities if their ex ante and ex post valuations of the test taking or differences in financial costs of attendance. returns to a selective college are roughly equal. Such costs could offset the wage benefits for some students, The evidence from the literature is more consistent with but it is unlikely that very many students view the choice as my interpretation. First, in their Boston experiment, Avery a knife-edge decision, so removing $150 from the marginal and Kane (2004) find that exam-taking rates are systemati- cost of selective enrollment makes the benefit exceed the cally lower among disadvantaged students, which cannot be cost. A more plausible explanation is that many compliers explained by differences in perceived returns. In fact, stu- held biased estimates of their chances of earning high dents, including those from disadvantaged backgrounds, scores. tend to overestimate returns, and their perceived returns do not have much impact on the selectivity of the school they choose to attend. The authors surmise it more likely that VI. Conclusion disadvantaged students discount their own qualifications than the returns to college. In addition, while prior work I show that ACT mandates led to large increases in ACT demonstrates that high school students respond to informa- participation, the number of high-scoring students, and tion about colleges and the college application process selective college enrollment. About 40% to 45% of students through mentoring and tailored recommendations, non- induced to take the ACT exam under a mandate earned streamlined information like brochures and pamphlets competitive scores and many of these new high scorers— might be difficult for students and families to distill on their about 20%—ultimately enrolled in competitive schools. own (Avery, 2010; Carrell & Sacerdote, 2013; Hoxby & The fraction of all mandate compliers who achieved high Turner, 2013, 2015). Bettinger et al. (2012) found large scores and upwardly revised their enrollment plans—about effects of informing students of their aid eligibility only 10%—is well beyond the fraction that can be accommo- when they had assistance with forms (and none otherwise). dated by an unbiased model of the test-taking decision. Hoxby and Turner’s experiment produced large enrollment Under extremely conservative assumptions, such a fraction effects from targeted information even though their treat- would not be higher than 3%. 48 Some authors find that students lack the necessary information to 47 These conversions are approximate, since each estimate derives from form the ‘‘right’’ expectations (Manski, 2004; Orfield & Paul, 1994; a unique setting. Dale and Krueger’s estimate relies on a sample of col- Schneider & Stevenson, 1999). leges quite limited in scope, ranging only from ‘‘Competitive’’ to ‘‘Most 49 See also Stinebrickner and Stinebrickner (2012), Zafar (2011), and Competitive,’’ all of which I consider ‘‘selective’’ in my categorization. Stange (2012).

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/REST_a_00600 by guest on 25 September 2021 682 THE REVIEW OF ECONOMICS AND STATISTICS

ment and control groups were both eligible to receive mar- large effects on selective college enrollment appear imme- keting materials, suggesting that brochures alone might not diately after the implementation of the mandates. be very helpful; as confirmation, their survey found that Another concern might be the generalizability of the pol- untreated students dramatically underused information on icy effects: What could we expect if we were to institute college quality.50 Still, one might worry that mandate com- nationwide compulsory testing? Most of the estimates I pre- pliers in my setting receive additional materials that sent rely on two widely separated states that implemented improve their outcomes (e.g., scholarship offers), but mandates. While the total increase in selective enrollment Hoxby and Turner’s evidence suggests that they do not. In represented about 15% of prior selective enrollment among sum, nonspecific information that test takers receive does students from those states, it was to only 1% of total not appear to affect their preferences enough to generate the national selective enrollment and 5% of selective enroll- enrollment effects I find. ment from the mandate states and their neighbors. Still, one One concern is that the effects I detect derive not from hurdle to generalizing based on the experiences in Illinois the mandates themselves but from the accountability poli- and Colorado is that national policies may create congestion cies of which they were a part. In each of the mandate states in selective colleges, leading either to crowd-out of always- but Tennessee, the mandates were accompanied by new takers by compliers or to reduced enrollment of compliers standards of progress and achievement imposed on schools (Bound & Turner, 2007; Card & Lemieux, 2000).51 This and districts. If those reforms directly improved student kind of crowd-out would be particularly likely if the selec- achievement or admissions qualifications, the reduced-form tive college sector was unable to expand quickly enough to analyses would attribute those effects to the mandates. accommodate new mandate-induced demand. Over the There are several reasons, however, to believe that the sample period, national selective enrollment grew by about results derive from the mandates themselves. The first is the 2% each year, suggesting that projected growth under a absence of an overall enrollment effect of the mandate poli- national ACT mandate could be absorbed with under eight cies. One would expect that accountability policies that years of normal growth. The supply of selective college raise students’ preparedness would lead some to enroll in seats nationwide is likely elastic enough to avoid large or college who would not otherwise have. There is no sign of long-lasting crowd-out. this; effects appear to be concentrated among students who In sum, the evidence indicates that students are systema- planned to attend college. Second, National Assessment of tically underpredicting their suitability for selective schools. Educational Progress (NAEP) state testing results for math Many students are learning important information when and reading in both Colorado and Illinois mirror the they are required to take the ACT, and these better- national trends over the same period. These results are informed students, all else equal, are enrolling in better based on eighth-grade students so do not reflect the same schools. This pattern is not consistent with a model of cohorts. The stability of NAEP scores is evidence that the rational ignorance; students who can collect valuable infor- school systems in mandate states were not broadly improv- mation at low cost by taking the ACT frequently opt not to ing student performance. Third, the policies in the later- do so. Framed differently, a substantial share of high-ability adopting states and the different ways in which they were students prematurely and incorrectly rule out selective col- implemented provide some insight into the causal role of leges from their choice sets. Although I cannot directly esti- the testing mandate. Beginning in spring 2008, Tennessee mate the characteristics of these students, it seems likely mandated the ACT as part of a battery of required assess- that many come from low-income families. Low-income ments but did not implement other changes in accountabil- students are much less likely to take the test in the absence ity policy at the same time. A specification mirroring equa- of a mandate than their peers, and low-SES mandate com- tion (2) estimates that selective enrollment from Tennessee pliers are no less likely to earn high scores than high-SES rose by 15% in 2010, similar to the estimates for Illinois compliers. and Colorado. This strongly suggests that it is the testing Recent studies have demonstrated that college quality is mandates themselves, not accountability policies that an important determinant of later success. Black and Smith accompanied them, that account for the enrollment effects. (2006) and Cohodes and Goodman (2012), for example, Finally, one would expect accountability-driven school find that students at lower-quality schools earn less over improvement to take several years to produce increases in selective college enrollment. But my estimates reveal that 51 The share of the population from untreated states enrolled in selective schools did not fall after the early mandates were introduced, suggesting that they did not produce meaningful crowd-out. Since mandate compliers 50 Most of their control students reported they did not weight college from two states represent such a small share of national selective enroll- quality heavily when deciding where to apply. Also, they experiment ment and are fairly evenly distributed across types of schools, this experi- among students taking admissions exams, who are thus somewhat ment offers limited insight into the extent of crowd-out we might antici- engaged with the competitive higher education landscape. Such students pate from broader policies. Had effects been larger or more concentrated, are more likely than disengaged students to try to distill broad-based there might be more evidence of crowd-out. Given the large increases in information from promotional mailings. Their work reveals that even the selective out-of-state and private in-state enrollment (appendix table 13), engaged group is systematically underinformed and undermatches to mandate compliers might compete for admissions slots with always- selective colleges on receiving such promotional mailings. takers if these policies were scaled up.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/REST_a_00600 by guest on 25 September 2021 LEARNING FROM THE TEST 683

their lifetimes and are less likely to graduate. Since my Black, D., and J. Smith, ‘‘Estimating the Returns to College Quality with Multiple Proxies for Quality,’’ Journal of Labor Economics 24 results show that compulsory ACT testing leads many stu- (2006), 701–728. dents to enroll in more competitive, higher-quality colleges, Bound, J., and S. Turner, ‘‘Cohort Crowding: How Resources Affect Col- mandating the ACT likely enables many students to vastly legiate Attainment,’’ Journal of Public Economics 91 (2007), 877– improve their lifetime earnings trajectories. Using the con- 899. Bowen, W., M. Kurzweil, and E. Tobin, Equity and Excellence in Ameri- tinuous quality measure I describe in appendix E, I can link can Higher Education (Charlottesville: University of Virginia my results to Black and Smith’s findings that an additional Press, 2005). standard deviation in college quality produces a 4.2% Cameron, A. C., J. Gelbach, and D. Miller, ‘‘Bootstrap-Based Improve- ments for Inference with Clustered Errors,’’ this REVIEW 90 (2008), increase in lifetime earnings. The increase in selective 414–427. enrollment that I estimate translates into a 0.247 standard Card, D., and A. Krueger, ‘‘Does School Quality Matter? Returns to deviation increase in average college quality. The roughly Education and the Characteristics of Public Schools in the United States,’’ Journal of Political Economy 100:1 (1992), 1– 4.5% of students who are induced to change their enroll- 40. ment status by the mandate could, on average, expect a Card, D., and T. Lemieux, ‘‘Dropout and Enrollment Trends in the Post- boost of 23% in their earnings.52 War Period: What Went Wrong in the 1970s?’’ (pp. 439–482), in J. Gruber, ed., An Economic Analysis of Risky Behavior among Pallais and Turner (2006) establish that underprivileged Youth (Chicago: University of Chicago Press, 2000). groups are underrepresented at top schools, which they Clark, M., Rothstein, J., and D. Schanzenbach, ‘‘Selection Bias in College attribute to a combination of information constraints, credit Admissions Test Scores,’’ Economics of Education Review 28 (2009), 295–307. constraints, and precollegiate underachievement. My analy- Carrell, S. E., and B. Sacerdote, ‘‘Why Do College Going Interventions sis provides support for their first explanation: that lack of Work?’’ NBER working paper 19031 (2013). information can explain some missing low-income students. Clotfelter, C., and J. Vigdor, ‘‘Retaking the SAT,’’ Journal of Human Resources 38:1 (2003), 1–33. Increasing information flow between universities and stu- Cohodes S., and J. Goodman, ‘‘First Degree Earns: The Impact of College dents from underrepresented groups so that the potential Quality on College Completion Rates,’’ Harvard Kennedy School high scorers know they are indeed suitable candidates could Faculty Research working paper RWP12-033 (2012). College Board, ACT and SAT1 Concordance Tables (RN-40). erase some of the enrollment gap found at top schools. This Conley T., and C. Taber, ‘‘Inference with ‘Difference in Differences’ with is a potentially fruitful area for policy. Expanding mandates a Small Number of Policy Changes,’’ this REVIEW 93 (2011), 113– appears to be a desirable policy on its own, but there may 125. Dale, S., and A. Krueger, ‘‘Estimating the Payoff to Attending a More be additional policies that would complement mandates in Selective College: An Application of Selection on Observables targeting students’ information shortages. and Unobservables,’’ Quarterly Journal of Economics 117 (2002), 1491–1527. ——— ‘‘Estimating the Return to College Selectivity over the Career

52 Using Administrative Earnings Data,’’ NBER working paper According to figures on average lifetime earnings of B.A. holders 17159 (2011). (Pew Research Center Social and Demographic Trends, 2011), 23% of Deming, D., and S. Dynarski, ‘‘Into College, out of Poverty? Policies to lifetime earnings amounts to about $760,000. Increase the Postsecondary Attainment of the Poor’’ (pp. 283– 302), in Phil Levine and David Zimmerman, eds., Targeting Investments in Children: Fighting Poverty When Resources Are Limited (Chicago: University of Chicago Press, 2010). REFERENCES Dinkelman, T., and C. Martıńez, ‘‘Investing in Schooling in Chile: The Role Abadie, A., and J. Gardeazabal, ‘‘The Economic Costs of Conflict: A of Information about Financial Aid for Higher Education,’’ Centre Case Study of the Basque Country,’’ American Economic Review for Economic Policy Research discussion paper DP8375 (2011). 93 (2003): 113–132. Dynarski, S. ‘‘Does Aid Matter? Measuring the Effect of Student Aid on Angrist J., G. Imbens, and D. Rubin, ‘‘Identification of Causal Effects College Attendance and Completion,’’ American Economic Review Using Instrumental Variables,’’ Journal of the American Statistical 93:1 (2003), 279–288. Association 91 (1996), 444–455. Hansen, C., ‘‘Asymptotic Properties of a Robust Variance Matrix Estima- Autor, D., ‘‘Outsourcing at Will: The Contribution of Unjust Dismissal tor for Panel Data When T Is Large,’’ Journal of Econometrics 141 Doctrine to the Growth of Employment Outsourcing,’’ Journal of (2007), 597–620. Labor Economics 21 (2003), 1–42. Hill, C., and G. Winston, ‘‘Access to the Most Selective Private Colleges Avery, C., ‘‘The Effects of College Counseling on High-Achieving, Low- by High Ability, Low-Income Students: Are They Out There?’’ Income Students,’’ NBER working paper 16359 (2010). Williams Project on the Economics of Higher Education discus- Avery, C., and T. Kane, ‘‘Student Perceptions of College Opportunities: sion paper 69 (2005). The Boston COACH Program,’’ in C. Hoxby, ed. College Choices: Hoekstra, M., ‘‘The Effect of Attending the Flagship State University on The Economics of Where to Go, When to Go, and How to Pay for Earnings: A Discontinuity-Based Approach,’’ this REVIEW 91 It (Chicago: University of Chicago Press, 2004). (2009), 717–724. Bailey, M., and S. Dynarski, ‘‘Inequality in Postsecondary Attainment,’’ Hoxby, C., ‘‘The Changing Selectivity of American Colleges,’’ Journal of (pp. 117–132), in G. Duncan and R. Murnane, eds., Whither Economic Perspectives 23 (2009), 95–118. Opportunity: Rising Inequality, Schools, and Children’s Life Hoxby, C., and C. Avery, ‘‘The Missing ‘One-Offs’: The Hidden Supply Chances (New York: Russell Sage Foundation, 2011). of High-Achieving, Low Income Students,’’ NBER working paper Barron’s College Division, Barron’s Profiles of American Colleges: 24th 18586 (2012). Edition, 2001 (New York: Barron’s Educational Series, 2000). Hoxby, C., and S. Turner, ‘‘Expanding College Opportunities for High- Baum, Sandy, and Jennifer Ma, ‘‘Education Pays,’’ College Board trends Achieving, Low Income Students,’’ Stanford Institute for Eco- in higher education series (2007). nomic Policy Research discussion paper 12-014 (2013). Bettinger, E., B. Long, P. Oreopoulos, and L. Sanbonmatsu, ‘‘The Role of ——— ‘‘What High-Achieving Low-Income Students Know about Col- Application Assistance and Information in College Decisions: lege,’’ NBER working paper 20861 (2015). Results from the H&R Block FAFSA Experiment,’’ Quarterly Hurwitz, M, J. Smith, J. Howell, and S. Niu, ‘‘The Maine Question: How Journal of Economics 127 (2012), 1205–1242. Is Four-Year College Enrollment Affected by Mandatory College

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/REST_a_00600 by guest on 25 September 2021 684 THE REVIEW OF ECONOMICS AND STATISTICS

Entrance Exams?’’ Educational Evaluation and Policy Analysis Pallais, A., ‘‘Small Differences That Matter: Mistakes in Applying to Col- 37 (2014), 138–159. lege,’’ Journal of Labor Economics 33 (2015), 493–520. Jacob, B., and T. Wilder, ‘‘Educational Expectations and Attainment,’’ Pallais A., and S. Turner, ‘‘Opportunities for Low Income Students at Top NBER working paper 15683 (2010). Colleges and Universities: Policy Initiatives and the Distribution Jensen, R., ‘‘The (Perceived) Returns to Education and the Demand for of Students,’’ National Tax Journal 59 (2006), 357–386. Schooling,’’ Quarterly Journal of Economics 125 (2010), 515– Pew Research Center Social and Demographic Trends, ‘‘Is College Worth 548. It?’’ (May 16, 2011) http://www.pewsocialtrends.org/category Kane, T., ‘‘Rising Public College Tuition and College Entry: How Well /reports/2011/page/2/. Do Public Subsidies Promote Access to College?’’ NBER Working Saavedra, J., ‘‘The Returns to College Quality: A Regression Discontinu- Paper 5164 (1995). ity Approach,’’ unpublished manuscript, Harvard University ——— ‘‘Misconceptions in the Debate over Affirmative Action in College (2008). Admissions,’’ (pp. 17–31), in G. Orfield and E. Miller, eds, Chilling Schneider, B., and D. Stevenson, The Ambitious Generation: American Admissions: The Affirmative Action Crisis and the Search for Alter- Teenagers Motivated but Directionless (New Haven, CT: Yale natives (Cambridge, MA: Harvard Education Publishing Group, University Press, 1999). 1998). Stange, K., ‘‘An Empirical Investigation of the Option Value of College Manski, C., ‘‘Measuring Expectations,’’ Econometrica 72 (2004), 1329– Enrollment,’’ American Economic Journal: Applied Economics 4 1376. (2012), 49–84. McPherson, M., ‘‘Low-Income Access through Different Lenses,’’ address Stinebrickner, T., and R. Stinebrickner, ‘‘Learning about Academic Abil- to WISCAPE, University of Wisconsin, Madison, February 1, 2006. ity and the College Dropout Decision,’’ Journal of Labor Econom- Oreopoulos, P., and R. Dunn, ‘‘Information and College Access: Evidence ics 30 (2012), 707–748. from a Randomized Field Experiment,’’ NBER working paper Zafar, B., ‘‘How Do College Students Form Expectations?’’ Journal of 18551 (2012). Labor Economics 29 (2011), 301–348. Orfield, G., and F. Paul, ‘‘High Hopes, Long Odds: A Major Report on Zimmerman, S., ‘‘The Returns to College Admission for Academically Hoosier Teens and the American Dream’’ (Indianapolis: Indiana Marginal Students,’’ Journal of Labor Economics 32 (2014), 711– Youth Institute, 1994). 754.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/REST_a_00600 by guest on 25 September 2021