Who Benefits from KIPP?

Who Beneﬁts from KIPP?

Josh Angrist, MIT Susan Dynarski, University of Michigan Thomas Kane, Harvard GSE Parag A. Pathak, MIT Chris Walters, MIT

April 2010 Background: The Achievement Gap

Whatever pathology may exist in Negro families is far exceeded by this social pathology in the school system that refuses to accept a responsibility that no one else can bear and then scapegoats Negro families for failing to do the job . . . The job of the school is to teach so well that family background is no longer an issue. – Martin Luther King (1968)

Black and Hispanic students still score substantially lower than Whites on achievement tests in all grades Similar gaps scross income and parental education Public schools struggle to close these gaps Many wonder if schools alone can ever close these achievement gaps Schools and the Achievement Gap: Three Skeptical Views

Hernstein and Murray’s (1994) Bell Curve put forth a kind of genetic determinism Rothstein (2004) argues that social forces are over-whelming: . . . there is nothing illogical about a belief that schools, if well-operated, can raise lower-class achievement without investing in health, social, early childhood, after-school, and summer programs. But while the belief is not illogical, it is implausible, and the many claims made about instructional heroes or methods that close that gap are, upon examination, unfounded. Heckman (e.g., 2005) lays out a sophisticated critique of the school-centered approach: Late remediation, no matter how extensive, cannot restore children from disadvantaged environments to the level of performance they would have attained had they received economically eﬃcient early interventions that compensate for disadvantage in the early years. The Charter Model

Massachusetts charters originated with a 1993 education reform law Charter schools are publicly funded, but operate with minimal government intervention X Charters - the right to operate a public school - are granted by the state X Each charter school functions as its own school district X Charters are granted to nonproﬁts, universities, teachers, or parents X Some charter schools are part of a Charter Management Organization (CMO), such as KIPP Charter schools are funded by sending districts, who pay tuition when one of their students opts to attend a charter X Tuition is approximately average per-pupil spending in the sending district X Districts’ charter school tuition costs are initially reimbursed by the state Key Charter Features

Charter schools are intended to be accountable X A Massachusetts charter is subject to ﬁve-year review; it may be suspended, revoked, or non-renewed X Accountability criteria: success of academic program; organizational viability; faithfulness to charter X Of 75 Massachusetts charters granted, 9 have been lost, the charter schools closed and children sent elsewhere

Most Massachusetts charter schools are outside local collective bargaining agreements X Charter schools hire, fire, and have loose work rules much like private schools X Charter teachers need not be certified, but must pass the state ed test in their first year of work The Knowledge is Power Program

The Knowledge is Power Program (KIPP) CMO operates about 80 schools in 19 states Most KIPP students are low-income, Black or Hispanic KIPP Lynn is the only KIPP school in New England and the only charter in Lynn KIPP Lynn opened in Fall 2004, now serves about 300 students in grades 5-8 Like all Massachusetts charter schools, KIPP Lynn uses a lottery when over-subscribed X KIPP Lynn was under-subscribed in 2004, marginally over-subscribed in 2005 X More recently, 200+ applications for about 90 seats X Applicants must be residents of Lynn, MA X KIPP aggressively seeks out students, recruits from laundromats, homeless shelters KIPP Lynn

Follows the No Excuses Instructional Model: X Long school year: starts in August, some Saturdays X Long school days: 7:30-5:00 X Long hours for teachers: carry cell phones and are on call at home X Student Discipline: students walk single ﬁle, talk only when called upon in class X Student Incentives: students receive ”paychecks,” points awarded for good work X Home visits: KIPP parents sign a ”KIPP contract,” committing to help their children succeed Staﬃng Model: X KIPP teachers are young: 90% under 40 vs 29% in Lynn Public Schools(LPS) X Few KIPP teachers are licensed: 26% vs 98% in LPS X Many KIPP teachers are TFA graduates Demographics of KIPP and Lynn

Lynn is a poor city North of Boston: 90,000 residents Former industrial town with high crime, unemployment Schools are overwhelmingly nonwhite, low-income: 13,000 students X Lynn Public School students are 46% Hispanic, 13% Black, 10% Asian Like many charter schools, KIPP Lynn serves a mostly minority population X KIPP Lynn is 55% Hispanic, 22% Black, and 3% Asian KIPP is unusual among charters in enrolling high proportion of Hispanics Why Study KIPP Lynn?

KIPP is emblematic of the No Excuses paradigm KIPP is a large and fast-growing CMO KIPP Lynn is unique in serving a large population of Hispanic, LEP and special education students KIPP Lynn has few local competitors; lottery losers are likely to remain in Lynn Public Schools Evidence for KIPP success is widely debated: KIPP students . . . enter with substantially higher achievement than the typical achievement of schools from which they came. ...[T]eachers told us either that they referred students who were more able than their peers, or that the most motivated and educationally sophisticated parents were those likely to take the initiative to . . . enroll in KIPP (Economic Policy Institute, 2005) Lottery Research Design

First Stage: Estimate the eﬀect of winning the lottery on time spent at KIPP (measured in years)

X 0 sigt = λt + κg + µj dij + Γ Xi + πZi + ηigt , (1) j Second Stage: Estimate the eﬀect of time spent at KIPP on test scores

X 0 yigt = αt + βg + δj dij + γ Xi + ρsigt + igt , (2) j Data

We study the cohorts who applied to KIPP from Fall 2005 to Fall 2008 Outcomes are MCAS scores for 5th, 6th, 7th, and 8th graders tested Spring 2006-9 Typically, a 4th grade applicant contributes 5th and higher-grade scores X 2005-6 applicants in 5th, 6th, 7th, and 8th X 2006-7 applicants in 5th, 6th, 7th X 2007-8 applicants in 5th, 6th X 2008-9 applicants in 5th Admissions lotteries, the key to our research design, are summarized in Table 2 We matched applicant records to state administrative records (SIMS) and test score (MCAS) data; details in paper Descriptive Stats and Covariate Balance

Descriptive statistics for the linked research file appear in Table 1 Student characteristics X Lynn PS and Kipp are mostly nonwhite; KIPP has more Hispanic and Black students X Lynn PS students are poor; KIPP students a little poorer X Lynn PS students have low 4th grade MCAS scores; KIPP students do a little worse Balance between winners and losers X Our lottery reconstruction may be imperfect, but most difs are small and insignificant X Overall F looks good X Baseline scores are (insig.) higher, LEP rates lower for winners; this does not seem to drive results We check for differential attrition here: Table 3 Lottery Results

We measure effects in standard deviation units X .5σ, a large effect, is a move from the 40th to 60th percentile, or from the 30th to the median Table 4 reports lottery results X First stage estimates: almost 1.25 years longer in KIPP for winners at test time X Large score gains for winners in Math; 2SLS estimates about .35σ per year X Smaller gains in ELA, but still substantial and (mostly) significant X Covariates (esp. lagged scores) increase precision These are remarkably similar to our Boston findings OLS estimates come out close to 2SLS; little selection bias in the Lynn applicant population OLS s.e.s are about half as large as 2SLS Special-Needs Subgroups

Charter schools in Boston serve fewer LEP and Special Education students than BPS Ravitch (2010) comments on this in her summary of our Boston study: This analysis by no means diminishes the accomplishments of Boston’s top charter schools . . . but it leaves open the question of how to educate the neediest students and which schools will do so. Table 5 looks at special-needs subgroups (cols 1-4) X Math effects are larger for LEP and SPED; X ELA effects come almost entirely from these groups Although boys are not by definition special needs, they are over-represented in special needs groups X Table 5 shows similar math effects by sex; LEP gains come entirely from boys Ability Interactions

KIPP has also been said to target high-achieving, motivated students Rothstein (2004) again: They select from the top of the ability distribution those lower class children with innate intelligence, well-motivated parents, or their own personal drives, and give these children educations they can use to succeed in life. Table 6 show how the KIPP eﬀect varies with baseline (4th grade) score quartiles X KIPP Lynn beneﬁts weaker students more, especially in ELA These results weigh against Hernstein and Murray’s (1994) genetic determinism as well as social determinism MCAS Performance Levels and School Switchers

The Massachusetts Department of Education splits MCAS scores into 4 categories: Advanced, Proficient, Needs Improvement, and Warning To meet the Adequate Yearly Progress (AYP) standard under NCLB, a school’s overall average and averages in various subgroups must be proficient or better (roughly) Table 7 reports IV estimates of the effect of KIPP attendance on the probability of falling into each category KIPP reduces the probability of a Warning score by 10 percent for math and 6 percent for ELA (per year) The effects imply a rightward shift of the math score distribution; For ELA, KIPP moves only the lowest group up

School switchers: Does KIPP drive em out? X Table 8 suggests not X Ignoring the 5-to-6 transition, KIPP kids are no more likely to change schools Summary

KIPP Lynn appears to have generated substantial score gains for lottery winners Gains of about .35σ for Math, .12σ for ELA per year These are similar to our findings for Boston charters (mostly also No Excuses schools) KIPP Lynn effects are especially large for special education and LEP students KIPP Lynn benefits the weakest students most; Language gains come entirely from sp. needs/low achievers The bottom line: In three years of middle school, KIPP Lynn more than closes the achievement gap in Math and makes major inroads in ELA It’s only one school, but KIPP schools are prototypes, much like Perry Preschool for Headstart [label=lotterystats] Table 2: KIPP Academy Lynn Lotteries Number of Calendar years Number of applicants in lottery Average years at Lottery Cohort observed Grades observed applicants sample Percent offered Percent attended KAL (winners) (1) (2) (3) (4) (5) (6) (7) (8) 2005-2006 2006-2009 5-8 138 107 0.925 0.673 2.556 2006-2007 2007-2009 5-7 117 86 0.674 0.535 2.293 2007-2008 2008-2009 5-6 167 127 0.654 0.567 1.711 2008-2009 2009 5 207 137 0.540 0.401 0.703 All cohorts 2006-2009 5-8 629 457 0.687 0.536 1.847 Notes: This table reports characteristics of the four lotteries conducted at KIPP Academy Lynn from 2005 to 2008. The lottery sample excludes sibling applicants, late applicants, repeat applicants, applicants without baseline demographics, applicants who could not be matched to the MCAS data, and applicants who had completed 6th or 7th grade prior to the lottery.

←- [label=covbalance] Table 1: Descriptive Statistics and Covariate Balance Means Balance regressions Lynn Public 5th KIPP Lynn 5th KIPP Lynn lottery Demographic No controls graders graders applicants controls (1) (2) (3) (4) (5) Hispanic 0.418 0.565 0.538 -0.052 - (0.053) Black 0.173 0.235 0.254 0.027 - (0.044) White 0.296 0.168 0.182 -0.010 - (0.040) Asian 0.108 0.021 0.022 0.026* - (0.015) Female 0.480 0.474 0.484 -0.010 - (0.054) Free/reduced price lunch 0.770 0.842 0.825 -0.030 - (0.041) Special Education 0.185 0.189 0.197 -0.013 - (0.042) Limited English Proficiency 0.221 0.172 0.206 -0.075 -0.060 (0.047) (0.044) Baseline Math Score -0.307 -0.336 -0.390 0.097 0.066 (0.114) (0.109) Baseline Verbal Score -0.356 -0.399 -0.438 0.054 0.028 (0.118) (0.109) Fourth Grade Applicant 0.768 0.056 0.068 (0.046) (0.047) F-value from joint test 0.820 0.998 p-value from F-test 0.621 0.409

N for demographics 3964 285 457 457 457 N for baseline Math 3808 284 446 446 446 N for baseline ELA 3805 284 447 447 447 Notes: Columns (1)-(3) report 4th grade means for students in 5th grade in Lynn PS or KIPP Lynn from 2005-2008. Column (4) reports coefficients from regressions of the variable indicated in each row on an indicator variable equal to one if the student won the lottery. Column (5) adds all of the demographic controls with the exception of LEP to the regressions for LEP, baseline scores, and 4th grade application. F-tests are for the null hypothesis that the coefficients on winning the lottery in all regressions are equal to zero. These tests statistics are calculated for the subsample that has non-missing values for all variables tested. * significant at 10%; ** significant at 5%; *** significant at 1%

←- [label=attrition] Table 3: Attrition Differential Follow-up (winner - loser) Proportion of non-offered with Demographics and Basic controls Demographics MCAS scores baseline scores Subject (1) (2) (3) (4) Math 0.851 0.052* 0.041 0.044 (0.032) (0.031) (0.030) 971 971 957

ELA 0.855 0.048 0.031 0.041 (0.031) (0.032) (0.031) 971 971 958 Notes: This table reports coefficients from regressions of an indicator variable equal to one if the outcome test score is non-missing on an indicator variable equal to one if the student won the lottery. The sample is restricted to cohorts for which we should observe follow-up scores and excludes applicants with sibling priority. Robust standard errors (clustered at the student level) are reported in parentheses. * significant at 10%; ** significant at 5%; *** significant at 1%

←- [label=mainlotto] Table 4: Lottery Results all applicants First Stage Reduced Form 2SLS OLS Subject Controls (1) (2) (3) (4) Math Basic 1.222*** 0.431*** 0.353*** 0.304*** (0.063) (0.116) (0.095) (0.048) 865 865 865 865 Demographics 1.232*** 0.392*** 0.318*** 0.316*** (0.065) (0.105) (0.084) (0.041) 865 865 865 865 Demographics & 1.228*** 0.425*** 0.346*** 0.317*** Baseline Scores (0.066) (0.066) (0.052) (0.032) 856 856 856 856

ELA Basic 1.223*** 0.183 0.150 0.170*** (0.063) (0.117) (0.094) (0.049) 866 866 866 866 Demographics 1.235*** 0.118 0.095 0.172*** (0.066) (0.097) (0.077) (0.041) 866 866 866 866 Demographics & 1.234*** 0.149** 0.120** 0.172*** Baseline Scores (0.066) (0.073) (0.058) (0.031) 856 856 856 856 Notes: Basic controls include year dummies, grade dummies, and risk sets. Demographics include dummies for female, black, hispanic, asian, other race, special education, limited english proficiency, free/reduced price lunch, and a female*minority interaction. Robust standard errors (clustered at the student level) are reported in parentheses. * significant at 10%; ** significant at 5%; *** significant at 1%

←- Figure 1

1.2

0.8

0.6 2005

0.4 2006

0.2 2007 KAL Effect 0 2008 4th 5th 6th 7th 8th -0.2

-0.4

-0.6

A. Math Reduced Form

1.2

0.8

0.6 2005 0.4 2006 0.2 2007 KAL Effect 0 2008

-0.2 4th 5th 6th 7th 8th

-0.4

-0.6

B. ELA Reduced Form

Notes: This figure plots the coefficients from a regression of test scores on the lottery offer dummy interacted with dummies for grade of test*application [label=cohortrfs] year. Basic and demographic controls are included. ←- [label=subgroups]

Table 5: Subgroupss Effects by Subgroup LEP Non-LEP SPED Non-SPED Male Female Subject Controls (1) (2) (3) (4) (5) (6) Math Demographics 0.628*** 0.254*** 0.527** 0.271*** 0.323*** 0.290** (0.197) (0.093) (0.215) (0.087) (0.111) (0.126) 132 733 175 690 444 421 Demographics and Baseline 0.451*** 0.312*** 0.441*** 0.325*** 0.322*** 0.385*** Scores (0.155) (0.056) (0.146) (0.053) (0.071) (0.079) 131 725 174 682 439 417

ELA Demographics 0.416** 0.019 0.220 0.038 0.140 0.010 (0.183) (0.084) (0.216) (0.079) (0.104) (0.116) 131 735 176 690 442 424 Demographics and Baseline 0.384*** 0.051 0.298* 0.049 0.152* 0.061 Scores (0.140) (0.062) (0.162) (0.058) (0.079) (0.086) 130 726 174 682 436 420 Notes: Columns (1)-(4) report 2SLS estimates in subsets of the lottery sample. The sample for each regression is restricted to individuals who were classified as limited english proficient (LEP), special education (SPED), or male in columns (1), (3), and (5), compared to those who were not in columns (2), (4), and (6) respectively. The LEP estimation sample includes 79 students, while the non-LEP sample includes 319. The SPED estimation sample includes 78 students, while the non-SPED sample includes 320. The male estimation sample includes 205 students, while the female sample includes 196. Columns (7) and (8) report results from models interacting baseline test score with years at KIPP Academy Lynn. Main effects are at the mean. The interaction * significant at 10%; ** significant at 5%; *** significant at 1%

←- [label=baselinequartiles]

Table 6: Distribution Effects Lowest Group Second Lowest Second Highest Highest Group Subject (1) (2) (3) (4) Panel B. Effects by Baseline Score Quartile Math 0.508*** 0.464*** 0.456*** 0.198*** (0.158) (0.108) (0.101) (0.056) 215 225 202 214

ELA 0.477*** 0.046 0.012 -0.076 (0.143) (0.121) (0.096) (0.075) 218 211 214 213 Notes: Panel A reports the coefficients from 2SLS regressions of indicator variables for each of the 4 MCAS performance levels on years in KIPP Lynn instrumented by the lottery offer dummy. Panel B reports 2SLS estimates of test score effects by baseline score quartile. Grades are stacked. Controls include demographics and baseline scores. Baseline quartile effects are estimated with one regression per subject using interaction terms. Robust standard errors (clustered at the student level) are reported in parentheses. * significant at 10%; ** significant at 5%; *** significant at 1%

←- [label=perﬂevels] Table 7: Effects on MCAS Performance Levels Warning Needs Improvement Proficient Advanced Subject (1) (2) (3) (4) Math -0.100*** -0.019 0.016 0.103*** (0.028) (0.038) (0.039) (0.026) 856 856 856 856

ELA -0.055*** 0.068* -0.005 -0.003 (0.020) (0.037) (0.037) (0.017) 856 856 856 856 Notes: Coefficients are from 2SLS regressions of indicator variables for each of the 4 MCAS performance levels on years in KIPP Lynn instrumented by the lottery offer dummy. Controls include demographics and baseline scores. Robust standard errors (clustered at the student level) are reported in parentheses. * significant at 10%; ** significant at 5%; *** significant at 1%

←- [label=switching] Table 8: School Switching Regressions Differential Follow-up (winner - loser) Mean for non-offered Demographics and Basic controls Demographics students baseline scores (1) (2) (3) (4) 0.504 -0.278*** -0.291*** -0.294*** Any switch (0.044) (0.044) (0.045) 419 419 412

0.855 -0.495*** -0.503*** -0.509*** 6th grade school is (0.061) (0.060) (0.059) different from 5th 294 294 291

0.081 -0.004 -0.006 -0.004 Any switch excluding 5th (0.033) (0.033) (0.034) to 6th transition 419 419 412 Notes: This table reports coefficients from regressions of an indicator variable equal to one if a student switched schools on an indicator variable equal to one if the student won the KIPP Academy Lynn lottery. The dependent variable in the first row is one if a student ever moves from one observed school to another from 4th to 8th grade, either within a school year or between school years. The dependent variable in the second row is one if a student switches schools between 5th and 6th grade; only observations where both schools are observed are used. The dependent variables in the the third row is 1 if a student switches schools at any time besides the transition from 5th to 6th grade. Robust standard errors are reported in parentheses. * significant at 10%; ** significant at 5%; *** significant at 1%

←-