Connecting Social-Emotional Development, Academic

CONNECTING SOCIAL-EMOTIONAL DEVELOPMENT, ACADEMIC ACHIEVEMENT, AND ON-TRACK OUTCOMES: A MULTI-DISTRICT STUDY OF GRADES 3 TO 10 STUDENTS SUPPORTED BY CITY YEAR AMERICORPS MEMBERS

MAY 2020

ROBERT BALFANZ AND VAUGHAN BYRNES EVERYONE GRADUATES CENTER AT THE JOHNS HOPKINS UNIVERSITY SCHOOL OF EDUCATION A REPORT BY: The Everyone Graduates Center at the Johns Hopkins University School of Education

AUTHORED BY: Robert Balfanz Vaughan Byrnes

This manual may contain Internet website IP (Internet Protocol) addresses. At the time this manual was printed, any website and/or email address was checked for both validity and content as it relates to this report’s corresponding topic. Johns Hopkins University, and its licensers, are not responsible for any changes in content, IP addresses, pop advertisements, or redirects. CONTENTS

EXECUTIVE SUMMARY 3 INTRODUCTION 7 THE GROWING RELEVANCE OF SOCIAL-EMOTIONAL LEARNING IN THE FIELD OF EDUCATION 8 CITY YEAR’S WHOLE SCHOOL WHOLE CHILD APPROACH AND THEORY OF CHANGE 9 DATA SAMPLE 10 ACADEMIC OUTCOMES 11 SOCIAL-EMOTIONAL MEASURES 11 DEMOGRAPHICS 12 IMPLEMENTATION DATA 12 COMPASS ACADEMY SUB-SAMPLE 12 RESEARCH QUESTIONS 13 ANALYSIS 13 RELATIONSHIP OF SOCIAL-EMOTIONAL SKILLS TO STUDENT OUTCOMES 14 THE CITY YEAR WHOLE SCHOOL WHOLE CHILD APPROACH AND STUDENT OUTCOMES 17 LIMITATIONS, COMPLEXITIES, AND NEXT STEPS 24 REFERENCES 27

EXECUTIVE SUMMARY

There is a growing understanding that an integrated measures of education outcomes (attendance, course approach to social, emotional, and academic develop- grades, and achievement tests) on a large scale, employ- ment provides the best path toward ensuring all students ing a multi-state and city, multi-grade (3 - 10) dataset that graduate from high school prepared for postsecondary includes elementary, middle, and high schools. The main and adult success. Recent findings from the Aspen Com- dataset includes 38,131 students in grades 3 through mission and the Science of Learning and Development 10 , attending 326 schools in 28 cities spread across 20 Project show that social and emotional factors interact different states. It also includes data from three different with cognitive actions to shape learning experiences. tools used to measure students’ social-emotional skills: the Holistic Student Assessment (HSA), the Devereux Students Social-emotional learning concepts began to draw consid- Strengths Assessment (DESSA), and a measure of Growth erable interest in the field of education in the first decade Mindset (GMS). of the 2000s, with much of the discussion centered around concepts of perseverance such as ‘grit’ and ‘tenacity’. There are two other unique features of this data set. All Reviews and meta-analyses of the many smaller studies of the collected data is from students who attend schools conducted this decade and before indicate that increases with high poverty rates and who, through their atten- in social-emotional skills can lead to improved academic dance, behavior, or course grades, signaled that they were outcomes. on course to fall off the path to high school graduation. As a result, all of the students in the data set received sup- The role social-emotional learning can play in educational ports from City Year Americorps members, including tutor- advancement moved from individual school or school dis- ing, mentoring, and near-peer supportive relationships. trict efforts to more broad-based efforts in 2013 when the California Office to Reform Education (CORE) districts, a Thus, unlike the few other large-scale studies of the consortium of nine school districts in California that serve connection between social-emotional development and over one million students in more than 1,500 schools, de- academic outcomes, this study is the first large-scale cided to integrate measures of students’ social-emotional attempt to look at an effort to support the students most skills directly into school accountability metrics. Many of in need of additional supports to succeed in school with the recent studies on the relationship between students’ a human-centered, relationship-driven approach. Nearly social-emotional skills and their academic outcomes all prior studies of social-emotional development and ed- have been based on the analyses of the CORE data. These ucation outcomes have examined the impact of curricular studies continue to find evidence that strengthening approaches, a prescribed set of lessons or experiences, or social-emotional skills leads to improved attendance, the broader impact of a school’s climate. behavior, and test scores amongst students; and further, that students’ social-emotional skills can be influenced by Using this unique dataset, available through City Year, the schools they attend. we aim to further the evidence base regarding the link between social-emotional skills and academic outcomes. With a few notable exceptions, including the CORE analyses, work done by the Collaborating Districts Initiative, and City Year, an education non-profit founded in 1988, places the recent national study of growth mindset, which have a team of eight to fifteen diverse AmeriCorps members in been based on large samples, most studies of social-emo- chronically under-resourced urban elementary, middle, tional learning have been small in scale. They have also and high schools, where they serve full-time as near-peer employed varying conceptualizations of what social-emo- tutors, mentors, and role models. City Year’s Whole School tional skills are and a wide assortment of tools for measur- Whole Child (WSWC) approach is informed by three ing them, and different measures of academic outcomes. decades of human development experience dedicated to The following study is one of the first to examine the cultivating lifelong dispositions for civic and community connections between social-emotional skills and multiple engagement. AmeriCorps members partner with class- (continued on page 5) KEY FINDINGS IN THE STUDY INCLUDE:

Students’ levels of SOCIAL-EMOTIONAL SKILLS ARE NOT FIXED POINTS but do in fact vary over the course of time 1 and schooling. Results found STATISTICALLY SIGNIFICANT AND CONSISTENT RELATIONSHIPS BETWEEN STUDENTS’ SOCIAL-EMOTIONAL SKILLS AND THEIR ACADEMIC OUTCOMES. The effects of moving students’ SEL skills either from a Need for In- struction to a Typical level, or from a Typical level to an area of Strength, range from 0.08 to 0.42 in terms of effect sizes, with many ranging between one quarter to one third of a standard deviation (0.25-0.33). Effects of such sizes are considered to be large and substantial shifts in the context of comprehensive school reform and student achievement. To illustrate the magnitude of these impacts, multiple studies have estimated such effect sizes as the equivalent of an entire school year of academic achievement 2 growth in mathematics or English for students in grades 3-10. Another method of translating the practical importance of effect sizes is the What Works Clearinghouse’s “Improvement Index,” which weighs effect sizes in the 0.25-0.33 range as equivalent to raising the average student by 10-13 percentile points. Further, a student who moves from an area of Strength to merely Typical, or from Typical to a Need for Instruction, is roughly twice as likely to be have low attendance, receive a low course grade, or receive a low test-score.

The findings affirm that STUDENTS’ SOCIAL-EMOTIONAL LEVELS ACCOUNT FOR A SUBSTANTIAL AMOUNT OF THE VARIATION IN THEIR ACADEMIC OUTCOMES, an impact comparable to that of their family background. This tells research- 3 ers and practitioners that addressing students’ social-emotional skills is a viable path to improving their academic outcomes.

Also, the MORE HOURS STUDENTS SPENT WORKING WITH A CITY YEAR AMERICORPS MEMBER, THE LESS LIKELY THEY WERE TO STRUGGLE WITH THE VARIOUS SOCIAL-EMOTIONAL COMPETENCIES at the end of the year (controlling for start-of-year social-emotional levels). For students who received the median number of hours support, the effects sizes 4 on their overall social-emotional skills was between .06 and .08. According to the WWC’s improvement index, this is equivalent to moving the average student up 2-3 percentiles in the population’s distribution of SEL skills. The analyses revealed that THE MORE HOURS A STUDENT SPENT RECEIVING SUPPORT FROM A CITY YEAR AMERICORPS MEMBER IN EITHER ENGLISH OR MATH, THE HIGHER WERE THE STUDENT OUTCOMES – IN NOT ONLY THE RELATED SUBJECT, BUT ALSO IN ATTENDANCE. For students who received the median number of hours of support from a City Year AmeriCorps member for English or math, the related increase in their course grade would be 0.1, equivalent to one tenth of a grade level (A-F) and an effect size of 0.08-0.09. The effect can be translated 5 to roughly 2-4 months of learning in academic achievement growth, and an improvement index gain of 3-4 percentiles. Further, the more hours a student received support in a subject, the less likely he or she was to be off-track either in that course or attendance at school. Students receiving the median amount of support from a City Year AmeriCorps member would also be 42% less likely to be off-track in English class (odds-ratio = 0.58) and one-third less likely to be off-track in math class (odds-ratio = 0.66). Lastly, results found that THE LOWER A STUDENT’S PRIOR ACADEMIC OR SOCIAL-EMOTIONAL LEVEL WAS, THE STRONGER THE RELATIONSHIP BETWEEN THE CITY YEAR INTERVENTION and that student’s spring outcomes. In other words, the students who began with the lowest attendance rates or course marks, and those with the lowest 6 social-emotional skills, were the ones who benefited the most from receiving one-on-one support from an AmeriCorps member.

4 room teachers and school principals to employ a holistic measures of social-emotional skills provides an excellent and positive approach that integrates the academic with opportunity to examine evidence on the relationship be- the social-emotional (SEAD) and places relationships at tween students’ social-emotional development and their the center of the practice. There is an equal emphasis academic behaviors and outcomes. on social-emotional mindset and skill development, the creation of a positive whole-school learning environment, In summary, the study’s findings show a significant and and directly supporting progress of individual students. In substantial relationship between students’ social-emo- 2018-19, 3,000 City Year AmeriCorps members served in tional skills and academic outcomes across grades 3 350 schools in 29 communities across the United States. to 10 in multiple states and districts. Further, available implementation data for the City Year program suggests In addition to providing data from its network of schools that human-centered, relationship-focused, school-based that includes both student academic data and measures interventions can be successful on a wide scale in devel- of students’ social-emotional skills, City Year also provided oping students’ social-emotional skills, as well as their some rough measures of its program’s implementation academic outcomes. The results show that the more time and engagement with individual students. Using this a student spent working directly with a City Year Ameri- information on the amount of time City Year AmeriCorps Corps member, the greater the student’s social-emotional members spent with students, our study also examines skills; students with stronger social-emotional outcomes whether more time with a City Year AmeriCorps member obtained greater academic outcomes in terms of course is associated with improved student outcomes, both grades, achievement tests, and predictive indicators of academic and social-emotional. By exploring the orga- high school graduation. Taken together, these points nized effort to improve student outcomes through the further validate City Year’s Whole School, Whole Child City Year AmeriCorps program, the study aims to deepen approach, show that human-centered, relationship-driven our understanding of practitioners’ abilities to both affect approaches to social-emotional development may be as, social-emotional skill development amongst students and if not more, impactful than curricular or programmatic improve their academic outcomes. approaches. The results also support policymakers’ efforts to adopt and support social-emotional development as a City Year primarily partners with systemically under-re- part of basic K-12 education. Not only are social-emotional sourced schools in large urban school districts. Within outcomes important for students’ educational success, but those schools, the students who receive support as part of they are susceptible to change through the investment of City Year’s program, and for whom we have data, are those school practitioners and community partners. that teachers have identified as struggling in one or more areas (math, English, attendance, behavior). Thus, the These findings, moreover, were drawn from a large sample is not representative of the national population of multi-district sample across elementary, middle, and early schools and students; we cannot assume that any of the high school grades. This suggests that they are not the exploratory results presented in this report will hold true result of extraordinary efforts in a unique setting or limit- for all students. However, because City Year works with ed to one particular age-band of students, but rather can the high schools with the lowest graduation rates and the occur at a range of high-needs schools within high-needs elementary and middle schools that feed into them, the school districts: the very settings whose populations sample is very representative of the types of schools and struggle the most and where support is typically focused. students that state and federal agencies most typically These results intensify the call to action for educators and identify as needing support in order to raise student policymakers to support the expansion and integration outcomes. Therefore, the patterns highlighted below of social-emotional development in schools across the between students’ socio-emotional learning levels and nation. academic outcomes are likely to be representative of the sub-populations that public and non-profit organizations are most interested in supporting. While the City Year data clearly focuses on specific sub-populations, its large sample size and its inclusion of both academic records and

5 “THERE IS A GROWING UNDERSTANDING THAT AN INTEGRATED APPROACH TO SOCIAL, EMOTIONAL, AND ACADEMIC DEVELOPMENT PROVIDES THE BEST PATH TOWARDS ENSURING ALL STUDENTS CAN GRADUATE FROM HIGH SCHOOL PREPARED FOR POST-

SECONDARY AND ADULT SUCCESS.” ~ASPEN INSTITUTE, 2019

6 6 INTRODUCTION

“There is a growing understanding that an integrated ap- Recently, a few studies have begun to look at the rela- proach to social, emotional, and academic development tionship between social-emotional development and provides the best path towards ensuring all students can academic outcomes on a larger scale across multiple graduate from high school prepared for post-secondary school districts: most notably, the analyses of the Califor- and adult success” (Aspen Institute, 2019). This is support- nia Office to Reform Education (CORE) data, the Collab- ed by the science of learning, which shows that social and orating Districts Initiative (AIR, 2015) and the national emotional factors interact with cognitive actions to shape study of growth mindsets “(Yeager, Hanselman, Walton, et learning experiences and outcomes (Immordino-Yang, al., 2019).To date, much of the analysis of the CORE data Darling-Hammond, & Krone 2018). In short, social-emo- conducted by the Policy Analysis for California Education tional and environmental factors can either impede or (PACE) center and by Panorama Education have found enable the effort, focus, and time that learning requires evidence that stronger social-emotional skills are associat- (Cantor, et al., 2018). ed with improved attendance, better behavior, and higher test scores among students, and further, that students’ These findings from the Aspen Commission and the Sci- social-emotional skills can be influenced by the schools ence of Learning and Development Project suggest that they attend (West, 2016; West et al., 2016; West et al., there is a connection between social-emotional outcomes 2018; Panorama, 2018, Claro & Loeb, 2019; Claro & Loeb, and academic achievement, as traditionally measured by 2019b). The Collaborating Districts study found that the schools through course grades and standardized tests. A systemic implementation of SEL at the district level led to counter-argument is that social, emotional, and academ- improvements in school climate, reductions in exclusion- ic skills may all be important to adult success, but may ary discipline, and consistent improvements in student operate independently of each other (Whitehurst, 2019). achievement across subjects. For example, interpersonal relations and perseverance can play substantial roles in enabling educational attain- Despite a wave of recent research to match the growing ment without having a direct impact on achievement test interest in SEL skill development, questions remain about scores. which social and emotional skills are most critical to students’ school success and life success; how best to mea- Most existing studies on the relationship between stu- sure the development of such skills; whether skill develop- dents’ social-emotional skills and their academic out- ment can be affected by educators’ efforts; and what types comes have focused on small samples of students and of educator actions are most impactful, for which types of schools, and on a sub-set of grade levels. Moreover, they social-emotional skills and outlooks? Empirical answers to have mainly examined curricular or programmatic efforts whether, how, and which social-emotional skills operate to improve social-emotional outcomes in which students to advance student academic success and how best to are taught a series of lessons or engaged in a planned set develop them are, in part, awaiting datasets of sufficient of activities. These studies have also employed varying size and breadth to enable analysis of multiple factors and conceptualizations of what social-emotional skills are complex relationships across varied school environments. and a wide assortment of education outcome measures (Hart et al. 2020). When the main effects of these studies This study is one of the first to examine the connections are brought together in meta-analysis, we see evidence between social-emotional skills and measures of academic of a connection between social-emotional skills broad- outcomes on a large scale, employing a multi-state and ly defined and academic outcomes, but are not able to city, multi-grade dataset. The dataset includes data for parse many details (Durlak, et al., 2011; Taylor, et al., 2017, 38,131 students in grades 3 through 10 attending 326 Corcoran et al. 2018). Finally, when only studies based on schools in 28 cities spread across 20 different states. high quality randomized control trials are examined, more mixed impacts are found (Jones et al. 2017). 7 THE GROWING RELEVANCE OF SOCIAL-EMOTIONAL LEARNING IN THE FIELD OF EDUCATION

Long studied in the field of psychology, social-emotion- based largely on disciplinary incidents, attendance, and al learning (SEL) concepts began to draw considerable measures of school climate and culture, the districts also interest in the field of education in the first decade of the piloted a student survey to capture measurements of stu- 2000s (Farrington, et. al., 2012). Much of the discussion dents’ SEL skills, with an eye to eventually incorporating centered around the works of psychologists Angela Duck- this as a component in their school accountability system. worth and Carol Dweck indicating that SEL skills could be Designed in partnership with Transforming Education predictive of student outcomes; it initially focused largely and Panorama Education, the SEL survey used in the on concepts of perseverance, such as ‘grit’ and ‘tenacity’ CORE districts targeted four conceptual social-emotional and growth mindset or the importance of believing that areas: self-management; social awareness; self-efficacy; academic outcomes are not determined by fixed aptitudes and growth mindset. Given the size of the CORE district’s but could be improved through effort (Dweck, 1999; student population, the survey represented the first Duckworth & Seligman, 2005; Dweck, 2006; Duckworth, et. large-scale collection of social-emotional learning data. al., 2007; Dweck, Walton, & Cohen, 2011). Reviews and me- Efforts to more formerly integrated social-emotional ta-analyses of the many smaller studies conducted during development into educational practice, which began in this decade and before of largely curricular and program- the CORE districts and Illinois which was the first state to matic efforts to improve social-emotional development adopt state-wide SEL standards, has subsequently spread indicated that: increases in social-emotional skills do lead nationwide, with x states now having established state SEL to improved academic outcomes amongst students (Far- standards. rington, et. al., 2012), but just as importantly, interventions designed to increase students’ SEL levels have the desired A series of nationally representative surveys conducted effect of increasing their course grade and achievement on behalf of the Collaborative for Academic, Social, and levels in tandem (Durlak, et. al., 2011). Emotional Learning (CASEL) found that students, principals, and teachers all believe in the inherent value of This era also say the development of a growing number social-emotional skills, recognize that these play a key role of frameworks, to organize social-emotional skills into an in supporting student achievement, and believe that SEL integrated set of components. Most notably, the CASEL skills should be integrated into schooling through curric- framework. (CASEL, 2019) ulum and accountability (Bridgeland, et. al., 2013; DePaoli, et. at., 2017; DePaoli, et. at., 2018; Atwell & Bridgeland, The intersection of social-emotional learning research 2019). Despite their belief in the value of SEL skills and and the wider adoption of social-emotional development their desire to adopt them, principals and teachers also in the field of education shifted and accelerated in 2013 indicated in the survey that they wanted more evidence when the CORE districts, a consortium of nine school of the link between SEL skills and student outcomes, and districts in California that serve over one million students more guidance on how to incorporate them into everyday in more than 1,500 schools, decided to integrate measures school practices and teaching. of students’ social-emotional skills directly into school accountability metrics. At that time, six of the CORE districts The current study seeks to address these questions by received a waiver from the U.S. Department of Education examining the connection between SEL development to include a non-achievement-based component in their and a range of student academic outcomes, including school accountability metric. While this component is course grades, test scores, and predictive indicators of

8 educational attainment, in elementary, middle and high One example of a theoretical framework that City Year schools (grade 3 to 10) across multiple states and school AmeriCorps members use to maintain this positive devel- districts. It also explores the organized effort to improve opmental mindset and apply it to everyday practice is the SEL outcomes through the City Year AmeriCorps program Clover Model from the PEAR Institute. The Clover Model and aims to deepen our understanding of practitioners’ highlights four essential elements, or leaves, that people abilities to both affect SEL skill development in students of all ages need in order to thrive, learn, and grow: Active and, in turn, improve their academic outcomes. A unique Engagement, Assertiveness, Belonging, and Reflection. feature of this study, is that the SEL supports provided The model posits that, although there are key times by City Year corps members, are not done primarily via during childhood and young adulthood for specialization the delivery of a curriculum or program, but rather a across the four leaves, a person’s development is dynam- human-centered, relationship driven approach to provide ic and recursive over the lifespan. City Year AmeriCorps customized skill development and at the right time sup- members apply this lens to student interactions, learning ports consistently over the school year. As such it offers environment creation, academic and social-emotional potentially a third way, along with curricular/programmat- instructional practices, and individual and group engage- ic approaches and efforts to integrate SEL into academic ment. This positioning offers a consistent alternative way course instruction and classroom and school climates, to to interpret student behaviors and address the needs develop the social-emotional skills and outlooks we find associated with them. For instance, instead of labeling are important to school success. fourth grade student Alice as “fidgety and unable to control her body,” AmeriCorps member Dan could reframe CITY YEAR’S WHOLE SCHOOL WHOLE CHILD Alice’s behavior as indicative of her “having a strong need for active engagement.” He would then change his ped- APPROACH AND THEORY OF CHANGE agogical and interpersonal practice with her to include City Year, an education non-profit founded in 1988, places frequent physical breaks and lessons beginning with an a team of eight to fifteen diverse AmeriCorps members in active game or a walk and talk. systemically under-resourced K-12 urban elementary, mid- City Year’s services are, by design, comprehensive. Not dle, and high schools, where they serve full-time as near- only do AmeriCorps members support small group peer tutors, mentors, and role models. The AmeriCorps instruction and provide whole-school events and af- members partner with classroom teachers and school ter-school programs for all students, they also work principals to employ a holistic and positive approach that regularly with and monitor performance for students on integrates the academic with the social-emotional (SEAD) “focus lists.” and places developmental relationships at the center of the practice. There is an equal emphasis on social-emo- These are students who are recommended by their teach- tional mindset and skill development, the creation of a ers for intervention because they exhibit one or more whole-school learning environment and directly support- “early-warning indicators” (Bruce, et al., 2011) that place ing the progress of individual students. In 2018-19, 3,000 them at increased risk of dropping out: low attendance, City Year AmeriCorps members served in 350 schools in 29 poor behavior/social-emotional skills, or course failure in communities across the United States. English or mathematics. City Year measures its social-emotional development work using the Devereux Student AmeriCorps members apply a consistent asset-based Strengths Assessment (DESSA), a standardized, observa- lens and purposefully promote authentic relationships, tional, strengths-based assessment of student compe- integrated academic and social-emotional growth, and in- tencies. City Year AmeriCorps members are also starting creased student motivation and engagement. AS the crit- to administer the Holistic Student Assessment (HSA), a ical mass of the AmeriCorps cohort uses research-based data-driven tool that captures students’ perceptions of strategies and curricula to reinforce these conditions their own social-emotional growth, school experience, throughout the school day and after school, students resiliencies, and engagement. are better able to make academic and social-emotional strides, reach responsible decisions, construct meaning Guided by this data, City Year provides customized from their experiences, reflect on their identity and learn- supports to ensure student and school success such as tu- ing, and feel a sense of safety and belonging. toring in academics or social-emotional skill development,

9 building relationships that help students stay engaged in DATA SAMPLE class, organizing school-wide events that foster a positive culture and climate, and running after-school programs For this study, City Year provided access to data from the that both reinforce the core curriculum and expose stu- 2017-18 school year for 38,131 students from 326 schools dents to expanded curricular options and potential career across the nation. Students ranged from grades 3-10, tracks. and attended elementary, middle, and high schools in 28 school districts across 20 different states. As described City Year’s Theory of Change assumes that these holistic, above and seen in the descriptive statistics in Table 1, City integrated services and intentional formative learning Year primarily partners with schools in large urban school events, delivered in a positive and developmentally appro- districts that predominantly serve students of color and priate environment by near peers, lead to student identity students from low-income backgrounds1. Within those formation, sense of agency, and durable skills. schools, the students who received support as part of City Year’s program, and for whom we have data, are those IDENTITY FORMATION: Students develop a self-narra- that teachers identified as struggling in one or more areas tive about who they are as learners and leaders and (math, English, attendance, behavior) and requested that the meaning they’ve gained from their experiences they be placed on a focus list. Thus, the sample is not rep- AGENCY: Students believe in their ability to succeed, resentative of the national population of schools and stu- advocate for themselves, and become agents of dents; we cannot assume that any of the exploratory re- change in their schools and communities sults presented in this report will hold true for all students, or even students who attend schools City Year Partners DURABLE SKILLS: Students develop durable and inte- with, however, since City Year works with the high schools grated social-emotional and academic skills, skills that with the lowest graduation rates and the elementary and fuel their identity formation and sense of agency and middle schools that feed into them, this large sample of set the stage for more complex learning and career convenience is also a very functional one, representing choices the types of schools and students that state and federal City Year AmeriCorps members are key members of a agencies most typically identify as needing support to network of caring adults in schools who are dedicated to improve student outcomes. Therefore, the patterns found ensuring students are prepared for college, career, and between students’ socio-emotional development levels civic success. As students engage in experiences that help and academic outcomes are likely to be representative of them construct a narrative about who they are and what the subpopulations that public and nonprofit organiza- their place is in their community, they also know that they tions are most interested in supporting. While the city year matter, feel a sense of personal efficacy, and can confi- data clearly focuses on specific sub-populations, its large dently grow their foundational skills. Students are then sample size and its inclusion of both academic records equipped to access the grade-level core curriculum and and measures of social-emotional skills provide an excel- are prepared to successfully graduate from high school lent opportunity to examine evidence on the relationship and navigate college and career. By gaining confidence between students’ social-emotional development and through skill acquisition, and self-assurance through their academic behaviors and outcomes. learning and making meaning of their lived experiences in community and supported by a City Year AmeriCorps member, the aim is to better position students to succeed through an integrated development of their social, emotional, and academic competencies.

1 The sites with which City Year partnered with for the 2017-18 school year are predominantly large urban city areas, including: Little Rock (AR); Los Angeles, Sacramento, San Jose (CA); Denver (CO); Washington (D.C.); Jacksonville, Miami, Orlando (FL); Chicago (IL); Kansas City (KS); Baton Rouge, New Orleans (LA); Detroit (MI); Columbia (MO); Manchester (NH); New York City (NY); Cleveland, Columbus (OH); Tulsa (OK); Philadelphia (PA); Providence (RI); Memphis (TN); Dallas, San Antonio (TX); and Seattle (WA).

10 students’ math and English achievement outcomes, the TABLE 1 – SAMPLE DESCRIPTIVES FOR SCHOOLS assessments given to students participating in City Year AND DISTRICTS IN CITY YEAR DATA SET programs vary widely among schools, and even within grade levels in a single school. In some cases, sample sizes SCHOOLS DISTRICTS were too small to create standardized (or sample normed) scores for each test and grade level. Therefore, for achieve- N 326 51 ment outcomes, we analyze only the ordinal measures CHARTER 8% - where students were identified as on or off track.

ELEMENTARY 49% - For each outcome measure, a prior measure from either MIDDLE 25% - the prior school year or from the fall is used as a co-variate, HIGH 24% - helping to address any selection bias inherent in this purposive sampling of students by controlling for students’ OTHER/OVERLAPPING 2% - initial attendance and academic levels at the start of the FREE/REDUCED LUNCH ELIGIBLE 88% year. One caveat to the dataset is that student outcome data is only available for students who were on that indi- ASIAN 3% 5% cator’s focus list in terms of City Year programming and HISPANIC 38% 28% support. Therefore, while the overall sample includes data BLACK 50% 53% for over 38,000 students, the analyses for each set of outcomes are smaller (Math – 15,187 total students; English WHITE 6% 13% – 18,923; Attendance – 9,517; Behavior/Social-Emotional OTHER 3% 3% – 11,979) and based largely on independent subsamples SPECIAL EDUCATION - 15% (only 364 students appear on all four focus lists). ENGLISH-LANGUAGE-LEARNER - 12% AVERAGE ENROLLMENT 661 60,275 SOCIAL-EMOTIONAL MEASURES AVERAGE NUMBER OF SCHOOLS - 102 The primary measure of students’ social-emotional learning (SEL) skills is their score on the Devereux Student Data obtained from 2016-17 Common Core of Data (CCD) Strengths Assessment (DESSA), which was administered to almost all students in the sample (30,043 students in fall, and 24,433 in winter/spring). The DESSA, a norm-ref- ACADEMIC OUTCOMES erenced behavior rating scale, includes information for eight separate SEL competency areas as well as an overall Within the City Year data set, measures of students’ composite measure (LeBuffe et al., 2014). Raw scores academic outcomes include attendance rates (0-100%), range from 28-72, while categorized scores range from English and math course grades (A-F; 0-4); and math and 1-3 where a score of 1 signifies that the area is a student English achievement (raw test scores). For each outcome, strength, 2 represents a typical student level, and 3 means City Year staff also codes students into a three-tier mea- it is an area in which a student requires support. Thus, the sure of on-track status: “on-track”, “sliding”, or “off-track”. higher a student scores in a DESSA category, the worse For attendance purposes, students are considered on we would expect academic outcomes to be, if there is a track if their attendance rates are at 90% or above, sliding relationship between SEL levels and academic outcomes. if their attendance is between 80-89%, and off-track if their attendance rates are below 80%. For course out- The DESSA covers the social-emotional competency areas comes, students are on track if their mark is a “C”or higher, identified as most important by CASEL: self-awareness, sliding if their mark is a “D”, and off track if they are failing self-management, social awareness, relationships, deci- the course. For achievement outcomes coding, each sion making, and responsibility, among others. With the implementation site uses publisher’s guidelines for the exception of growth mindset, the DESSA competency assessment employed to identify cut-of scores indi- areas also overlap with those found on the CORE assess- cating students are below, at, or above grade level. For ments and CASEL surveys. A key difference between the

11 DESSA and the social-emotional measures employed by Year’s Whole School Whole Child approach and students’ CORE is that the DESSA is administered by adults who academic outcomes and social-emotional learning levels. observe the students, while the CORE SEL measures are In so doing, it may also provide insight into the impact of based on student self-assessments. human-centered, relationship driven approach to improving SEL outcomes and enable comparisons with curricular DEMOGRAPHICS and learning climate approaches. For the main City Year sample, no demographic data was COMPASS ACADEMY SUB-SAMPLE available outside of gender, age, and English Language Learner (ELL) status; even these were only available for Also available for analysis is a separate, supplementary a sub-sample of school districts. Since gender and ELL data set that includes only students from a single school, status were available for so few students, and age cor- Compass Academy in Denver, Colorado. While smaller in relates very highly with students’ grade level, which is size, this sample allows to explore questions the larger available for all students, no demographic measures are data set cannot answer. These include testing the results included in the analyses below as covariates. While the of the larger study, with a data set which primary sample and analyses lack demographic data, a secondary sample from the 2018-19 school year for one a) enables , us to conduct longitudinal analyses, ob- school district alone included the key academic and SEL serving the measurement of students’ academic measures as well as demographic information such as outcomes and social-emotional skills over time, ELL status, gender, race and age. Though not as robust a b) add a wide range of demographic controls into sample, analyses of this secondary data set, it enabled us the analyses, and to re-examine any variation in the relationship between SEL skills and academic outcomes, based upon students’ c) examine impacts of SEL development on aca- demographic characteristics. demic outcomes using different measures of social-emotional skills and outlooks.

IMPLEMENTATION DATA Compass Academy is a middle school serving students in 6th-8th grades that partners with both City Year and Through City Year’s internal data collection efforts, some the Center for Social Organization of Schools (CSOS) at supplemental data on implementation is also available in Johns Hopkins University. Although Compass is a part of three different formats. For students who were placed on the larger City Year network and is included in our prima- the English, mathematics or behavior focus lists, imple- ry analytic sample, the school’s special partnership with mentation is measured in terms of the number of one-on- City Year and CSOS involves a more in-depth collection of one hours of support they received from an AmeriCorps student data, providing a richer set of information based member. For students placed on a behavior for attendance on more years of observation. The sample includes data focus list, records show the number of days they remained for 6th graders in 2015-16, 6th and 7th graders in 2016- on those lists. Lastly, for students on the behavior focus 17, and 6th-8th graders in 2017-18: a total of 661 annual list, records show the number of minutes they spent work- measurements for 338 unique students, 88 of whom were ing with AmeriCorps members on specific SEL skills. None tracked longitudinally across all three years of middle of these measures is ideal. The number of days a student school (97 students have data for two years, and another spent on a focus list is as representative of their level of 203 students have only one year of observed data). need as it is of the amount of support they received. The number of hours of direct one-on-one support from an This outcome data includes students’ attendance rates AmeriCorps member does not capture the AmeriCorps and math and English achievement scores on the Colo- members’ involvement in students’ regular classroom, rado annual state assessment (during the data collection after-school activities, or other school-wide aspects of period, a faithful replication of the PARCC assessment), the City Year program. However, these proximal measures including both pre- and post-measurements. The data of the amount of one-on-one time each student spent set’s also provides student demographics, including: race; with a corps member, or dosage, allow us to tentatively special education status; English Language Learner status; explore the relationship between a key aspect of City

12 gifted status; and gender. While students’ eligibility status for the federal Free/Re- TABLE 2 – COMPARING GMS SCORES (RANGES 1-5) TO HSA duced Lunch program is also available, it PYRAMID SCORES (RANGES 1-3) is missing for some students; almost all GROWTH FALL HSA WINTER HSA SPRING HSA observed students were eligible (95%). MINDSET 2017-18 2017-18 2017-18 Given the missing data and the near-total uniformity of the available data, it is not GROWTH MINDSET 1 .144* .120 .190** included in analyses below. FALL HSA 2017-18 .144* 1 .614*** .522***

Measures of students’ social-emotional WINTER HSA 2017-18 .120 .614*** 1 .619*** skills, the Compass Academy data includes SPRING HSA 2017-18 .190** .522*** .619*** 1 fall and winter Holistic Student Assessment *. is significant at the 0.05 level; **. is significant at the 0.01 level; ***. is significant at the (HSA) scores. Like the DESSA, the HSA is a 0.001 level norm-referenced rating scale that includes raw scores as well as tiered/category scores for 14 competency areas along with an 2) Is there any evidence that student involvement overall composite measure (Allen et. al., 2017). The Com- in an effort like City Year’s Whole School Whole pass Academy data also includes growth mindset (GMS) Child program, which takes a human-centered, scores for only the 2017-18 school year. Growth mindset, relationship driven approach to build students’ based largely on the work of Carol Dweck (2006), captures social-emotional and academic skills and keep a student’s belief that abilities (intelligence or talent) are them on track to school success, is related to not predetermined but can change over time through stronger student outcomes (both social-emotion- effort and challenge. For Compass Academy, the con- al and academic)? cept was measured through students’ responses to four questions (Appendix C), taken from the same survey that ANALYSIS was administered to students in the CORE districts (Claro & For our measure of social-emotional skills we first looked Loeb, 2017). While the DESSA scores available in the larger at students’ scores across different time-points to see City Year data set are based upon the observations of within each assessment if students’ scores from the be- adult raters, both the HSA and GMS are based on student ginning of the school year to later in the school year were self-reports through surveys. similar. Correlations between students’ pre- and post-DES- SA scores overall were 0.425. These correlations were RESEARCH QUESTIONS consistent across elementary, middle and high school grade levels (Appendix Tables B.1-B.4). A correlation of This study takes advantage of the unique aspects of the 0.425 represents a medium level of correlation, short of datasets described above to extend the research base on approaching high levels in the 0.6-0.8 or higher range. social-emotional learning. Specifically, the hypotheses we That the measures of students’ social-emotional skills are wish to test are: significantly related to each other is as expected. However, that the levels of the correlation is only in a middle range 1) Are students’ social-emotional learning skills is in itself an important point, as it shows that students’ significantly related to their academic outcomes levels of social-emotional skills are not fixed points but and predictive indicators of school success? ? do in fact vary over the course of time and schooling. This Do these relationships vary significantly across matches the findings of a recent analysis of CORE data school districts or grade-level, for students who that compared to math and reading skills, students’ so- signal the need for additional support in high cial-emotional skills were much less stable over time and poverty environments? more susceptible to contextual factors, such as changes in classroom or school environment (Soland et. al, 2019).

Similar results are also found in the supplementary data set for Compass Academy students in grades 6-8. Correla-

13 tions between HSA scores taken at different time-points the DESSA or HSA means a student is struggling more in are highly correlated but with substantial variations for that area, moving either from a strength to a normative/ students between time-points (0.5-0.6). For the two SEL typical level, or from a normative/typical level to a chal- measures in the Compass data set, correlations between lenge/need for support. For student outcomes, raw atten- the growth mindset score and the HSA are also very low, dance rates range from 0-100, and math and English raw ranging between 0.1-0.2. In this case, both measures are grades range 0-4 (F-A). Thus, with regard to raw outcomes, a based upon student self-reports. The growth mindset negative relationship or coefficient in the table below means survey, however, clearly captures a different SEL concept that when students struggle more on a SEL measure, their from those measured by the HSA, which focuses more academic outcomes are lower. on relationships, resiliencies, and academic and school engagement. For achievement tests, raw scores are not reported, since different schools used many different tests (even within schools and grade levels) that sample sizes were too small RELATIONSHIP OF SOCIAL-EMOTIONAL SKILLS TO to reliably standardize. While greater attention is typically STUDENT OUTCOMES paid to standardized test scores, research has shown that’ course grades and attendance are often stronger predic- MAIN ANALYSES tors of students high school and postsecondary outcomes Analyses of the relationship between students’ DES- (Farrington et. al., 2012). However, for achievement scores, SA scores and their academic outcomes relied upon as well as for attendance rate and course performance multi-level regression models to account for the nest- outcomes, we were able to report the results of models ed nature of students within schools, and the schools predicting the odds of a student being ‘off track’ in the giv- themselves within districts. Multi-level modeling is similar en outcome. These models were logistic models for binary to regression modeling but takes into account the fact outcomes where students were coded as ‘1’ if they were that with nested data, students within the same school off track in the related academic outcome, and ‘0’ if they will have shared similar experiences and thus they will were on track or sliding. In the tables below, estimates not be independent of each other, violating a statistical from logistic models for off track outcomes are odds-ratios assumption of standard regression modeling (Snijders & that can be interpreted as the odds of being off track for Bosker, 1999; Bryk & Raudenbush, 2002). Models using the a student whose DESSA or HSA score worsens by one-tier. Compass Academy data set are also multi-level (e.g. the Odds-ratios above 1.0 mean that a student is more likely DESSA models), but as opposed to observing students to be academically off track, while odds-ratios below 1.0 within schools, the Compass study analyses investigate mean a student is less likely to be off-track. students’ academic outcomes longitudinally over time for MAIN FINDINGS given students. All models control for students’ grade level as well as a prior measure of the focus outcome, which as Results for DESSA scores found statistically significant and a covariate helps to control for some of the selection bias consistent relationships between students’ social-emotional inherent in our purposive sample of City Year schools and skills, as recorded by the DESSA, and their academic out- students. Estimations of SEL academic impacts that do not comes. The results were highly significant across all out- include prior measures of outcomes, moreover, have also comes, and across all sub-score areas. The DESSA compe- been shown to produce over-estimates (Hart et. al. 2020) tency areas that more strongly relate to student outcomes are Personal Responsibility and Goal-Directed Behavior, In the tables below, the estimates reported (model coeffi- followed by Self-Management and Decision Making. How- cients) represent the effect that moving up one tier level ever, in general, the composite measure for the entire SEL on the DESSA or HSA domain has on the identified stu- scale is stronger than the individual competency domains dent outcome. In all cases, moving up one tier on either (Table 3)2.

2 There was statistically significant variation in the relationship across school and districts, such that SEL measures had stronger ties to student outcomes in some schools and districts and weaker relationships in others. However, the variation was random and not linked to any known characteristics of schools and districts. Further, the variation was inconsistent across outcomes; particular schools or districts might have a stronger-than-average relationship between SEL measures and student attendance, but a weaker-than-average relationship between SEL measures and student course grades.

14 For raw outcomes, the impacts of moving up one tier on the DESSA range from 0.15 to 0.42 in terms of effect size, with many ranging between one quarter to one EXAMINING THE ROLE OF third of a standard deviation (0.25-0.33). In general, effects of such sizes are considered to be large and sub- STUDENT DEMOGRAPHICS stantial shifts in the context of comprehensive school reform and student achievement (Borman et. al., 2003; An additional sample of data was available from Dynarski, 2017). To put the magnitude of these effects one school district for the 2018-19 school year, in another context, multiple studies have estimated one year after the sample in our main analyses. such effect sizes as the equivalent of an entire school While this sample is much smaller and less ro- year of academic achievement growth for students in bust than our main sample, it included all the grades 3-10 in mathematics or English (Bloom et. al., 2008; Hanushek, Woessmann, & Peterson, 2012; Lipsey same student data with the addition of stu- et. al. 2012). Another method of translating the practical dent demographic information. This allowed importance of an intervention’s effect is the What Works us to re-test the findings in our main sample, Clearinghouse’s “Improvement Index” (U.S. Department and additionally to examine whether relation- of Education, 2014). The improvement index can be ships between students’ SEL skills and academ- explained as the expected change in percentile rank ic outcomes varied by students’ demographic for the average student if that student had received the backgrounds. Additionally controlling for gen- intervention. Effect sizes in the 0.25-0.33 range are considered equivalent to raising the average student 10-13 der, English-language learner, race, and over- percentiles in a normal population. In terms of the age-for-grade status in our statistical models, probability of being off track, a student who moves up showed the relationship between SEL skills and one tier on the DESSA (from a Strength to merely Typi- academic outcomes remained statistically sig- cal, or from Typical to a Need for Instruction) is twice as nificant and positive, across outcomes and SEL likely to be off track in any of the academic outcomes. skills. There was no consistent evidence that the It is also worth noting that, including the composite relationship between SEL skills and academic measure of the DESSA in the statistical models accounts outcomes differed between male and female on average for 5% of the student-level variation in out- students, by race, or for English-language learn- comes, after controlling for students’ prior measures on ers. For overage students, SEL skills were even these outcomes and their grade level. In comparison, more strongly linked to their attendance at students’ prior measures on the outcomes and their grade levels accounted for an average 16% of the varia- school than for students who were age appro- tion between students in academic outcomes. Similarly, priate for their grade (but not more strongly for research conducted by the developers of the DESSA course mark and achievement test outcomes). found that its measures of students’ social-emotional Results can be seen in Appendix tables D.1-D.4. skills accounted for roughly 15-16% of the variation in students’ math and reading achievement scores respectively, while family income, often considered a key de- terminant, accounted for only 8% (LeBuffe et. al., 2014). These findings affirm that students’ social-emotional levels account for a substantial amount of the variation in their academic outcomes, comparable in size to the effect of their family background, and suggests that addressing students’ SEL skills is a viable path to raising their academic outcomes.

With regard to the timing of students’ SEL skills assessment, our results suggest that their winter/spring TABLE 3 – POST DESSA MEASURES AND STUDENT ACADEMIC OUTCOMES

ENGLISH MATH ATTENDANCE ENGLISH COURSE MATH COURSE ACHIEVEMENT ACHIEVEMENT

OFF- SEL MEASURE RATE GRADE OFF-TRACK GRADE OFF-TRACK OFF-TRACK OFF-TRACK TRACK -2.5*** -0.3*** -0.3*** SELF-AWARENESS 2.0*** 2.2*** 2.0*** 1.5*** 1.6*** (0.21) (-0.25) (-0.28) -2.7*** -0.4*** -0.4*** DECISION MAKING 2.0*** 2.4*** 2.0*** 1.4*** 1.5*** (-0.22) (-0.30) (-0.31) GOAL-DIRECTED -2.4*** -0.4*** -0.4*** 1.9*** 2.4*** 2.6*** 1.7*** 1.9*** BEHAVIOR (-0.20) (-0.35) (-0.36) -2.5** -0.4*** -0.4*** SELF-MANAGEMENT 1.8*** 2.4*** 2.3*** 1.4*** 1.9*** (-0.21) (-0.33) (-0.32) -2.1*** -0.3*** -0.3*** OPTIMISTIC 1.7*** 2.1*** 1.9*** 1.4*** 1.5*** (-0.17) (-0.24) (-0.26) -1.8*** -0.3*** -0.3*** RELATIONSHIP SKILLS 1.6*** 2.0*** 1.7*** 1.3*** 1.5*** (-0.15) (-0.21) (0.21) PERSONAL -2.8*** -0.5*** -0.5*** 2.2*** 2.9*** 2.9*** 1.4*** 1.6*** RESPONSIBILITY (-0.23) (-0.40) (-0.42) -2.3*** -0.3*** -0.3*** SOCIAL AWARENESS 1.8*** 1.9*** 1.9*** 1.4*** 1.4*** (-0.19) (-0.25) (-0.22) -2.8*** -0.4*** -0.4*** COMPOSITE SCORE 2.1*** 2.6*** 2.6*** 1.5*** 1.8*** (-0.23) (-0.34) (-0.36)

*. is significant at the 0.05 level; **. is significant at the 0.01 level; ***. is significant at the 0.001 level (Effect Sizes in parentheses)

DESSA scores were somewhat stronger in predicting EXPLORATORY/SUPPLEMENTAL ANALYSIS end-of-year academic outcomes than were fall DESSA scores. Both fall and spring/winter measures, each of these In the supplementary Compass data set, the HSA mea- cross-sectional measures taken at one point in time, were sures of SEL based on student self-reports have significant stronger and more significant predictors than was growth correlations to student outcomes, though these are not as in SEL skills, as measured by change in students’ DESSA strong or as consistent as the DESSA adult rating measures scores between the two time points (Tables B.5-B.7). in the larger City Year data set. Again, mid-year or winter Relationships between DESSA levels and student out- scores are more strongly related to end of year outcomes comes were also consistent across grade levels, though than are fall scores (results for fall scores can be seen in the magnitude of the relationship between DESSA scores Appendix Table B.19), and cross-sectional measures of SEL and student academic outcomes is slightly larger for 9th skills are stronger than measures of growth between the grade students than for students in grades 3-8 (Tables fall and winter time-points. Examining the relationships B.11-B.13). One explanation for this is that in 9th grade, between the various HSA competency areas and student when students encounter more challenging curriculum outcomes reveals that the relationship to test scores is and higher teacher expectations, social-emotional skills almost singularly driven by the Academic Motivation such as self-management, decision making, and personal sub-category, a finding that is similar to other research on responsibility are critical to keeping up with increased the HSA which has found that Academic Motivation was academic rigor and responsibilities like turning in assign- the competency area most related to students’ academic ments, taking notes, and studying for tests. outcomes as measured by course grades (Allen et. al.,

16 TABLE 4 – WINTER HSA AND GROWTH MINDSET MEASURES AND STUDENT ACADEMIC OUTCOMES, COMPASS ACADEMY

ELA MATH

ATTENDANCE SCALE PERFORMANCE GROWTH SCALE PERFORMANCE GROWTH RATE SCORE LEVEL PERCENTILE SCORE LEVEL PERCENTILE

ACTION ORIENTATION -0.5 -1.3 0.0 -1.3 0.9 0.0 0.1 EMOTIONAL CONTROL 0.2 -1.6 -0.1 -1.9 -2.1 -0.1 -4.1* ASSERTIVENESS -0.7 -1.8 -0.1* -1.9 1.6 0.0 0.5 TRUST -1.0* -0.6 0.0 -2.5 0.5 0.0 -0.4 EMPATHY -1.8*** -2.9* -0.1 -3.7* -1.6 0.0 -3.0 REFLECTION -1.3** -2.5 -0.1 -2.7 -2.1 -0.1 -3.7 OPTIMISM -1.6*** -1.5 -0.1 -1.7 -0.7 0.0 -1.1 RELATIONSHIP WITH PEERS -1.6** -1.5 0.0 -1.2 -1.7 -0.1 -2.1 RELATIONSHIP WITH ADULTS -0.7 0.4 0.0 0.9 -1.7 -0.1* -4.1* LEARNING INTEREST -1.3** -2.8 -0.1 -4.1* -0.9 -0.1 -0.9 CRITICAL THINKING -0.9* -2.8 -0.1 -3.9* -2.5 -0.1 -4.2* PERSEVERANCE -1.1* -1.7 -0.1 -2.3 -2.4 -0.1* -4.1 ACADEMIC MOTIVATION -1.5*** -6.1*** -0.2** -6.2** -5.9*** -0.2*** -10.7*** SCHOOL BONDING -0.9* -1.9 -0.1 -3.4 -1.5 -0.1 -3.6 PYRAMID -0.8* -3.1** -0.1*** -3.9** -2.4* -0.1** -3.5* FALL-WINTER -0.2 -1.6 0.0 -2.2 0.9 0.0 1.2 PYRAMID GROWTH GROWTH MINDSET 0.0 -3.4* -0.1* -4.3* -6.6*** -0.2*** -6.2**

*. is significant at the 0.05 level; **. is significant at the 0.01 level; ***. is significant at the 0.001 level Models for sub-scores based on 2016-17 and 2017-18 data only

2019). Students’ attendance rates are also significantly test a second and related question. Given that SEL skills related to several competency areas on the HSA. Growth are tied to students’ academic outcomes, can students’ mindset scores also show a significant correlation to stu- social-emotional skills be influenced by their schools and dent achievement test scores, especially in mathematics teachers? City Year’s program in schools focuses on both confirming the findings of a recent PACE study conducted students’ academic performance and social-emotional with CORE data (Claro & Loeb, 2019b). development; implementation data from its program can be used to explore whether greater involvement with the THE CITY YEAR WHOLE SCHOOL WHOLE CHILD program and the AmeriCorps members was associated with stronger outcomes for students. APPROACH AND STUDENT OUTCOMES Measurement data of each student’s involvement with Having found evidence of a positive and significant City Year and the AmeriCorps members takes three forms. relationship between students’ social-emotional skills and For students on the math, English, or behavior focus their academic outcomes, we use the City Year data to lists, staff provided a detailed record of the number of

17 TABLE 5 – DESCRIPTIVE STATISTICS OF CITY YEAR DOSAGE

N* MEAN MEDIAN STD. DEV. MINIMUM MAXIMUM ELA HOURS 19,432 17.6 16.3 10.5 0.083 158.25 MATH HOURS 15,808 17.3 16.1 11.3 0.083 145.17 BEHAVIOR HOURS 9,246 4.4 3.2 4.5 0.067 63.33 BEHAVIOR DAYS 12,362 228.5 229 61.9 1 412 ATTENDANCE DAYS 9,727 220.9 221 65.9 3 424 CRITICAL THINKING 9,690 1.7 0 11.5 0 330 DECISION MAKING 9,690 24.0 0 53.1 0 1,840 EMPATHY 9,690 6.6 0 24.8 0 420 GOAL DIRECTED BEHAVIOR 9,690 47.5 20 74.7 0 945 LEARNING INTEREST 9,690 12.9 0 45.9 0 1,020 OPTIMISTIC THINKING 9,690 16.0 0 34.5 0 745 PERSONAL RESPONSIBILITY 9,690 24.4 10 40.4 0 465 REFLECTION 9,690 28.4 0 61.2 0 957 RELATIONSHIP SKILLS 9,690 28.6 0 61.5 0 820 SELF-AWARENESS 9,690 17.1 0 32.6 0 465 SELF-MANAGEMENT 9,690 19.5 0 37.7 0 618 SOCIAL AWARENESS 9,690 12.1 0 30.8 0 570 TRUST 9,690 10.8 0 45.7 0 1,400

* Many students who received treatment according to the dosage measures, were not on the intervention ‘watch’ lists.

hours each student received support from an AmeriCorps that is because the dosage variables are measured in member as part of the City Year program. For students on terms of hours, days and minutes. So, the effect of dosage the focus list for either attendance or behavior, City Year on students’ raw academic outcomes being reported is for staff captured the number ofdays each student spent on the effect of ‘one hour’, ‘one day’ or ‘one minute’ only. The the focus list. Finally, and only for students who were on descriptive statistics for those measures of dosage (Table the behavioral focus lists, City Year recorded the number 5) show a wide range in terms of how much time students of minutes students spent with AmeriCorps members spent receiving one-on-one support from an Ameri- working on specific social-emotional skills. In all formats, Corps member. The median, or middle, amount of time this implementation data captures only the individual spent working directly with an AmeriCorps member is 16 or small group work of City Year AmeriCorps members hours in math or English, and three hours for behavioral with students; it does not reflect indirect effects from support. All of these dosage measures have substantial whole-classroom and whole-school supports provided by standard deviations, indicating considerable variation in City Year AmeriCorps members. the amount of time different students, spent with City Year corps members. Multi-level models similar to those described in prior analyses were run, including the measures for City Year The first relationship examined is between City Year corps dosage. While some of the results/estimates in the tables member dosage and educational outcomes. If there is no may seem very small in scale (to several decimal places), relationship between time spent with a City Year corps

18 members and student on education outcomes (achieve- The relationship between time spent working one on one ment, on-track rates etc.), then it is unlikely we will find a or in small groups on math and English skills on atten- relationship between City Year dosage, social-emotional dance, could result through several mechanism. It could development and education outcomes. be, that students who feel better about their academic skills are more likely attend regular or less likely to miss This analysis revealed that the more hours a student spent days, because they do want to reprimanded for not having receiving support from an AmeriCorps member in either done an assignment or called out for not knowing some- English or math, the higher were the student outcomes, not thing or being able to perform in class. Alternatively, or only in the target subject, but also in attendance. Similarly, in conjunction, the strong relationships formed with City the more hours a student received support in a subject, the Year corps members, through one on one work in Math less likely they were to be off track, in either that course or or English, could have spill over effects on attendance, as attendance. For students who received the median num- students know there is an adult at the school, who cares ber of hours of support from an AmeriCorps member for about them and is there to help them succeed, and whom English or math, the related increase in their course grade they trust, which in turn propels them to attend more would be 0.1, equivalent to one tenth of a grade level (A-F) frequently (Balfanz and Byrnes 2018). and an effect size of 0.08-0.09. This is the same increase in course grades as that found by a recent randomized study For students receiving support for behavioral reasons, on the impact of a growth mindset intervention, conduct- there was no behavioral or disciplinary outcome to link ed by the National Study of Learning Mindsets (Yeager, directly to support. For these students, there was no asso- Hanselman, Walton, et al., 2019). The effect can be translat- ciation between hours spent with an AmeriCorps member ed to roughly 2-4 months of learning in academic achieve- and attendance at school, Compared to math and English, ment growth (Lipsey, et. al. 2012), and an improvement in- the average dosage of CY corps member time, is limited, dex gain of 3-4 percentiles. Students receiving the median three hours verse 16 hours. Thus, the dosage itself may amount of support from an AmeriCorps member would not have been sufficient to establish the strength and also be 42% less likely to be off-track in English class type of relationship needed to impact attendance. In (odds-ratio = 0.58) and one-third less likely to be off-track terms of the number of days spent on either behavioral in math class (odds-ratio = 0.66). The strong relationship or attendance focus lists, we see a similar result in that between dosage and off-track rates, indicates that the the number of days spent on the lists was significantly City Year supports may be most impactful on the students associated with lower course and achievement outcomes most likely to fall off the path to high school graduation. and an increased odds of being off track in those areas.

TABLE 6 – CITY YEAR DOSAGE AND STUDENT OUTCOMES

ENGLISH MATH ATTENDANCE ENGLISH COURSE MATH COURSE ACHIEVEMENT ACHIEVEMENT OFF- OFF- OFF- RATE GRADE GRADE OFF-TRACK OFF-TRACK TRACK TRACK TRACK ELA DOSAGE 0.05*** 0.98*** 0.006** 0.97*** 0.001 0.99* 0.99 1.00 (HOURS) MATH DOSAGE 0.08*** 0.98*** 0.000 1.00 0.007*** 0.97*** 1.01*** 0.99 (HOURS) BEHAVIOR DOSAGE 0.07 .098 -0.015** 1.03 -0.011* 1.03* 1.01 1.01 (HOURS) BEHAVIOR DOSAGE 0.00 1.00 -0.001*** 1.01** -0.001*** 1.01*** 1.01*** 1.01*** (DAYS) ATTENDANCE DOSAGE 0.01** 0.99*** -0.001*** 1.01*** -0.001*** 1.01*** 1.00 1.00 (DAYS)

*. is significant at the 0.05 level; **. is significant at the 0.01 level; ***. is significant at the 0.001 level

19 TABLE 7 – CITY YEAR DOSAGE AND DESSA OUTCOMES

BEHAVIOR BEHAVIOR ATTENDANCE DESSA DOMAIN ELA HOURS MATH HOURS HOURS DAYS DAYS

SELF-AWARENESS -0.0032*** -0.0019** -0.0121*** -0.0001 0.0002*** DECISION MAKING -0.0028*** -0.0025*** -0.0098*** 0.0001 0.0002*** GOAL-DIRECTED -0.0026*** -0.0019*** -0.0109*** 0.0000 0.0002*** BEHAVIOR SELF-MANAGEMENT -0.0031*** -0.0018** -0.0079*** 0.0001 0.0002*** OPTIMISTIC -0.0022*** -0.0017** -0.0136*** -0.0001 0.0002*** RELATIONSHIP SKILLS -0.0028*** -0.0021* -0.0153*** -0.0001 0.0002***

PERSONAL -0.0030*** -0.0017* -0.0072*** 0.0000 0.0002*** RESPONSIBILITY

SOCIAL AWARENESS -0.0024*** -0.0029*** -0.0066*** 0.0001* 0.0001* COMPOSITE SCORE -0.0028*** -0.0023*** -0.0138*** -0.0001 0.0002***

However, days spent on the attendance focus list, was pos- students’ social-emotional skills. These analyses found the itively and significantly associated with the directly related more hours students spent working with a City Year Ameri- outcome of attendance rates. For students who received Corps member, the less likely students were to struggle with the median number of days of support from an Ameri- the various social-emotional competencies at the end of the Corps member for attendance, the related increase in their year (controlling for start-of-year social-emotional levels). attendance rate would have been 2.2% (effect size = 0.17), For students who received the median number of hours or roughly four days of school a year, while they would be support, the effects sizes on the composite measure of the 41% less likely to be off track in attendance (odds-ratio = DESSA would be .08, .06, and .08, respectively for English, 0.59). math and behavior. According to the WWC’s improvement index, this would be equivalent to moving the average Thus, for academic outcomes, we see a direct and pos- student up 2-3 percentiles in the population’s distribution itive link between City Year support in Math, English, of SEL skills. Thus, the more direct, holistic support they and Attendance, and those student outcomes. Students received from a City Year AmeriCorps member for math, receiving support in Math and English were then also con- English, or behavior, the stronger their social-emotional nected to better secondary outcomes in their attendance skills later in the school year (Table 7). In regard to the at school. However, we also see that students receiving length of time spent on a focus list, there was no asso- support for either Behavioral or Attendance reasons were ciation between the days spent on the behavior list and connected to lower course and achievement outcomes. It social-emotional skills, while more days spent on the at- may be that those students determined to be in need of tendance focus list was significantly associated with lower Behavioral or Attendance support were also struggling in social-emotional levels. These findings, could reflect that Math and English, and therefore pre-selected with lower more time on the attendance or behavior focus list does course marks and achievement scores but received no not necessarily imply greater dosage. In fact, they could direct support in those areas. signal students for whom interventions and supports Having examined the relationship between City Year corps are not working as more days on the focus list translates member dosage and academic and on-track outcomes, to more days of students demonstrating attendance or we next looked at the relationship between dosage and behavior challenges.

20 2018 - 19 SCHOOL DISTRICT DATA

Where the main study sample included academic outcome data only for those students who were engaged directly with City Year, the additional sample of data from the following year for one school district includes academic outcome data for all students in the six district schools working with City Year. This allowed us to do follow-up analyses for all students, not just those who received one-on-one intervention. While the inclusion of students who did not receive direct support from City Year is not the perfect counterfactual, it does provide a comparison group and allows us to re-test our original findings showing a relationship between the number of hours spent with an AmeriCorps member and students’ academic outcomes. The results of the follow-up analyses match those of the original study, confirming them and establishing a consistent pattern of results – students who spend more hours working with AmeriCorps members on English and math, had higher end-of-year outcomes in those subjects (course grades and test scores), along with higher attendance rates. Using the student demographic information available in this additional sample, there is also no consistent evidence that the relationship between hours spent with an AmeriCorps member and students’ academic or SEL outcomes varies by gender, race, English-language learner or overage-status. These results can be seen in Appendix tables D.5-D.15.

21 FIGURE 1 – PATH DIAGRAM OF CITY YEAR EFFECT

Further models, for both academic and SEL outcomes, together. To explore this, we combined the two sets of tested for an interaction between the hours spent with regression models in a structural equation model (SEM; an AmeriCorps member and students’ initial starting Bollen, 1989; Long, 1983; Mueller, 1996), using the AMOS points (fall attendance rates, course marks, and SEL levels). software package and its maximum likelihood method of Results found that the lower a student’s initial level was, estimation (Arbuckle, 2005). The path diagram from the for academic or social-emotional indicators, the stronger model for students’ English course marks can be seen in was the relationship between City Year intervention and Figure 1 and visually displays the multiple effects of hours students’ spring outcomes (Appendix Tables B.21-B.22). In of additional support by an AmeriCorps members on stu- other words, the students who began with the lowest atten- dents’ English grades as well as on their social-emotional dance rates or course marks, and those with the lowest SEL skills as measured by the composite score of the DESSA3. skills, were the ones who benefited the most from receiving one-on-one support from an AmeriCorps member. Additional support from City Year was a highly significant predictor of both, with a direct effect on students’ English With strong relationships established between City Year marks of .04 (effect size), and an indirect effect on grades Corps member dosage or time spent with students and of .02 through its effect on the DESSA (or social-emotion- both academic and social-emotional outcomes, as well so- al development), for a total effect of .06. The results of a cial-emotional development and academic performance model for students’ math course marks were similar, as the final question looked at is how do all these pieces fit support from City Year was a highly significant predictor of

3 According to several indices of fit that provide a means for determining if a model is a good fit or not (Gerbing & Anderson, 1993), our path model is a good fit to our sample and the observed data, and a close approximation to the true model. The path model in Figure 1 has a Goodness of Fit Index (GFI) of 0.98, and Adjusted Goodness of Fit Index (AGFI) of 0.92, a Normed Fit Index (NFI) of 0.84, and a Comparative Fit Index (CFI) of 0.84. For all the above indices, a statistic of 1.0 represents a perfect fit of the model to the data, and as rules of thumb a model must at least meet a standard of 0.90 to be considered an acceptable fit, and at or above 0.95 to be considered a good fit. For Math, the index measures were 0.98 (GFI), 0.92 (AGFI), 0.86 (NFI), and 0.86 (CFI).

4 Some time was also spent working on skills beyond social-emotional ones, such as homework, geometry, and other enrichment activities.

22 TABLE 8 – CITY YEAR IMPLEMENTATION AREA AND STUDENT OUTCOMES

ENGLISH MATH ATTENDANCE ENGLISH COURSE MATH COURSE ACHIEVEMENT ACHIEVEMENT

RATE OFF-TRACK GRADE OFF-TRACK GRADE OFF-TRACK RAW OFF-TRACK RAW OFF-TRACK CRITICAL 0.06 0.99 0.001 1.00 -0.002*** 1.00 -0.02 1.01** 0.08 0.99*** THINKING DECISION MAKING 0.00 1.00 -0.001 1.00 -0.001 1.00 0.06 1.00 0.00 1.00 EMPATHY 0.00 1.00 -0.002 1.01** 0.001 1.00 -0.11 1.00 0.03 1.00 GOAL DIRECTED 0.00 1.00 -0.001** 1.01** 0.000 1.00 0.03 1.00 0.02 1.00 BEHAVIOR LEARNING 0.01 1.00 0.000 1.00 0.000 1.00 0.11 1.00 -0.06 1.00 INTEREST OPTIMISTIC 0.01 1.00 -0.001 1.00 0.000 1.00 0.10 0.99* -0.09 1.00 THINKING PERSONAL 0.00 1.00 -0.001* 1.00 -0.001 1.01* 0.00 1.00 0.01 1.00 RESPONSIBILITY

REFLECTION 0.01 1.00 -0.001*** 1.01*** -0.001* 1.01*** -0.02 1.00 -0.02 1.00

RELATIONSHIP 0.01 1.00 0.000 1.00 -0.001 1.01* -0.02 1.00 0.00 1.00 SKILLS SELF-AWARENESS 0.00 1.00 -0.001 1.00 ** 1.01** 0.01 1.00 -0.03 1.00 SELF- 0.00 1.00 -0.002 1.00 -0.001 1.01* -0.09 1.00 -0.03 1.00 MANAGEMENT SOCIAL 0.01 1.00 -0.001 1.00 -0.001 1.01 -0.02 1.00 -0.09 1.00 AWARENESS TRUST 0.01 0.99* 0.000 1.01* -0.002* 1.00 0.02 1.00 -0.13 1.00

*. is significant at the 0.05 level; **. is significant at the 0.01 level; ***. is significant at the 0.001 level

both students’ math grades and their spring DESSA Com- areas from the DESSA4. As seen in Table 8 below, the time posite score, with a direct effect on students’ math grades spent working with AmeriCorps members on specific of .03 (ES) and an indirect effect of .03, for a total effect of social-emotional skills was not consistently related to stu- .06. In translated terms, the total effects are equivalent to dents’ academic outcomes. However, time spent working 1-3 months of learning in terms of achievement growth on a given social-emotional skill was statistically linked to and a gain of 1-2 percentiles as per the improvement improvements in that same competency area, at least for index. those that were measured by the DESSA tool. Similarly, time spent working on individual social-emotional skills The third and final source of implementation data, one was also significantly related to improvements on stu- that was only available for those students who had been dents’ composite DESSA scores (Table 9). That more time on the program’s focus list for behavior, examined the working on specific social-emotional skills was related to specific social-emotional skill they had worked on during improvement in those skills, but not academic outcomes, one-on-one sessions with an AmeriCorps member. For again may reflect the relatively small amounts of City those students, the data measured the amount of time, Year corps member time devoted to this particular part of in minutes, they had spent working on social-emotional their holistic support. With the exception of goal directed skills, some of which overlapped directly with competency behavior, where mean dosage was 45 minutes, the mean

23 well-led district or limited to one particular age-band of TABLE 9 – CITY YEAR IMPLEMENTATION AREA students, but rather can occur at a range of high-needs AND STUDENT SEL MEASURES schools, in high-needs school districts.

RELATED DESSA The strong associations found between spending more SUB-SCORE COMPOSITE one on one or small group time with a City Year Corps member, receiving math or English supports through City CRITICAL THINKING N/A -0.0005 Year’s whole child approach, with its stress on establish- DECISION MAKING 0.0000 -0.0001 ing positive developmental relationships between the EMPATHY N/A -0.0011*** Corps member and student, and both academic and social emotional outcomes provides insight into how the GOAL DIRECTED BEHAVIOR -0.0005*** -0.0006*** connection between social emotional development and LEARNING INTEREST N/A -0.0008*** academic outcomes occurs. The overall findings suggest, OPTIMISTIC THINKING -0.0006** -0.0009*** that the combination of building positive development relationships, while working on improving students math PERSONAL RESPONSIBILITY -0.0002 -0.0007*** or English skills, is building social-emotional skills that are REFLECTION N/A -0.0005*** beneficial in themselves, but also have an impact on the RELATIONSHIP SKILLS -0.0006*** -0.0006*** student’s academic outcomes, over and above those re- sulting from the direct work to improve them. Moreover, SELF-AWARENESS -0.0008*** -0.0010*** these interactions done over a long enough period of SELF-MANAGEMENT -0.0002 -0.0006** time over the course of school year (a median of 16 hours SOCIAL AWARENESS -0.0007*** -0.0012*** of interaction) has stronger impacts on the relationship between social-emotional development and academic TRUST N/A -0.0005** outcomes, then shorter focused attempts to more directly *. is significant at the 0.05 level; **. is significant at the 0.01 level; ***. improve key social-emotional skills. is significant at the 0.001 level LIMITATIONS, COMPLEXITIES, AND NEXT STEPS

There are several limitations and complexities associated dosage levels for all the other social-emotional skills was with this analysis and findings. First, there are the inherent under 30 minutes, and in some case, only 10 minutes. A measurement challenges regarding social-emotional skill key question for further study then, is would increased development. Both adult ratings of student behaviors, dosage in developing specific social-emotional skills lead as done in the DESSA in the main City Year sample, and to relationship with academic outcomes. student self-reports, used by the Holistic Student Assess- CONCLUSION ment and growth mindset measures employed in the supplemental Compass Academy data, have biases and In summary, the available implementation data for the limitations. In addition, social-emotional development City Year program suggests that school-based interven- is not typically a linear or even process and this, along tions can be effective in developing students’ social-emo- with compounding measurement error, may explain why tional skills. The data indicates that the more time a stu- growth measures did not have the same consistent and dent spent working directly with a City Year AmeriCorps significant relationship with academic outcomes as point- member the greater the student’s social-emotional skills in-time status measures did. as measured by the DESSA; students with stronger social-emotional skills obtained greater academic outcomes A second set of limitations and complexities stems from in terms of course grades, achievement tests, and im- the nature of the City Year data sets. City Year only collect- provements in predictive indicators of high school gradua- ed social-emotional and academic behavior data (attention. These findings, moreover, were drawn from a large dance, behavior, and course performance and academic multi-district sample including elementary, middle, and achievement measures) for students on its focus lists. To early high school grades. This suggests that they are not be on a City Year focus list, students had to signal some the result of extraordinary efforts in an innovative and/or level of challenge or distress at school. They had to have

24 low attendance, behavioral infractions, or struggles with what strategies and practices are most effective at raising their courses or academic assessments. Some students students’ SEL levels, perhaps through A/B testing. A were selected for focus lists based on teacher observation second follow-up question is: how does the overall school of struggle or need for additional support. Thus, while the environment contribute to students’ social-emotional sample represents the very students for whom the quest skill development? We will seek to better understand to find effective interventions and supports is the most the connection between the overall school climate and intense and needed, it is not a representative sample of settings in which SEL supports are provided, and stu- all students or even all students attending high-need dents’ SEL skill development and aligned improvement in schools. In addition, the students in the sample are those academic outcomes, through analysis of school climate who could be expected to receive additional supports data, district data, City Year dosage data, City Year practice beyond those typically provided to all students. In this mapping data, and school conditions. A third question light, the City Year sample represents a large, multi-dis- for future research is: how do students’ SEL skills develop trict, multi-grade sample of the connection between SEL over time, and how does their relationship to academic and academic outcomes among students receiving extra outcomes operate over time? Using longitudinal data supports in these dimensions. The education field is rife sets for several consecutive years, we would like to study with interventions that do not work at scale; at some level, how students’ SEL skills become fluent and stable over most students who attend systematically under-resourced time, and whether specific skills are associated with either schools are taking part in some form of intervention or shorter or longer term effects on academic outcomes. All another. Thus, the mere fact that the sample represents of these anticipated research areas could probably best students receiving interventions does not make it unique be explored with a mixed methods approach, including nor more likely to engineer a connection between SEL both qualitative and quantitative methods. In either these skills and academic outcomes. In interpreting the find- proposed studies, or others, it would also be desirable to ings, it is important to keep in mind that it is a sample of confirm the findings of this initial study and observation students being directly supported in the outcomes being tools, but using a more conclusive form of counterfactual. measured (though at some level the same can be said of This would primarily entail including districts, schools, and many advantaged students). students who have no involvement with City Year, and in- volve the administration of the DESSA by non-AmeriCorps Finally, it is important to know that the adults doing the members to measure students’ SEL skills. ratings on the DESSA scales are the City Year AmeriCorps members. Knowing the students and their academic performance in school could lead to corresponding ratings of their social-emotional skills. However, the fact that fall DESSA scores were also significantly and consistently correlated to students’ end-of-year academic outcomes makes this theory less plausible. Additionally, research on the DESSA has found that raters accounted for only 16% of the variation in students’ DESSA scores, and less when the raters where trained properly in its administration (Shap- iro et. al., 2016). Thus, raters’ influence on students’ DESSA scores are unlikely to create a substantial bias in the results of this study’s large-scale analyses, which include thousands of students from hundreds of schools across the country.

The findings reported here lead us to several immediate areas for further research. The first is: given that we have found that one level of improvement on the DESSA is associated with up to a year is worth of achievement growth, what are effective ways of raising students’ social emotional skills? Specifically, research should examine

25 26 26 REFERENCES

Allen P.J., Thomas, K., Triggs, B., & Noam, G.G. (2017). The holistic student assessment (HSA) technical report. Belmont, MA: The PEAR Institute: Partnerships in Education and Resilience.

Allen P.J., Triggs, B., & Noam, G.G. (2019). Using self-assessment for social-emotional learning: Case study of a high-needs urban school. [Unpublished manuscript.] The PEAR Institute: Partnerships in Education and Resilience.

American Institutes for Research. (2015). Collaborating districts initiative: 2015 outcome evaluation report. Washington, D.C. https://www.air.org/sites/default/files/downloads/report/Cleveland-Cross-District-Outcome-Evaluation-Re- port-2014.pdf.

Arbuckle, J. (2005). AMOS 6.0 user’s guide. SPSS.

Baggs, J. (1994). Development of an instrument to measure collaboration and satisfaction about care decisions. Journal of Advanced Nursing, 20, 176–182. https://doi.org/10.1046/j.1365-2648.1994.20010176.x.

Aspen Institute. (2019). From a nation at risk to a nation at hope: Recommendations from the national commission on social, emotional, & academic development. http://nationathope.org/report-from-the-nation/.

Atwell, M.N. & Bridgeland, J. (2019). Ready to lead: A 2019 update of principals’ perspectives on how social and emotional learning can prepare children and transform schools. Civic Enterprises with Peter D. Hart Research Associates. https://casel.org/wp-content/uploads/2019/10/Ready-to-Lead_FINAL.pdf.

Balfanz, R. & Byrnes V. (2018). Using data and the human touch: Evaluating the NYC inter-agency campaign to reduce chronic absenteeism. Journal for Education Students Placed at Risk, 23(1-2) 107-121. https://www.tandfonline. com/doi/full/10.1080/10824669.2018.1435283.

Bloom, H. S., Hill, C. J., Black, A. B., & Lipsey, M. W. (2008). Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions.Journal of Research on Educational Effectiveness, 1(4), 289–328. https://doi.org/10.1080/19345740802400072.

Bollen, K.A. (1989). Structural equations with latent variables. John Wiley and Sons.

Borman, G. D., Hewes, G. M., Overman, L. T., & Brown, S. (2003). Comprehensive school reform and achievement: A meta-analysis. Review of Educational Research, 73(2), 125-230. https://doi.org/10.3102/00346543073002125.

Bridgeland, J., Bruce, M., & Hariharan, A. (2013). The missing piece: A national survey on how social and emotional learning can empower children and transform schools. Civic Enterprises with Peter D. Hart Research Associates. https:// www.casel.org/wp-content/uploads/2016/01/the-missing-piece.pdf.

Bryk, A. S., & Raudenbush, S. W. (2002). Hierarchical linear models. Sage Publications.

Cantor, P., Osher D., Berg, J., Steyer, L., & Rose, T. (2018). Malleability, plasticity, and individuality: How children learn and develop in context. Applied Developmental Science, 23(4), 307-337. https://doi.org/10.1080/10888691.2017.1398649.

CORE SEL Competencies. (n.d.). Retrieved from https://casel.org/core-competencies/.

27 Claro, S., & Loeb, S. (2017). New evidence that students’ beliefs about their brains drive learning. Evidence Speaks Reports 2(29). Brookings. https://www.brookings.edu/research/new-evidence-that-students-beliefs-about-their-brains- drive-learning/.

Claro, S. & Loeb, S. (2019a). Self-management skills and student achievement gains: Evidence from California’s CORE districts. PACE. https://edpolicyinca.org/publications/self-management-skills-and-student-achievement-gains-evidence-california-core-districts.

Claro, S. & Loeb, S. (2019b). Students with growth mindset learn more in school: Evidence from California’s CORE school districts. PACE. https://files.eric.ed.gov/fulltext/ED600488.pdf.

Corcoran R. P., et al. (2018) Effective universal school-based social and emotional learning programs for improving academic achievement: A systematic review and meta-analysis of 50 years of research. Educational Research Review, 25, 56-72. https://www.sciencedirect.com/science/article/pii/S1747938X17300611.

DePaoli, J.L., Atwell, M.N., & Bridgeland, J. (2017). Ready to lead: A national principal survey on how social and emotional learning can prepare children and transform schools. Civic Enterprises with Peter D. Hart Research Associates. https://files.eric.ed.gov/fulltext/ED579088.pdf.

DePaoli, J.L., Atwell, M.N., Bridgeland, J., & Shriver, T.P. (2018). Respected: Perspectives of youth on high school & social and emotional learning. Civic Enterprises with Peter D. Hart Research Associates. https://casel.org/wp-content/uploads/2018/11/Respected.pdf.

Duckworth, A.L., & Seligman, M.E.P. (2005). Self-discipline outdoes IQ in predicting academic performance of adolescents. Psychological Science, 16(12), 939-44. https://doi.org/10.1111/j.1467-9280.2005.01641.x.

Duckworth, A.L., Peterson, C., Matthews, M.D., & Kelly, D.R. (2007) Grit: Perseverance and passion for long-term goals. Jour- nal of Personality and Social Psychology, 92(6), 1087-1101. https://doi.org/10.1037/0022-3514.92.6.1087

Duckworth, A. L., & Yeager, D. S. (2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44(4), 237-251. https://doi.org/10.3102/0013189X15584327.

Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82(1), 405–432. https://doi.org/10.1111/j.1467-8624.2010.01564.x.

Dweck, C. S. (1999). Self-theories: Their role in motivation, personality and development. Psychology Press.

Dweck, C. (2006). Mindset: The new psychology of success. Random House

Dweck, C.S., Walton, G.M., and Cohen, G.L. (2014). Academic tenacity: Mindsets and skills that promote long-term learning. Bill and Melinda Gates Foundation. https://files.eric.ed.gov/fulltext/ED576649.pdf

Dynarski, S. M. (2017, August 10). For better learning in college lectures, lay down the laptop and pick up a pen. The Brookings Institution. https://www.brookings.edu/research/for-better-learning-in-college-lectures-lay-down-the-laptop- and-pick-up-a-pen/.

Farrington, C.A., Roderick, M., Allensworth, E., Nagaoka, J., Keyes, T.S., Johnson, D.W., & Beechum, N.O. (2012). Teaching adolescents to become learners: The role of noncognitive factors in shaping school performance: A critical literature review. The University of Chicago Consortium on Chicago School Research. https://consortium.uchicago.edu/ sites/default/files/2018-10/Noncognitive%20Report_0.pdf.

28 Gerbing, D., & Anderson, J. (1992). Monte Carlo evaluations of goodness of fit indices for structural equation models.So - ciological Methods and Research, 21(2), 132–160. https://doi.org/10.1177/0049124192021002002

Hanushek, E. A., Woessmann, L., & Peterson, P. E. (2012). Is the US catching up? International and state trends in student achievement. Education Next, 12(4), 25-33. https://www.educationnext.org/is-the-us-catching-up/.

Hart, S.C., et al. (2020) Nothing lost, something gained? Impact of a universal social-emotional learning program on future state test performance. Educational Researcher, 49(1) 5-20. https://journals.sagepub.com/doi/ full/10.3102/0013189X19898721.

Immordino-Yang, M.H., Darling-Hammond, L., & Krone C. (2018). The brain basis for integrated social, emotional, and academic development: How emotions and social relationships drive learning. Aspen Institute: National Commission on Social, Emotional, and Academic Development. https://www.aspeninstitute.org/publications/the-brain-basis-for-integrated-social-emotional-and-academic-development/.

Jones, S.M. (2017) Promoting social and emotional competencies in elementary school. The Future of Children, 27 (1), 49- 72. https://eric.ed.gov/?id=EJ1144815.

LeBuffe, P.A., Shapiro, V.B., & Naglieri, J.A. (2014).Devereux Student Strengths Assessment (DESSA): Assessment, technical manual, and user’s guide. Aperture Education.

Long, J.S. (1983). Confirmatory factor analysis. Sage Publications.

Lipsey, M.W., Puzio, K., Yun, C., Hebert, M.A., Steinka-Fry, K., Cole, M.W., Roberts, M., Anthony, K.S., Busick, M.D. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms (NCSER 2013-3000). Washington, DC: National Center for Special Education Research, U.S. Department of Educa- tion. http://ies.ed.gov/ncser/.

Mueller, R.O. (1996). Basic principles of structural equation modeling: An introduction to LISREL and EQS. Springer.

Panorama Education (2018). What new research tells us about the connections between social-emotional learning & the ABCs of student success. https://go.panoramaed.com/sel-abc-research.

Shapiro, V. B., Kim, B. K. E., Robitaille, J. L., & LeBuffe, P. A. (2016).Protective factor screening for prevention practice: Sen- sitivity and specificity of the DESSA-Mini.School Psychology Quarterly, 32(4), 449-464. https://doi.org/10.1037/ spq0000181

Snijders, T. A. B. & Bosker, R. J. (1999). Multilevel analysis. Sage Publications.

Soland, J., Kuhfeld, M., Wolk, E. & Bi, S. (2019). Examining the state-trait composition of social-emotional learning con- structs: Implications for practice, policy, and evaluation. Journal of Research on Educational Effectiveness, 12(3), 550-577, https://doi.org/10.1080/19345747.2019.1615158.

Taylor, R. D., Durlak, J. A., Oberle, E., & Weissberg, R. P. (2017). Promoting positive youth development through school-based social and emotional learning interventions: A meta-analysis of follow-up effects.Child Development, 88(4), 1156- 1171. https://doi.org/10.1111/cdev.12864.

U.S. Department of Education, Institute of Education Sciences. (2014). What Works Clearinghouse: Procedures and standards handbook version 3.0.

West, M. R., Kraft, M. A., Finn, A. S., Martin, R. E., Duckworth, A. L., Gabrieli, C. F. O., & Gabrieli, J. D. E. (2016). Promise and paradox: Measuring non-cognitive traits of students and the impact of schooling. Educational Evaluation and Policy Analysis, 38(1), 148–170. https://doi.org/10.3102/0162373715597298.

29 West, M. R. (2016). Should non-cognitive skills be included in school accountability systems? Preliminary evidence from California’s CORE districts. Evidence Speaks Reports 1(13). Brookings. https://www.brookings.edu/research/ should-non-cognitive-skills-be-included-in-school-accountability-systems-preliminary-evidence-from-californi- as-core-districts/

West, M. R., Fricke, H., & Pier, L. (2018). Trends in student social-emotional learning: Evidence from the CORE districts. PACE. https://files.eric.ed.gov/fulltext/ED591084.pdf

Whitehurst, G. (2019). A prevalence of “policy-based evidence making.” Education Next, 19(3), 68-74. https://www.educationnext.org/prevalence-policy-based-evidence-making-forum-should-schools-embrace-social-emotional-learning/

Yeager, D.S., Hanselman, P., Walton, G.M., Murray, J.S., Crosnoe, R., Muller, C., Tipton, E., Schneider, B., Hulleman, C.S., Hinojo- sa, C.P., Paunesku, D., Romero, C., Flint, K., Roberts, A., Trott, J., Iachan, R., Buontempo, J., Yang, S.M., Carvalho, C.M., … Dweck, C.S. (2019). A national experiment reveals where a growth mindset improves achievement. Nature, 573, 364–369. https://doi.org/10.1038/s41586-019-1466-y.

30 ACKNOWLEDGMENTS

This report is based on research funded by (or in part by) the Bill & Melinda Gates Foundation. The findings and conclu- sions contained within are those of the authors and do not necessarily reflect positions or policies of the Bill & Melinda Gates Foundation.

We would like to acknowledge the efforts of the staff at City Year in providing the data used for conducting this research, as well as thank them for their continued support throughout the project in working with that data and in providing more information about how City Year’s program is implemented in schools throughout the nation.

We would also like to thank the following people for their time and consideration in reviewing previous drafts of this report and providing us with constructive feedback: Pam Cantor (Harvard School of Education and Founder, Turnaround for Children); Bethany Little (Education Counsel); David Osher (AIR); Lisa Quay & Shanette Porter (Mindset Scholars Network); Robert Sherman (Independent Consultant); Bethiel Girma Holton (Oak Foundation); Rob Jagers (CASEL); Sara Krachman (Transforming Education); Karen Pittman (Forum for Youth Investment); Camille Farrington (Chicago Consortium on School Research); Rich Lerner (Institute for Applied Research in Youth Development, Tufts); Dave Paunesku (PERTS); John Gomperts (America Promise Alliance); Rick Hess (American Enterprise Institute). 2800 North Charles Street, Suite 420 • Baltimore, Maryland 21218 www.every1graduates.org