India Findings Report

Do Early Warning Systems and Student Engagement Activities Reduce Dropout?

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India

Volume 1: Main Findings

Contract No. EDH-I-00-05-00029-00 Task Order AID-OAA-TO-10-00010

September 2015

This document was produced for review by the United States Agency for International Development. It was prepared by Creative Associates International.

Do Early Warning Systems and Student Engagement Activities Reduce Dropout? Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India

Volume 1: Main Findings

Submitted to:

United States Agency for International Development Washington, DC

Submitted by:

Creative Associates International, Inc. Washington, DC

September 2015

This report was made possible by the American People through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of Creative Associates International and do not necessarily reflect the views of USAID or the United States Government.

DEC Submission Requirements Contract No. EDH-I-00-05-00029-00 a. USAID Award Number Task Order AID-OAA-TO-10-00010 USAID Objective Title b. Investing in People (IIP) and Number USAID Project Title and USAID Asia and Middle East Regional School Dropout c. Number Prevention Pilot (SDPP) Program USAID Program Area and Education (program area 3.2) d. Program Element Basic Education (program element 3.2.1) Findings from the School Dropout Prevention Pilot e. Descriptive Title Program Impact Evaluation in India, Volume 1: Main Findings Creative Associates International and Mathematica f. Author Name(s) Policy Research Creative Associates International, Inc. 5301 Wisconsin Avenue, NW, Suite 700 g. Contractor name Washington, DC 20015 Telephone: 202 966 5804 Fax: 202 363 4771 Contact: [email protected] Sponsoring USAID Asia/TS h. Operating Unit and COTR Eric Bergthold, COTR i. Date of Publication September 2015 j. Language of Document English, Hindi

TABLE OF CONTENTS

List of Tables ...... iv List of Figures ...... iii Acronyms ...... vi Acknowledgements ...... vii Executive Summary ...... viii I. Introduction ...... 1 II. SDPP Program in India ...... 3 A. Targeting grades and geographic areas for intervention ...... 3 B. Interventions ...... 5 1. Selecting SDPP interventions ...... 5 2. Early Warning System (EWS) ...... 9 3. Enrichment Program (EP) ...... 10 C. Program implementation ...... 10 D. Fidelity of implementation ...... 12 III. Evaluation design ...... 14 A. SDPP theory of change ...... 14 B. Research questions ...... 16 C. Evaluation design ...... 16 1. Study eligibility ...... 17 2. Primary impact analysis ...... 17 3. Impact analysis for at-risk students...... 18 4. Additional subgroup analyses ...... 19 5. Primary and additional measures of SDPP’s effectiveness ...... 20 IV. Sampling and data collection ...... 22 V. Characteristics of the sample prior to implementation ...... 24 VI. Impacts of SDPP ...... 28 A. Impacts on teacher outcomes ...... 29 1. Impacts on teacher take-up of dropout prevent practices ...... 29 2. Impact on additional teacher outcomes...... 32 3. Impacts on additional school administrator outcomes ...... 36

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page i

B. Impacts on at-risk students’ attitudes toward school ...... 39 1. Impact on primary measures of student attitudes toward school ...... 39 2. Impact on additional measures of student attitudes toward school...... 47 C. Impacts on student engagement in school ...... 52 1. Impact on student’s daily attendance in school ...... 52 2. Impact on additional measures of student engagement in school ...... 56 D. Impacts on school dropout ...... 58 1. Impact on primary measure of school dropout ...... 58 2. Impact on additional measures of school dropout ...... 63 VII. School-level dropout trends ...... 67 VIII. Discussion ...... 70 A. Overview of findings ...... 70 B. Contextual factors related to findings ...... 73 C. Conclusion ...... 75 References ...... 76

LIST OF TABLES

Table ES.1. SDPP Program impacts on primary measures of program effectiveness in India ...... xix Table ES.2. SDPP Program impacts on additional outcome measures in India ...... xix Table III.C.1. Primary and additional subgroup analyses ...... 20 Table III.C.2. Primary and additional measures of the India SDPP Program’s effectiveness at endline (SY 2012–2013, SY 2013–2014, and SY 2014–2015) ...... 21 Table IV.1. Study sample sizes ...... 23 Table V.1. Average school and target grade teacher characteristics before intervention (5th grade, SY 2012–2013) (percentage unless indicated otherwise) ...... 26 Table V.2. Average student characteristics before intervention (5th grade, SY 2012– 2013) (percentage of students unless otherwise indicated)...... 27 Table VIII.A.1. SDPP program impacts on primary measures of program effectiveness in India ...... 72 Table VIII.A.2. SDPP Program impacts on additional outcome measures in India ...... 72

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page ii

LIST OF FIGURES

Figure ES.1. Impacts of the India SDPP Program on teacher dropout prevention practices at endline (SY 2012–2013, SY 2013–2014, and SY 2014–2015) ...... xii Figure ES.2. Impacts of the India SDPP Program on at-risk student attitudes toward school at endline (SY 2012–2013, SY 2013–2014, and SY 2014–2015) ...... xiv Figure ES.3. Impacts of the India SDPP Program on daily attendance at endline (SY 2012–2013, SY 2013–2014, and SY 2014–2015), overall and by at-risk status ...... xvi Figure ES.4. Impacts of the India SDPP Program on between-grade school dropout at endline (SY 2012–2013 and SY 2013–2014), overall and by at-risk status ...... xvii Figure II.A.1. National dropout rates by grade (2009–2010) ...... 4 Figure II.A.2. Target region of the SDPP program ...... 4 Figure II.B.1. Reported causes of dropout ...... 6 Figure II.B.2. Absenteeism reported by dropouts and at-risk students ...... 7 Figure II.B.3. Treatment by teachers reported by at-risk students ...... 7 Figure II.B.4. At-risk student engagement in school ...... 8 Figure II.C.1. Rollout of the interventions in India ...... 11 Figure II.D.1. EWS: proportion of schools meeting or exceeding 80% FOI threshold ...... 12 Figure II.D.2. EWS: proportion of schools meeting or exceeding FOI threshold by component ...... 13 Figure II.D.3: Open Houses: proportion of schools meeting or E=exceeding 80% overall FOI threshold ...... 13 Figure II.D.4. Enrichment program: proportion of schools meeting or exceeding 80% FOI threshold ...... 14 Figure III.A.1. India SDPP conceptual model ...... 15 Figure III.C.1. SDPP Randomized Control Trial design in India ...... 17 Figure VI.A.1. SDPP Program impacts on teacher dropout prevention practices at endline (SY 2012–2013, SY 2013–2014, and SY 2014–2015) ...... 31 Figure VI.A.2. SDPP Program impacts on teachers’ dropout prevention practices at endline, by subgroup (SY 2012–2013, SY 2013–2014, and SY 2014–2015) ...... 32 Figure VI.A.3. SDPP Program impacts on teacher sense of self-efficacy at endline (SY 2012–2013, SY 2013–2014 and SY 2014–2015) ...... 34 Figure VI.A.4. SDPP Program impacts on teacher sense of responsibility at endline (SY 2012–2013, SY 2013–2014 and SY 2014–2015) ...... 35 Figure VI.A.5. SDPP Program impacts on administrators’ dropout prevention practices scale...... 36 Figure VI.A.6. SDPP Program impacts on administrator sense of self-efficacy ...... 37

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page iii

Figure VI.A.7. SDPP Program impacts on administrator sense of responsibility ...... 38 Figure VI.B.1. SDPP Program impacts on at-risk students’ emotional attitudes toward school at endline (SY 2012–2013, SY 2013–2014, and SY 2014–2015) ...... 40 Figure VI.B.2. SDPP’s impacts on students’ emotional attitudes toward school at endline, by subgroup (SY 2012–2013, SY 2013–2014, and SY 2014–2015) ...... 41 Figure VI.B.3. SDPP Program impacts on at-risk students’ cognitive attitudes toward school at endline (SY 2012–2013, SY 2013–2014, and SY 2014–2015) ...... 44 Figure VI.B.4. SDPP Program impacts on students’ cognitive attitudes toward school at endline, by subgroup (SY 2012–2013, SY 2013–2014, and SY 2014–2015) ...... 45 Figure VI.B.5. SDPP Program impacts on at-risk students’ behavioral attitudes toward school at endline (SY 2012–2013, SY 2013–2014, and SY 2014–2015) ...... 46 Figure VI.B.6. SDPP Program impacts on students’ behavioral attitudes toward school at endline, by subgroup (SY 2012–2013, SY 2013–2014, and SY 2014–2015) ...... 47 Figure VI.B.7. SDPP Program impact on students’ perceptions of teacher support at endline (SY 2012–2013, SY 2013–2014, and SY 2014–2015) ...... 48 Figure VI.B.8. SDPP Program impact on students’ perceptions of parental support at endline (SY 2012–2013, SY 2013–2014, and SY 2014–2015) ...... 50 Figure VI.C.1. SDPP Program impact on daily attendance at endline, overall and by at- risk status (SY 2012–2013, SY 2013–2014, and SY 2014–2015) ...... 53 Figure VI.C.2. SDPP Program impact on daily attendance at endline, by subgroup (SY 2012–2013, SY 2013–2014, and SY 2014–2015) ...... 54 Figure VI.C.3. SDPP Program impacts on math performance at endline (SY 2012–2013 and SY 2013–2014) ...... 57 Figure VI.C.4. SDPP Program impacts on Hindi language performance at endline (SY 2012–2013 and SY 2013–2014) ...... 57 Figure VI.D.1. SDPP Program impact on between-grade dropout rates at endline, overall and by at-risk status (SY 2012–2013 and SY 2013–2014) ...... 59 Figure VI.D.2. SDPP Program impacts on dropout rates, by school year (SY 2012–2013, SY 2013–2014, and SY 2014–2015) ...... 60 Figure VI.D.3. SDPP Program impact on between-grade dropout rates at endline, by subgroup (SY 2012–2013 and SY 2013–2014) ...... 61 Figure VI.D.4. SDPP Program impact on within-grade dropout (SY 2012–2013, SY 2013–2014, and SY 2014–2015) ...... 63 Figure VI.D.5. SDPP Program impact on grade progression (SY 2012–2013 and SY 2013–2014...... 64 Figure VII.1. School between-grade dropout rates by grade and school year ...... 68 Figure VII.2. School within-grade dropout rates by grade and school year ...... 69 Figure VIII.A.1. SDPP conceptual model with program impacts ...... 71

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page iv

VOLUME 2: TECHNICAL APPENDICES

APPENDIX A. Evaluation Design A.1

APPENDIX B. Analytic Methods A.12

APPENDIX C. Analyzing Teacher and School Administrator Outcomes A.19

APPENDIX D. Analyzing At-Risk Student Attitudes A.26

APPENDIX E. Analyzing School Engagement A.29

APPENDIX F. Analyzing School Dropout A.31

APPENDIX G. Supplemental Tables A.35

APPENDIX H. School–Level Trend Analysis A.49

APPENDIX I. Propensity Score Analysis of Students Identified by the EWS A.55

APPENDIX J. Data Collection Instruments A.64

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page v

Acronyms

CARE Care International

EP Enrichment Program

EWS Early Warning System

ITT Intent-to-Treat

KAPE Kampuchean Action for Primary Education

MDI Minimum Detectable Impact

OH Open House Program

PTA Parent Teacher Association

QUEST Quest Alliance

RCT Randomized Control Trial

RTE Right of Children to Free and Compulsory Education

SDPP School Dropout Prevention Pilot Program

STS School-to-School International

SY School Year

TOT Treatment-on-the-Treated

USAID United States Agency for International Development

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page vi

Acknowledgements

These findings are the result of the work of the collective SDPP team in the United States and India in its stages from research design and instrument development to data collection and entry to analysis and reporting. The following persons played a major role in the process:

Karen Tietjen (Creative Associates)

Nancy Murray (Mathematica Policy Research) Quinn Moore (Mathematica Policy Research) Emilie Bagby (Mathematica Policy Research) Ali Protik (Mathematica Policy Research) Mark Strayer (Mathematica Policy Research) Kristine Johnston (Mathematica Policy Research) Kathy Buek (formerly Mathematica Policy Research) Larissa Campuzano (Mathematica Policy Research) Jane Fortson (Mathematica Policy Research) Ebo Dawson-Andoh (Mathematica Policy Research) Owen Schochet (Mathematica Policy Research) Ron Palanca (Mathematica Policy Research) Jeremy Page (Mathematica Policy Research) Linda Molinari (Mathematica Policy Research) Serge Lukashanets (Mathematica Policy Research) Brendan Kirwan (Mathematica Policy Research) Carol Razafindrakoto (Mathematica Policy Research) Jonathan McCay (Mathematica Policy Research) Ivonne Padilla Espinosa (Mathematica Policy Research)

Aakash Sethi (IDEAL with Quest Alliance) Amitav Nath (IDEAL with Quest Alliance) Vir Narayan (formerly IDEAL with Quest Alliance)

Pranav Chaudhary (Sunai Consultancy, India)

The final report was prepared by Nancy Murray, Quinn Moore, Emilie Bagby, Ali Protik, Kristine Johnston, Mark Strayer, Owen Schochet, Jonathan McCay, Anitha Sivasankaran and Karen Tietjen.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page vii

Executive Summary

Over the past two decades, considerable progress has been made in increasing school enrollment. However, many children do not complete primary or secondary cycles once they enroll. In many countries and regions, a greater percentage of out-of-school children have dropped out of school than have never enrolled in school. Interventions have been conducted in the United States and abroad to prevent dropout, but there is limited evidence on how well they work, particularly in developing countries.

SDPP Program in India

India’s SDPP Program had two main components: (1) an EWS and (2) an Enrichment Program (EP) consisting of language, arts and crafts, and sports and games (Creative Associates International 2012a, 2012b). The SDPP Program was targeted to 5th-grade students in thirteen blocks of the Samastipur district in Bihar state, as both the grade and geographic area exhibited the highest dropout rates and would benefit most from a dropout prevention program (Shin, Jennifer, Rajani Shrestha, and Karen Tietjen 2011a).

The EWS consists of three components: (1) identification of students at risk of school dropout; (2) first response strategies; and (3) community engagement. For Component 1, the SDPP Program in India worked with teachers to identify at-risk students based on six measures of dropout risk. Component 2, SDPP helped teachers use a “track and trigger” approach to closely monitor the attendance, behavior and coursework of at-risk students and initiate “first response” activities when students show signs of struggling, ranging from in-class attention to contact via letter, phone call and/or home visit to case management meetings with school staff. Component 3 focused on raising awareness within the community about the importance of schooling and the problem of dropout, working with parent-teacher associations (PTAs) and other community groups on advocacy activities and enlisting their support for the first response activities, working closely with the school.

The EP consisted of four in-school sessions per week for all target-grade students, with special encouragements for students identified as at-risk through the EWS (Creative Associates

1 SDPP is implemented by Creative Associates International, with international partners Mathematica Policy Research and School-to-School International and local partners Kampuchean Action for Primary Education (KAPE) in Cambodia, Quest Alliance (QUEST) in India, and Care International (CARE) in Timor-Leste. Creative Associates has a corporate office in Tajikistan, which covers the responsibilities of a local partner in that country.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page viii

International 2012a). The activities were scheduled during the last class of the day, which had been designated for life skills. Each week, students participated in two sessions each of sports and language, arts, and crafts. This EP session were conducted by 5th-grade teachers and locally-hired community champions in each school. The activities were intended to provide students a chance to have fun while at school, at the same time challenging them to acquire and use new skills. The intervention was developed with cost and sustainability considerations in mind to increase the likelihood of continued implementation in India beyond the funding period.

The SDPP Program was active in schools during three school years (SY), 2012–2013, 2013–2014, and 2014–2015, exposing three cohorts of 5th-grade students to the program. For the evaluation, SDPP is able to estimate impacts on some outcomes for all three of these cohorts. However, for other outcomes, the analysis is limited to the 2012-2013 and 2013-2014 cohorts, due to the timing of data collection in the last year of the study. The teachers and school administrators were first trained during June 2012, at the beginning of SY 2012–2013 prior to the onset of the intervention in July. Subsequent trainings occurred before each academic quarter. The EWS began in July 2012 and the enrichment program sessions began in September 2012 for the current 5th graders identified as at-risk in SY 2012–2013. For entering 5th graders in SY 2013–2014 and SY 2014– 2015, both the EWS and enrichment program sessions began at the start of each year in April.

SDPP evaluated the fidelity of implementation of the EWS and EP interventions during field visits conducted in May, June, and September of 2014. Overall, schools appeared to implement the EWS as intended, scoring high levels of implementation fidelity (Creative Associates International and School-to-School International 2015). The results suggested that the EP was also implemented with fidelity (Creative Associates International and School-to-School International 2015).

Evaluation Design

SDPP hypothesized that enrichment and recreational activities, along with changes in teacher practices and increased community involvement, would improve student attitudes and behavior, particularly for at-risk students, translating into increased student engagement and reduced school dropout. Guided by this conceptual model, SDPP designed the impact evaluation to address five primary research questions:

1. Does SDPP improve teacher behavior and attitudes? 2. Does SDPP improve student attitudes? 3. Does SDPP improve student engagement in school associated with retention, such as attendance? 4. Does SDPP reduce the improve rate? 5. What are SDPP Program impacts for students most at risk of dropping out of school?

The SDPP evaluation addressed these research questions using a Randomized Control Trial (RCT) in which SDPP compared the outcomes of students and teachers in 113 schools randomly assigned to provide SDPP services to those of teachers and students in 107 schools randomly assigned to a control group providing business-as-usual services. With random assignment, exposure to the SDPP Program is not directly related to the choices or pre-existing characteristics of study

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page ix

participants, allowing attribution of any observed treatment-control differences in outcomes to the SDPP Program.

Data Collection

The data used in this report were collected from school records and through interviews with teachers and at-risk students. SDPP collected data four times from all 220 schools in the study over four school years, between SY 2011–2012 and SY 2014–2015, to follow the three cohorts of 5th- graders who were exposed to the SDPP intervention. However, SDPP was unable to observe between-grade dropout for the SY 2014-2015 cohort, due to timing of data collection. SDPP collected information from school records for 40,250 students, and conducted interviews with more than 9,900 at-risk students and 1,150 teachers and administrators.2

SDPP and control group schools had comparable characteristics at baseline, with no evidence of systematic differences between the groups. A typical school in the SDPP group enrolled about 374 students, compared with about 372 students in control group schools. The average SDPP group school had 58 students 5th-grade students, compared with 57 students in the control group. Both groups had about 7 5th-grade teachers per school. Most study schools have no active external school programs, although the Right To Education (RTE) act program was being rolled out to schools as SDPP was starting it work.

Impacts of SDPP

A conceptual model of SDPP Program activities and how they might affect student and teacher outcomes guided the design of the program and the impact evaluation. This model posits that teacher and parent knowledge and practices—shared and reinforced by the larger community— are inputs into students’ attitudes toward school and educational aspirations. These student attitudes translate into student engagement in school, including their attendance, behavior, and academic performance. The complex, cumulative interaction of these factors are inputs into the student’s ability, desire and decision to remain in school or drop out.

The evaluation estimated the SDPP Program’s impacts on teacher outcomes, student attitudes, student engagement in school, and school dropout. In each domain, SDPP focuses on a small set of key outcomes, identified before the analysis began.

The text boxes and figures below summarize the primary measures of effectiveness and findings for them in domains related to teacher practices, at-risk students’ attitudes toward school, student engagement in school, and school dropout. Estimates of the impact of the SDPP Program are based on differences in average outcomes for SDPP and control group students and teachers. These impact estimates represent the difference in the outcome of interest at endline that is attributable to the SDPP Program relative to the status quo. The estimates are expressed as percentage point differences between the treatment and control group; we also present percentage increases or decreases in the primary outcomes across the treatment and control groups. These “percentage changes” should not be interpreted as the percentage “change” that might be calculated in a pre-

2 Please see Appendix A for further details about data collection. An eligible subset of these total students and teachers collected were used in the analysis.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page x

post or baseline/endline change, but rather as the increase or decrease in the treatment group’s outcome measure in relation to the control group at endline.

Statistical tests were conducted to assess whether each impact is significantly different from zero. Impacts estimates are described as statistically significant if there is less than a 5 percent probability that it is due to chance (and not to the SDPP Program). Impact estimates are described as marginally significant if the probability that it is due to chance (and not to the SDPP Program) is between 5 and 10 percent.

SDPP effectiveness in influencing teacher outcomes

Primary research question Did SDPP affect teacher dropout prevention practices?

Primary measure of SDPP Program’s effectiveness  Teacher take-up of dropout prevention practices Additional measures of SDPP Program’s effectiveness  Teacher sense of self-efficacy  Teacher sense of responsibility  Administrator dropout prevention practices, sense of self-efficacy, and sense of responsibility

SDPP did not have a statistically significant impact on teacher dropout prevention practices (Figure ES.1). The teacher scores on the dropout prevention scale—which measures whether teachers report engaging in behavior that might help at-risk students succeed in school—were very high for teachers in both the SDPP (7.40 out of 8) and the control groups (7.45 out of 8). Thus, even without SDPP, teachers may be employing most of the dropout prevention practices measured by this scale; or, the scale scores could be artificially high in both groups because they are based on self-reports rather than direct observation. Alternatively, the impact of SDPP on teachers may have been lessened because of the important role played by the locally-hired community champions. Although teachers were the primary responsible person for implementing the EWS and enrichment activities, community champions were active in supporting most activities and were often responsible for important tasks such as contacting parents and providing support to the most at-risk students. It is also possible that some relevant teacher behaviors are not captured by this measure.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page xi

Figure ES.1. Impacts of the India SDPP Program on teacher dropout prevention practices at endline (SY 2012–2013, SY 2013–2014, and SY 2014–2015) 8 7.45 7.40

point scale - 4

2 Score eightonan

SDPP group Control group

Sources: SDPP baseline and follow-up teacher self-administered questionnaires and school records data collection; July 2012, May 2013, February 2014, and November 2014. Note: The analysis is based on 5th-grade homeroom, math and language teachers during SY 2012–2013, SY 2013–2014, and SY 2014–2015. The sample includes 329 teachers for the SDPP group and 306 teachers for the control group. Differences between SDPP and control group means were tested using two-tailed t-tests. Mean values are adjusted for baseline characteristics. The analysis accounts for clustering of teachers within schools and school-year fixed effects. For a tabular presentation of these findings, see Appendix Table G.5. ***/**/* Impact estimate is statistically significant at the .01/.05/.10 level.

The evaluation also examined teachers’ and administrators’ sense of self-efficacy (ability to respond to factors related to dropout) and sense of responsibility (whether they perceived it was their role to prevent at-risk students from dropping out). The SDPP Program had no impact on these outcomes in India. The contrast between the high sense of self-efficacy felt by teachers in both the control and treatment groups and their relatively low sense of responsibility for at-risk students is striking.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page xii

SDPP effectiveness in influencing student attitude outcomes

Primary research question Did SDPP affect at-risk students’ attitudes toward school? Primary measures of SDPP Program’s effectiveness  Emotional attitudes toward school. (i.e., student likes school)  Cognitive attitudes toward school. (i.e., student adopts better study habits)  Behavioral attitudes toward school. (i.e., student observes school requirements and rules)

Additional measures of SDPP Program’s effectiveness  Students’ perceptions of teacher support  Students’ perceptions of parental support

SDPP had a positive, statistically significant impact on at-risk students’ emotional attitudes toward school, but had no impacts on cognitive or behavioral attitudes toward school (Figure ES.2). These measures of student attitudes toward school—which were captured by surveying students identified as being at risk of dropout based on baseline characteristics—could have changed because of changes in teacher or parent attitudes and practices. 3 On the emotional attitudes toward school scale, at-risk students in SDPP schools agreed with 93 percent of the scale items, compared to 91.3 percent for students in control schools, a statistically significant difference.4

3 The three measures of student attitudes are constructed from responses to a survey administered to two cohorts of grade 9 students identified as at risk based on their 8th grade characteristics. The survey was administered at the end of the 2012/2013 and 2013/2014 school years. This survey is explained in further detail in the Technical Appendix – Section A. 4 Higher scores indicate more positive feelings about school.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page xiii

Figure ES.2. Impacts of the India SDPP Program on at-risk student attitudes toward school at endline (SY 2012–2013, SY 2013–2014, and SY 2014–2015)

100 93.0*** 91.3 91.5 91.0 90 82.6 82.6 80

10 Percentage of scale items with which students withstudents whichscale agreed items Percentage of 0 Emotional attitudes toward school Cognitive attitudes toward school Behavioral attitudes toward school

SDPP group Control group

SDPP also measured at-risk students’ perceptions of the support provided by their parents and their teachers. The SDPP Program had a statistically significant positive impact on at-risk students’ perceptions of parental and teacher support.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page xiv

SDPP effectiveness in influencing student engagement Primary research question Did SDPP affect attendance, overall or for at-risk students? Primary measures of SDPP Program’s effectiveness  Student average daily attendance Additional measures of SDPP Program’s effectiveness  Student performance in school  Student behavior in school

SDPP improved attendance for the three 5th grade cohorts exposed to the program, overall and for the at-risk students (Figure ES.3). The average rate of daily attendance was fairly low— 65.2 percent for target-grade students in the SDPP group compared to 63.5 percent for students in the control group; this difference was statistically significant. For students at-risk of dropping out of school, these rates were even lower—62.7 percent for students in the SDPP group compared to 60.8 percent for students in the control group; this difference was also statistically significant.

Both elements of SDPP—the EWS and the enrichment and recreation activities— are likely to have contributed to SDPP Program impacts on student emotional attitudes toward school and attendance. Contacting parents if students were absent may have induced at-risk students to attend school more. Fun, recreational activities may have attracted students to attend school more frequently than they would without them.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page xv

Figure ES.3. Impacts of the India SDPP Program on daily attendance at endline (SY 2012–2013, SY 2013–2014, and SY 2014–2015), overall and by at-risk status

100

90 81.7 80.9 80

70 65.2*** 63.5 62.7*** 60.8 60

30 Percentage days Percentage attended of 20

0 All students At-risk students Not-at-risk students

SDPP group Control group

Sources: SDPP baseline and follow-up school records data collection, July 2012, May 2013, February 2014, and November 2014. Note: The analysis is based on SY 2012–2013, SY 2013–2014, and SY 2014–2015 5th-grade students. The sample includes 19,018 students for the SDPP group (16,311 at-risk and 2,707 not at-risk) and 18,018 students for the control group (15,590 at-risk and 2,428 not at-risk). Differences between SDPP and control group means were tested using two-tailed t-tests. Mean values are adjusted for baseline characteristics. The analysis accounts for clustering of students within schools, and cohort fixed effects. For a tabular presentation of these findings, see Appendix Table G.5. ***/**/* Impact estimate is statistically significant at the .01/.05/.10 level. SDPP estimated impacts on students’ math and language performance, measured at the end of the school year. The SDPP Program did not affect academic performance in India.

SDPP effectiveness in influencing school dropout Primary research question Did SDPP affect school dropout, overall or for at-risk students? Primary measures of SDPP Program’s effectiveness  Between-grade school dropout Additional measures of SDPP Program’s effectiveness  Student progression in school  Alternative measures of dropout

SDPP had no impact on between-grade dropout for the two cohorts of 5th grade students that experienced the program during SY 2012–2013 and SY 2013–2014, for whom between grade dropout could be observed (Figure ES.4). Students who did not enroll in the school in the

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page xvi

following school year were considered to have dropped out of school. Students in both SDPP schools and control schools dropped out at a rate of 8.3 percent. At-risk students dropped out at a higher rate, at 9.3 percent in both SDPP and control schools. It did have a significant impact on between-grade dropout reduction for students in schools with low percentages of at-risk students.

Figure ES.4. Impacts of the India SDPP Program on between-grade school dropout at endline (SY 2012– 2013 and SY 2013–2014), overall and by at-risk status 100

Percentage students of dropped out 8.3 8.3 9.3 9.3 10 2.8 2.6 0 All students At-risk students Not-at-risk students

SDPP group Control group

Sources: SDPP baseline and follow-up school records data collection, July 2012, May 2013, February 2014, and November 2014. Note: The analysis is based on SY 2012–2013 and SY 2013–2014 5th-grade students. The sample includes 13,149 students for the SDPP group (11,283 at-risk and 1,866 not at-risk) and 12,413 students for the control group (10,606 at-risk and 1,807 not at-risk). Differences between SDPP and control group means were tested using two-tailed t-tests. Mean values are adjusted for baseline characteristics. The analysis accounts for clustering of students within schools, and cohort fixed effects. For a tabular presentation of these findings, see Appendix Table G.5. ***/**/* Impact estimate is statistically significant at the .01/.05/.10 level.

SDPP also measured grade progression, or whether a student enrolled in the next grade or higher in the following school year, and between-grade dropout, or whether a student enrolled in school in any grade during the following school year. For example, if a student repeated 5th grade, he/she would not be considered a between-grade dropout, but he/she would not be considered as having progressed to the next grade or higher. SDPP found no impact on progression from grade 5 to grade 6 in India.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page xvii

What do beneficiaries say about the SDPP Program interventions? Insight into how beneficiaries interacted with the SDPP interventions was obtained through a Qualitative Research Study. Responses from parents, students, teachers and administrators were very positive about both the Early Warning System (EWS) and the Enrichment Program (EP). EWS: Although most parents thought that the dropout problem was getting worse, few (10%) thought their child was at risk of dropping out of school. Nearly all had been contacted by the school. Most parents were happy about it, albeit surprised. They especially appreciated the home visits. Parents said that they received helpful information to support their child. They more closely monitored their children’s attendance and paid more attention to helping them with school--washing clothes and providing meals punctually so the child could get to school on time, and checking homework. Students and dropouts reacted differently to the contact: at-risk students were happy, while dropouts either remained silent or were afraid and noted the school could have done more to help them stay in school. Teachers selectively implemented EWS, but most (80%) indicated they called parents about student attendance, resulting in improved student behavior, motivation and parental support. Head masters felt EWS helped organize school records, and increased teacher responsibility and understanding of children’s needs. They appreciated the support of the community champions in “bridging the gap” with parents, something teachers were not always able or willing to do. Both head masters and teachers were concerned about continuing EWS without community champion support. Teachers suggested the EWS forms be streamlined and school management committees assume responsibility for home visits. EP: EP sessions were especially popular with at-risk children: all participated in EP and most (90%) averaged more than two sessions per week. Most (80%) at-risk students shared it improved their studies and 68 percent said it motivated them to attend on days when there were EP classes. Dropouts were less enthusiastic: 80 percent attended, but only two days per week; only 38 percent said it helped their studies; and 15 percent said they attended more on EP days. All EP teachers participated in the enrichment activities, with 15 percent of EP teachers leading sessions and 75 percent assisting the community champions. EP teachers said that they used the EP methods or activities in their regular classes. They noted the EP activities contributed to easier classroom management: students were friendlier with one another, attended more and more punctually, had greater curiosity and confidence, and showed an improvement in Hindi. Head masters and parents seconded these observations agreeing that students were better behaved and more motivated to come to school. Parents (85%) would like to see more EP activities, and 97% of them said it was not a problem if their children stayed at the school longer for the activities. When asked directly if they thought they would be able to continue with the EP classes, 70 percent of the head masters said they planned to try.

Discussion

Although SDPP did not have an impact on teacher or administrator prevention practices, it did effect improvements in key student attitudes and behaviors that can lead to dropout: at-risk students’ emotional attitudes toward school, and daily attendance both overall and among at-risk students. SDPP also led to improved perceptions of parent and teacher support among at-risk students. Despite improvement in student engagement to school as measured through attitudes and attendance, SDPP did not reduce the primary measure of school dropout—between-grade dropout for 5th-grade students—either overall or among at-risk students. However, it did have a significant impact on between-grade dropout reduction for students in schools with low percentages of at-risk students.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page xviii

Table ES.1. SDPP Program impacts on primary measures of program effectiveness in India

Impact Teacher dropout prevention practices ○ At-risk student attitudes toward school Emotional attitudes toward school +++ Cognitive attitudes toward school ○ Behavioral attitudes toward school ○ Attendance Overall +++ At-risk +++ Dropout Overall ○ At-risk ○ + + +/+ +/+ Statistically significant positive impact at the .01/.05/.10 level. — — —/— —/— Statistically significant negative impact at the .01/.05/.10 level. ○ Impact is not statistically significant.

Table ES.2. SDPP Program impacts on additional outcome measures in India

Impact Teacher outcomes Teacher self-efficacy ○ Teacher sense of responsibility ○ Administrator dropout prevention practices ○ Administrator self-efficacy ○ Administrator sense of responsibility ○ Student attitudes toward school At-risk student perceptions of parent support +++ At-risk student perceptions of teacher support +++ Student engagement Math performance ○ Hindi performance ○ Dropout Progression ○ + + +/+ +/+ Statistically significant positive impact at the .01/.05/.10 level. — — —/— —/— Statistically significant negative impact at the .01/.05/.10 level.○ Impact is not statistically significant.

There are several contextual factors that might explain why SDPP did not reduce dropout in India. Each of these factors are briefly discussed below.

Duration of exposure. SDPP focused on a single grade, allowing for maximum exposure of only one year for participating students. In addition, exposure time was limited by interruptions in the school calendar. It should be noted that the student attitudes toward school and behaviors patterns are formed over time, starting with initial enrollment. Introducing SDPP interventions—which combine targeted adult attention and support, a welcoming school environment and activities that build student engagement—at the start of a student’s academic career, as well as continuing

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page xix

throughout their schooling, could help create positive perceptions of school and better behaviors. That the program had its intended effect on key intermediate student outcomes suggests that it would have its intended effect on dropout with longer exposure to the intervention.

Enforcement of compulsory education through age 14. The potential for impacts on dropout to be observed was likely affected by concurrent changes in the education sector. The Right of Children to Free and Compulsory Education (RTE) Act of 2009 stated that children aged 6-14 should not be held back or expelled until they completed primary school. Many schools interpreted this to mean that no student’s name was to be removed from its registers, which would thus affect measurement of dropout. Furthermore, the operationalization of RTE was accompanied by a series of subsidy and support programs--such as provision of uniforms and books, school lunches and remedial lessons--mandated for all primary schools, albeit unevenly implemented. This caused an increase in the numbers of “ghost” students, i.e. students who remained on the school register, but did not attend the school. Schools were motivated to keep their student rolls inaccurately high in order to collect per student subsidies; parents, who had sent their children to private schools or essentially withdrawn them from school, registered them to collect some of the subsidies. This also affected measurement of dropout. Bihar state is the only state in India that defines dropout officially; this clear articulation of the definition of dropout occurred shortly after SDPP began work in Bihar, suggesting that a focus on dropout could have been affected by SDPP’s presence there. Based on the rich student-level information collected as part of this evaluation, SDPP found that dropout was a smaller problem in study schools (8.3 percent) than had been expected based on aggregate administrative data—collected prior to the rollout of RTE--when the program targeted the geographical areas in which it would work (26.8 percent in the target state and 33.5 percent in the target district). These reductions in the dropout rate may have made it more difficult for SDPP to further reduce dropout, and contributed to the challenges of measurement.

Complexity of factors related to dropout. Given the complex pathways and inputs that lead to school dropout, it is possible that the impacts on the intermediate outcomes SDPP services can affect are not sufficient to influence the dropout decision. In particular, economic factors that SDPP was contractually prohibited from addressing directly—such as school-related expenses, and the need to supplement income through household chores or domestic work5—were found to be particularly relevant in decisions to drop out of school (Creative Associates International 2014a).

This study shows that the SDPP Program in India was successful in improving some important intermediate outcomes—particularly students’ emotional attitudes towards school and daily attendance—but not in reducing dropout. It is useful to compare the impacts of SDPP in India to other countries where SDPP was able to target students in contexts where dropout was much higher, and where students of different ages and grades were exposed to the program for a longer period of time. Additional discussion of the impacts of the SDPP Program across all SDPP countries is presented in a separate, four-country summary report (Creative Associates International and Mathematica Policy Research 2015).

5 The SDPP dropout prevention interventions were contractually prohibited from including conditional cash transfers or economic incentives, which have already been demonstrated to be effective by prior research.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page xx

I. Introduction

Over the past two decades, considerable progress has been made in increasing school enrollment. From 2000 to 2011, the number of children out of school worldwide has decreased from 102 million to 57 million—a reduction of almost 45 percent (Millennium Development Goals Report 2013). This effort has been supported by extensive research evaluating interventions aimed at increasing access to schooling in developing countries (Petrosino, Anthony, Claire Morgan, Trevor Fronius, Emily Tanner-Smith, and Robert Boruch 2012). However, many children do not complete primary or secondary cycles once they enroll; out of the 137 million children worldwide who entered first grade in 2011, 34 million are likely to leave school before reaching the last grade in primary school (Millennium Development Goals Report 2013). In many countries and regions, a greater percentage of out-of-school children have dropped out of school than have never enrolled in school. Interventions have been conducted in the United States and abroad to prevent dropout; however, there is limited evidence on how well they work, particularly in developing countries.

The School Dropout Prevention Pilot (SDPP) Program, a five-year multi-country program funded by the U.S. Agency for International Development (USAID), is designed to identify successful means of decreasing student dropout rates in primary and secondary schools.6 Its goal is to provide evidence-based programming guidance to USAID missions and countries in Asia and the Middle East (AME) on student dropout prevention by piloting and testing the effectiveness of dropout prevention interventions in four countries: Cambodia, India, Tajikistan, and Timor-Leste. To better understand ways of mitigating dropout in these countries, SDPP used a three-stage process: (1) reviewing the literature to identify international best practices in preventing dropout; (2) analyzing dropout trends and identifying risk factors and conditions associated with dropout in each country as part of a situational analysis; and (3) designing, implementing, and evaluating interventions to keep students at risk of dropout in school. Earlier reports describe findings from the first two stages of the project (Brush, Lorie, Jennifer Shin, Rajani Shrestha, Karen Tietjen 2011; Creative Associates International 2014a, 2014b; Shin, Jennifer, Rajani Shrestha, and Karen Tietjen 2011a, 2011b; Shrestha, Rajani, Jennifer Shin, and Karen Tietjen 2011a, 2011b).

Based on the findings from the literature review and situational analyses, as well as input from key stakeholders in the four countries, SDPP worked with the Ministry of Education in each country to identify two interventions to address dropout. In all four countries, SDPP introduced an Early Warning System (EWS) and a student engagement intervention to motivate students to stay in school.

Early Warning Systems (EWS) is a dropout prevention strategy that has shown promise in the United States, but for which there is very little evidence internationally.7 EWS interventions use teacher assessments and student records data to identify students who are at risk of dropping out of school and to target them for assistance. In reviews of dropout prevention research conducted

6 SDPP is implemented by Creative Associates International, with international partners Mathematica Policy Research (Mathematica) and School-to-School International (STS) and local partners Kampuchean Action for Primary Education (KAPE) in Cambodia, Quest Alliance (QUEST) in India, and Care International (CARE) in Timor-Leste. Creative Associates has a corporate office in Tajikistan, which covers the responsibilities of a local partner 7 The SDPP dropout prevention interventions were contractually prohibited from including conditional cash transfers or economic incentives, which have already been demonstrated to be effective by prior research.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 1

by the What Works Clearinghouse, U.S. Department of Education, two versions of the EWS, the Check & Connect program and ALAS program, were found to help keep middle and high school students from dropping out and potentially help them progress in school (American Institutes of Research 2006a, 2006b). However, existing research does not tell us whether EWSs would have similar impacts in developing countries, given the different educational and cultural contexts in these countries. Similarly, the effectiveness of these types of programs on students of younger ages and lower grades is not known.

In developing countries, there is evidence that other types of interventions designed to mitigate the factors that affect dropout, such as cash transfers (in specific contexts), scholarships, and school construction, can improve enrollment, attendance, and retention (see, for example, Angrist, Joshua, Eric Bettinger, Erik Bloom, Elizabeth King, and Michael Kremer 2002; Levy, Dan, Matt Sloan, Leigh Linden, and Harounan Kazianga 2009; Schultz, T. Paul 2001). Interventions that target specific groups of students, such as girls (see, for example, Friedman, Willa, Michael Kremer, Edward Miguel, and Rebecca Thornton 2011; Oster, Emily, and Rebecca Thornton 2011) and students in rural areas (see, for example, Miguel, Edward, and Michael Kremer 2004) have also proven successful. However, to our knowledge, the SDPP evaluations present the first rigorous evidence on the effectiveness of EWSs in the developing country context. Studies of interventions that have incorporated academic activities, such as tutoring, computer labs, and other after-school activities, have had mixed results, though there is little rigorous evidence from evaluations that focus specifically on these activities (Banerjee, Abhijit, Shawn Cole, Esther Duflo, and Leigh Linden 2007; Brush, Lorie, Jennifer Shin, Rajani Shrestha, and Karen Tietjen 2011).

The SDPP Program for each of the four project countries included an EWS combined with additional activities in the schools, which varied depending on the country. 8 The additional activities were designed to motivate greater student engagement, better attendance, and increase the desire to stay in school. The interventions were rolled out to the target grades in each country at various times during 2012.

Although all of SDPP’s country programs included the EWS, they were distinct enough to merit independent evaluations in each country. The implementation of the EWS in four diverse countries allowed experimental evaluation of the effectiveness of these interventions in several contexts with different populations, strengthening the external validity of the evaluation’s findings.

SDPP conducted rigorous evaluations of the effectiveness of the SDPP Program in each of the four countries. In each country, schools that were eligible to receive the program were identified, recruited, and asked to consent to participate in the study. Eligible schools in targeted regions were then randomly assigned to either a SDPP treatment group, which offered the SDPP intervention package, or a control group, which did not. For each country, program effects were estimated by comparing the outcomes of students and teachers in SDPP program schools with the outcomes of those in control schools.

This report presents findings from the impact evaluation of the SDPP Program in India. In India, the program included an EWS in combination with a daily enrichment program for 5th-grade

8 The grades targeted in each country are as follows: grades 7, 8, and 9 in Cambodia; grade 5 in India; grade 9 in Tajikistan; and grades 4, 5, and 6 in Timor-Leste.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 2

students. The impact evaluation draws on school records and survey data collected in 113 SDPP program schools and 107 control schools across thirteen education blocks in Samastipur district, Bihar State. The student sample consists of 5th-grade students from each of the three school years (SY) (2012–2013, 2013–2014, and 2014–2015) during which the SDPP program was in effect.

The findings and information on the impact evaluation for SDPP in India are presented in two volumes. Volume 1 presents the impact evaluation findings and Volume 2 details the methodology used.

This report—Volume 1—is organized as follows: Section II describes the SDPP Program and its implementation in India. Section III discusses the impact evaluation design and describes the types of outcome domains used to evaluate the program. Section IV discusses the sample and data collection, and Section V describes the characteristics of the sample prior to implementation. Sections VI present the impacts of the program on teacher outcomes, attitudes toward school, engagement in school, and school dropout. Section VII presents school level dropout trends. Section VIII discusses the findings and conclusions.

Volume 2—a technical appendix—provides more details on the study, including the further detail on the sample frame, data collection, estimation procedures, subgroup analyses, robustness checks, and additional exploratory analyses.

II. SDPP Program in India The SDPP program in India had two main components: (1) an EWS and (2) an enrichment program offered during school (Creative Associates International 2012a, 2012b). To the extent possible, these activities built on existing Ministry of Education procedures and policies to facilitate sustainability after the project ends.

A. Targeting grades and geographic areas for intervention Using data from the District Information System for Education, SDPP identified the population for whom dropout was most prevalent and who would benefit most from a dropout prevention program. Nationally, dropout was found to be highest in grade 5: the between- grade dropout rate was nearly 16 percent (Figure II.A.1).9 The state of Bihar reported one of the highest dropout rates in grade 5 at 27 percent. Samastipur district had one of the highest between-grade dropout rates in Bihar at 33.5 percent. Consequently, the India SDPP Program was targeted to 5th-grade students in 13 blocks of Samastipur district in the state of Bihar (Shin, Jennifer, Rajani Shrestha, and Karen Tietjen 2011a). In India, 5th grade is the last year of primary school.

9 The between-grade dropout rate is the proportion of students who no longer enroll in the following year; the in-grade dropout rate is the proportion of students who drop out of school before the end of the academic year.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 3

Figure II.A.1. National dropout rates by grade (2009–2010) 18.0 15.9 16.0 14.0 12.0 10.4 10.0 7.2 8.0 6.7 6.5 5.5 5.8 6.0 4.0 2.0 0.0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7

Source: Creative Associates International (2014b).

Figure II.A.2. Target region of the SDPP program

Source: Creative Associates International (2014a, 2014b).

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 4

B. Interventions

1. Selecting SDPP interventions

SDPP selected dropout prevention interventions to be implemented on the basis of: (1) a review of the existing domestic and international evidence on interventions designed to decrease school dropout; (2) an analysis of existing policies and programs in each country that could affect dropout; (3) situational analyses of the factors and conditions associated with school dropout in each country; and (4) input from key stakeholders in each country. The literature review found little rigorous evidence on dropout prevention interventions in an international context.10 Conditional cash transfer interventions showed consistently positive impacts on school dropout, but other evidence was mixed or focused on U.S.-based interventions.

SDPP conducted primary research in October 2011 in 13 blocks within the Samastipur district (Bibhutipur, Dalsinghsarai, Hasanpur, Khanpur, Morwa, Pattouri, Pusa, Shivajinagar, Ujjiarpur, Vidyapatinagar, Warisnagar, Mohiuddinagar, Kalyanpur) in order to identify key factors and conditions associated with school dropout in India in the Samastipur district. The situational analysis collected data from at-risk students, dropouts, their parents/guardians, school administrators and teachers, community members and local education officials in 30 school- communities. Findings from the SDPP Situational Analysis indicate that the top causes of student dropout among Grade 5 students in the target blocks fell into four categories: (1) economic, (2) academic or school-related factors, (3) illness, and marriage. (Figure II.B.1).11 Economic reasons are most commonly cited by children and their families: over half of the at-risk students, dropout and parent/guardians named the need to help with chores or the family business, and 30-40 percent cited the inability to pay for school–related expenses. Approximately 40 percent cited illness. About one-third of at-risk and dropouts cited marriage as a reason, although a much smaller percentage (10%) of their parents did. Many Grade 5 students also drop out of school for academic and school-related reasons: they perform poorly and fail exams, fall behind in their lessons, are asked to leave by school authorities, experience poor school quality and are discouraged by their teachers. About 10 percent simply did not like school (Creative Associates International 2014b).

10 See Brush, Lorie, Jennifer Shin, Rajani Shrestha, and Karen Tietjen (2011) for complete findings from the literature review. 11 See Creative Associates International (2014b) for complete findings from the situational analysis.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 5

Figure II.B.1. Reported causes of dropout

70 61 At-risk students 59 60 56 At-risk parents 53 Dropout children 50 47 Dropout parents 42 43 39 38 40 36 37 37 34 30 30 30 30 28 28

20 20 1716 14 14 12 13 10 11 1010 11 9 9 9 88 8 10 6 6 7

Source: Creative Associates International (2014a).

These reasons led to or were exacerbated by high student absenteeism. (Figure II.B.2.) The majority (55 percent) of dropouts and nearly half (40 percent) of at-risk students missed more than 15 consecutive days of school over the academic year. Thirty-six percent of at risk students and 61 percent of dropouts missed 3 or more days per month. Forty percent of parents/guardians were not or seldom aware of their child’s absences.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 6

Figure II.B.2. Absenteeism reported by dropouts and at-risk students

70 61 60 55 50 39 40 36 30 20

10 Percent respondentsof 0 Absent 2+ days/month Absent 15+ consecutive days Dropouts At-risk students

Source: Creative Associates International (2014a). Many at-risk students reported poor treatment by teachers and an unsupportive school environment (Figure II.B.3). Fifty percent of the students said teachers use corporal punishment and nearly as many reported they were criticized by their teachers for wrong answers. Nearly 40 percent felt that teachers did not care about them and did not think they were intelligent. Girls felt particularly vulnerable with over a third reporting they felt unsafe on their commute to and from school and threatened while they were at school. Student engagement was low, with little more than half of at-risk students participating in school activities. An equally low percentage never completed their required homework. Over half said they were bullied (Figure II.B.4).

Figure II.B.3. Treatment by teachers reported by at-risk students

90 80 80

60 50 50 44 39 39 40 36

Percent Respondents of 20

0 Criticizes student Uses physical Does not organize Teacher does not Teacher treats Teacher does not for wrong answer punishment on study think I am/was some students care about my student groups/extra intelligent better than others problems tutoring

Source: Creative Associates International (2014a)

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 7

Figure II.B.4. At-risk student engagement in school

60 56 56

50 44

40 38

Percent respondentsof 20

10 4

0 Rarely or never Spends less than School is not fun No participation Is bullied or picked on completes required 1 hour on homework in school activities homework each night

Source: SDPP situational analysis data collection; 2011

In January 2012, SDPP convened a consultation workshop on program design in India to solicit ideas for and determine the school-based dropout prevention interventions with the greatest chance of both success and sustainability. SDPP led stakeholders—including nongovernmental organizations and education authorities—through the main findings from the literature review and situational analyses, and discussed intervention options for the SDPP program. The SDPP contract parameters—which excluded economic subsidies, vocational training, construction/infrastructure improvements and general teacher training—were reviewed. 12 Design workshop participants ranked a set of intervention options. SDPP then selected the interventions for India using these rankings along with additional program considerations. Two complementary inventions were planned, based on power calculations, sample size parameters and timeline.13

One of the complementary interventions was aimed at reducing negative student behaviors associated with dropout, such as attendance, and the other aimed at making school a more welcoming and interesting environment to increase student motivation, based on the findings of

12 While USAID’s AME Regional Bureau recognized the role that economic, infrastructure/construction, and school quality barriers could play in families’ decisions not to send their children to school, these types of interventions were excluded from experimentation for a variety of reasons. A solid research base already existed for economic subsidies and cost alleviation measures. Infrastructure improvements and construction exceeded USAID manageable interests. Desire to focus on dropout-specific interventions eliminated general teacher training for instructional improvement, which was already funded under other programs. Finally, funding for USAID Basic Education prohibited expenditure of SDPP budget on vocational education activities. 13 The SDPP contract specified two recommendations for interventions. SDPP was originally a three-year program, and did not provide sufficient time to design, develop, and implement multiple interventions in each country.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 8

the situational analysis, which found low student engagement. Stakeholders in India selected EWS as the primary program component and an enrichment program as the supplemental component. 14 2. Early Warning System (EWS)

The purpose of the EWS was to identify and provide targeted support to students at risk of dropping out of school. The EWS used existing data in schools as well as teacher input to identify at-risk students, closely monitor them, and target them for additional support. It was intended to enhance the capacity of schools to address the needs of at-risk pupils, strengthen the partnership between the parents/guardians and school personnel to monitor and improve school attendance and performance, and raise awareness among parents/guardians and the community about the value of children staying in school and what parents/guardians can do to support their children. The EWS consisted of three components: (1) identification of students at risk of school dropout; (2) first response strategies; and (3) community engagement.

In Component 1, the SDPP Program in India worked with teachers to identify at-risk students based on six measures of dropout risk. These included the globally recognized ABCs of dropout – attendance, behavior, and coursework—and were augmented with other contextually specific indicators—how often the student completes his or her homework, and whether the student’s plans after 9th grade include further schooling (Creative Associates International 2014b). SDPP worked with teachers to score and rank students’ relative risk level. Students with the highest score were deemed at-risk students.15

Component 2 used a “track and trigger” approach to closely monitor the progress of at-risk students and initiate “first response” activities when students showed signs of struggling. Teachers recorded and tracked key student behaviors, such as attendance, behavior and coursework. Signs of problems—for example, frequent absences or failed classes—triggered an immediate set of response actions, ranging from in-class attention to contact via letter, phone call and/or home visit with parents to case management meetings with school staff to develop an individualized program of intervention.

Component 3 focused on raising awareness within the community about the importance of schooling and the problem of dropout. SDPP worked with parent-teacher associations and other community groups on advocacy activities and enlisted their support in implementing some of the first response activities, working closely with the school. For many school communities, this was the first time that community or school organizations and their members engaged in student support activities, not limited to fund raising or infrastructure improvement. This component also included outreach activities, such as school events (Open Houses) to discuss how parents can support their child in school and a voice messaging system to increase awareness and inform parents of upcoming school events. These activities were designed to directly change the behaviors, community members, parents, and students themselves.

14 See Shrestha, R. and K. Tietjen, SDPP Pilot Design Plan: India (February 2012).

15 Please see Appendix A for more details on the SDPP determination of dropout risk.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 9

3. Enrichment Program (EP)

An enrichment program, aimed at enhancing students’ school experience by making school more attractive and fun, was offered to grade 5 students. It consisted of two 45-minute and two 90- minute in-school sessions per week, scheduled during the final, un-programmed period of the school day (Creative Associates International 2012a). All 5th-grade students participated in the enrichment activities, with special encouragement for students identified as at risk through the EWS. SDPP program staff divided students into two groups each session; one group participating in sports and games, while the other in language, arts, and crafts. This enrichment program was conducted jointly by 5th-grade teachers and locally-hired community “champions” in each school. The sessions took place in classrooms and on the school playground. Sports equipment and arts- and-crafts supplies were provided (Creative Associates International 2012a).

C. Program implementation

The SDPP Program was active in schools during three school years, 2012–2013, 2013–2014, and 2014–2015 (Figure II.C.1). Three cohorts of 5th-grade students were exposed to the program. The teachers and school administrators were first trained during June of 2012, at the beginning of the 2012–2013 prior to the start of the intervention. Subsequent trainings occurred before each academic quarter. The EWS began in July 2012 and the EP sessions began in September 2012 for the current 5th graders identified as at-risk in the 2012–2013 school year. For the entering 5th graders in the 2013–2014 and 2014–2015 school years, both the EWS and enrichment program sessions began at the start of each year in April. The SDPP program in India concluded in March 2015 (although final data for the impact evaluation were collected in November 2014).

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 10

Figure II.C.1. Rollout of the interventions in India

Note: The school year in India runs from April to March. T = Teacher and school administrator training begins; E = EWS intervention rolled out to students; A = Additional intervention/intervention component rolled out to students; EOA = end of activities; X = Impact evaluation data collection in schools

2012–2013 school year 2013–2014 school year 2014–2015 school year

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 11

D. Fidelity of implementation

As part of the research design, Fidelity of Implementation (FOI) was measured to determine the extent to which the SDPP interventions were being implemented as designed. SDPP evaluated the FOI of the EWS and the EP interventions. Round 1 data collection for the EWS, Open Houses (OH) and EP interventions took place between May and September 2014. Round 2 data were collected from September to November 2014 in those schools that had scores below 80 percent in round 1. Additionally, a third round of data was collected in February 2015 for the OH program in three schools that were below the 80 percent threshold in round 2.

To assess the fidelity of the EWS implementation, SDPP examined how well schools and teachers identified at-risk students, tracked their attendance, communicated with parents, and provided follow-up support.

Overall, schools appeared to implement the EWS as intended, having high levels of implementation fidelity (Creative Associates International and School-to-School International 2015). Ninety-nine percent of schools met or exceeded the 80 percent threshold (Figure II.D.1). In round 2, FOI data were only gathered for the two schools that were identified for follow-up from round 1. (Figure II.D.1). Both schools met the threshold (100 percent). Nearly all schools meant the thresholds for each EWS component (Figure II.D.2).

Figure II.D.1. EWS: Proportion of schools meeting or exceeding 80% FOI threshold

99% 100%

Early Warning Response System Early Warning Response System Round 1 Round 2 (n=113) (n=2)

Source: Creative Associates International and School-to-School International (2015).

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 12

Figure II.D.2. EWS: proportion of schools meeting or exceeding FOI threshold by component

100% 100% 99% 100% 100% 91% 73%

50%

Component 1 Component 2 Component 3 Component 4 (Identify At-Risk (Track At-Risk (Communication with (Follow-up Action) Students) Students) Parents)

Round 1 (n=113) Round 2 (n=2)

Source: Creative Associates International and School-to-School International (2015).

As part of the EWS outreach activities, schools invited parents and community members to Open Houses to showcase student work and discuss how to prevent dropout. This component consisted of six essential activities: agendas, attendees list, enrichment displays, family journals, posters and voice messages to announce the OH events. The FOI data are reported as a single score because some of the components consisted of a single question.

Overall, the OH program received high FOI scores and a high proportion of schools exceeded the threshold score of 80 percent in all three rounds. (Figure 11.D.3) The FOI tool was administered in all 113 SDPP schools for rounds 1 and 2, since different aspects of the OH program were being tracked. For example, in round 1, the distribution of materials such as games and posters was tracked; in round 2, the use of these materials by families was tracked. Round 3 of FOI focused only on those schools which were below the 80 percent threshold in round 2. The round 2 tool was used in Round 3.

Figure II.D.3: Open Houses: Proportion of schools meeting or exceeding 80% FOI threshold

98% 97% 100%

Open House Open House Open House Round 1 Round 2 Round 3 (n=113) (n=113) (n=3) Source: Creative Associates International and School-to-School International (2015).

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 13

To assess the implementation of the Enrichment Program (EP), SDPP looked at whether the EP activities were prepared, conducted classroom observations to determine how EP were carried out (on assigned days, on time, teacher engaged), assessed if there was meaningful participation (i.e. energizer completed, introduction, etc.) and assessed the student participation. FOI was high (between 86 and 100 percent) in round 1. (Figure II.D.4). The school with the lowest score was revisited in round 2, even though it had already exceeded the threshold in round 1 at 86 percent. In round 2, however, this school did not meet the 80 percent threshold for overall FOI score.

Figure II.D.4. Enrichment Program: Proportion of schools meeting or exceeding 80% FOI threshold

100% 100% 99% 92%

0% 0% 0% 0%

Component 1 Component 2 Component 3 Component 4 (Preparation) (Classroom Observation) (Meaningful (Students' Participation) Participation)

Round 1 (n=113) Round 2 (n=1)

Source: Creative Associates International and School-to-School International (2015).

III. Evaluation Design

A. SDPP theory of change

A conceptual model of SDPP Program activities and how they might affect student and teacher outcomes guided the design of the impact evaluation (Figure III.A.1). This model posits that teacher and parent knowledge and practices—shared and reinforced by the larger community— are inputs into students’ attitudes toward school and educational aspirations. These student attitudes translate into student engagement in school, including their attendance, behavior, and academic performance. The complex, cumulative interaction of these factors are inputs into the student’s ability, desire and decision to remain in school or dropout.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 14

Figure III.A.1. India SDPP conceptual model

The ultimate goal of these activities is to reduce school dropout. SDPP activities involved working directly with teachers, administrators, community champions, and parents with this goal in mind. Teachers received extensive training to influence their attitudes toward and practices used with at- risk students, as well as instruction on how to use a new EWS. This EWS was designed to improve student attendance and attitudes toward school, both directly (through interactions with students) and indirectly (through interactions with teachers and parents). The idea behind this system is that, by changing teacher, administrator and parent knowledge and behaviors toward students, student attitudes toward school should change. This improvement in attitudes should lead to more student engagement in and attachment to school, which in turn should reduce school dropout. Likewise, the enrichment program activities were designed to improve teacher-student interactions and, as a consequence, student attitudes toward school by offering recreational activities to encourage attendance.

Recognizing the complex processes that lead to dropping out, mechanisms through which the SDPP interventions aim to reduce dropout are varied and focused on influencing intermediate outcomes—specifically the attitudes, practices, and behaviors of teachers, parents, and students—that that may be related to dropout.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 15

B. Research questions

Based on this conceptual model, SDPP designed the impact evaluation to address five primary research questions: 16

1. Does SDPP improve teacher behavior and attitudes? 2. Does SDPP improve student attitudes toward school? 3. Does SDPP improve student engagement in school associated with retention, such as attendance? 4. Does SDPP improve the dropout rate? 5. What are SDPP’s impacts for students most at risk of dropping out of school?

C. Evaluation design

To answer these questions, SDPP used a randomized controlled trial design, as depicted in Figure III.C.1. SDPP randomly assigned schools to either an SDPP group that provided the SDPP program, or a control group that operated as usual. SDPP estimated program impacts by comparing relevant outcomes for students and teachers in schools offering the SDPP Program (the SDPP group) to the outcomes of students and teachers in schools offering regular services (the control group). What is an impact evaluation? A randomly assigned control group is a crucial element of a rigorous impact evaluation because it allows the  Goal: an evaluation designed to evaluator to estimate what would have happened in the provide answers to policy or absence of the program. With well-implemented random program effectiveness aims to assignment, the students and teachers in treatment generate rigorous evidence to schools will be similar to those in control schools in terms answer questions about program of their pre-existing characteristics. The only systematic impact difference between these groups is that the students and  Purpose: establish the causal teachers in the treatment group were offered the SDPP effect of an intervention through Program, and the students and teachers in the control the use of a counterfactual group were not. The result is that any observed treatment-  Mechanism: Comparison of control differences in outcomes can be attributed to the outcomes for those randomly SDPP Program and not to pre-existing differences in the assigned to SDPP and control characteristics of students, teachers, and schools in the groups sample.

16 See Murray, Nancy, Quinn Moore, Larissa Campuzano, Kathy Buek, Emilie Bagby, and Mark Strayer 2012 for details on the evaluation design.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 16

Figure III.C.1. SDPP Randomized Control Trial design in India

Eligible Schools

Random Assignment

SDPP EWS + Enrichment Control group Program group

From 220 eligible schools in India, 107 schools were randomly assigned to the control group and 113 schools were randomly assigned to the SDPP group. Over the course of the project data were collected on 19,541 and 1,199 individual students and teachers in control schools, and 20,713 and 1,295 individual students and teachers in SDPP schools.

1. Study eligibility

To be eligible for the evaluation, blocks within Samastipur district had to meet four criteria: (1) accepted the randomized controlled trial (RCT); (2) were located in a non-flood and non-crime prone area; (3) experienced high dropout rates in grade 5; and (4) had a sufficient number of schools offering grade 5. Within each eligible block, schools were eligible if they: (1) were located in a rural area: (2) had a sufficient number of grade 5 students; and (3) had playground space available for the enrichment program activities. SDPP identified 220 schools that met these criteria17; 113 were randomly assigned to the SDPP group and 107 were randomly assigned to the control group. Random assignment was conducted in block-wide meetings attended by administrative officials from each district and each school.

2. Primary impact analysis

Given the RCT design, the assessment of the SDPP Program’s effectiveness focuses on the difference in average outcomes at our final follow-up between students and teachers randomly assigned to the SDPP group and those randomly assigned to the control group. Because random assignment means that there should be no systematic differences in baseline characteristics between the SDPP and control groups, a simple difference in outcomes across groups provides a rigorous, unbiased estimate of the SDPP Program’s impact. However, we are able to increase the precision of the impact estimates and our ability to identify impacts as statistically significant by using statistical models that adjusted for small differences in the initial characteristics of the study groups that may have arisen by chance or because of survey nonresponse.18 In this way, the impact estimates adjust for baseline values of the outcomes of interest, as well as individual and school-

17 With this sample size, and a 33.5 percent dropout rate, power calculations indicated that the study would be able to detect an impact of 7 percentage points or higher. Please see Appendix A for more information. 18 Statistical significance is explained in Section IV; see text box on “statistical significance”.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 17

level characteristics. RCT impact estimates are considered the gold standard in evaluating program effectiveness.19

These impact estimates represent the difference in the outcome of interest at endline that is attributable to the SDPP program relative to the status quo. The estimates are expressed as percentage point differences between the treatment and control group. The impact estimates reported in this study should be interpreted as the difference in outcomes that resulted from exposure to SDPP. For example, an “X” percentage point favorable impact on school dropout indicates that, on average, the dropout rate under SDPP is “X” percentage points lower than it would have been under business-as-usual operations. We also present percentage increases or decreases in the primary outcomes across the treatment and control groups. These “percentage changes” should not be interpreted as the percentage “change” that might be calculated in a pre- post measure or baseline/endline change, but rather as the increase or decrease in the treatment group’s outcome measure in relation to the control group at endline.

The impact analysis includes all students in the targeted grades in SDPP and control schools, regardless of whether the students in the SDPP schools actually participated in SDPP activities. Therefore, these estimates represent the average impact of SDPP on all students in the enrolled schools. These are called intent-to-treat (ITT) estimates—they reflect the fact that not every school or student intended to be treated (via program services) actually participated in the program. The ITT estimates therefore answer the policy-relevant question—do programs make a difference for schools that choose to enroll? These types of estimates are widely used in large-scale evaluations and preserve the integrity of the random assignment design.

SDPP worked with 5th graders from SY 2012–2013, SY 2013–2014 and SY 2014–2015. Therefore the analyses look at outcomes for all three of these cohorts of students. SDPP schools and control school outcomes are compared, and differences are estimated while controlling for school level and individual level information regarding the school year during which they received SDPP.20

3. Impact analysis for at-risk students

The SDPP program was intended to affect outcomes more strongly for students at risk of dropout than for students not at risk, since the goal of the EWS was to train teachers to identify and work with students that are at risk of dropping out of school. Therefore, SDPP analyzed the impacts on students at risk of dropout in two ways. First, students in SDPP Program and control schools were identified as at risk based on student characteristics before they entered the 5th grade, mimicking the EWS identification process as closely as possible with available data. SDPP conducted a subgroup analysis of the students who were identified as at-risk. This subgroup analysis is part of SDPP’s primary assessment of whether the program effectively accomplished its goals (or was effective).

19 See Appendix B for more details on SDPP’s impact estimation methods. 20 Because the last cohort from SY 2014–2015 did not receive a full year of treatment prior to final data collection, and endline data collection occurred prior to the start of the following school year, between-grade dropout is not measured for this cohort.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 18

Importantly, the students identified as at-risk for the subgroup analysis in SDPP schools based on baseline characteristics are not necessarily the same as those identified by the EWS in SDPP schools. However, there is substantial overlap in these groups. SDPP focused the primary analysis on the subgroup of students identified as at risk of dropout based on their baseline characteristics because it would allow us to identify at-risk students in the control group using an analogous process. More critically, preserving the integrity of the random assignment design requires that subgroup analysis be based on baseline characteristics observed before receiving program services, and students in both SDPP and control schools are thus identified as being at-risk in the same manner.

4. Additional subgroup analyses

In addition to looking at impacts separately for at-risk students, SDPP also conducted several additional subgroup analyses. The SDPP program was not designed to have different impacts for different subgroups of students (other than students at risk of dropout); however, the literature suggests that outcomes and impacts might vary for different types of students. There may be differential impacts for students with certain characteristics (such as differences by gender or for those over-age for their grade), for different types of schools (such as the percentage of at-risk students in a school and the school’s distance to the district capital), or for different types of teachers (such as differences by gender or by full-time teaching status) (Table III.C.1). These subgroup analyses provide interesting context for interpreting the main findings but are not part of the evaluation’s primary assessment of program effectiveness. Because these subgroup analyses were exploratory, SDPP did not adjust statistical significance thresholds for multiple comparisons when there were multiple subgroup comparisons being made. These additional subgroup analyses provide interesting context for our interpretation of the main findings and insight into the groups for which the SDPP program may be more or less successful.21

21 Because these subgroup analyses were exploratory, SDPP did not adjust statistical significance thresholds for multiple comparisons when there were multiple subgroup comparisons being made (Schochet, Peter Z. 2009).

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 19

Table III.C.1. Primary and additional subgroup analyses

Subgroup Analysis Subgroup Definition Type Type At-risk Students Students were identified as being most at risk of dropping out Primary students/Not- based on information on their characteristics available in school at-risk students records at baseline before entering the target grade. Female/male Students; Student sex was determined from school records. Additional Teachers Full-time Teachers Teachers were determined to be full-time or not full-time (part- Additional teaching status time, contract, volunteer, or other types of teachers that are not full time employees) Over-age/not Students A student is considered to be over-age if he or she is two years Additional over-age older than the appropriate age for his or her grade, compared to those who were within two years of the appropriate age. In India, 5th graders would typically be 12 years old, so SDPP looked at students that were 14 or older but still in 5th grade. Low caste/not Students Students were assigned to a caste group depending on their Additional low caste reported caste or caste category, in which “low caste” is defined as a minority, scheduled caste, or schedule tribe, and “not low caste” is defined as general or other backward caste. High % at- Schools Schools were divided into a group that was at the 70th percentile Additional risk/low % at- or higher in percentage of at-risk students at baseline among risk control group schools (the high percentage group) and a group that was below the 70th percentile (the low percentage group). Distance to Schools SDPP defined schools that were below the control school Additional school median for distance to the district capital as schools “close” to the district capital (the “low” group) and schools that were at or above the median as schools “far” from the district capital (the “high” group).

5. Primary and additional measures of SDPP’s effectiveness

To select outcomes for the impact study, SDPP identified the key domains that were expected to be affected by the SDPP program as indicated by the program theory of change: (1) teacher behavior and attitudinal outcomes; (2) student attitudes; (3) student engagement in school; and (4) school dropout. Within each of these domains SDPP identified key outcomes that the program was intended to affect; these primary outcomes can be used to assess whether the program achieved its goals. In addition to the primary outcomes, the evaluation presents findings for additional outcomes to provide context to the primary analysis and more detail on how students and teachers may have been affected by SDPP.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 20

Table III.C.2. Primary and additional measures of the India SDPP Program’s effectiveness at endline (SY 2012–2013, SY 2013–2014, and SY 2014–2015)

Primary measures of program Secondary measures* Domain effectiveness Teacher Teacher dropout prevention practice scale Teacher’s sense of responsibility for at-risk outcomes (range: 1 to 8), SY 2012–2013, SY 2013– students scale (range: 1 to 4), SY 2012– 2014, and SY 2014–2015, grade 5 teachers 2013, SY 2013–2014, and SY 2014–2015, grade 5 teachers Teacher’s sense of self efficacy scale (range: 1 to 5), SY 2012–2013, SY 2013–2014, and SY 2014–2015, grade 5 teachers Administrator dropout prevention practice scale (range: 1 to 8), SY 2012–2013, SY 2013–2014, and SY 2014–2015, school administrators Administrator’s sense of responsibility for at- risk students scale (range: 1 to 4), SY 2012–2013, SY 2013–2014, and SY 2014– 2015, school administrator Administrator’s sense of self efficacy scale (range: 1 to 5), SY 2012–2013, SY 2013– 2014, and SY 2014–2015, school administrators At-risk student Emotional attitudes toward school, SY Student perception of teachers, SY 2012– attitudes 2012–2013, SY 2013–2014, and SY 2014– 2013, SY 2013–2014, and SY 2014–2015, 2015, grade 5 grade 5 Cognitive attitudes toward school, SY 2012– Student perception of parental engagement, 2013, SY 2013–2014, and SY 2014–2015, SY 2012–2013, SY 2013–2014, and SY grade 5 2014–2015, grade 5 Behavioral attitudes toward school, SY 2012–2013, SY 2013–2014, and SY 2014– 2015, grade 5 Engagement in Attendance, April 2012-March 2013, SY Performance in Hindi, SY 2012–2013 and school 2012–2013; April 2013-March 2014, SY SY 2013–2014, grade 5 2013–2014; April 2014-October 2014; SY Performance in Math, SY 2012–2013 and 2014–2015, grade 5 SY 2013–2014, grade 5 Dropout Between-grade dropout based on enrollment Within-grade dropout rate based on the following school year, SY 2012–2013 attendance during last three months of and SY 2013–2014, grade 5 school year, SY 2012–2013, SY 2013– 2014, and SY 2014–2015, grade 5 Progression from grade 5 to grade 6 or higher, SY 2012–2013 and SY 2013– 2014, grade 5 * Exploring the impact of SDPP on these additional outcomes of interest is meant to be descriptive in nature, to provide context and better understand the impacts on the primary measures by which SDPP’s effectiveness is determined. In these exploratory analyses on additional outcomes, SDPP did not adjust statistical significance thresholds for multiple comparisons when presenting impacts of the program. SY = school year.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 21

Within the teacher outcomes domain, the primary outcome measure is a scale representing teacher dropout prevention practices (Table III.C.2). SDPP worked directly with teachers to improve their knowledge and practices related to preventing dropout, and they received training on how better to interact with students through the enrichment program. Since teachers inspire and shape student attitudes and behaviors, changes in teacher practices may represent catalysts for student change. Changes in student attitudes should then result from these changes in teacher attitudes and practices. Therefore, SDPP looks at outcomes within the student attitudes domain. The three primary outcomes in this domain are: student emotional, cognitive, and behavioral attitudes toward school.22 Changes in student attitudes should then result in changes in student engagement in school, including in their attendance and performance. Low attendance or performance may indicate that students are less active participants in their own education, which could be a sign that students are on their way to dropping out of school. Finally, SDPP analyzes dropout after 5th grade, or between-grade dropout, for 5th grade students.

In addition to the primary outcomes, the evaluation analyzes additional information to provide context to the primary analysis and increase understanding of the ways in which the program was and was not effective in influencing a particular domain. Table III.C.2 lists these additional outcomes, which include teacher attitudes and practices, student perceptions of teachers and parents, academic performance, behavior, and grade progression.

IV. Sampling and Data collection

This report draws on data from three sources: (1) student records collected from schools; (2) surveys conducted with at-risk students; and (3) surveys conducted with school administrators and teachers teaching 5th-grade homeroom, math, and Hindi language courses. SDPP collected these data at four points in time, collecting data for four school years, from the school year before the rollout of the program (SY 2011–2012) to the school year after completion of the program (SY 2014–2015) (Figure II.C.1). Across these four rounds of data collection, SDPP followed the three cohorts of 5th graders over time, gathering different information about them and their schools and teachers at each point. Because the final round of data collection began before the end of the SY 2014-2015, between-grade dropout cannot be estimated for this cohort. Therefore, impacts of the SDPP program on dropout are estimated for only two cohorts.

Across all data collection rounds SDPP gathered data from 220 schools--113 SDPP and 107 control – that are used in the impact analyses (Table IV.1). SDPP gathered information used in the student analyses from school records for 19,721 students from the SDPP program group and 18,669 students from the control group. Similarly, SDPP analyzed information collected from interviews with 3,695 students from the SDPP group and 3,526 students from the control group, and 420 5th-grade teachers and administrators in the SDPP program group and 393 5th-grade teachers and administrators in the control group.

22 SDPP used a multiple-comparison adjustment procedure (the Benjamini-Hochberg method) for student attitudinal outcomes to account for the fact that there were three primary outcomes in this domain (Benjamini, Yoav and Yosef Hochberg 1995).

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 22

Table IV.1. Study sample sizes SDPP group Control group Total Schools 113 107 220 Student records 19,721 18,669 38,390 Student survey 3,695 3,526 7,221 Teacher survey 420 393 813 Administrator survey 208 198 406 Sources: SDPP baseline and follow-up student records, school questionnaire, student questionnaire, and teacher self-administered questionnaire; July 2012, May 2013, February 2014, and November 2014. Note: Sample sizes refer to the number of unique data points from each data collection source that are used in the impact analyses. Data on different cohorts come from different rounds of data collection. Overall sample sizes for each data collection effort were larger, since some students were included in the school records that are not eligible to be included in the analysis.

SDPP administered school questionnaires to the headmaster or deputy headmaster to gather information about school characteristics, enrollment and teacher characteristics during each data collection round. SDPP administered teacher questionnaires to headmasters; deputy headmasters; and 5th-grade math, Hindi, and homeroom teachers at the beginning of SY 2012–2013, at the end of SY 2012–2013 and SY 2013–2014, and during SY 2014–2015. The teacher questionnaire included questions related to respondents’ experience and training, awareness of risk factors related to dropout, and attitudes toward and practices used with at-risk students. Eligible teachers responded at a rate of 99 percent, 98 percent, 96 percent, and 95 percent at each data collection point (SY 2012–2013 beginning and end, SY 2013–2014, and SY 2014–2015 respectively).

During each round of data collection, SDPP obtained official student records to glean information on the attendance, school performance, caste and demographics, and enrollment of the 2012–2013 and 2013–2014 5th grade cohorts during their 4th, 5th and 6th grades, and the 2014–2015 5th grade cohort during their 4th and 5th grade years.23 During each round of data collection SDPP also interviewed students who were at risk of dropping out of school (based on their baseline characteristics) to assess their attitudes about school. A subset of at-risk students from each cohort was sampled for interviews and included in the analyses. The actual sampling process in the field varied by cohort because of the data available to identify student risk status at the time of sampling. Appendix A describes this process in detail.

As mentioned earlier, to identify at-risk students for this evaluation, SDPP only used data that were available in school records for all SDPP program and control group schools. This is a different method than the one used to identify at-risk students via the EWS in SDPP schools. To identify students based on school records, the evaluation used two analogs of the six at-risk components used in India’s EWS at-risk identification process: (1) most recent attendance rates; and (2) the average of annual math and Hindi exam scores (course performance).24

23 Since the last round of data collection occurred during the last year of the SDPP program, SDPP does not have 6th grade information for the 2014-2015 5th grade cohort. 24 For the SY 2012–2013 cohort, SDPP used the average of February and May 2012 attendance rates and the average of SY 2011–2012 exam scores in math and Hindi. For the SY 2013–2014 cohort, SDPP used the average of February

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 23

At-risk students who were selected for the interview responded to questions about demographics, cognitive, behavioral and emotional attitudes toward school, and perceptions of teachers and parents. Sampled students responded at a rate of 81 percent, 73 percent, and 74 percent at each data collection point (at the end of SY 2012–2013 and SY 2013–2014 and during SY 2014–2015) respectively.

V. Characteristics of the Sample Prior to Implementation Having data on the sample members before they are exposed to the intervention is a crucial element of a rigorous impact evaluation; it provides information on the sample’s baseline characteristics and allows us to check for equivalence between the treatment and control groups. In the case of the SDPP Program in India, the characteristics of schools, teachers, and students gathered before the intervention started in July 2012 serve as the reference point for all subsequent measurement and analysis.

SDPP and control group schools had comparable characteristics at baseline (SY 2012–2013), with no statistically significant differences (Table V.1, top panel). Most study schools had no active external school programs. (This means that SDPP Program activities were being implemented in schools without other active programs targeting school dropout, which strengthens any claims of SDPP Program effectiveness.) Schools in both SDPP and control groups were about 23 kilometers from the district headquarters. The typical SDPP group school enrolled about 374 students, compared with about 372 students in control group schools. The average SDPP group school had 58 5th-grade students, compared with 57 in the control group. Both groups had about seven 5th- grade teachers per school.

SDPP also compared the characteristics of 5th-grade teachers in the SDPP group versus the control group (Table V.1, bottom panel), and found three marginally significant differences out of 13 comparisons made.25 Forty-one percent of SDPP group teachers were female, compared with nearly 50 percent of control group teachers. Teachers in the control group reported adopting 7.2 of 8 dropout prevention practices while teachers in the treatment group reported adopting 7.0 of 8. In addition, the distribution of teachers’ teaching experience overall differed between the treatment and control groups. Each of these differences was marginally significant at the 10 percent level. Although the number of differences was more than might be expected based on chance, none of the differences are sufficiently large to survive corrections for multiple comparisons.

and May 2013 attendance rates. For the SY 2014–2015 cohort, SDPP used the average of December 2013 and January 2014 attendance rates. SDPP used only attendance to identify at risk students for the SY 2013-2014 and SY 2014- 2015 cohorts because there were too many missing values in the annual test score data to reliably use it in the at-risk identification process. 25 SDPP compared the SDPP program and control school teachers on a larger set of characteristics in Appendix Table H.3. Out of 21 comparisons, 2 were statistically significant at the 5 percent level. In particular, teachers’ ability to identify 6 or more of 8 risk factors for school dropout, their sense of self-efficacy and their score on the teacher dropout prevention practice scale differed between SDPP and control group schools. These differences may imply real baseline differences between the groups, but they may also exist because the baseline teacher interviews were conducted after random assignment. Because SDPP teachers were aware that they were participating in a dropout prevention program, they may have respondent differently to dropout related questions. In either case, SDPP controls for these differences in the impact estimates.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 24

The average teacher in both groups was about 39 years old. Higher secondary or graduate-level schooling was the highest level of education for the vast majority of teachers. Seventy-five percent of SDPP group teachers had less than 10 years teaching experience, compared to 82 percent of teachers in the control group schools. In both groups, fewer than 5 percent of teachers had 30 years or more of teaching experience. Very few (3.5 percent) teachers had ever received training related to at-risk students, and even fewer (0.9 percent) had received this training within the past year.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 25

Table V.1. Average school and target grade teacher characteristics before intervention (5th grade, SY 2012–2013) (percentage unless indicated otherwise)

Outcome SDPP group Control group School characteristics prior to intervention Offer grades 1 through 6 100.0 100.0 Enrollment (mean number of students) Grade 5 (target grade) 57.8 56.7 Overall (grades 1-5) 373.5 371.5 Grade 6 93.4 93.0 Teacher Count Number of teachers 7.4 7.1 Number of grade 5 homeroom teachers 1.0 1.0 Attendance rate at time of headcount (% of grade 5 students) 51.2 52.6 Active external school programsa No active external school programs 97.3 96.3 Infrastructure 0.9 2.8 Other programsa 1.8 0.9 Distance to district headquarter (kilometers) 23.0 22.5 Grade 5 math, Hindi, and homeroom teacher characteristics prior to intervention Age (years) 39.2 38.6 Female 40.8* 49.5 Highest level of educationb Post-graduate or higher 13.3 16.5 Graduate 40.3 36.5 Higher secondary 39.9 41.0 Secondary school or less 6.4 6.0 Teaching experience overallb † Less than 10 years 74.6 82.0 10 years to less than 20 years 13.4 9.0 20 years to less than 30 years 9.5 4.5 30 years or more 2.6 4.5 Received training related to at-risk students Ever 3.5 4.0 In the past year 0.9 1.0 Teacher dropout prevention practice scale (range: 1 to 8)c 7.0* 7.2 Sample size Schools 113 107 Teachers 233 200 Sources: SDPP baseline teacher self-administered questionnaire and school questionnaire, July 2012. Note: Differences between SDPP and control group means were tested using two-tailed t-tests unless otherwise indicated. The teacher analysis accounts for clustering of teachers within schools. Sample sizes for some characteristics may be smaller due to missing responses. a External programs are those funded and implemented by organizations other than the school system. More than one external program can operate in a school. Examples of these other programs include disaster reduction awareness programs, teacher training programs, and programs providing scholarships. b Differences between SDPP and control group distributions were tested using a chi-squared test. c This scale represents the sum of teacher responses to eight items that indicate whether the teacher reports recording daily attendance, taking action if the student is absent three days in a month, giving weak students feedback, discussing support for weak students with other teachers, developing plans to support weak students, communicating with parents of weak students, meeting with weak students, and being willing to come early or stay late to help weak students. ***/**/* Difference between SDPP and control group means is statistically significant at the .01/.05/.10 level. †††/††/† Difference between SDPP and control group distributions is statistically significant at the .01/.05/.10 level.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 26

Students who were in 5th grade in SY 2012–2013 (at the beginning of the school year, before the intervention) were also similar across the SDPP program and control groups (Table V.2). About 50 to 52 percent of both groups were female and about 12 to 13 percent were over-age for their grade. In both groups, about 13 percent were in the general caste grouping, 30 to 33 percent were in the scheduled caste/scheduled tribe grouping, 46 to 49 percent were in the OBC/BC caste grouping, and about 8.5 percent were classified as a minority. About 83 percent of the treatment group students were characterized as at risk of school dropout at baseline, compared to 82 percent of control group students. At-risk students had lower attendance and academic performance grades than the full sample. The daily attendance rate for at-risk students was about 60 percent, compared to 62-64 percent for the full sample. There were no statistically significant differences in student characteristics between research groups.

Table V.2. Average student characteristics before intervention (5th grade, SY 2012–2013) (percentage of students unless otherwise indicated)

Full sample At-Risk sample

SDPP Control SDPP Control group group group group Demographic characteristics Female 50.6 52.1 50.4 51.6 Overage for gradea 12.8 11.7 13.0 12.4 Caste Categoryb General 12.5 13.0 11.9 11.9 SC/ST 30.1 32.6 30.7 33.2 OBC/BC 48.9 45.8 49.1 46.1 Minority 8.5 8.6 8.3 8.8 Factors related to risk of dropout Categorized as At-Risk based on Baseline Informationc 83.4 82.1 100.0 100.0 Daily Attendance during Prior School Year 62.5 63.8 59.8 60.7 Academic Performance during Prior School Year (Range: 1-100) Mathematics 62.5 66.4 56.6 56.6 Hindi 61.5 65.0 55.7 55.3 Overall subject scores 54.1 55.3 52.9 53.5 Sample size Schools 113 107 113 107 Students 6,607 6,205 5,511 5,097 Sources: SDPP baseline school records data collection; July 2012 Note: Differences between SDPP and control group means were tested using two-tailed t-tests. The analysis accounts for clustering of students within schools. Sample sizes for some characteristics may be smaller due to missing responses. a A student is considered overage for his or her grade if he or she is two years older than the appropriate age for their grade. b Differences between SDPP and control group distributions were tested using a chi-squared test. c A student was identified as at-risk if (1) the student received a score of 2 on the indicator related to attendance, (2) the sum of the student’s at-risk indicators (at-risk index) was greater than or equal to 3, or (3) the student’s at-risk index was in the upper 67th percentile of the distribution of the at-risk index for the student’s class. ***/**/* Difference between SDPP and control group means is statistically significant at the .01/.05/.10 level. †††/††/† Difference between SDPP and control group distributions is statistically significant at the .01/.05/.10 level.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 27

VI. Impacts of SDPP

The evaluation estimated the SDPP Programs’ impacts on teacher outcomes, student attitudes, student engagement in school, and school dropout. This chapter discusses the extent to which the SDPP program was successful in improving the primary measures of program effectiveness in each of these domains. The assessment of the SDPP Program’s effectiveness focuses on program impacts measured as the difference in average outcomes at final follow-up between students and teachers randomly assigned to the SDPP group and those randomly assigned to the control group, adjusting for baseline characteristics. The impact estimates reported in this study should be interpreted as the difference in outcomes that resulted from exposure to SDPP. For example, an “X” percentage point favorable impact on school dropout indicates that, on average, the dropout rate under SDPP is “X” percentage points lower than it would have been under business-as-usual operations. In some places, to provide additional context we also present percentage increases or decreases in the primary outcomes across the treatment and control groups. These “percentage changes” should not be interpreted as the percentage “change” that might be calculated in a pre- post measure or baseline/endline change, but rather the increase or decrease in the treatment group’s outcome measure in relation to the control group at endline.

The discussion is organized by outcome domain, as follows: (1) teacher behavior and attitudinal outcomes; (2) student attitudes; (3) student engagement in school; and (4) school dropout. Impact findings for primary and additional measures of program effectiveness are presented with bar charts corresponding to the mean outcome level by random assignment group status. Differences that are statistically different than zero are indicated with asterisks.

Statistical significance

Estimates of the impact of the SDPP Program are based on differences in average outcomes for SDPP and control group students and teachers. In interpreting these estimates, it is important to evaluate whether they are sufficiently large that it is unlikely that the difference is due to chance (indicating that the SDPP Program did have an impact). With this in mind, statistical tests were conducted to assess whether each impact is significantly different than zero. Impacts estimates are described as statistically significant if there is less than a 5 percent probability that they are due to chance (and not to the SDPP Program). Impact estimates are described as marginally significant if the probability that they are due to chance (and not to the SDPP Program) is between 5 and 10 percent. In tables and figures, the statistically significant impacts at the 1 percent, 5 percent, and 10 percent levels are denoted with asterisks, as ***, ** or *. The chance (1%, 5% or 10%) that the reported findings are falsely reporting an impact increases as additional tests are conducted. Therefore, caution should be used when interpreting the meaning of our exploratory analyses of additional outcomes and subgroups because we do not correct for the total number of comparisons being made. Individual tests of these additional contrasts of program effects for other subgroups are provided as additional context for the main findings.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 28

A. Impacts on teacher outcomes

Teachers can shape the attitudes and actions of students through their own actions. As shown Teacher Outcomes in the theory of change conceptual model Primary measure of program effectiveness (Figure III.A.1), a teacher’s actions might  Teacher take-up of dropout prevention represent the first level of change in reducing practices school dropout. SDPP worked directly with teachers to improve their knowledge and Additional measures awareness of dropout, training them to identify  Teachers sense of self-efficacy in dealing at-risk students and to work with those students with dropout and their families to strengthen their  Teachers’ sense of responsibility for attachment to school. Teachers were taught to addressing dropout use the EWS to improve their dropout prevention practices, and received training on  Administrators’ dropout prevention how to better deal with students through the practices, self-efficacy and sense of complementary enrichment program, which, in responsibility turn, should improve student attitudes and behaviors. Thus, changes in teacher actions can be early indicators of changes in student behaviors.

This section discusses the impacts of SDPP on teachers’ knowledge, attitudes, and practices related to dropout.

1. Impacts on teacher take-up of dropout prevent practices

The primary measure of SDPP’s influence on teacher outcomes is the teacher dropout prevention scale. This scale combines responses to eight questions posed to homeroom teachers, who were the focus of the EWS intervention, and to teachers of math and language. The questions focus on teacher behavior that might help at-risk students succeed in school, such as whether the teacher records daily attendance and whether he or she takes action if the student is absent three days in a month. (See Appendix C for details on the creation of the scale.)

Teacher dropout prevention practices scale (primary measure)

Teachers responded to a questionnaire. Scoring is based on an 8-point scale corresponding to eight survey items that indicate whether the teacher has adopted dropout prevention practices. The items include:

 Recording daily attendance  Taking action if the student is absent three days in a month  Giving weak students feedback  Discussing support for weak students with other teachers  Developing plans to support weak students  Communicating with parents of weak students  Meeting with weak students  Willingness to arrive early or stay late to help weak students

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 29

SDPP did not have a statistically significant impact on teachers’ practices related to dropout prevention, (Figure VI.A.1). SDPP and control group teachers both scored 7.40 and 7.45 on the 8-point dropout prevention scale, respectively. The difference between the two was not statistically significant. Given the high level of this measure in both the control group and at baseline, there may not be much room for improvement, so it is perhaps not surprising that there is no impact.

This finding might suggest that even without the SDPP Program, teachers may be employing most of the dropout prevention practices measured by this scale. Alternatively, the scale scores could be artificially high because they are based on self-reports rather than on direct observation. Control group teachers might be more likely to over-report use of dropout prevention practices, perhaps because, unlike SDPP group teachers, they have not received training on what these practices entail. SDPP group teachers might be more likely to reduce the amount they report, perhaps because they and others in the school are more in tune with what these actions entail and are more truthful after having received the training. In either case, impacts of the SDPP Program on teacher dropout prevention practices are less likely to be registered with the self-reported dropout prevention practice scale.

Other reasons for the lack of teacher take-up could include the limited exposure to training, high teacher turnover, and insufficient time to master dropout prevention practices. Although SDPP conducted teacher training over three school years, only 43 percent of 5th-grade homeroom, math, and language teachers in SDPP schools reported having received training related to at-risk students. Of those teachers that received training, 78.1 percent of the teachers received two years of SDPP training and 21.9 percent received only one year, given high levels of teacher turnover.

There was high turnover of homeroom teachers in both SDPP and control schools—only 51 percent of Grade 5 homeroom teachers were still serving as homeroom teachers in the second year of the SDPP intervention, and only 41 percent were still homeroom teachers in the third year of the SDPP intervention. This means that many homeroom teachers implemented SDPP for only one year or two years, which may not have been enough time for them to become comfortable with and implement specified dropout prevention practices. Nevertheless, teacher knowledge of dropout prevention significantly increased in SDPP schools. Knowledge was assessed during teacher interviews. SDPP teachers identified about 50 percent of the risk factors identified for school dropout, compared to control school teachers who identified only 37 percent of risk factors. This represents a statistically significant difference, indicating that the training SDPP teachers received resulted in increased knowledge about school dropout.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 30

Figure VI.A.1. SDPP Program impacts on teacher dropout prevention practices at endline (SY 2012– 2013, SY 2013–2014, and SY 2014–2015)

8 7.45 7.40

point scale -

Score eightonan 2

SDPP group Control group

SDPP also explored the effects of the SDPP Program on additional subgroups that were not directly targeted by SDPP but are still of interest.26 These comparisons provide useful context and can suggest pathways through which SDPP might be working.

SDPP examined impacts on the teacher dropout prevention practices scale for four pairs of subgroups (Figure VI.A.2). In particular, SDPP estimated impacts separately by the teacher’s gender, the percentage of at-risk students in a school, the school’s distance to the district capital, and the teacher’s full-time teaching status. Impacts on the teacher dropout prevention practices scale differed only by teaching status. The SDPP Program had a marginally significant, positive impact on teacher dropout prevention practices for full-time teachers, and the difference from the impact for teachers who were not full-time was statistically significant.27

26 As described in Section III, because these analyses are exploratory, the results are not adjusted for multiple comparisons, despite the large number of comparisons. 27 Differences between impacts for each subgroup were tested. There was a statistically significant difference between the impact estimates for full-time and not full-time teachers.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 31

Figure VI.A.2. SDPP Program impacts on teachers’ dropout prevention practices at endline, by subgroup (SY 2012–2013, SY 2013–2014, and SY 2014–2015)

0.5 0.39*

0.15 0.16 0.08

0.03 point scale point - 0

-0.05

-0.08 -0.06 scores on an eightan on scores

-0.5

Female Male Full-time Not full-time High % At- Low % At-risk Remote Not remote Difference in SDPP and control group means in means group control and SDPP in Difference †† risk schools schools

SDPP group

Sources: SDPP baseline and follow-up teacher self-administered questionnaires and school records data collection; July 2012, May 2013, February 2014, and November 2014. Note: The analysis is based on 5th-grade homeroom, math, and language teachers during SY 2012–2013, SY 2013–2014, and SY 2014–2015. Differences between SDPP and control group means were tested using two-tailed t-tests. Mean values are adjusted for baseline characteristics. The analysis accounts for clustering of teachers within schools and school-year fixed effects. Because these are additional outcomes and the analyses are exploratory, statistical significance thresholds were not adjusted for multiple comparisons. ***/**/* Impact estimate is statistically significant at the .01/.05/.10 level. †††/††/† Statistically significant difference between the subgroup impact estimates at the .01/.05/.10 level.

2. Impact on additional teacher outcomes

Besides the primary measure of SDPP’s effect on teachers’ dropout-related practices, SDPP used two other scales to examine teachers’ sense of self-efficacy and sense of responsibility for at-risk students. SDPP provides these findings here because they may be of interest and could help paint a more complete picture of teachers’ experiences.28

a. Teacher sense of self-efficacy

An additional indicator of the SDPP Program’s effectiveness in addressing dropout is influencing teachers’ belief that they are capable of affecting factors associated with student dropout, such as poor behavior, disinterest in class, and absenteeism. This outcome is measured according to the teacher sense of self-efficacy scale, which SDPP adapted from Tschannen-Moran and Hoy (2001). The questions focus on teacher beliefs that they can help at-risk students succeed in school, such

28 As described in Section III, because these analyses are exploratory, we do not adjust the results for multiple comparisons, despite the large number of comparisons being made.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 32

as whether the teacher thinks he or she can encourage students to value learning and provide assistance to families in helping their child succeed in school.

Teacher sense of self-efficacy scale Teachers were asked how much they can do to prevent or address the following 12 factors associated with dropout:  Disruptive behavior in the classroom  Motivation of students with low interest in school  Encouragement of students to believe they are capable of succeeding in school  Help students value learning  Make lessons interesting for students  Enforcing classroom rules  Encouragement of active participation among students not engaged  Identification of students needing extra support  Student attendance  Modification of teaching and learning activities to help weak or poorly performing students  Assistance to families in helping their children do well in school  Help for poor-performing students to do better in school

Responses to the questions about teachers’ self-efficacy ranged from 1 (nothing) to 5 (a great deal) and were compiled into a scale in which higher values corresponded to a higher sense of self- efficacy.

Teachers in both research groups tended to have a high sense of self-efficacy (Figure VI.A.3). Teachers in both the SDPP and control group schools had an average score of 4.76 on the teacher’s sense of self-efficacy scale. The mean value indicates that teachers typically felt that they could do “a great deal” to respond to factors associated with risk of dropout.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 33

Figure VI.A.3. SDPP Program impacts on teacher sense of self-efficacy at endline (SY 2012–2013, SY 2013–2014 and SY 2014–2015)

5 4.76 4.76

point scale - 3

Score onfivea 2

SDPP group Control group

Source: SDPP baseline and follow-up teacher self-administered questionnaires and school records data collection; July 2012, May 2013, February 2014, and November 2014. Note: The analysis is based on 5th-grade homeroom, math, and language teachers during SY 2012–2013, SY 2013–2014, and SY 2014–2015. Differences between SDPP and control group means were tested using two-tailed t-tests. Mean values are adjusted for baseline characteristics. The analysis accounts for clustering of teachers within schools and school-year fixed effects. Because these are additional outcomes and the analyses are exploratory, statistical significance thresholds were not adjusted for multiple comparisons. ***/**/* Impact estimate is statistically significant at the .01/.05/.10 level.

b. Teacher sense of responsibility

In addition to the teacher sense of self-efficacy scale, SDPP developed a scale to measure teachers’ sense of responsibility for students at risk of dropping out. The questions in this scale focus on teachers’ opinions about ways to prevent students from dropping out of school.

Responses for this scale ranged from 1 (strongly agree) to 4 (strongly disagree). The scale score is the mean of the five items. Higher values correspond to a higher sense of teacher responsibility for at-risk students.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 34

Teacher sense of responsibility for at-risk students scale

This scale is based on teacher agreement with the following five statements about at-risk students.:

 Students at risk of dropping out of school should work harder  Little can be done by the teacher or school to help students at risk of dropping out  If a student is at risk of dropping out, it is mainly the fault of the parent/guardian or family  At-risk students face too many challenges to succeed in school  At-risk students need more help than teachers have time or resources to provide

Teachers in both research groups tended to have a low sense of responsibility for at-risk students (Figure VI.A.4). The average score for teachers in the SDPP schools was 2.10 (out of 4), indicating that teachers tended to agree that they bore little responsibility for at-risk students. The average score for teachers in the control schools was 2.08, indicating that control school teachers have about the same sense of responsibility for at-risk students. It is interesting to note that teachers feel they are able to greatly influence many of the factors associated with dropout (self-efficacy), but they do not feel particularly responsible for doing so as shown by the teacher’s sense of responsibility scale. Appendix C presents further information about the findings for additional teacher outcomes as well as for administrators.

Figure VI.A.4. SDPP Program impacts on teacher sense of responsibility at endline (SY 2012–2013, SY 2013–2014 and SY 2014–2015) 4

point scale -

2.10 2.08

2 Score onfoura

SDPP group Control group

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 35

3. Impacts on additional school administrator outcomes

Impacts were also estimated for school administrators’ dropout-prevention practices, sense of self- efficacy, and sense of responsibility for at-risk students. Each of these outcomes was measured in the same way as the teacher outcomes.

a. Administrator dropout prevention practices

As with teachers, the SDPP Program did not have a statistically significant impact on administrators’ dropout prevention practices in India (7.37 versus 7.25) (Figure VI.A.5).

Figure VI.A.5. SDPP Program impacts on administrators’ dropout prevention practices scale

8 7.37 7.25 7

point scale - 5

3 Score eightonan 2

SDPP group Control group

Sources: SDPP baseline and follow-up teacher self-administered questionnaires. Note: Analysis accounts for clustering of administrators within schools. Differences between SDPP and control group means were tested using two-tailed t-tests. Mean values are regression adjusted. Because these additional analyses are exploratory, statistical significance thresholds were not adjusted for multiple comparisons. ***/**/* Impact estimate is statistically significant at the .01/.05/.10 level.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 36

b. Administrator sense of self-efficacy

Consistent with the findings for teachers’ sense of self-efficacy, there were no statistically significant impacts on administrators’ sense of self-efficacy (Figure VI.A.6).

Figure VI.A.6. SDPP Program impacts on administrator sense of self-efficacy

5 4.74 4.73

point scale -

3 Score onfivea 2

SDPP group Control group

Sources: SDPP baseline and follow-up teacher self-administered questionnaires. Note: Analysis accounts for clustering of teachers within schools. Differences between SDPP and control group means were tested using two-tailed t-tests. Mean values are regression adjusted. Because these additional analyses are exploratory, statistical significance thresholds were not adjusted for multiple comparisons. ***/**/* Impact estimate is statistically significant at the .01/.05/.10 level.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 37

c. Administrator sense of responsibility

Similarly to the results for teachers, SDPP had no statistically significant impacts on administrator sense of responsibility (Figure VI.A.7).

Figure VI.A.7. SDPP Program impacts on administrator sense of responsibility

point scale -

2.11 2.06

2 Score onfoura

SDPP group Control group

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 38

B. Impacts on at-risk students’ attitudes toward school

The SDPP theory of change conceptual model At-risk student attitudes toward school (Figure III.A.1) suggests that a change in teacher outcomes knowledge and practices could lead to a change in student attitudes toward school. While SDPP did Primary measure of program effectiveness not find a statistically significant impact on the teacher knowledge and practices measured, the  Emotional, cognitive, and behavioral EWS and enrichment activities could directly attitudes toward school improve student attitudes through other pathways. Additional measures: SDPP captured these attitudes by surveying students whom were identified—via the available  Students’ perceptions of teacher support baseline data as described in Section IV—as being  Students’ perceptions of parental support at risk of dropout.

This section presents findings on the impact of the program on attitudes of at-risk students.

1. Impact on primary measures of student attitudes toward school

As described earlier, SDPP used three primary measures to determine the effectiveness of the program on student attitudes: emotional, cognitive, and behavioral attitudes toward school.29

The emotional attitudes toward school scale measures how a student feels towards school. Each question required a yes or no response, and the scales reflect the percent of items with which the student agreed. A higher score represents more positive attitudes towards school. (See Appendix D for details on the construction of the scales.)

Emotional attitudes toward school Students identified as being at risk of dropping out of school responded to a questionnaire. This scale is based on student responses to the following six survey items related to how students feel about school:  School is a fun place to be.  There are teachers I can talk to.  I participate in school activities after school.  I enjoy participating in class activities.  I look forward to school.  A tutoring program would help me with my studies.

29 SDPP constructed the three measures of student attitudes from responses to a survey administered to three cohorts of 5th-grade students identified as at risk based on their 4th-grade characteristics. The survey was developed through developing and testing new questions and adapting existing student engagement, cognitive and behavioral attitudes surveys (see Fredericks, Jennifer A., Phyllis Blumenfeld, Jeanne Friedel, and Alison Paris, 2005; and Finlay, Krystina A. 2006.) and was administered at the end of the 2012–2013 and 2013–2014 school years and during the 2014-2015 school year. For more details on the survey, see Appendix A.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 39

For emotional attitudes toward school, at-risk students in SDPP schools agreed with 93.0 percent compared to 91.3 percent for students in control schools, a statistically significant difference (Figure VI.B.1). This represents a 1.9 percent improvement for the SDPP group relative to the control group. It is interesting that while the control group and baseline levels of this measure were quite high and there was not much room for improvement, there was an impact on emotional attitudes toward school. Both the EWS and the enrichment activities could have contributed to this change. Appendix D presents findings using alternative constructions of these primary measures.

Figure VI.B.1. SDPP Program impacts on at-risk students’ emotional attitudes toward school at endline (SY 2012–2013, SY 2013–2014, and SY 2014–2015)

100 93.0*** 91.3 90

10 Percentage of scale items with which student agreed withstudent whichscale items Percentage of

SDPP group Control group

Sources: SDPP baseline and follow-up school records data collection: July 2012, May 2013, February 2014, and November 2014; baseline and follow-up student surveys, May 2013, February 2014, and November 2014. Note: The analysis is based on SY 2012–2013, SY 2013–2014, and SY 2014-2105 5th-grade at-risk students. The sample includes 3,695 students for the SDPP group and 3,526 students for the control group. Differences between SDPP and control group means were tested using two-tailed t-tests. Mean values are adjusted for baseline characteristics. The analysis accounts for clustering of students within schools, and cohort fixed effects. Impacts were adjusted for multiple comparisons using the Benjamini-Hochberg method. For a tabular presentation of these findings, see Appendix Table G.5. ***/**/* Impact estimate is statistically significant at the .01/.05/.10 level.

SDPP targeted students who were at risk of school dropout. However there are additional subgroups of students for whom understanding the effects of SDPP on student attitudes would provide useful further context in understanding the intervention and the potential pathways through

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 40

which it has effects. Here, SDPP explores the impacts of the program on the emotional attitudes of at-risk students toward school using additional subgroups of interest (Figure VI.B.2).30

Consistent with our main findings, SDPP Program impacts on emotional attitudes toward school were statistically significant for at-risk students in several subgroups. These include: females, students at the appropriate age for their grade, students from the low caste grouping, students from schools with a low percentage of at-risk students, and students from schools both close to and far from the district capital.

Figure VI.B.2. SDPP Program impacts on students’ emotional attitudes toward school at endline, by subgroup (SY 2012–2013, SY 2013–2014, and SY 2014–2015)

3 2.8*** 2.6*** 2.3***

2 1.9*** 1.8** 1.8**

1 0.8 0.8 0.6

0.0

items items withwhich student agreed 0

Improvements Improvements in percentagescaleof Female Male Low Not low Overage Not High % Low % Remote Not ††† caste††† caste overage At-risk At-risk schools remote SDPP group schools

Sources: SDPP baseline and follow-up school records data collection; July 2012, May 2013, February 2014, and November 2014; baseline and follow-up student surveys, May 2013, February 2014, and November 2014. Note: The analysis is based on SY 2012–2013, SY 2013–2014, and SY 2014–2015 5th-grade at-risk students. Sample sizes are 3,792 (female) and 3,429 (male), 724 (overage) and 6,473 (not overage), 1,973 (SC/ST/Minority caste group) and 5,219 (General/OBC caste group), 2,429 (schools with high percentage of at-risk students) and 4,792 (schools with low percentage of at-risk students), and 3,382 (schools below median distance to district capital) and 3,839 (schools at or above median distance to district capital). Differences between SDPP and control group means were tested using two-tailed t-tests. Mean values are adjusted for baseline characteristics. The analysis accounts for clustering of students within schools, and cohort fixed effects. Because these subgroup analyses are exploratory, statistical significance thresholds were not adjusted for multiple comparisons. A student is over-age for his or her grade if he or she is two years older than the appropriate age for the grade. Students were assigned to a caste category depending on their reported caste, and then further grouped into two subgroups, in which “low caste” is defined as a minority, scheduled caste, or schedule tribe, and “not low caste” is defined as general or other backward caste. Schools with a high percentage of at-risk students are those with an at-risk percentage at or higher than the 70th percentile of control group schools at baseline. Schools with a low percentage were below the 70th percentile. Schools labeled as below the median were located closer to the district capital than the median of the control group, while schools at or above the median were located further from the district capital than the median of the control group. ***/**/* Impact estimate is statistically significant at the .01/.05/.10 level.

30 As described in Section III, because these analyses are exploratory, the results are not adjusted for multiple comparisons, despite the large number of comparisons.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 41

Increasing student emotional engagement in India

Being a teenage girl still in primary school is not easy and many over-age girls drop out of school before finishing. Shy, 16- year-old Phoolo had a lot stacked against her: a history of irregular attendance; the burden of caring for her younger siblings; low self-confidence; and the need to work as a part-time maid instead of studying. Much older than her other classmates, Phooloo had no friends at school. She never felt like she fit in. Having to return home to take care of her younger brothers and sisters did not leave much time for friendship or to be involved in the activities Friends enjoy Enrichment Program games at school. Her parents did not see her unhappiness and frequent absences as a problem. Instead of encouraging her to do her school work and take part in school activities, her parents were trying to get her married. But Phooloo’s future took a different turn when her school was targeted to take part in the Student Dropout Prevention Pilot (SDPP) Program. SDPP worked with schools, families and communities to increase awareness about the benefits of schooling, taught teachers how to use an Early Warning System (EWS) to identify and support students at risk of dropping out of school, and how to build partnerships with parents to help their children stay in school. When the teachers in her school used the EWS and identified Phoolo as at high risk of dropping out of school, it triggered a support system into action to keep her in school. The EWS enabled teachers, parents and school administrators to give Phooloo extra attention. As part of the EWS first response strategies, her teacher regularly tracked her attendance, made sure she participated in class and even scheduled regular home visits to meet with her parents. Phooloo participated in SDPP Enrichment Program sessions that encouraged creative expression through arts and crafts, story writing and group games and sports. As she got more involved in the enrichment sessions, Phooloo started building stronger relationships with her classmates and teachers and became more confident in her scholastic and creative abilities. Initially subdued and hesitant, something shifted in Phooloo the day a classmate made a presentation about their friendship. Phooloo realized that despite being older than the other students she did fit in and had a right to be in school. She belonged. She went from missing one in three days of school to a better than 90 percent attendance rate. Her teacher’s ongoing communication with her parents paid off, too. Her mother recounts with tears in her eyes the first time she attended the Anan Utsav (the school Open House) and saw her daughter’s crafts and stories hanging on the wall. “I am so proud of her. Instead of planning her wedding, I see Pholoo needs to be in school.” Today, Pholoo is happy that she is going to school. She hopes she’s paving the way for other girls—especially her younger sister—to go to school.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 42

The cognitive attitudes toward school measures how a student thinks about school and school work. Each question required a yes or no response, and the scales reflect the percent of items with which the student agreed. A higher score represents more positive attitudes towards school. (See Appendix D for details on the construction of the scales.)

Cognitive attitudes toward school

Students identified as being at risk of dropping out of school responded to a questionnaire. This scale is based on student responses to the following nine survey items related to how students think about school:  I will complete grade 9.  Completing grade 9 will be useful to my family and me.  Missing school affects my performance in school.  Doing homework helps me do well in school.  I am interested in the work I get to do in my classes.  I check my school work for mistakes.  I need extra help with my studies or homework.  I have difficulty paying attention in school.  I try to do my best at school, even if it is not perfect.

There were no statistically significant impacts of SDPP on at-risk students’ cognitive attitudes toward school. At-risk students in both the SDPP and control group scored 82.6 percent on the scale (Figure VI.B.3).

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 43

Figure VI.B.3. SDPP Program impacts on at-risk students’ cognitive attitudes toward school at endline (SY 2012–2013, SY 2013–2014, and SY 2014–2015)

100

90 82.6 82.6 80 70 60 50 40

student student agreed 30 20

10 Percentage of scale items Percentagescaleof items withwhich 0 SDPP group Control group

As with emotional attitudes toward school, we explore the impacts of SDPP on the cognitive attitudes of at-risk students using additional subgroups of interest. Consistent with the overall findings, SDPP did not find a statistically significant impact of the program on at-risk students’ cognitive attitudes toward school in any subgroup (Figure VI.B.4).

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 44

Figure VI.B.4. SDPP Program impacts on students’ cognitive attitudes toward school at endline, by subgroup (SY 2012–2013, SY 2013–2014, and SY 2014–2015)

1.5

0.5 0.2 0.2 0.2 0.1 0.0 0.0 0.0 0

-0.2 -0.5 -0.4

-1 with which student whichagreed student with -1.0 -1.5 Improvements scale items of in Improvements percentage Female Male Not low Low caste Overage Not High % Low % Remote Not caste overage At-risk At-risk schools remote schools SDPP group

Sources: SDPP baseline and follow-up school records data collection, July 2012, May 2013, February 2014, and November 2014; baseline and follow-up student surveys, May 2013, February 2014, and November 2014. Note: The analysis is based on SY 2012–2013, SY 2013–2014, and SY 2014–2015 5th-grade at-risk students. Sample sizes are 3,792 (female) and 3,429 (male), 724 (overage) and 6,473 (not overage), 1,973 (SC/ST/Minority caste group) and 5,219 (General/OBC caste group), 2,429 (schools with high percentage of at-risk students) and 4,792 (schools with low percentage of at-risk students), and 3,382 (schools below median distance to district capital) and 3,839 (schools at or above median distance to district capital). Differences between SDPP and control group means were tested using two-tailed t-tests. Mean values are adjusted for baseline characteristics. The analysis accounts for clustering of students within schools, and cohort fixed effects. Because these subgroup analyses are exploratory, statistical significance thresholds were not adjusted for multiple comparisons. There were no statistically significant differences between subgroup impact estimates. ***/**/* Impact estimate is statistically significant at the .01/.05/.10 level.

The behavioral attitudes toward school scale measures how a student reports behaving in relation to school activities. Each question required a yes or no response, and the scales reflect the percent of items with which the student agreed. A higher score represents more positive attitudes towards school. (See Appendix D for details on the construction of the scales.)

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 45

Behavioral attitudes toward school

Students identified as being at risk of dropping out of school responded to a questionnaire. This scale is based on student responses to the following 10 survey items related to how students act at school:  I have thought about dropping out.  I attend school regularly.  I reach school on time.  I stay home from school even if I am not sick.  I skip classes during school.  I skip school or miss classes without telling my parents.  I do the homework assigned to me.  I follow the rules at school.  I get in trouble at school.  I have difficulty getting along with other students.

At-risk students in SDPP schools scored 91.5 percent on the behavioral attitudes toward school scale, compared to 91.0 percent among at-risk students in control schools. This difference was not statistically significant (Figure VI.B.5).

Figure VI.B.5. SDPP Program impacts on at-risk students’ behavioral attitudes toward school at endline (SY 2012–2013, SY 2013–2014, and SY 2014–2015) 100 91.5 91.0 90 80 70 60 50

40 student student agreed 30 20

Percentage of scale items Percentagescaleof items withwhich 10 0 SDPP group Control group

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 46

Similarly, SDPP had no statistically significant impacts on at-risk students’ behavioral attitudes toward school among any of the subgroups (Figure VI.B.6).

Figure VI.B.6. SDPP Program impacts on students’ behavioral attitudes toward school at endline, by subgroup (SY 2012–2013, SY 2013–2014, and SY 2014–2015)

1 0.9 0.9 0.8 0.7 0.6 0.5 0.5 0.2 0 0

-0.3 -0.5

with which student whichagreed student with -0.6

-1 Improvements scale items of in Improvements percentage Female Male Not low Low caste Overage Not High % Low % Remote Not caste overage At-risk At-risk schools remote schools SDPP group

2. Impact on additional measures of student attitudes toward school

In addition to measuring impacts on the three primary student attitudinal outcomes described above, SDPP also assessed the program’s impacts on two other attitudinal outcomes for at-risk students: their perceptions about teachers’ and parents’ support of their success in school.31 Higher scores on these scales represent more positive student perceptions of the support they receive from teachers and parents. These scales were also based on responses to the at-risk student survey.

31 As described in Section III, because these analyses are exploratory, the results are not adjusted for multiple comparisons, despite the large number of comparisons.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 47

Student perceptions of teacher support Students identified as being at risk of dropping out of school responded to a questionnaire that included the following 11 survey items related to how students perceive the support provided by their teachers:

 I have had difficulty getting along with my teacher(s).  My teacher(s) cares about how I’m doing.  My teacher(s) talks to me about how I did on my homework and/or exams.  My teacher(s) helps me if I am having problems with a lesson.  I feel comfortable asking my teacher(s) for help with my lessons.  My teacher(s) talks to me if I miss school or class.  My teacher(s) thinks I am capable of completing my current grade.  My teacher(s) has talked to me about my future plans.  My teacher(s) has contacted my parents about my school work.  My teacher(s) has contacted my parents about my attendance.  My teacher(s) and my classmates encourage me not to drop out.

SDPP found statistically significant impact on student perceptions of teacher support. Students from the SDPP group agreed with 89.6 percent of the students’ perception of teachers scale items, while students in the control group agreed with 87.2 percent (Figure VI.B.7).

Figure VI.B.7. SDPP Program impact on students’ perceptions of teacher support at endline (SY 2012– 2013, SY 2013–2014, and SY 2014–2015) 100 89.6*** 90 87.2 80 70 60 50

40 student student agreed 30 20

Percentage of scale items Percentagescaleof items withwhich 10 0 SDPP group Control group

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 48

“My Teacher Made the Difference”

As the youngest in his family of seven, ten-year old Maya Kumar in India dreaded going to school. His father, an agricultural laborer, could hardly support his family because of his ill health. This left all the responsibility to take care of the family on his mother. In addition to doing all the housework and caring for her children, she also worked as a farmer. Maya was shy and quiet. His older siblings’ demanded the little attention his parents could spare. He was afraid of the teachers, who often criticized or beat students, and anxious about his performance in class. Embarrassed by his low marks, Maya rarely attended school. Maya’s teacher used the Early Warning System (EWS) to evaluate Maya’s case. When she looked at Maya’s attendance and scores it was clear that he was at high risk of dropping out of school. The teacher was determined to take action. One of the first things that needed to be done was to make Maya feel welcomed in the classroom, create a place that was interesting and where he wanted to be. The teacher also knew that the A confident and engaged student. school must work in partnership with Maya’s family to provide a holistic support system for him. In class, Maya was one of the quietest boys. He rarely interacted with his peers and was hesitant to engage in learning activities. When his teacher and the community champion began working with him, he confessed he was scared of asking questions in class and did not understand his lessons. They needed to handle his case with great sensitivity. He was moved to the front of the class and paired with a friendly, higher performing classmate. He was encouraged to ask questions. He was also encouraged to participate in enrichment activities that use arts, crafts and body movement to make the learning process more engaging. The sports activities helped Maya—a good athlete--to open up and build stronger relationships with his peers. Soon he was participating more actively, asking questions about his class work. He also discovered he liked school and enjoyed seeing his friends every day. He didn’t want to stay at home anymore. Meanwhile, the headmaster, his teacher and the community champion met with Maya’s parents to take action to keep their son in school. They visited his home and made phone calls to talk with his family. They encouraged and assisted his family to maintain the EWS family journal so that they could track Maya’s attendance at home. At first, their conversations addressed reasons for his absences and hesitant participation in school. But over time, the discussions evolved into conversations focused on Maya’s progress and how to support his school work.. Both Maya and his parents point to the actions taken by his teacher and other school personnel that made the difference, “We were considering sending Maya to live with his sister in Delhi for work. But not anymore.” Not wanting to squander his newfound interest and progress, they confirmed that Maya would keep going to school.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 49

Student perceptions of parental support

Students identified as being at risk of dropping out of school responded to a questionnaire that included the following 10 survey items related to how students perceive the support provided by their parents:  My parents know when I have not completed my homework and assignments.  My parents have talked with my teacher about my exam scores or absences.  My parents have talked with my teacher about my attendance.  My parents make sure I go to school every day.  It is important to my parents that I do well in school.  My parents attend school events.  My parents talk to me about improving my grades.  My parents try to support me with my studies.  My parents free up my time for schoolwork.  My parents want me to complete my current grade.

For the students’ perception of parental support scale, students in the SDPP group agreed with 92.1 percent of scale items and students in the control group agreed with 89.0 percent of scale items (Figure VI.B.8). This impact is statistically significant.

Figure VI.B.8. SDPP Program impact on students’ perceptions of parental support at endline (SY 2012– 2013, SY 2013–2014, and SY 2014–2015) 100 92.1*** 89.0 90 80 70 60 50 40

student student agreed 30 20 10

Percentage of scale items Percentagescaleof items withwhich 0 India

SDPP group Control group

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 50

Parental Support—An essential ingredient for preventing dropout Khushboo was withdrawn and quiet in class. She did not participate actively and would regularly miss school. The days she did come to school, she would usually come late or leave early. When her teachers talked to her mother about her attendance patterns they were told, “I want my daughters to do all the household work so that by the time they get married they know how to manage their own household.” Her father was too busy in his shop to pay attention to his daughter’s education. The SDPP situational analyses found that low parental expectations and support for their children’s schooling was a factor associated with dropout. One of the key components of the Early Warning System introduced by SDPP into Khushboo’s school was to strengthen parental involvement in and support of their child’s schooling. Under SDPP’s guidance, teachers and community champions used a number of activities to increase the parents’ awareness about the importance of staying in school and how to help students succeed. They worked with the parents to become actively involved in their children’s education. Khushboo’s teachers began a dialogue with her parents. When she was absent or had problems at school, they would call her parents or go to her house to discuss what the school and Khusboo’s parents could do. Over time, Khushboo’s teachers and parents built a strong relationship. The result was that her parents began to appreciate how important a good education was for Khusboo’s future. Voice messages sent to parents’ mobile phones provided guidance on how to help students, informative mini-dramas about households dealing with dropout-related issues, and invitations to school events. In response, Khushboo’s parents also joined others at quarterly school Open Houses at the school to see their children’s work, skits and demonstrations, and learn about support methods by playing a chutes- and-ladders dropout board game. After attending two Open Houses, Mothers play the Stay-in-School game at an Khushboo’s father, Sanjeet Mahto, said, “I open house never got an opportunity to connect with my daughter’s school and her education in this way before.” He suggested that the event should become a monthly practice and said he now realizes the value in knowing what the children are learning at school. Her mother, Sanju Devi, learned how to fill out the SDPP family journal and is now tracking Khushboo’s attendance. Since her school has been a part of the SDPP Program, Khushboo reports that her parents support her studies: her workload at home has been reduced and they provide school supplies. Her parents want her to finish her education and her teachers are helping to make that happen.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 51

C. Impacts on student engagement in school

The SDPP theory of change conceptual model Student engagement in school suggests that an improvement in student outcomes attitudes towards school will result in an improvement in their engagement in school. Primary measure of program effectiveness SDPP used three outcomes to measure student  Daily attendance engagement in school: attendance, academic performance, and behavior. SDPP used school Additional measures records to construct the outcome measures.  Student performance in school Because the program was intended to improve  Student behavior in school student attendance, daily attendance is the primary measure by which SDPP determines its effectiveness in this domain.

This section presents findings on the impact of SDPP on student engagement in school.

1. Impact on student’s daily attendance in school

Because SDPP was intended to improve student attendance, it is the primary measure used to determine SDPP’s effectiveness in this domain. Students at risk of dropping out may miss excessive amounts of school, causing them to do poorly on their school work and fall behind in their studies, discouraging and alienating them from their studies, school, and classmates. SDPP estimated impacts on attendance for all students as well as for students identified as at risk of dropout based on their baseline characteristics.

Student attendance

The attendance measure is average daily attendance for the full school year. The daily attendance rate is the percentage of school days a student attended during the school year, constructed by averaging the monthly percentages for the most recent school year.

SDPP had a positive, statistically significant impact on attendance (Figure VI.C.1). Fifth- grade students in SDPP schools had an average daily attendance rate of 65.2 percent, compared with 63.5 percent for students in control schools, a 2.7 percent change improvement in attendance for the SDPP group as compared to the control group.

Average attendance rates differed greatly by at-risk student status. At-risk students had an average attendance rate of 62.7 percent in SDPP schools, and 60.8 percent in control schools. The difference was statistically significant. In contrast, not-at-risk students in SDPP schools

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 52

had an average attendance rate of 81.7 percent, compared to a rate of 80.9 percent among not-at- risk students in control schools. However, this difference was not statistically significant.

Improved attendance: what does this mean? Positive impacts on attendance mean that students were in school and able to receive instruction more often than they would have without SDPP. Impacts on attendance can be translated into the number of additional school days attended by all students who received the SDPP Program. The more school days attended, the lower the wastage of educational resources

In India, schools are open for 225 days, on average, during a school year. A student typically misses around 82 days of school per school year. Overall, a student who participated in SDPP attended four additional days at school compared to a student who did not participate in SDPP. Given that around 19,000 students were exposed to the intervention in the SDPP schools, the number of additional school days attended is up to 76,000 among students who participated in SDPP in India. Among at-risk students in India, the number of additional school days attended is up to 65,000.

Figure VI.C.1. SDPP Program impact on daily attendance at endline, overall and by at-risk status (SY 2012–2013, SY 2013–2014, and SY 2014–2015) 100

90 81.7 80.9 80

70 65.2*** 62.7*** 63.5 60.8 60

30 Percentagedaysof attended 20

0 All students At-risk students Not-at-risk students

SDPP group Control group

Sources: SDPP baseline and follow-up school records data collection; July 2012, May 2013, February 2014, and November 2014. Note: The analysis is based on SY 2012–2013, SY 2013–2014, and SY 2014–2015 5th-grade students. The sample includes 19,018 students for the SDPP group (16,311 at-risk and 2,707 not at-risk) and 18,018 students for the control group (15,590 at-risk and 2,428 not at-risk). Differences between SDPP and control group means were tested using two-tailed t-tests. Mean values are adjusted for baseline characteristics. The analysis accounts for clustering of students within schools, and cohort fixed effects. For a tabular presentation of these findings, see Appendix Table G.5. ***/**/* Impact estimate is statistically significant at the .01/.05/.10 level.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 53

The SDPP Program was not designed Attendance data to have different impacts for different subgroups of students (other than at- SDPP collected aggregate attendance data via headcounts risk of dropout); however, the during each data collection effort during mid-morning. These values were similar in SDPP and control schools. literature suggest that outcomes and impacts might vary for different types The attendance rate from these data was then compared to of students. 32 Impacts of the SDPP attendance measures derived from student level data from Program on attendance were focused school records. The attendance rate based on the headcount on particular subgroups (Figure was lower than the rate determined through school records in both research groups. VI.C.2).

Among both females and males, the attendance rate between SDPP and control school students was higher by a statistically significant amount. For students who were 2 years older than the appropriate age for their grade and for students who were an appropriate age, the SDPP Program had a statistically significant positive impact on attendance. SDPP increased attendance among both caste groupings. The difference between these subgroup impacts is not statistically significant. Among schools with few at-risk students, students in SDPP schools had a statistically significant higher attendance than control school students. For students in schools close to the district capital, the difference for SDPP versus control school students was statistically significant.

Figure VI.C.2. SDPP Program impact on daily attendance at endline, by subgroup (SY 2012–2013, SY 2013–2014, and SY 2014–2015)

3 2.5*** 2.7*** 2.1** 2.2** 1.7*** 1.7** 1.6*** 2 1.4** 1.3 1

-1 -0.3

SDPP and control and group control SDPP -2 percentage attended days percentage of Percentage point difference difference Percentage in point -3 Female Male Low caste Not low Overage Not High % At- Low % At- Remote Not remote caste overage risk risk schools schools SDPP group ††

Sources: SDPP baseline and follow-up school records data collection; July 2012, May 2013, February 2014, and November 2014. Note: The analysis is based on SY 2012–2013 and SY 2013–2014 grade 5 students. Differences between SDPP and control group means were tested using two-tailed t-tests. Mean values are adjusted for baseline characteristics. The analysis accounts for clustering of students within schools, and cohort fixed effects. Because these are additional outcomes and the analyses are exploratory, statistical significance thresholds were not adjusted for multiple comparisons. ***/**/* Impact estimate is statistically significant at the .01/.05/.10 level. †††/††/† Statistically significant differences among the subgroup impact estimates at the .01/.05/.10 level.

32 As described in Section III, because these analyses are exploratory, the results are not adjusted for multiple comparisons, despite the large number of comparisons.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 54

Enrichment Program Motivates Regular Attendance

Chhotu is a student in Samastipur, Bihar, India. His father runs a small bike repair shop next to his school. Instead of going to school, Chhotu preferred spending most of his time in his father’s shop. His parents were anxious to see their son become well educated and get the reading and writing skills needed to help his father run his business. But Chhotu didn’t like school and rarely attended classes. When he did, he generally started trouble. His parents did not know what to do. When the SDPP Program started in his school, Chhotu’s teacher identified him as a student at- risk of dropping out of school. It was clear he Students enjoying EP activities. was falling behind due to his poor attendance, disengaged behavior, and poor class performance. Following Early Warning System (EWS) “track and trigger” steps, his teacher and the community champion carefully tracked his attendance and coursework. They also spoke with his family to learn more about the reasons for his behavior and to identify how they can address his absenteeism. His father informed the school that he had tried to get Chhotu to attend class. “I have visited the school and spoken to his teachers about it. But nothing that we did helped to get him to school.” One day, one of the teachers saw Chhotu peering through the school fence, watching his classmates participate in the SDPP Enrichment Program activities. He asked Chhotu to join the group, but Chhotu was not sure. Finally after several days, Chhotu made up his mind to participate. Initially Chhotu only wanted to attend the sports sessions. But as his relationship began to blossom with classmates and teachers, he started attending more, and soon everyone discovered Chhotu’s knack for arts and crafts. He enjoyed drawing and painting and experimenting with different mediums. As his interest grew in attending the extracurricular activities, his attendance in regular classes improved, too. The teachers and the community champion gave him more responsibilities in the EP sessions, putting him in charge of bringing out the sports equipment and distributing art material. This not only gave Chhotu a sense of responsibility and ownership, but also helped him build stronger bonds with his peers. He gained even more confidence when he showcased his artwork at the school’s Open Houses, attended by parents and community members. Within a few months Chhotu’s stopped skipping class. His head master explained, “Chhotu takes great interest in his work now. Earlier I would always see him fighting with his classmates, but now he is playing with them.” His mother agrees. “Before, he ran away even if we took him to the school,” she says, “Now he cleans his uniform himself and goes to school neatly without us insisting.” Instead of sitting in his father’s shop watching other children go to school, Chhotu now sits at the front of the class every day.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 55

2. Impact on additional measures of student engagement in school

In addition to measuring impact on daily attendance described above, SDPP assessed the program’s impacts on two other engagement outcomes: students’ academic performance in math and language and their behavior in school.

Math and language performance

In India, schools use two different performance measures. SDPP created an aggregate measure that calculates whether a student scored above average in math. The outcome variable is the percentage of students who scored above the mean score of the relevant performance measure among students who took both tests in math. Academic performance measures combine Child profile and Pragati Patrak scores of those students who took both tests in each subject. Scores are calculated in each performance measure in each subject among the students who took both tests. Outcome reflects the percentage of students who scored above the mean score of the relevant performance measure among the students who took both tests in that subject.

a. Academic performance in math and language

SDPP also examined the program’s impacts on academic performance since an increase in attendance, targeted by both the EWS and EP, could result in an overall improvement in performance in school. SDPP did not attempt to directly influence academic performance, although the enrichment program did incorporate activities related to cooperative learning and language skills, as described in Section II. The analysis of student performance does not affect the evaluation’s primary determination of program effectiveness.33 The measures SDPP examined are Hindi language and math performance from annual scores for students. Because schools in India use two different performance measures, SDPP created an aggregate measure that calculates whether a student scored above average in each subject. The outcome variable is the percentage of students who scored above the mean score of the relevant performance measure among students who took both tests in that subject.

SDPP found a marginally significant impact of SDPP on math performance among not-at- risk students (Figure VI.C.3). However, SDPP found no impact on math or Hindi performance among all students or among at-risk students (Figure VI.C.3 and Figure VI.C.4). Appendix E presents additional findings for the engagement domain based on alternative measures of attendance.

33 Moreover, impact estimates for academic performance may be biased if the program was successful in keeping low-performing students in school. Also, as described in Section III, because these analyses are exploratory, the results are not adjusted for multiple comparisons, despite the large number of comparisons.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 56

Figure VI.C.3. SDPP Program impacts on math performance at endline (SY 2012–2013 and SY 2013– 2014) 100 90 76.3* 80 70.4 70 59.1 60 56.3 55.7 53.5 50 40 30 20 10 0

Percentagescoring aboveaverage All students At-risk students Not-at-risk students SDPP group Control group

Figure VI.C.4. SDPP Program impacts on Hindi language performance at endline (SY 2012–2013 and SY 2013–2014) 100 90 75.0 80 70.9 70 57.4 53.8 60 53.6 50.2 50 40 30 20 10 0

Percentagescoring aboveaverage All students At-risk students Not-at-risk students SDPP group Control group

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 57

Appendix E presents additional findings for the engagement domain based on alternative measures of attendance, including an alternative approach to accounting for missing values from school records.

D. Impacts on school dropout

Reducing school dropout is the primary goal of School dropout outcomes the SDPP intervention. Students who drop out of school prematurely forgo a considerable Primary measure of program effectiveness amount in yearly earnings over their working  Between-grade dropout lives (Duflo, Esther 2001), limit their access to further education and training, reduce their Additional measures ability to contribute to the well-being of their  Student progression in school community, and add to wastage of the resources  Alternative measures of dropout invested in their schooling.

For India, SDPP defined the primary dropout measure as dropout between school years (between- grade) for the targeted 5th-grade students. Students who did not enroll in the school in the following school year were considered to have dropped out of school. In this section, SDPP used school records to measure dropout from school. Because the last round of data collection occurred during the 2014–2015 school year, when the last cohort was still in the 5th grade, between-grade dropout information is available for only the SY 2012–2013 and SY 2013–2014 cohorts.

This section presents findings on the impact of SDPP on student dropout from school.

1. Impact on primary measure of school dropout

Between-grade school dropout

Dropout is based on whether the student had dropped out at the time of follow-up data collection. Between-grade dropout for the targeted 5th graders; students who did not enroll in the school in the following school year were considered to have dropped out of school.

SDPP had no impact on between-grade dropout for the SY 2012–2013 and SY 2013–2014 5th-grade cohorts (Figure VI.D.1). SDPP was not able to assess impact on between-grade dropout for the SY 2014-2015 5th-grade cohort, due to the timing of the last data collection. Students in both SDPP and control schools dropped out at a rate of 8.3 percent.

At-risk students dropped out at a higher rate than not-at-risk students. In both SDPP and control schools, at-risk students dropped out of school at a rate of 9.3 percent.34

34 We also estimated the treatment-on-the-treated (TOT) impacts for students who were identified through the EWS for SDPP services using quasi-experimental techniques. The TOT estimates are consistent with the impacts for at-risk students presented here (See Appendix I for details).

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 58

In SDPP schools, students not at risk dropped out at a rate of 2.8 percent, while in control schools students not at risk dropped out at a rate of 2.6 percent; the difference between SDPP and control groups for not at risk students was not statistically significant. Given the low level of dropout in study schools, as well as the compulsory schooling laws, it is not surprising that the SDPP Program did not significantly reduce dropout.

Figure VI.D.1. SDPP Program impact on between-grade dropout rates at endline, overall and by at-risk status (SY 2012–2013 and SY 2013–2014) 100 90 80 70 60 50 40 30 20 8.3 8.3 9.3 9.3 Percentage students of dropped out 10 2.8 2.6 0 All students At-risk students Not-at-risk students

SDPP group Control group

Sources: SDPP baseline and follow-up school records data collection; July 2012, May 2013, February 2014, and November 2014. Note: The analysis is based on SY 2012–2013 and SY 2013–2014 5th-grade students. The sample includes 13,149 students for the SDPP group (11,283 at-risk and 1,866 not at-risk) and 12,413 students for the control group (10,606 at-risk and 1,807 not at-risk). Differences between SDPP and control group means were tested using two-tailed t-tests. Mean values are adjusted for baseline characteristics. The analysis accounts for clustering of students within schools, and cohort fixed effects. For a tabular presentation of these findings, see Appendix Table G.5. ***/**/* Impact estimate is statistically significant at the .01/.05/.10 level.

Because three cohorts of 5th-grade students received the intervention, SDPP estimated the impacts on each cohort by school year (Figure VI.D.2).35 Dropout rates differed slightly by school year, but there were no statistically significant differences between SDPP and control group students in any school year. In SY 2012–2013, students in SDPP schools had a between-grade dropout rate of 9.6 percent, compared to a rate of 10.4 percent for students in control schools. In SY 2013–2014, in contrast, 7.0 percent of students in SDPP schools dropped out of school between grades, compared to 6.3 percent of students in control schools. Finally, in SY 2014–2015 SDPP has a within-grade measure of whether students who enrolled at the start of the 2014–2015 school year were still enrolled at the time of data collection in October 2014. In SDPP schools 10.5 percent of

35 As described in Section III, because these analyses are exploratory, the results are not adjusted for multiple comparisons, despite the large number of comparisons.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 59

students dropped out by October 2014, compared to 10.0 percent in the control schools; this difference was not statistically significant.

Figure VI.D.2. SDPP Program impacts on dropout rates, by school year (SY 2012–2013, SY 2013–2014, and SY 2014–2015)

100 90 80 70 60 50 40 30 20 9.6 10.4 10.5 10.0

10 7.0 6.3 Percentage students of dropped out 0 SY 12-13 SY 13-14 SY 14-15 SDPP group Control group

Sources: SDPP baseline and follow-up school records data collection; July 2012, May 2013, February 2014, and November 2014. Note: The analysis is based on SY 2012–2013, SY 2013–2014, and SY 2014–2015 5th grade students. The sample includes 19,721 students for the SDPP group (6,607 in SY 2012–2013, 6,542 in SY 2013–2014, and 6,572 in SY 2014–2015) and 18,669 students in the control group (6,205 in SY 2012–2013, 6,208 in SY 2013–2014, and 6,256 in SY 2014– 2015). Differences between SDPP and control group means were tested using two-tailed t-tests. Mean values are adjusted for baseline characteristics. The analysis accounts for clustering of students within schools, and cohort fixed effects. Because these subgroup analyses are exploratory, statistical significance thresholds were not adjusted for multiple comparisons. For a tabular presentation of these findings, see Appendix Table H.7. ***/**/* Impact estimate is statistically significant at the .01/.05/.10 level.

SDPP also explored the SDPP Program’s impacts on between-grade dropout for the five additional subgroups of interest: gender, over-age status, caste grouping, whether the school has a low or high percentage of students at risk, and school distance to the district capital.36 Despite finding no effect on dropout overall, SDPP did find impacts for the subgroups related to the school’s percentage of at-risk students (Figure VI.D.3). In schools with a few at-risk students, SDPP decreased dropout by 2.0 percentage points. On the other hand, in schools with a large number of at-risk students, dropout increased by 3.8 percentage points. Both differences between the SDPP and control groups were statistically significant, and the difference in impacts between schools with a low percentage of students at risk and schools with a high percentage of student at risk was also statistically significant at the 1 percent level.

36 As described in Section III, because these analyses are exploratory, the results are not adjusted for multiple comparisons, despite the large number of comparisons.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 60

Figure VI.D.3. SDPP Program impact on between-grade dropout rates at endline, by subgroup (SY 2012–2013 and SY 2013–2014)

4 3.8*** 3 2 1 0.6 0.4 0.7 0.0 0 -0.1 -1 -0.6 -0.6

dropout rates -2 -1.3 -2.0** -3

Percentagepoint difference in -4 SDPP and controlgroup school Female Male Low Not low Overage Not High % Low % Remote Not † caste caste overage At-risk†††At-risk schools remote schools SDPP group

Sources: SDPP baseline and follow-up school records data collection; July 2012, May 2013, February 2014, and November 2014. Note: The analysis is based on SY 2012–2013 and SY 2013–2014 5th-grade students. Differences between SDPP and control group means were tested using two-tailed t-tests. Mean values are adjusted for baseline characteristics. The analysis accounts for clustering of students within schools, and cohort fixed effects. Because these subgroup analyses are exploratory, statistical significance thresholds were not adjusted for multiple comparisons. ***/**/* Impact estimate is statistically significant at the .01/.05/.10 level †††/††/† Statistically significant differences among the subgroup impact estimates at the .01/.05/.10 level.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 61

Dropout Averted for At-Risk Student

Sujit Kumar is a class 5 student and has four siblings. Even though his parents did not go to school, his father understands the importance of a good education. He is determined to send all his children to school, but Sujit was rebellious. Each morning, his parents would send him off to school, but he often went to play instead. Even on the days he did go to school, once attendance was taken, he would sneak out, especially after the school lunch was served. He would return home early, complaining about the teacher or the class work. In Sujit’s school, he belonged to a group of boys who were considered trouble-makers. In fact, most of Sujit’s friends also struggled with their course work or often missed class. A high percentage dropped out. When his friends skipped school, Sujit happily went along with them. So, in spite of his father’s urging him to go to school and stay there all day, Sujit did not comply. When the Early Warning System was introduced in his school, Sujit was identified as one of many students at-risk of dropping out. His teacher worked with the school’s community champions to carefully monitor his attendance, behavior and course performance. They regularly visited Sujit’s home to talk to his parents and discuss how to solve Sujit’s attendance problem. They also invited Sujit to take part in the Enrichment Program during the last class of the day, telling him he would have fun with his classmates. Curious, Sujit decided to give it a chance. To his surprise, he enjoyed the activities, especially the sports and outdoor games. At first, he came to school only to attend the Enrichment Program, but he gradually started coming to school for his regular classes as well. On the days when Sujit came to school, his teacher made sure he received extra attention. He welcomed him and told him how happy he was Sujit was there; he made sure Sujit received help with his school work. Over time, Sujit’s behavior improved. His attendance jumped from 50% to over 95% by the end of the year. His demeanor had changed, too. He helped other students with their studies and was assigned as a group leader. The headmaster observed, “Earlier he used to bully the other children and pick fights, but now he cooperates with the other students.” When his mother came to an Open House at the school, she profusely thanked his teacher, “If you

did not pay attention to my child’s studies, he would have dropped out from school long ago.” Sujit has become a model student and a favorite among his teachers. His studies have improved, especially Hindi. He now rushes home to share with his parents all the wonderful things that are happening at school. The only regret his parents have about the SDPP Program is that it was not in place earlier so that Sujits’ older brother might have stayed in school.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 62

2. Impact on additional measures of school dropout

a. Within-grade dropout

Within-grade dropout is another measure of interest in India.37 It is calculated using monthly attendance data. This measure is included because it is the only dropout measure available for the SY 2014–2015 5th-grade cohort.

Within-grade school dropout

Within-grade dropout is defined by whether a student who was enrolled in school at the beginning of the school year and was exposed to the intervention for at least one year is still enrolled at the year’s end. A student is considered to have dropped out of the school if the student did not attend in last three months of the school year. For the 2014-2015 5th grade cohort, who were only exposed to the intervention for half a year at the time of endline data collection, SDPP constructed a mid-year dropout variable. This measure reflects whether the student dropped out before October 2014 (the last point at which enrollment information is available for this cohort), using the last three months of attendance data available.

SDPP had no impact on within-grade dropout overall (Figure VI.D.4).

Figure VI.D.4. SDPP Program impact on within-grade dropout (SY 2012–2013, SY 2013–2014, and SY 2014–2015) 100 90 80 70 60 50 40

dropped out 30 20 Percentage students of 10.4 10.9 10 0 SDPP group Control group

Sources: SDPP baseline and follow-up school records data collection; July 2012, May 2013, February 2014, and November 2014. Note: The analysis of within-grade dropout is based on SY 2012–2013, SY 2013–2014, and SY 2014–2015 5th-grade students. Differences between SDPP and control group means were tested using two-tailed t-tests. Mean values are adjusted for baseline characteristics. The analysis accounts for clustering of students within schools, and cohort fixed effects. Because these are additional outcomes and the analyses are exploratory, statistical significance thresholds were not adjusted for multiple comparisons. ***/**/* Impact estimate is statistically significant at the .01/.05/.10 level.

37 As described in Section III, because these analyses are exploratory, results were not adjusted for multiple comparisons, despite the large number of comparisons.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 63

b. Grade progression

SDPP also calculated a measure of grade progression after the target grade year. It is measures whether a 5th-grade student enrolled in 6th grade or higher in the year following exposure to SDPP. Grade progression differs from between-grade dropout because grade repeaters are not counted. A student who repeats the grade he/she is in is not considered to have progressed. Although not included in the school dropout domain, as described in Table IV.1, this measure provides additional information on students’ successful completion of 5th grade.

Grade progression

A student is considered to have progressed if they are enrolled in 6th grade or higher in the following school year.

SDPP had no impact on grade progression in school (Figure VI.D.5).

Figure VI.D.5. SDPP Program impact on grade progression (SY 2012–2013 and SY 2013–2014 100 90 85.4 85.1 80 70 60 50 40 30 20

10 Percentage students of progressed 0

SDPP group Control group

Sources: SDPP baseline and follow-up school records data collection; July 2012, May 2013, February 2014, and November 2014. Note: The analysis of grade progression is based on SY 2012–2013 and SY 2013–2014 5th grade students. Differences between SDPP and control group means were tested using two-tailed t-tests. Mean values are adjusted for baseline characteristics. The analysis accounts for clustering of students within schools, and cohort fixed effects. Because these are additional outcomes and the analyses are exploratory, statistical significance thresholds were not adjusted for multiple comparisons. ***/**/* Impact estimate is statistically significant at the .01/.05/.10 level. See Appendix F for findings related to the dropout domain based on alternative measures of between-grade dropout and within-grade dropout. In addition, Appendix G provides findings for an additional measure of progression, which is defined as a student enrolling in 6th grade (but not higher) the school year following exposure to SDPP.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 64

Qualitative Research Findings in India During the second year of intervention implementation in the treatment schools, the SDPP team conducted a Qualitative Research Study in India to (1) better understand why changes in student and teacher behaviors and attitudes had happened, and (2) how beneficiaries and targets (students, teachers, school directors, parents and community members) responded and reacted to the interventions. Some finding are highlighted below. What do beneficiaries say about the Early Warning System? The need to help support their families was the primary reason students considered dropping out and the reason why some had dropped out. Additionally, 60 percent of students who dropped out said it was their own decision, while 40 percent did it at the urging of their parents and family. Nearly half of the dropouts indicated more could have been done to keep them in school, including improving the quality of education, adding their names to the register, talking with their parents, addressing the teasing and bullying they were experiencing, and lastly, offering scholarships. But, despite this, about 85 percent of students and dropouts confirmed their family received a home visit or phone call from a friend, teacher, or community champion to discuss their attendance and staying in school. Although school directors did not contact their families, students said they encouraged them to stay in school and provided support, such as seeking medical attention when they were sick. At-risk students and dropouts did not share the same view about school contact with their families. When asked to select an emoticon to represent their feelings about the contact, at-risk students selected “happy,” while dropouts selected “silent,” followed by “sad to leave school”, then “afraid” and lastly, “surprised.” Although most parents noted that dropout was increasing, few (10%) parents thought that their child was at risk of dropping out of school. Nearly all parents had been contacted by the school about their child. Most parents were happy about being contacted by the school, and nearly all said that they received helpful information to support their child. They especially appreciated the home visits, followed by phone calls, Open Houses, and recorded messages. School contact general resulted in closer parental monitoring of attendance. They report they pay more attention, such as washing clothes and providing meals punctually, to enable the child to get to school on time, and checking homework. Only half of the teachers confirmed they consistently implemented the EWS in their classrooms – a number probably influenced by the stronger role community champions played in the implementation of the EWS. Despite this, most (80%) indicated they called parents about student attendance and found that generally student behaviors improved, they were more motivated, and it fostered greater involvement from the parents. Overall, nearly all (90%) the teachers EWS was effective at mitigating school dropout. The majority (60%) said EWS made their job easier, but over a third (38%) said it made it harder due to paperwork and dealing with “ignorant” parents. Head masters liked the EWS. The felt it helped organize school records, and increased teacher responsibility and understanding of children’s needs. They especially appreciated the support of the community champions in “bridging the gap” between the school and parents, something teachers were not always able or willing to do. When asked about continuing with the EWS, head masters said that teacher work load, teacher shortages, and teacher absenteeism presented challenges to EWS implementation. Most did not feel they could continue without community champions. Teachers hoped to continue tracking attendance, communicating with parents, scoring students, and assessing performance in Hindi, in that order. They suggested the EWS forms be streamlined and school management committees assume responsibility for home visits.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 65

What do beneficiaries say about the Enrichment Program? The Enrichment Program (EP) session were a required part of the school schedule. All at risk students and 80 percent of dropouts participated in the EP. Attendance varied: nearly all at-risk children (90%) attended and averaged more than two days per week. A lesser percentage (60%) of dropouts attended, averaging two days per week. Eighty percent of at-risk students said the EP improved their studies and 68 percent of at- risk students said it motivated them to attend on days when there were EP classes. Dropouts were not as enthusiastic: only 38 percent said it helped their studies and 15 percent said they attended more on EP days. Almost all the parents reported their children enjoyed the EP classes and supported their child’s participation. However, nearly 40 percent of students who dropped out reported their parents told them not to participate. Community champions, who worked with EP teachers in conducting the EP classes, found the lessons for the EP classes helpful and noted strong support from the head masters. They also reported that about 75 percent of teachers enjoyed and were involved in implementing the classes. When asked about the changes in the students from participating in the EP classes, they pointed to more regular attendance, improved punctuality, greater curiosity and confidence, better gender integration in all the activities, and an improvement in Hindi. They believed that the students were better disciplined and listened to direction better. They also observed many of the teachers using the instructional techniques in their classrooms. All of the EP teachers participated in the enrichment activities, though they differed in their level and type of involvement. About 15 percent of EP teachers led the EP sessions, while three-fourths of them assisted the community champion. When the EP teachers did lead an activity, it was generally about 60 percent of the sessions. Almost all of the teachers—even those who did not actively participate--found the materials very clear and simple to use. More importantly, all of the EP teachers said that they now use some of the EP methods or activities in their regular classes. In particular, they have embraced the interactive nature of the pedagogy, have worked to create a friendlier environment that welcomes children and sometimes use energizers to start activities. They noted the EP activities contributed to easier classroom management since they tempered the behavior issues of some of the students. They observed the students listened much better to instructions during the EP classes. They also felt the students were friendlier with one another, had greater respect for one another, and became less shy by engaging in open discussions and participating in groups. As one teacher shared, “Even other teachers enjoy the EP activities. They come out of class to see how the activities are done. Students in other classes want to participate and try to imitate the EP activities.” Head masters seconded these observations agreeing that students were better behaved and more motivated to come to school. They also noted that many of their teachers were using techniques from the EP classes in their regular teaching. Parents confirm the value added of the EP classes as well. In fact, almost 85 percent of them would like to see more EP activities at the school, and 97 percent of them said it would not be not a problem if their children stay at the school longer for the activities. They noted their children were better behaved, had improved attendance, were more attentive to their appearance, communicated better with them and exhibited fewer gender issue. When asked about continuing with EP classes after the close of SDPP, most teachers were doubtful. Head masters were more optimistic with 80 percent of them noting it would be worthwhile. However, they acknowledged that the student-teacher ratios were already high, and teachers had little time for such additional responsibilities, although the EP was conducted during scheduled class time. When asked directly if they thought they would be able to continue with the EP classes, 70 percent of the head masters said they planned to try. In sum, students liked the classes, wanted to be a part of them, and were drawn to come to school as a result. As one teacher observed, “Students of lower classes stay in school to watch the sessions. Students also have an idea that EP activities are for class 5, so class 4 students feel excited to enter class 5.”

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 66

VII. School-level dropout trends

The analyses presented thus far use student-level measures of dropout. This section presents different types of data to explore if alternative measure can provide additional insight or support to the findings presented earlier. SDPP has complementary grade-level measures of dropout using data on enrollment and transfer counts from school records, as is often done in education studies.38 Constructing grade-level rates based on counts requires a much smaller data collection effort than gathering details about all individual students in a grade or school. School systems often report aggregate measures of enrollment and dropout as part of regular reporting, and stakeholders often find these measures to be easier to understand and interpret than other measures. Therefore, for readers more accustomed to these measures, they are presented in addition to the student-level estimates. These grade level measures are available for 5th grade as well as for grades that were not the focus of the interventions (grades 1-6). SDPP presents aggregate between-grade and within-grade dropout rates over time, across three complete school years from SY 2011–2012 to SY 2013–2014 and one partial school year (SY 2014–2015), for grades 1-6 in SDPP and control schools.

Grade-level measures, although much easier to collect than individual-level data, typically suffer from substantial measurement error because it is difficult to distinguish between enrolled students, dropouts, transfers, repeaters, and newly enrolled students. Thus, aggregate dropout measures actually include students in all of these situations. The magnitude of the aggregate dropout measure is therefore sensitive to whether more students transferred in or repeated the grade than dropped out or transferred out; the magnitude could even change signs to show a negative dropout rate. As a result, depending upon the levels of transfer, repetition, and dropout, there may or may not be observable trends in dropout when using measures such as these that are constructed from aggregate grade-level counts.

Across grades, there were no clear trends in between-grade dropout in either the SDPP group or the control group (Figure VII.1). The between-grade dropout rate for grades 1 to 4 ranged widely across grade and school year, averaging between -30.1 percent and 13.4 percent across SDPP and control schools. Between-grade dropout rates were generally negative for 1st grade (which likely represents a large influx of students to first grade during the school year), and positive for 2nd to 4th grade.

The between-grade dropout rate for 5th grade was largely negative, ranging between -44.1 percent and -70.2 percent. In no case were the difference between SDPP and control groups statistically significant. The rates presented here for 5th grade differ greatly from the overall rates shown in Figure VI.D.1. This difference arises because the measure counts the number of students enrolled at specific points in time, in particular at the start and end of the school year, and uses these grade level values to derive the dropout rate, instead of following individual students over time to observe what happens to them. As such, students who transfer in and out of schools during the year are not properly accounted for the measure, leading to error. Because schools were required to have a 6th grade to be eligible for SDPP, it is possible that these negative between-grade dropout rates from 5th to 6th grade are the result of an influx of students transferring into the 6th grade from schools

38 The number of students enrolled at the start of each school year, and the number of students enrolled during the last month of each school year (March) were collected and used to calculate within-grade and between-grade dropout.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 67

that did not offer a 6th-grade class. Only including aggregate measures introduces error to the dropout measure. In Appendix F SDPP presents an alternative measure of school between-grade dropout that includes transfers, however there are still no clear trends in dropout with that alternate measure.39

Figure VII.1. School between-grade dropout rates by grade and school year

20.0 11.6 12.6 13.4 12.1 12.7 8.6 10.0 5.0 2.4 2.9 3.63.9 0.40.7 2.11.2 0.51.2 0.0 -0.3 -3.5 -10.0 -7.4 -12.5 -20.0 -16.2 -15.7

-30.0 -30.1 -40.0

-50.0 -46.3-44.1

Percentage students of dropped out -60.0 -57.7 -62.7 -63.4 -70.0 -70.2

-80.0

2013-2014 2011-2012 2012-2013 2013-2014 2011-2012 2012-2013 2013-2014 2011-2012 2012-2013 2013-2014 2011-2012 2012-2013 2011-2012 2012-2013 2013-2014 School year School year School year School year School year Grade 1 Grade 2 Grade 3 Grade 4 Grade 5

SDPP group Control group

Sources: SDPP baseline and follow-up school records data collection; July 2012, May 2013, February 2014, and November 2014. Note: Between-grade dropout is measured by the number of students enrolled at the beginning of an initial school year and the number enrolled at the beginning of the following year. The analysis is based on aggregate enrollment count data for SY 2011–2012, SY 2012–2013, SY 2013–2014, and SY 2014–2015 1st through 6th-grade students. Differences between SDPP and control group means were tested using two-tailed t-tests. ***/**/* Statistically significant difference at the .01/.05/.10 level. SDPP also looked at trends in within-grade dropout rates using grade-level measures, which varied across grade and school year as with the between-grade measure (Figure VII.2). They generally decreased from SY 2011–2012 to SY 2014–2015 in both SDPP and control group schools, with the exception of 1st grade, where dropout rates were negative across all school years. Within-grade

39 It is possible to incorporate counts of transfer students in such a measure; however, records of transfers often are not kept consistently in schools. We discuss these limitations in Appendix F.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 68

dropout rates for 1st grade ranged between -5 percent and -26 percent. These negative dropout rates likely reflect the entry of new students into the 1st grade during the school year. Across grades 2-6, SY 2011–2012 dropout rates were between 7.7 and 10.5 percent. In SY 2012–2013, dropout rates were between 0.6 and 5.4 percent, and in the remaining two school years dropout rates were close to zero, with one exception. Within-grade dropout rates for students in the SDPP group schools was -8.1 percent in SY 2013–2014. In no case was the difference between SDPP and control groups statistically significant.

Figure VII.2. School within-grade dropout rates by grade and school year

15 10.5 10.1 9.7 9.0 9.2 10 7.8 8.6 7.7 6.5 5.4 4.1 4.0 5 3.8 3.3 3.6 2.9 2.0 2.3 1.6 1.7 2.2 0.6 1.5 1.1 0.5 0 -0.2 -0.6 -0.2 -0.1 -0.7 -1.7 -1.9 -1.1-1.9 -1.0 -1.0 -5 -2.6 -5.0 -3.8 -1.9 -10 -8.1 -9.4* -15 -13.0 -14.4 -15.9* -20 -17.8

-18.4 Percentage students of dropped out -25 -25.8

-30

2011-2012 2012-2013 2013-2014 2014-2015 2011-2012 2012-2013 2013-2014 2014-2015 2011-2012 2012-2013 2013-2014 2014-2015 2011-2012 2012-2013 2013-2014 2014-2015 2011-2012 2012-2013 2013-2014 2014-2015 2011-2012 2012-2013 2013-2014 2014-2015 GRADE 1 GRADE 2 GRADE 3 GRADE 4 GRADE 5 GRADE 6 SDPP group Control group

Sources: SDPP baseline and follow-up school records data collection; July 2012, May 2013, February 2014, and November 2014. Note: Within-grade dropout is defined by the number of students who were enrolled in school at the beginning of the school year and the number enrolled at year’s end. The analysis is based on aggregate enrollment count data for SY 2011–2012, SY 2012–2013, SY 2013–2014, and SY 2014–2015 1st through 6th-grade students. Differences between SDPP and control group means were tested using two-tailed t-tests. ***/**/* Statistically significant difference at the .01/.05/.10 level.

Given the measurement issues in these dropout rates, it is challenging to draw conclusions about the effectiveness of the program using grade-level estimates. These estimates can be useful in understanding how the level of dropout might change across grades (for example, it is at a similar level across all grades 1-6), but cannot be used to estimate an accurate level of dropout.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 69

SDPP also examined trends using similar information for enrollment and for attendance, as measured by headcounts. During each round of data collection, the SDPP team counted the number of students present in grade 5 classrooms on the day of data collection by entering classrooms during third period and counting the number of students present. The headcounts collected by SDPP during each data collection round show that the average number of grade 5 students present at the time of data collection was similar in SDPP and control schools. An attendance rate was then derived from these headcounts and compared to attendance as measured through student records to determine how similar the two data sources were. At each round of data collection from SY 2012-2013 to SY 2014-2015 the attendance rate as determined through headcounts was lower than the attendance rate determine through school records, although the differences were not statistically significant. This could be caused by overstatement of student attendance by teachers in the school records, or because students attend school for only part of the day and were not in class during the third period, when head counts were collected. Appendix I presents these findings in more detail.

VIII. Discussion

The SDPP Program in India aimed to reduce school dropout among students in 5th grade—the last grade in the primary cycle before students transition to upper primary education in India. SDPP interventions included an EWS to identify and target support to students at risk of dropout and sponsored activities to improve student learning and social skills. SDPP hypothesized that enrichment and recreational activities, along with changes in teacher practices and increased community involvement, would improve student attitudes and behavior, particularly for at-risk students, translating into increased student engagement and reduced school dropout. The SDPP evaluation in India examined the effectiveness of SDPP using a randomized controlled trial in which the outcomes of students and teachers in 113 schools randomly assigned to provide SDPP services were compared to those of teachers and students in 107 schools randomly assigned to a control group providing business-as-usual services. The evaluation of the SDPP Program in India is part of a broader study of similar dropout prevention interventions implemented in four Asian countries.

A. Overview of findings

SDPP examined the effectiveness of SDPP on the primary and additional outcomes the program was intended to affect related to teacher practices, at-risk student’s attitudes toward school, student engagement in school, and school dropout. Figure VIII.A.1 shows SDPP’s impacts according to its theory of change. The boxes banded in red show the outcomes on which SDPP had statistically significant or marginally statistically significant impacts as measured by the impact evaluation.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 70

Figure VIII.A.1. SDPP conceptual model with program impacts

a Dropout School Dropout

Attendanceb Behaviorc Performanced Student Engagement in School

Student Student Attitudes Attitudes and Aspirationse

Teacher Behavior and Attitudinal Teacher Prevention Parent Outcomes Practices and Supportg Supportf

Teacher, Parent, and Community Knowledge and Awarenessh

Notes: The thickness of the border around each box indicates the number significant or marginally significant favorable impacts on the primary and secondary outcomes within the given domain. Impacts on subgroups are not included.

Significant impacts for this domain were found on one outcome

a Dropout includes the primary measure of dropout, within-grade dropout. b Attendance is daily attendance averaged over the entire school year. c Behavior data was not collected in India. d Math and language performance is measured on a range of 2–5, with 2 being the lowest grade and 5 being the highest. e Student attitudes and aspirations are measured only for at-risk students and include students’ emotional, cognitive, and behavioral attitudes toward school. f Teacher and administrator prevention practices and at-risk student perceptions of teacher support. g Parent support is measured by the at-risk students’ perceptions of parent support scale. h Teacher and administrator self-efficacy and sense of responsibility.

Overall, SDPP effected improvements in key student attitudes and behaviors that can lead to dropout. These include positive, statistically significant changes in:

 At-risk students’ emotional attitudes toward school;  At-risk students perceptions of parental and teacher support;  Daily attendance overall and among at-risk student; and  Increased math performance for students not at-risk of dropout.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 71

Despite improvement in student engagement in school as measured through attitudes and attendance, SDPP did not reduce school dropout or dropout among students identified as at-risk based on their baseline characteristics. SDPP also did not affect teacher or administrator dropout prevention practices, the primary measure of interest, or their sense of self-efficacy and responsibility, secondary measures of interest.

Tables VIII.A.1 and VIII.A.2 summarize the impacts.

Table VIII.A.1. SDPP program impacts on primary measures of program effectiveness in India Impact Teacher dropout prevention practices ○ At-risk student attitudes toward school Emotional attitudes toward school +++ Cognitive attitudes toward school ○ Behavioral attitudes toward school ○ Attendance Overall +++ At-risk +++ Dropout Overall ○ At-risk ○

+ + +/+ +/+ Statistically significant positive impact at the .01/.05/.10 level. — — —/— —/— Statistically significant negative impact at the .01/.05/.10 level. ○ Impact is not statistically significant.

Table VIII.A.2. SDPP Program impacts on additional outcome measures in India Impact Teacher outcomes Teacher self-efficacy ○ Teacher sense of responsibility ○ Administrator dropout prevention practices ○ Administrator self-efficacy ○ Administrator sense of responsibility ○ Student attitudes toward school At-risk student perceptions of parent support +++ At-risk student perceptions of teacher support +++ Perception of computer training N/A Student engagement Math performance ○ Hindi performance ○ Behavior ○ Dropout Progression ○ + + +/+ +/+ Statistically significant positive impact at the .01/.05/.10 level. — — —/— —/— Statistically significant negative impact at the .01/.05/.10 level. ○ Impact is not statistically significant.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 72

B. Contextual factors related to findings

In interpreting findings from the evaluation, it is useful to consider the policy environment in which SDPP was implemented and other contextual factors that may have influenced its effectiveness.

SDPP worked directly with teachers at school, community members affiliated with the schools, and parents at home to improve knowledge and awareness of dropout. This work was intended to improve the dropout prevention practices of teachers, but SDPP did not improve scores on the teacher dropout prevention practice scale used by the evaluation. However, scores on this scale were near the maximum for teachers in both the SDPP and the control groups (7.40 and 7.45 out of 8, respectively). Even without SDPP, teachers may be maintaining a high intensity of the dropout prevention practices measured by this scale. Alternatively, the scale scores might be artificially high in both groups because they are based on self-reports rather than direct observation. Finally, the impact of SDPP on teachers may have been minimal because of the important role played by the SDPP community champions. Although teachers were the primary individuals responsible for implementing the EWS and enrichment activities, community champions were active in supporting most EWS activities and were typically responsible for important tasks such as contacting parents and providing support to the most at-risk students. Findings from the qualitative analysis confirmed that teachers felt that community champions played an important role in implementing the EWS. In contrast to the findings here, both headmasters and community champions indicated that teachers felt a greater sense of responsibility, although many teachers reported that the EWS made their jobs harder (Creative Associates International 2015).

As noted earlier, SDPP had a positive impact on at-risk students’ emotional attitudes toward school and daily attendance. Both elements of SDPP—the EWS and enrichment activities—are likely to have contributed toward SDPP’s success in impacting these outcomes. It is possible that the enrichment and recreation activities improved student’s emotional attitudes toward school and attracted them to attend more frequently than they would have without the appeal of these activities.

As part of the EWS, homeroom teachers and community champions were asked to monitor at-risk students’ attendance and contacted parents if students were consistently absent. Findings from the qualitative research suggest that students and parents felt stronger connections with schools (Creative Associates International 2015). The EWS ensured that their parents were aware of their attendance and performance, and engaged in monitoring them through calendars and journals. Schools hosted “Open Houses” as a way of communicating with parents. A voice message system, which delivered recorded messages to parents, kept them informed of school events and provide tips on supporting their child in school. This part of the SDPP Program could have induced at- risk students to attend school more than they would have in the absence of the program, leading to the positive impact on attendance for this group of students. In considering pathways by which these positive impacts emerged, it should be kept in mind that the implementation analysis found that the EWS was implemented as intended, although the evaluation of program effectiveness in influencing teacher behavior did not detect a statistically significant impact on dropout prevention practices. This could be due to the important role played by the community champions in the EWS activities.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 73

There are several factors that might explain why SDPP was not successful in reducing on dropout in India. Each of these factors is discussed in more detail below.

Duration of exposure. Although SDPP did not reduce dropout, it was successful in improving some key intermediate outcomes, namely emotional attitudes toward school and attendance. Because SDPP focused on a single grade, this allowed for maximum exposure of only one year for participating students. In addition, from an operation standpoint, exposure time was limited by interruptions in the school calendar, including unscheduled schools closures due to excess heat in India. It should be noted that the student attitudes towards school and behaviors patterns are formed over time, starting with initial enrollment. Introducing SDPP interventions--which combine targeted adult attention and support, a welcoming school environment and activities that build student engagement—at the start of a student’s academic career, as well as continuing throughout their schooling, could help create positive perceptions of school and better behaviors. That the program had its intended effect on intermediate outcomes suggests that it could have its intended effect on longer term outcomes with longer exposure to the intervention. This evaluation examined outcomes after one year or less of exposure to the intervention for three cohorts; the original project period did not allow for measuring outcomes across multiple years of exposure and the intervention and evaluation designs reflected this limitation. Future research should assess whether SDPP’s positive impacts on intermediate outcomes translate into impacts on dropout with extended exposure to the intervention (for instance, if the intervention started prior to and continued through 5th grade).

Enforcement of compulsory education through age 14. The potential for impacts on dropout to be observed was likely affected by concurrent changes in the education sector. The Right of Children to Free and Compulsory Education (RTE) Act of 2009 stated that children aged 6-14 should not be held back or expelled until they completed primary school. Many schools interpreted this to mean that no student’s name was to be removed from its registers, which would thus affect measurement of dropout. Furthermore, the operationalization of RTE was accompanied by a series of subsidy and support programs--such as provision of uniforms and books, school lunches and remedial lessons--mandated for all primary schools, albeit unevenly implemented. This caused an increase in the numbers of “ghost” students, i.e. students who remained on the school register, but did not attend the school. Schools were motivated to keep their student rolls inaccurately high in order to collect per student subsidies; parents, who had sent their children to private schools or essentially withdrawn them from school, registered them to collect some of the subsidies. This also affected measurement of dropout. Bihar state is the only state in India that defines dropout officially; this clear articulation of the definition of dropout occurred shortly after SDPP began work in Bihar, suggesting that a focus on dropout could have been affected by SDPP’s presence there. Based on the rich student-level information collected as part of this evaluation, SDPP found that dropout was a smaller problem in study schools (8.3 percent) than had been expected based on aggregate administrative data—collected prior to the rollout of RTE--when the program targeted the geographical areas in which it would work (26.8 percent in the Bihar state and 33.5 percent in Samastipur district). (Shin, Jennifer, Rajani Shrestha, and Karen Tietjen 2011a). These reductions in the dropout rate may have made it more difficult for SDPP to further reduce dropout, and contributed to the challenges of measurement.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 74

Complexity of factors related to dropout. Given the complex pathways and inputs that lead to school dropout, it is possible that the impacts on the intermediate outcomes SDPP services can affect are not sufficient to influence the dropout decision for all students. In particular, economic factors that SDPP was contractually prohibited from addressing directly—such as school-related expenses, and the need to supplement income through household chores or domestic work—were found to be particularly relevant in decisions to drop out of school (Creative Associates International 2014).

C. Conclusion

This study shows that the SDPP Program in India was successful in improving some key intermediate outcomes—particularly emotional attitudes toward and daily attendance—but not in reducing dropout. It is useful to compare the impacts of SDPP in India to other countries where SDPP was able to target students in contexts where dropout was much higher, and where students of different ages and grades were exposed to the program for a longer period of time. Additional discussion of the impacts of the SDPP Program across all SDPP countries is presented in a separate, four-country summary report (Creative Associates International and Mathematica Policy Research 2015).

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 75

References

American Institutes of Research. “ALAS: Intervention Report.” Retrieved from http://ies.ed.gov/ncee/wwc/interventionreport.aspx?sid=22, 2006a. American Institutes of Research. “Check & Connect: Intervention Report.” Retrieved from http://ies.ed.gov/ncee/wwc/interventionreport.aspx?sid=78, 2006b. Angrist, Joshua, Eric Bettinger, Erik Bloom, Elizabeth King, and Michael Kremer. “Vouchers for private schooling in Colombia: Evidence from a randomized natural experiment.” The American Economic Review, vol. 92, no. 5, 2002, pp. 1535-1558. Banerjee, Abhijit, Shawn Cole, Esther Duflo, and Leigh Linden. “Remedying Education: Evidence from Two Randomized Experiments in India.” Quarterly Journal of Economics, vol. 122, no. 3, 2007, pp. 1235–1264. Benjamini, Yoav and Yosef Hochberg. “Controlling the false discovery rate: A practical and powerful approach to multiple testing.” Journal of the Royal Statistical Society Series B Methodological, vol. 57, no. 1, 1995, pp. 289–300. Bloom, Howard S. “Accounting for No-Shows in Experimental Evaluation Designs.” Evaluation Review, vol. 8, 1984, pp. 225–246. Brush, Lorie, Jennifer Shin, Rajani Shrestha, and Karen Tietjen. “School Dropout Prevention Pilot Program Review of the Literature.” Report submitted to the U.S. Agency for International Development. Washington, DC: Creative Associates International, Inc., 2011. Creative Associates International. “School Dropout Prevention Pilot Program; India Enrichment Program: School Manual.” Report submitted to the U.S. Agency for International Development. Washington, DC: Creative Associates International, Inc., 2012a. Creative Associates International. “School Dropout Prevention Pilot Program; India Early Warning System: School Manual (Phase 1).” Report submitted to the U.S. Agency for International Development. Washington, DC: Creative Associates International, Inc. 2012b. Creative Associates International. “Decision Rules for the Cut Point for At-Risk Students.” Memorandum to SDPP Local Country Offices, 2012c. Creative Associates International. “India Situational Analysis: Factors and Conditions that Affect Dropout.” Report submitted to the U.S. Agency for International Development. Washington, DC: Creative Associates International, Inc., 2014a. Creative Associates International. “School Dropout Prevention Pilot Program; Findings From the Situational Analysis: Factors and Conditions that Affect Dropout in Cambodia, India, Tajikistan, and Timor-Leste.” Report submitted to the U.S. Agency for International Development. Washington, DC: Creative Associates International, Inc., 2014b.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 76

Creative Associates International. “What Works: Beneficiary Perspectives on dropout Prevention Interventions in India” Report submitted to the U.S. Agency for International Development. Washington, DC: Creative Associates International, Inc., 2015. Creative Associates International and Mathematica Policy Research. “Findings from the Four-Country School Dropout Prevention Pilot Program Impact Evaluation.” Report submitted to the U.S. Agency for International Development. Washington, DC: Creative Associates International, Inc., 2015.

Creative Associates International and School-to-School International. “School Dropout Prevention Pilot Program: Summary of Findings, Fidelity of Implementation (FOI) of SDPP Interventions in Cambodia, India, Tajikistan and Timor-Leste.” Report submitted to the U.S. Agency for International Development. Washington, DC: Creative Associates International, Inc., 2015. Duflo, Esther. "Schooling and Labor Market Consequences of School Construction in Indonesia: Evidence from an Unusual Policy Experiment." American Economic Review, vol. 91, no. 4, 2001, pp. 795-813. Finlay, Krystina A. “Quantifying School Engagement: Research Report.” National Center for School Engagement, 2006. Fredericks, Jennifer A., Phyllis Blumenfeld, Jeanne Friedel, and Alison Paris. “School engagement.” In What do children need to flourish?: Conceptualizing and measuring indicators of positive development, edited by Kristin Anderson Moore and Laura H. Lippman. New York, NY: Springer Science and Business Media, 2005. Friedman, Willa, Michael Kremer, Edward Miguel, and Rebecca Thornton. “Education as Liberation?” No. w16939. National Bureau of Economic Research, 2011. Levy, Dan, Matt Sloan, Leigh Linden, and Harounan Kazianga. “Impact evaluation of Burkina Faso’s BRIGHT program. Final Report.” Washington, DC: Mathematica Policy Research, Inc. submitted to Millennium Challenge Corporation, 2009. Miguel, Edward, and Michael Kremer. “Worms: Identifying impacts on education and health in the presence of treatment externalities.” Econometrica, vol. 72, no. 1, 2004, pp. 159-217. The Millennium Development Goals Report: 2013. New York, New York: United Nations, 2013. Murray, Nancy, Quinn Moore, Larissa Campuzano, Kathy Buek, Emilie Bagby, and Mark Strayer. “School Dropout Prevention Pilot Program: Impact Evaluation Research Plan.” Report submitted to the U.S. Agency for International Development. Washington, DC: Creative Associates International, Inc., 2012. Oster, Emily, and Rebecca Thornton. “Menstruation, sanitary products, and school attendance: Evidence from a randomized evaluation.” American Economic Journal: Applied Economics, 2011, pp. 91-100.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 77

Petrosino, Anthony, Claire Morgan, Trevor Fronius, Emily Tanner-Smith, and Robert Boruch. “Interventions in Developing Nations for Improving Primary and Secondary School Enrollment of Children: A Systematic Review.” Campbell Systematic Reviews, vol. 8, no. 19, 2012. Schochet, Peter Z. “An Approach for Addressing the Multiple Testing Problem in Social Policy Impact Evaluations.” Evaluation Review, vol. 33, no. 6, 2009. Schochet, Peter Z., and John Burghardt. “Using Propensity Scoring to Estimate Program-Related Subgroup Impacts in Experimental Program Evaluations.” Evaluation Review, vol. 31, no. 2, 2007, pp. 95-120. Schultz, T. Paul. “School subsidies for the poor: Evaluating a Mexican strategy for reducing poverty.” Washington, DC: International Food Policy Research Institute, 2001. Shin, Jennifer, Rajani Shrestha, and Karen Tietjen. “School Dropout Prevention Pilot Program Dropout Trend Analysis: India.” Final report submitted to the U.S. Agency for International Development. Washington, DC: Creative Associates International, Inc., 2011a. Shin, Jennifer, Rajani Shrestha, and Karen Tietjen. “School Dropout Prevention Pilot Program Dropout Trend Analysis: Timor-Leste.” Final report submitted to the U.S. Agency for International Development. Washington, DC: Creative Associates International, Inc., 2011b. Shrestha, Rajani, Jennifer Shin, and Karen Tietjen. “School Dropout Prevention Pilot Program Dropout Trend Analysis: Cambodia.” Final report submitted to the U.S. Agency for International Development. Washington, DC: Creative Associates International, Inc., 2011a. Shrestha, Rajani, Jennifer Shin, and Karen Tietjen “School Dropout Prevention Pilot Program Dropout Trend Analysis: Tajikistan.” Final report submitted to the U.S. Agency for International Development. Washington, DC: Creative Associates International, Inc., 2011b. Tschannen-Moran, Megan and Anita Woolfolk Hoy. “Teacher Efficacy: Capturing an Elusive Construct.” Teaching and Teacher Education, vol. 17, pp. 783-805, 2001. Tsigilis, Nikolaos, Athanasios Koustelios, and Vasilios Grammatikopoulos. “Psychometric Properties of the Teachers’ Sense of Efficacy Scale Within the Greek Educational Context.” Journal of Psychoeducational Assessment, vol. 28, no. 2, 2010, pp. 153-162.

Findings from the School Dropout Prevention Pilot Program Impact Evaluation in India Page 78