Evidence from a Conditional Cash Transfer Program Jed
Total Page：16
File Type：pdf, Size：1020Kb
The determinants of program effectiveness: evidence from a conditional cash transfer program Jed Friedman, Eeshani Kandpal, Margaret Triyana and Aleksandra Posarac Abstract This study extends methodology developed to extend results from one RCT site to another to instead study the determinants of heterogeneous treatment effects of a household conditional cash transfer program in the Philippines. Specifically, we extend the covariate balancing technique proposed in Hotz, Imbens, Mortimer (2005) to explore the sources of residual heterogeneity—variation in program impact that cannot be explained by differences in population and location characteristics. The paper also provides a conceptual framework that links systematic differences in implementation quality and community characteristics to intervention effectiveness. In the case of the Philippines, we first document substantial implementation unit-level variation in program impact on stunting. Next, we show that implementation quality, remoteness, and program saturation were correlated with program impact to varying degrees. However, residual heterogeneity was most strongly associated with variables that likely capture implementation quality, suggesting the scope to further strengthen the impact of the program. 1 Introduction Household conditional cash transfer (CCT) programs have been widely implemented to provide additional resources to poor households, conditional on investing in children’s education and health as well as obtaining maternal and child health services. These programs have been successful in improving the targeted health seeking behavior, but the results on health outcomes are mixed (Fizbein et al 2009). More importantly, treatment heterogeneity—the degree to which an intervention has differential causal effects on the relevant treated unit—is commonly found in CCT programs. By itself, this heterogeneity is not necessarily surprising or even cause for further inquiry: many types of programs have varying impacts that depend on the characteristics of the sub-populations. Even though it may be the case that some groups experience greater gains than others, this may be due, for example, to differing baseline levels in the outcome of interest or differences in preferences for human capital investment across groups. As such, impact heterogeneity is not necessarily a result of factors within the control of the program such as the quality of local implementation although implementation issues can also be important for local program effectiveness. This study seeks to link program effectiveness and residual heterogeneity—variations that cannot be explained by variations in population and location characteristics. Identifying the determinants of residual heterogeneity will allow us to understand the mechanisms that support program implementation. This study uses data from the Pantawid Pamilyang Pilipino Program. The program is a household CCT program that was launched in 2008 in the Philippines. The program seeks to promote human capital investments to break the intergenerational transmission of poverty by keeping children in school, improving children’s health, and investing in children’s future. Eligible poor households were identified using a proxy means test. Those with children under 14 2 years of age and/or pregnant woman would then receive cash transfers every two months. The amount ranges from PhP 500 (USD 10) to PhP 1,400 (USD 29) per household per month, depending on the number of eligible children. The program has been evaluated using two methods: Randomized Control Trial (RCT) and Regression Discontinuity Design (RDD). The impact evaluation found that the program reached most of its key objectives (Onishi et al 2016, Kandpal et al 2016). The Pantawid CCT program increased children’s enrollment and attendance of school, as well as reduced severe stunting among 0-5 year old children in the RCT areas of study. While these impacts suggest that, on average, the program reached significant achievements, an important component of policy learning is to understand where the program was most effective and to explore possible determinants for this success. More importantly, the impact evaluation finds considerable treatment heterogeneity. One possible cause of such heterogeneity may simply be that impacts tend to be higher when baseline levels themselves are lower and there is more “room for improvement”, presumably due to the lower marginal cost to improve these low indicators. This is an example of a cause ascribed to characteristics of the population and not to the local implementation quality of the program. While such exploratory analysis highlights the importance of examining treatment heterogeneity, even for a successful intervention like Pantawid, it tells us little about the causes of such heterogeneity. Population characteristics can determine a large proportion of this heterogeneity, which, while informative, does not provide policy-actionable information relating implementation quality with program impact. Therefore, the rest of this study examines the correlates of residual heterogeneity in Pantawid impact after accounting for the effect of population characteristics. 3 This study uses recently developed empirical tools to explore spatial variations in program impact. In particular, we measure the residual heterogeneity on two outcomes: food consumption and stunting. Residual heterogeneity is identified through matched covariate balancing. Once identified, the residual heterogeneity can be related to program data on implementation quality as well as fixed features of localities, such as remoteness, to identify the correlates of heterogeneity in program impact. This associative analysis constitutes the first step towards identifying the drivers of such divergence in program effectiveness. The initial analysis shows that differences in implementation quality, remoteness, and program saturation were all correlated with program impact to varying degrees. However, residual heterogeneity was most strongly associated with variables that likely capture implementation quality, suggesting the scope to further strengthen the program impact on stunting. Data and method Data Data from the Pantawid impact evaluation survey were used in the analysis. We use the RDD method across all provinces for comparability. Table 1 illustrates the variability in estimated impacts across municipalities. As an example of the findings, the impact of program participation on household food consumption ranges from 0.9 percent (0.009 log units) in Sibanga (Agusan Del Sur) to almost 100 percent (0.990 log units) in San Luis (also in Agusan Del Sur) with a median impact across all municipalities of 20 percent (0.202 log units). The variability across municipalities in another key targeted outcome—stunting—is similarly large. The ratio between the largest and smallest impact is largest for household food consumption (over 100). This site specific variability is due to both differential impact across sites as well as 4 sampling variability. For each outcome there are municipalities where the impact is statistically significant and where it is not (statistically precise impacts highlighted in yellow). Table 1: Cross-Municipality Heterogeneity in Program Impacts for Selected Outcomes1 Food Province Municipality cons.2 Stunting3 MEDIAN 0.202 -0.062 Overall MAXIMUM 0.990 -0.989 MINIMUM 0.009 0.868 SAN LUIS 0.990 -0.036 ESPERANZA 0.070 -0.398 Agusan del Sur LORETO 0.131 -0.617 SIBAGAT 0.009 0.211 SANTA JOSEFA 0.121 -0.083 CALANASAN 0.958 -0.810 Apayao KABUGAO 0.125 -0.770 CONNER 0.163 0.261 SALVADOR 0.491 0.425 Lanao del Norte LALA 0.245 -0.069 BONIFACIO 0.202 -0.019 PLARIDEL 0.054 -0.964 Misamis TUDELA 0.134 0.711 Occidental SAPANG DALAGA 0.654 0.351 LOPEZ JAENA 0.379 0.868 SADANGA 0.451 -0.989 Mountain Province PARACELIS 0.500 -0.295 BASAY 0.772 0.183 Negros Oriental JIMALALUD 0.780 -0.017 LOPE DE VEGA 0.839 -0.582 SILVINO LOBOS 0.726 0.728 North Samar MONDRAGON 0.016 0.120 SAN ROQUE 0.025 0.048 CATUBIG 0.951 -0.301 Occidental SANTA CRUZ 0.077 -0.078 Mindoro PALUAN 0.089 0.394 KALAWIT 0.061 0.785 Zamboanga Del KATIPUNAN 0.520 -0.673 Norte GUTALAC 0.028 -0.188 1 Statistically significant impacts highlighted in yellow. 2 Log units of consumption of all recorded food items. 3 Defined as the height-for-age Z score of children 6-36 months old as being at least two standard deviations below the international reference mean. 5 MANUKAN 0.064 -0.351 BACUNGAN 0.833 -0.062 Note: For comparability, the table reports estimates using Regression Discontinuity Design methodology in both “RCT provinces” and “RDD provinces.” RCT provinces are Lanao del Norte, Mountain Province, Negros Oriental, Occidental Mindoro. The Proposed Methodology In this study, we use a methodology inspired by Hotz, Imbens, Mortimer (2005) to investigate correlates of the residual heterogeneity in Pantawid impact. Hotz et al. (2005) extrapolated the results of a job-training program estimated in one site to different sites by adjusting the expected treatment impact for differences in the observable characteristics of participants. The authors balance the observed characteristics of the control population in the program area and the target area through a matching estimator. They then estimate a treatment effect for the target area with the matching (propensity) weights that reflect the characteristics of the target population. The key identifying assumption with this method is that unobservable determinants of program performance are