Guidelines for Impact Evaluations in the Water and Sanitation Sector
Christine Poulos, Ph.D. Sr. Economist, RTI International Subhrendu Pattanayak, Ph.D. Sr. Economist and Research Fellow, RTI International Associate Professor, NC State University RTI International June 12, 2007 Motivation
Numerous texts on impact evaluation (IE), in general Few rigorous IEs in the WSS sector The Guide: focuses on specific considerations in evaluating WSH programs and projects; and supplements existing IE guidelines. See Baker [2000], Ravallion [2001,2005] and others for a comprehensive overview of IE, evaluation designs, or statistical methods Roadmap
1. IE and IE in WSS Interpret/apply concepts and practices in context of WSS 2. Illustrative Examples of IEs in WSS 3. WSS IEs in practice What is Impact Evaluation?
IE measures impacts on beneficiaries that are caused by the intervention (programs and policies) Need to construct counterfactual Æ What would have happened to the beneficiaries in the absence of the intervention? Identify control or comparison group similar to beneficiaries but for the intervention Compare comparison group to beneficiaries to measure impact IE Steps
1. Decide to do an IE 2. Describe the intervention need/motivation, context, stage of implementation, inputs, and results 3. Design and implement evaluation The inner workings of a project Æ project-specific issues (Bamberger 2006) Why decide to do an IE?
Use findings to: Add to global evidence base Identify favorable conditions Identify effective components Build support Ensure accountability Why decide to do an IE?
Select candidates that provide opportunities for feasible, informative studies: IE has political and financial support, and/or intervention is well-defined Results generalizable – intervention is scalable, replicable Interventions that are innovative, controversial, resource intensive Describe the Intervention: Objectives of Intervention
WSS interventions are Rural Multi-dimensional and multi-sectoral Rural Development, Private Sector Urban Development, Urban and Local WSS Government, and Environment Business
Env. Goals guiding most WSS projects: Based on World Bank Water and Sanitation Sector Board guidelines 1. efficient access to safe drinking water and/or basic sanitation services; 2. sustainable access to safe drinking water and/or basic sanitation services; and 3. equitable access to safe drinking water and/or basic sanitation services. WSS Interventions
Three types of reform measures: 1. improving operator performance, 2. service provision by the private sector or small-scale independent providers, and 3. decentralized delivery, typically relying on community demand, participation and management. Outputs of sector reform initiatives: Hardware: new or improved WSH infrastructure and services Software: training, education, better provider performance Describe the Intervention: Features and Linkages
Inputs: Resources: financial, institutional, legal and regulatory Activities: what the intervention does Describe the Intervention: Features and Linkages
Inputs: Resources: financial, institutional, legal and regulatory Activities: interventions Results: Outputs: direct product of program activities Outcomes: short-term changes in beneficiaries’ behaviors & knowledge Impacts: long-term changes in beneficiaries’ wellbeing Indicators: direct measure of progress toward goals Also need to understand external influences Describe the Intervention: Rehabilitating UWSS
Resources Activities Outputs Outcomes Impacts
Funding Replacement No. of rehab % of pop. Individual and of pipes, WSS systems with access to household Staff pumps, meters WSS incomes No. of Technical Trainings for connections to % of pop. School Assistance staff rehab WSS served by enrollment and systems rehab WSS attendance Laboratory Water quality services testing Operating Coping costs Prev. of cost of diarrheal systems Lpcd disease consumed Water rehab Under 5 (hrs) Use of rehab mortality WSS WQ tests Design and Implement Evaluation, I
Question: is the intervention effective in increasing efficient, sustainable, and equitable access to improved WSS? Robust IE design:
Time Æ Pre-intervention Intervention Post- (baseline) intervention
Beneficiaries T1 XT2 (treatment group)
Control/compar C1 C2 ison group Design and Implement Evaluation, II
Question: is the intervention effective in increasing efficient, sustainable, and equitable access to improved WSS IEs use baselines, controls/comparisons, and covariates to ensure causal effects can be identified No baseline, no control – how measure change? what to compare with (i.e., counterfactual)? Baseline, no control – what to compare with (i.e., counterfactual that captures trends and history)? sufficiently account for selection bias? No baseline, control – how to sweep out pre-existing differences (behaviors, rates, trends)? sufficiently account for selection bias? All the above, but no covariates – are you sure nothing else matters? No other factors effect program selection and or modify or mediate treatment? Design and Implement Evaluation, III
Design determines where controls/comparisons come from Randomized trials Quasi-experiments – longitudinal or cross- sectional natural experiments matching (propensity score, covariate, pipeline) Analysis Brief Randomized Trial Example
Design – treatment assigned randomly (not purposive, strategic or selective) so that confounders (alternative causes) are balanced across treatment and control group Example: Do information treatments change hygiene behavior in Delhi, India? [Jalan & Somanathan, 2004] Tell 500 households (out of 1000) about quality of their drinking water, and check after 1 year if they change hygiene behavior Similar in education, health literacy and hygiene behaviors Informed household 11% more likely to purify water Challenges: difficult to control, ethical concerns, political issues, and limited external validity Brief Quasi-Experimental Example
Design: match treatments to controls based on observable factors Example: Does Jalswarajya – a public, community-driven rural WSH program in Maharashtra, India – improve access to improved WSH and improve children’s health outcomes? ~250 villages (2 propensity score matched controls for each treatment) Baseline and post-intervention data collection from ~10,000 households Household and community surveys Challenges: need lots of data, assumes unobservables are uncorrelated with exposure to intervention Detailed Example: Randomized Trial of TSC’s IEC in Orissa, India
N
Map Showing Bhadrak District W E
S 10 0 10 Km.
OR ISSA LEG END: Drainage RO AD NA TIONA L HIG HW AY MAJOR ROAD OTHER RO AD
River BLOCK BOUNDARY
ORISSA
SUNDARGARH
JHARSUGUDA MAYURBHANJ
DEOGARH KEONJHAR BA LA S O RE SA M B A LP U R BARAGARH
BH A D RA K SO N P UR ANUGUL JA JP UR DHENKANAL BO U D
NU A P AD A BOLANGIR KE N D RA P A DA CU T TA C K
NA Y A G AR H KHURDHA JAGATSINGHPUR PHULBANI
PU R I KALAHANDI
GA N J AM NABARANGAPUR RAYGADA
GA J AP A TI
KORAPUT
MALKANGIRI
BHADRAK Y# Y# Y# Y# Y# %[Y# # %[ Y# %[Y %[ Y# Y# %[ BONTH Y#%[ Y# Y# Y# Y# Y# Y# %[ %[ Y# Y# Y# Y# BHADR AK %[ %[ %[ Y# Y# Y# Y# %[ %[ %[ # Y# Y %[ %[ Y# Y# %[ %[ Y# %[ %[ %[ %[ Y# Y# %[ %[ Y# %[ Y# %[ Y# %[ %[ Y#Y# Y# BASUDEVPUR %[ Y# Y# Y# %[ %[ Y# Y# %[ %[ %[ Y# %[ %[ TIHIDI Y# Y#Y# Y# BHAND ARIPOKHAR I %[ %[ Y# %[%[ %[Y# Y# Y# %[ Y# Y# Y#%[ # %[ Y Y# DHAMN AGAR Y# %[ Y# %[ Y# Y# #%[ Y# %[ Y#Y# Y Y# CHAND ABALI%[ %[ %[ Y# Y# Y# Y# Y# %[ Y# Y# Y# %[ Y# Y# Y# Y# %[ %[ # Y# Y# Y# Y# %[ Y Y# %[ Y# Randomized Trial of TSC’s IEC in Orissa, India: Describe Intervention, I
Government of India’s Total Sanitation Campaign Goal: increase use of IHLs Backdrop: inadequate services, high child mortality, and MDGs “Community-Led Total Sanitation” Intensive IEC Change knowledge Change attitudes Community demand and plan Small subsidies to the poor; know-how and material to all Implement via local NGOs Randomized Trial of TSC’s IEC in Orissa, India: Describe Intervention, II
Resources Activities Outputs Outcomes Impacts Funding: Bank, IEC: walk of No. of focus % of Prevalence of GoI, GoO, shame, fecal groups and households diarrheal community calculation, completed IECs owing IHL disease, by age contributions defecation and gender mapping No. of % of pop. using Staff: GoO, community IHL, by gender Norms NGOs, Distribute agreements and age regarding OD community subsidies monitors Total subsidies Coping costs- Individual and Training in IHL time household Supply Chain construction No. of incomes households Awareness of Technical Establishment trained in environment- School assistance and/or stocking construction health link enrollment and of rural attendance Enabling sanitation marts No. of rural programming - sanitation marts TSC Randomized Trial of TSC’s IEC in Orissa, India: Design and Implement IE, I
Research Questions: Does the TSC’s IEC cause increases access to and use of IHL? decreases in child morbidity due to diarrheal disease? Features: Controls: Leverage phasing to randomly assign IEC to 20 out of 40 communities in 1st phase Baseline: pre-intervention surveys Covariates: household and village surveys Indicators: selection based on literature, pretests Sample: ~1000 households Randomized Trial of TSC’s IEC in Orissa, India: Design and Implement IE, II
IHL Ownership and U5 Diarrha Rate Use 35 40 30 25 30 20 % 2005 2005 % 20 15 2006 2006 10 10 5 0 0 Treatment Control Treatment Control Randomized Trial of TSC’s IEC in Orissa, India: Design and Implement IE, III
Medical Costs (Rs) Time Spent Walking (min) 400
300 30 200 2005 Rs. 20 2006 Min. 2005 100 10 2006
0 0 Treatment Control Treatment Control Detailed Example: Quasi- Experimental Study Galiani, Gertler, and Schargrodsky (2005): Describe Intervention, I
Intervention: Privatization of local water companies in Argentina In the 1990s, about 30% of the country’s municipalities covering almost 60% of the country’s population were privatized. The remaining municipalities continued receiving water services from either public companies or nonprofit cooperatives. Research question: While efficiency gains have been demonstrated, does privatization of water supply improve health outcomes and alleviate poverty? Galiani, Gertler, and Schargrodsky (2005): Describe Intervention, II
Resources Activities Outputs Outcomes Impacts Enabling PSP process Efficiency in Water and Under five regulation and contracts billing and sewerage mortality rate, collection connections , by Private sector New systems by municipality- resources for billing, Improved municipality- level poverty accounting, service level poverty rate Access to maintenance credit System rate† Installation or performance New replacement of (e.g., hours of incentives pipes, pumps, service) meters Training Galiani, Gertler, and Schargrodsky (2005): Design IE, I
Research Questions: Does privatization of water supply improve health outcomes? Features: Comparisons: Propensity score matching of municipalities Baselines: secondary data Covariates: secondary data on socioeconomic and political characteristics of municipalities Indicators: U5 mortality rate, water and sewerage connection rate, private ownership of water company serving majority of municipalities' population Sample: ~4000 municipalities Galiani, Gertler, and Schargrodsky (2005): Key Findings
Analysis: Difference-in-difference estimation with PSM Findings: Child mortality fell by 8% Effect was largest in poorest areas (26%) Connections to the water network increased by 4.2% Challenges
Short project cycles 1. Measuring sustainability within the project cycle “no less than 3-5 years are required for an intervention to show an impact.” (Habicht et al. 1999) Use program theory (White 2005) 2. Turnover, impatience, fatigue 3. Devarajan and Kanbur (2004) “In short, we probably need to scale up something that is in short supply, namely, patience.” Constraints – Bamberger (2006) Budget Time Data Bamberger discusses the acceptability of compromises Strengthen Overall Quality
Allow sufficient time to meet with clients and key stakeholders Build political and institutional support early and often – counteract turnover, fatigue Convince them to stay the course Get off to a quick start If possible, change program design to allow for less expensive IE methods If possible, make mid-course corrections Develop program theory/logic model Consensus on how it work, identify where it breaks down, determine what to measure Alternative data collection methods Conclusions
Call for increase in use of rigorous IE methods Address project-specific issues in selection of study design and features Baseline, controls/comparison, and covariates are key Endline is equally important! Engage and encourage stakeholders to stay the course The End
Contact information: [email protected] [email protected]