Evaluation of CDWA Balochistan Component P&DD/UNICEF Joint Evaluation 1 Evaluation of CDWA Balochistan Component P&DD/UNICEF Joint Evaluation Photo Credits: Survey team Disclaimer: The views and opinion expressed in this report are those of the consultants and do not necessarily reflect the views or policies of UNICEF and/or other organizations involved in the programme. 2 Evaluation of CDWA Balochistan Component P&DD/UNICEF Joint Evaluation TABLE OF CONTENTS Title Page no. ACKNOWLEDGEMENTS 6 EXECUTIVE SUMMARY 7 CHAPTER 1: CONTEXT AND BACKGROUND OF THE EVALUATION 15 1.1 Background to Clean Drinking Water in Balochistan 15 1.2 Object of the Evaluation: Clean Drinking Water for All Project 18 CHAPTER 2: EVALUATION PURPOSE, OBJECTIVES AND METHODOLOGY 23 2.1 Purpose and Objectives 23 2.2 Theory of Change 23 2.3 Evaluation Scope 25 2.4 Users of the Evaluation and Associated Dissemination 26 2.5 Evaluation Criteria 27 2.6 Evaluation Framework and Key Questions 27 2.7 Methodology overview 28 2.8 Challenges and Risks 35 CHAPTER 3: FINDINGS 38 3.1 Overview of Findings against the Hypothesis and Key Evaluation 38 Questions 3.2 Relevance 41 3.3 Effectiveness 45 3.4 Efficiency 51 3.5 Outcomes 56 3.6 Sustainability 57 3.7 HRBA, Equity and Gender 58 CHAPTER 4: CONCLUSIONS AND RECOMMENDATIONS 60 4.1 Conclusion 60 4.2 Lessons learnt 61 4.3 Recommendations 61 ANNEXURES Appendix 1:ToR Evaluation of the Clean Drinking for All (CDWA) Project in 65 Balochistan Appendix 2:Evaluation ToR Annex-1: List of 409 CDWA Water Filtration 77 Plants Appendix 3: Evaluation Matrix 88 Appendix 4-A: Cross-functional Evaluation Workflow 90 Appendix 4-B: Household Questionnaire 91 Appendix 4-C: Key Informant Interview (KII) 98 Appendix 4-D: Focus Group Discussions (FGDs) 101 Appendix 4-E: Water Filtration Plant Status Data Collection 104 Appendix 4-F: Data Quality Assurance Guidelines 106 Appendix 4-G: Data Validation Checklist 108 Appendix 4-H Quality Assurance Spot check Checklist 109 Appendix 5: Training Plan for Officials of the GoB and Field Supervisors 110 Appendix 6: Pre and Post Test for the Field Team 114 Appendix 7: List of Reference Material Reviewed and Further Required 115 Appendix 8: Plant Condition (Operational and Infrastructure) 116 Appendix 9-A: Attendance of the GoB and Field Staff Training Workshop 118 Appendix 9-B: Commitment Agreement 119 Appendix 9-C: Materials Provided to Training Participants 120 Appendix 9-D: Feedback from Participants of the Training 121 3 Evaluation of CDWA Balochistan Component P&DD/UNICEF Joint Evaluation LIST OF TABLES Table 1: CDWA Intervention Timeline in Balochistan ........................................................................................ 16 Table 2: Assessment of Water Filtration Plants ................................................................................................. 29 Table 3: Number of Filtration Plants by Divisions/Districts and Status (PHED)................................................... 31 Table 4: Sample Size and Its Allocation by Divisions /Sample Districts/ Sample Households ............................. 32 Table 5: Evaluation Limitations and Mitigation Measures ................................................................................. 36 Table 6: Overview of Findings Against the Key Evaluation Questions ................................................................ 38 Table 7: Change in Population Density by District ............................................................................................. 43 Table 8: Summary of the Information Available on the CDWA in Different Provinces ....................................... 44 Table 9: Distribution of Sampled WFPs by Infrastructure Condition and Operational Status ............................. 47 Table 10: Comparison of Costs between TPV and PHED Reports ....................................................................... 53 Table 11: SDG Targets Relevant to PHED .......................................................................................................... 53 Table 12: Legislative and Best Practice Framework Applicable to PHED ............................................................ 55 Table 13: Practice of Cleaning Water Collection Vessels by Type of Vessel Used ............................................... 57 Table 14: List of Key Recommendations and Associated Timelines ................................................................... 62 LIST OF FIGURES Figure 1: Distribution of 409 Functional and Non-Functional WFPs (PHED) ....................................................... 21 Figure 2: Access to Water by Source (PHED) ..................................................................................................... 21 Figure 3: Evaluation Theory of Change .............................................................................................................. 24 Figure 4: Scope of Fieldwork Completed ........................................................................................................... 25 Figure 5: Components of the Integrated Evaluation Approach .......................................................................... 27 Figure 6: Elements of the Rigorous Training of Field Supervisors ...................................................................... 34 Figure 7: Change in Population Density for Selected Districts of Balochistan 1998–2017 .................................. 42 Figure 8: Population Growth between Censuses 1998 and 2017 ....................................................................... 43 Table 9: Distribution of Sampled WFPs by Infrastructure Condition and Operational Status ............................. 47 Figure 10: Examples of Non-Functionality and Reasons for Poor Operations .................................................... 48 Figure 11: Examples of Best Practices ............................................................................................................... 49 Figure 12: Status of Programme Management Elements .................................................................................. 50 Figure 13: Frequency of Vessel Cleaning ........................................................................................................... 56 4 Evaluation of CDWA Balochistan Component P&DD/UNICEF Joint Evaluation ACRONYMS ADP Annual Development Plan BCC Behaviour Change Communication BESA Balochistan Engineering Services Agency C4D Communication for Development CDO Community Development Officer CDWA Clean Drinking Water for All CDWI Clean Drinking Water Initiative DAC Development Assistance Committee DHO District Health Office DRR Disaster Risk Reduction FGD Focus Group Discussion GoB Government of Balochistan GoP Government of Pakistan HHS Household Survey HRBA Human Rights-Based Approach KII Key Informant Interview M&E Monitoring and Evaluation MPA Member Provincial Assembly MOS Measure of Size NECD National Evaluation Capacity Development NGO Non-Governmental Organization O&M Operation and Maintenance OECD Organization for Economic Cooperation and Development P&DD Planning and Development Department PCRWR Pakistan Council of Research in Water Resources PHED Public Health Engineering Department PME Programme Monitoring and Evaluation PMERR Planning, Management, Evaluation, Reporting and Research POL Petroleum, Oil and Lubricants PPIU Provincial Project Implementation Unit PPS Probability Proportional to Size PSLM Pakistan Social and Living Standards Measurement QL Qualitative QT Quantitative RO Reverse Osmosis SDG Sustainable Development Goal TOC Theory of Change ToR Terms of Reference TPV Third-Party Validation UC Union Council UF Ultra-Filtration UNEG United Nations Evaluation Group UNICEF United Nations Children’s Fund WASH Water, Sanitation and Hygiene WFP Water Filtration Plant WHO World Health Organization XEN PHED District Engineer 5 Evaluation of CDWA Balochistan Component P&DD/UNICEF Joint Evaluation ACKNOWLEDGEMENTS On behalf of H&H Consulting and the team of evaluators I would like to acknowledge the support of senior management of Public Health Engineering Department (PHED), particularly Engr Javaid Ahmad, Engr Aga Imran, Mr Eric Sampson and Engr Abdul Qayyum. I also acknowledge the keen personal interest in the findings of the evaluation taken by Zahid Saleem, Secretary, PHED and Shaikh Nawaz Ahmed, Advisor, PHED. Since the evaluation was part of a broader effort to develop evaluation capacities of the Government of Balochistan, it would not have been possible without the support of the Planning and Development Department (P&DD), particularly Mr Mohammad Ali Kakar, Secretary, P&DD Implementation, and Mr Manzoor Ahmed, Director General, Monitoring and Evaluation. We are particularly indebted to community members who gave generously of their time to share information and insights. Key informants, comprising PHED District Engineers (XENs), District Health Officers and representatives of local government, were extremely gracious in sharing valuable insights. The evaluation would not be possible without the continuous support of UNICEF representatives, particularly Ms Mussarrat Youssuf, Mr Omar Durrani and Dr Masood Aslam. Finally, I would also like to express my gratitude to Mr Hussain Tawawalla, the team leader, for bringing additional rigour to the evaluation. I would also like to acknowledge our team members, Mr Barkat Ali, Mr Karim Baloch, Ms Nouma Hanif, Mr Saboor Siddique, Mr Samiuallah Hayat, Mr Zahid Jamal, and the entire field team comprising 46 male and female members