USAID Pakistan EVALUATION Final Performance Evaluation USAID Maternal And Child Health Project Evaluation Report June 13, 2012 This publication was produced for review by the United States Agency for International Development (USAID). It was prepared under the Russia Monitoring & Evaluation Project (RMEP) by International Business &Technical Consultants, Inc. (IBTCI) |RUSSIA FINAL PERFORMANCE EVALUATION USAID MATERNAL AND CHILD HEALTH PROJECT EVALUATION REPORT June 13, 2012 This publication was produced for review by the United States Agency for International Development (USAID). It was prepared under the Russia Monitoring & Evaluation Project (RMEP) by International Business &Technical Consultants, Inc. (IBTCI) MATERNAL AND CHILD HEALTH INITIATIVE FINAL EVALUATION REPORT JUNE 13, 2012 International Business & Technical Consultants, Inc. 8618 Westwood Center Drive Suite 220 Vienna, VA 22182 USA Contracted under RAN-I-00-09-00016-00, Order No. AID-118-TO-11-00004 Russia Monitoring and Evaluation Project Evaluation Team: Alexey Kuzmin (team leader) Annette Bongiovanni Natalia Kosheleva DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Russia Monitoring and Evaluation Project – MCH Evaluation Draft Report Table of Contents Boxes, Figures and Tables ....................................................................................................................... a Acronyms and Abbreviations................................................................................................................... b EXECUTIVE SUMMARY ...................................................................................................................... i 1. INTRODUCTION ........................................................................................................................... 1 1.1. Description of the Project Evaluated ......................................................................................... 1 1.2. Purpose of the Evaluation ......................................................................................................... 1 1.3. Evaluation Questions................................................................................................................ 2 1.4. Collaboration with the local partners and stakeholders .............................................................. 2 1.5. Methodology ............................................................................................................................ 2 2. MATERNAL CHILD HEALTH PROJECT DESIGN ...................................................................... 7 2.1. Clarification of terminology ..................................................................................................... 7 2.2. USAID’s Program, project and activity results frameworks ...................................................... 7 2.3. MCHP Activity models .......................................................................................................... 10 2.4. Reconstruction of activities’ logic. ......................................................................................... 13 2.5. Implementation Models .......................................................................................................... 14 3. PROJECT FINDINGS: GOALS AND DELIVERABLES .............................................................. 18 3.1. Focus of MCHP Interventions ................................................................................................ 18 3.2. Causality of the observed results ............................................................................................ 18 3.3. Results of “Improving Care for Mothers and Babies” Activity ................................................ 20 3.4. Results of “Institutionalizing Best Practices in Maternal and Child Health” Activity ............... 26 4. SUSTAINABILITY ...................................................................................................................... 32 4.1. Two aspects of sustainability .................................................................................................. 32 4.2. Introduction of Quality Improvement methodology ................................................................ 33 4.3. Institute for Family Health as a sustainability factor ............................................................... 34 5. LESSONS LEARNED................................................................................................................... 36 5.1. Program elements that worked particularly well ..................................................................... 36 5.2. Program elements that did not work particularly well and opportunities for improvement ....... 37 5.3. Unintended Consequences ...................................................................................................... 39 5.4. MCHP and Vulnerable Groups ............................................................................................... 41 5.5. MCHP and Civil Society ........................................................................................................ 42 5.6. MCHP and Good Governance ................................................................................................ 45 6. MAIN CONCLUSIONS AND RECOMMENDATIONS ............................................................... 47 6.1. Main conclusions ................................................................................................................... 47 6.2. Main recommendations .......................................................................................................... 49 Annex 1. Evaluation Scope of Work (Notification of Performance Evaluation) ...................................... 50 Annex 2. Evaluation Design and Work plan ........................................................................................... 57 Annex 3. List of persons consulted ........................................................................................................ 69 Annex 4. List of documents studied ....................................................................................................... 74 Annex 5. Bibliography of significant publications used to consult .......................................................... 75 Annex 6. Glossary ................................................................................................................................. 76 Annex 7. Participation in Quality Improvement Collaboratives (URC) ................................................... 77 Annex 8. New technical focus on Chronic Disease Prevention and Treatment ........................................ 79 Annex 9. Performance indicators: reach, dose and fidelity ..................................................................... 80 Annex 10. What is a think tank? ............................................................................................................ 81 Annex 11. Disclosure of Conflict of Interest .......................................................................................... 82 Russia Monitoring and Evaluation Project – MCHP Evaluation Report a Boxes, Figures and Tables Boxes page Box 1. MCHI Development Hypothesis 14 Figures Figure 1. USAID Program Results Framework 8 Figure 2. Logical connections between activity, project and program 10 Figure 3. ICMB activity model 11 Figure 4. IBMCH activity model 12 Figure 5. Activities’ logic model (reconstructed) 13 Figure 6. URC Collaborative Model for Improvement 15 Figure 7. IFH/JSI Model for Introducing Evidence-Based Best Practices 17 Figure 8. The lifecycle continuum of care and MCHI interventions 18 Figure 9. Changes in infant mortality (%) from baseline during the period of project 22 implementation (2009-2011) in target and comparison regions Figure 10. MCHI as an information channel: the bottle neck 33 Tables Table 1. ICMB Project key tasks and actual results 20 Table 2. Neonatal mortality at the Perinatal Center of the Kostroma Oblast Hospital in 2009-2011 23 Table 3. Project reach in Kostroma, Yaroslavl and Tambov regions. 24 Table 4. Maternal mortality rate in target regions in 2007 – 2010 25 Table 5. Number of abortions per 100,000 births in target and comparison oblasts in 2007 – 2010 26 Table 6. IBPMCH Project key tasks and actual results 27 Table 7. Infant mortality nationally and in the project regions 28 Table 8. Infant mortality rates in target oblasts in 2007 – 2010 28 Table 9. Maternal mortality nationally and in the project regions 29 Table 10. Maternal mortality rates in target oblasts in 2007 – 2010 29 Table 11. Fidelity to essential neonatal practices at selected medical facilities in target oblasts 30 Table 12. Activity reach to maternity hospitals in selected target oblasts 30 Table 13. Activity reach in the target Federal Districts 30 Table 14. Good governance and MCHI 46 Russia Monitoring and Evaluation Project – MCHP Evaluation Report b Acronyms and Abbreviations AAP American Academy of Pediatricians ACOG American College of Obstetricians and Gynecologists CPR contraceptive prevalence rate CVD cardiovascular disease ENMR early neonatal mortality rate ETOH ethyl alcohol FL facilitative leadership FP family planning GOR Government of Russia HCI health care improvement IBPMCH Institutionalizing Best Practices in Maternal and Child Health (Activity 2) IBTCI International Business