FINAL EVALUATION OF GAVI ALLIANCE’S SUPPORT TO BOSNIA AND HERZEGOVINA FINAL EVALUATION OF GAVI ALLIANCE’S SUPPORT TO BOSNIA AND HERZEGOVINA FINAL EVALUATION REPORT July 31, 2014 FINAL EVALUATION OF GAVI ALLIANCE’S SUPPORT TO BOSNIA AND HERZEGOVINA TABLE OF CONTENT ABBREVIATIONS iii EXECUTIVE SUMMARY ........................................................................................................................ i CHAPTER 1: INTRODUCTION ................................................................................................... 12 1.1 COUNTRY BACKGROUND .......................................................................................................................... 12 1.2 KEY ISSUES IN HEALTH SECTOR .............................................................................................................. 2 1.3 GAVI SUPPORT TO BOSNIA & HERZEGOVINA .................................................................................... 5 CHAPTER 2: EVALUATION PURPOSE, OBJECTIVE AND METHODOLOGY ..................................... 8 2.1 RATIONALE ........................................................................................................................................................ 8 2.2 EVALUATION OBJECTIVES .......................................................................................................................... 8 2.3 EVALUATION SCOPE, GEOGRAPHICAL COVERAGE & PHASES .................................................. 8 2.4 EVALUATION FRAMEWORK ....................................................................................................................... 9 2.5 EVALUATION METHODS ........................................................................................................................... 12 2.6 QUALITY ASSURANCE ................................................................................................................................ 13 2.7 DATA COLLECTION ...................................................................................................................................... 14 2.7 ETHICAL CONSIDERATIONS .................................................................................................................... 15 2.8 DATA ANALYSIS ............................................................................................................................................ 16 2.9 EVALUATION LIMITATIONS .................................................................................................................... 17 CHAPTER 3: EVALUATION FINDINGS ....................................................................................... 18 3.1 RELEVANCE AND EFFECTIVENESS OF PLANNING ....................................................................... 18 3.2 EFFECTIVENESS AND EFFICIENCY OF IMPLEMENTATION ..................................................... 24 3.3 OUTCOMES AND CONTRIBUTION TO SUSTAINABILITY ............................................................ 36 3.4 POSITIVE AND NEGATIVE CONSEQUENCIES OF GAVI SUPPORT........................................... 46 CHAPTER 4: CONCLUSIONS, DISCUSSIONS AND LESSONS LEARNED ........................................ 49 4.1 RELEVANCE AND EFFECTIVENESS OF PLANNING – PRE GAVI SUPPORT ........................ 49 4.2 EFFECTIVENESS AND EFFICIENCY OF IMPLEMENTATION – DURING GAVI SUPPORT 50 4.3 OUTCOMES AND RESULTS OF GAVI SUPPORT – AFTER GAVI SUPPORT ........................... 53 CHAPTER 5: LESSONS LEARNED ............................................................................................... 56 CHAPTER 6: RECOMMENDATIONS .......................................................................................... 58 6.1 COUNTRY SPECIFIC RECOMMENDATIONS ...................................................................................... 58 6.2 RECOMMENDATIONS TO GAVI .............................................................................................................. 60 ANNEXES 64 ANNEX 1: LIST OF DOCUMENTS REVIEWED ................................................................................................... 64 ANNEX 2: LIST OF PEOPLE MET ............................................................................................................................ 66 ANNEX 3: IMMUNIZATION SCHEDULES ............................................................................................................ 69 ANNEX 4: EVALUATION FRAMEWORK .............................................................................................................. 70 ANNEX 6: INTERVIEW GUIDE................................................................................................................................. 86 ANNEX 7: RFP/TOR ..................................................................................................................................................... 95 i FINAL EVALUATION OF GAVI ALLIANCE’S SUPPORT TO BOSNIA AND HERZEGOVINA Map of Bosnia and Herzegovina1 1 The map presented in this document does not imply the expression of any opinion whatsoever on the part of GAVI and CIF or its partners concerning the legal status of any country, territory, city or area or of its authorities or concerning the delimitations of its frontiers or boundaries. ii FINAL EVALUATION OF GAVI ALLIANCE’S SUPPORT TO BOSNIA AND HERZEGOVINA ABBREVIATIONS AD Auto-Disable AEFI Adverse Events Following Immunization APR Annual Progress Report AVC Anti Vaccine Campaign BCG The bacille Calmette-Guérin vaccine BD Brcko District BiH Bosnia and Herzegovina CDC Centers for Disease Control/Atlanta cMYP Comprehensive Multi-Year Plan OECD/DAC Organization for Economic Cooperation and Development/Development Assistance Committee DTP Diphtheria, Tetanus, and Pertussis vaccine DTPa-IPV-Hib Combined diphtheria-tetanus-acellular pertussis-inactivated poliovirus and Haemophilus influenzae type b vaccine EPI Expanded Program on Immunization ET Evaluation Team EU European Union FBiH Federation of Bosnia and Herzegovina FGD Focus Group Discussion FSP Financial Sustainability Plan GAVI Global Alliance for Vaccines and Immunization GDP Gross Domestic Product GNI Gross National Income HD Health Department HepB Hepatitis B Hib Haemophilus influenzae type B HIF Health Insurance Fund ICC Interagency Coordination Committee INS GAVI Injection Safety Support IRC Independent Review Committee ISS Immunization Services Support KAP Knowledge, Attitudes and Practice survey M&E Monitoring and Evaluation MICS Multiple Indicator Cluster Survey MMR Measles, Mumps, Rubella Vaccine MoCA Ministry of Civil Affairs MoH Ministry of Health MoHSW Ministry of Health and Social Welfare MYSP Multi-Year Strategic Plan NGO Non-governmental organization NVS New and underused vaccine support OPV Oral Polio Vaccine PHC Primary Health Care iii FINAL EVALUATION OF GAVI ALLIANCE’S SUPPORT TO BOSNIA AND HERZEGOVINA PHI Public Health Institute RAT Rapid Assessment RFP Request for Proposals RS Republika Srpska TB Tuberculosis TV Television UNFPA United Nations Population Fund UNICEF The United Nations Children's Fund UNICEF SD Supply Division of the United Nations Children's Fund VMA Vaccine Management Assessment VPD Vaccine Preventable Diseases WB The World Bank WHO World Health Organization iv FINAL EVALUATION OF GAVI ALLIANCE’S SUPPORT TO BOSNIA AND HERZEGOVINA EXECUTIVE SUMMARY INTRODUCTION Bosnia and Herzegovina (BiH) is a transition economy state in Eastern Europe, consisting of two entities (the Federation of Bosnia and Herzegovina (FBiH), Republika Srpska (RS)) and the independently administered Brcko District (BD). Its economy relies heavily on the export of metals, as well as on remittances and foreign aid. A highly decentralized government creates challenges for economic policy coordination and reform, while bureaucratic barriers and a segmented market discourage foreign investments. BiH experienced a difficult civil war during the 1990s. After restoring peace, outputs, including health services, recovered slowly. Decision-making in the BiH health system is decentralized to the entities and the BD, and in FBiH these powers are further devolved to the local (cantonal) level. According to the March 2003 Law on Ministries, the Ministry of Civil Affairs (MoCA) of BiH is in charge of the coordination of health issues at the state level. Over the past several years, both entities have initiated wide-ranging reforms in the health sector aimed at increasing financing, strengthening delivery systems and improving quality of care. However, key health system challenges – such as inequalities in access, institutional fragmentation, shortage of medical personnel, and financial sustainability – contribute to the ongoing need for external financial and technical assistance. During BiH’s post-war recovery, the country required external assistance to improve the national immunization programmes and introduce new vaccines. The Global Alliance for Vaccines and Immunization’s (GAVI) support to BiH was launched in 2002 with the Vaccine Introduction Grant for the Hepatitis B (HepB) monovalent vaccine. The support ended in 2011 with the last shipments of the GAVI supported Haemophilus influenzae type B (HiB) monovalent vaccine. GAVI also supported the introduction of Injection Safety policies in BiH. Over the years, GAVI’s support amounted to a total of US$2.27 million. GAVI’s graduation policy did not exist at the time that BiH funding ended. Therefore, BiH did not experience a graduation phase similar to current Phase III GAVI graduating countries.