Performance Evaluation of the Health Systems Strengthening (HS-STAR) Project Mid-term Evaluation Report May 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Karine Boyakhchyan and Susanna Onanyan through Social Impact, Inc. i Cover Photo: Courtesy of HS-STAR Project, May 2012. i HEALTH SYSTEMS STRENGTHENING PROJECT (HS-STAR): MID-TERM PERFORMANCE EVALUTION EVALUATION REPORT May 27, 2013 Contract # AID-RAN-I-00-09-00019 Task Order # AID-111-TO-12-00002 Evaluation of Five USAID/Armenia Activities DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii CONTENTS Acronyms .................................................................................................................................................................................................. v Executive Summary................................................................................................................................................................................. 1 Project Background ................................................................................................................................................................................ 5 Introduction: Overview Of Health Reform In Armenia .......................................................................................................... 5 Project Description And Implementation ................................................................................................................................... 6 Evaluation Purpose And Methodology ............................................................................................................................................ 10 Evaluation Purpose .......................................................................................................................................................................... 10 Evaluation Methodology & Limitations ....................................................................................................................................... 10 Findings & Conclusions ........................................................................................................................................................................ 13 Evaluation Question #1 .................................................................................................................................................................. 13 Findings .......................................................................................................................................................................................... 13 Conclusions .................................................................................................................................................................................. 22 Evaluation Question #2 .................................................................................................................................................................. 24 Findings .......................................................................................................................................................................................... 24 Conclusions .................................................................................................................................................................................. 25 Evaluation Question #3 .................................................................................................................................................................. 26 Findings .......................................................................................................................................................................................... 26 Conclusions .................................................................................................................................................................................. 27 Evaluation Question #4 .................................................................................................................................................................. 29 Findings .......................................................................................................................................................................................... 29 Conclusions .................................................................................................................................................................................. 29 Evaluation Question #5 .................................................................................................................................................................. 30 Findings .......................................................................................................................................................................................... 30 Conclusions .................................................................................................................................................................................. 30 Recommendations ................................................................................................................................................................................ 32 Lessons Learned .................................................................................................................................................................................... 33 Annexes ................................................................................................................................................................................................... 34 Annex A. Evaluation SOW ............................................................................................................................................................ 35 Annex B. Draft Work Plan And Methodology ......................................................................................................................... 37 Annex C. Sample Data Collection Tools ................................................................................................................................... 41 Annex D. Data Sources .................................................................................................................................................................. 45 Annex E. HS-STAR Organizational Chart ................................................................................................................................. 47 Annex F. Grant Table ..................................................................................................................................................................... 50 Annex G. Workplan by Implementation Status ....................................................................................................................... 51 Annex H. Disclosure of Any Conflicts of Interest .................................................................................................................. 53 Annex I. Statement of Differences from Abt Associates ....................................................................................................... 55 iv ACRONYMS BBP Basic Benefit Package CME Continued Medical Education CPG Clinical Practice Guidelines COP Chief of Party DEC Development Experience Clearinghouse ECA Europe Central Asia FP Family Planning FY Fiscal Year GOA Government of Armenia HS-STAR Health Systems Strengthening Project LOE Level of Effort LMI Low Middle Income MCH Maternal and Child Health MIDAS Medical Information Data Analysis System MOH Ministry of Health MTEF Medium Term Expenditure Framework NOVA Innovation in Support of reproductive health NCD Non-Communicable Disease NTP National TB Program PD Project description PHC Primary Health Care PHCR Primary Health Care Reform PMP Performance Management Plan PPP Public Private Partnership QA Quality Assurance QI Quality Improvement RFA Request for Application RH Reproductive Health SHA State Health Agency SOW Statement of Work TB Tuberculosis TOT Training of Trainers USAID U.S. Agency for International Development WHO World Health Organization v EXECUTIVE SUMMARY EVALUATION PURPOSE AND QUESTIONS In September 2012, Social Impact, Inc. was awarded a contract by USAID/Armenia to evaluate five projects in the Mission’s portfolio. Among them was the Health Systems Strengthening Project (HS- STAR), aimed at addressing key constraints in Armenian health financing, leadership and governance, human resources and information systems that were perceived to limit delivery and access to quality health care services. The purpose of this mid-term evaluation is to inform USAID’s determination on whether the set programmatic goals and targets are being achieved, and whether the initial project designs are still valid in fostering the achievement of the original objectives. Findings from the mid-term evaluation must inform future work plans of the relevant projects, as well as designs of similar future activities. The evaluation findings will be used primarily by USAID/Armenia, the respective implementing partners, and by interested government entities where applicable. The responsible Mission and project staff will