Mid-Term Evaluation of the SARRAH programme April 2014 Report authors: Simon Griffiths (Coffey) Samy Ahmar (Coffey) Prof John Seager (Coffey) Prof Leickness Simbayi (HSRC) Ntombizodwa Mbelle (HSRC) INMED TABLE OF CONTENTS EXECUTIVE SUMMARY 7 Introduction 7 Evaluation approach and methods 7 Key findings and conclusions 8 RELEVANCE 8 EFFICIENCY 8 EFFECTIVENESS 8 SUSTAINABILITY 9 Lessons learnt 9 Recommendations 10 1 INTRODUCTION 11 1.1 Purpose of the Mid-Term Evaluation (MTE) 11 1.2 Governance of the SARRAH Evaluation 12 1.2.1 DFID 12 1.2.2 SARRAH MANAGING AGENT (HLSP) 12 1.2.3 EVALUATION TEAM OF SARRAH (COFFEY & HSRC) 13 1.2.4 SARRAH EVALUATION STEERING COMMITTEE 14 1.3 Overview of SARRAH 15 1.3.1 BRIEF HISTORY OF SARRAH 15 1.3.2 THE SARRAH THEORY OF CHANGE 17 1.4 Methodology & Analytical Framework 23 1.4.1 ANALYTICAL FRAMEWORK 23 1.4.2 RESEARCH METHODOLOGY 24 1.5 Synthesis 27 1.5.1 NUMERICAL TOOL 27 1.5.2 SYNTHESIS 28 1.5.3 PREPARATION OF THE MTE REPORT 28 1.6 Research Challenges and Limitations 28 1.6.1 METHODOLOGICAL CHALLENGES AND LIMITATIONS 28 DIVERSITY OF WORK-STREAMS AND DATA SOURCES 28 DIFFICULTIES IN ASSESSING SARRAH’S CONTRIBUTION 29 EVIDENCE BASE 29 1.6.2 OPERATIONAL CHALLENGES AND LIMITATIONS 29 ADMINISTRATIVE DELAYS 29 LACK OF ACCESS TO NDOH DATA FOR CONFIDENTIALITY REASONS 29 2 EVALUATION FINDINGS 30 2.1 Coordination, oversight and advocacy 30 2.1.1 SOUTH AFRICAN NATIONAL AIDS COUNCIL (SANAC) 30 2.1.2 TREATMENT ACTION CAMPAIGN (TAC) 31 2.2 Health systems strengthening and NHI 32 2.2.1 ASSESSMENT OF CEOS AND DISTRICT HEALTH MANAGERS (DHMS) 32 2.2.2 ASSET MANAGEMENT 33 2.2.3 CONDITIONAL GRANT MANAGEMENT 33 2.2.4 PUBLIC FINANCIAL MANAGEMENT 35 2.2.5 HUMAN RESOURCE FOR HEALTH STRATEGY 36 2.2.6 ORGANISATIONAL REDESIGN OF NDOH 37 2.2.7 NATIONAL HEALTH INSURANCE 38 2.2.8 NHIRD 39 2.2.9 PREVENTION OF MOTHER-TO-CHILD TRANSMISSION OF HIV (PMTCT) 40 2.2.10 QUALITY ASSURANCE 41 2.2.11 SAHPRA 42 2.2.12 SERVICE TRANSFORMATION PLANS (STP) 43 2.3 Mid-term Evaluation Scoring Sheet 43 3 VALUE FOR MONEY 46 3.1 Overall Approach 46 3.2 Scope of this VFM Assessment at the Mid-Term Stage 46 3.3 VFM Assessment Methodology 47 3.4 Assessment of SARRAH’s Value for Money 49 3.4.1 ECONOMY AND EFFICIENCY 49 3.4.2 LEVERAGE 49 3.4.3 THEORY OF CHANGE 49 3.5 MTE Value for Money conclusion 50 4 CONCLUSIONS 52 4.1 Relevance: Full alignment with the NSDA and the Ten Point Plan 52 4.2 Efficiency 53 4.3 Effectiveness and Impact of SARRAH to date 53 4.4 Looking forward: sustainability of SARRAH 53 4.5 Strategic Added Value: SARRAH’s catalytic effect 54 5 LESSONS LEARNT 55 5.1 SARRAH’s key drivers of success 55 5.2 A uniquely flexible facility for the South African NDOH 55 6 OVERALL RECOMMENDATIONS 57 ANNEXES Annex 1 Historical Review of DFID’s Engagement with South Africa’s Health Sector Annex 2 Strategic Context of SARRAH Annex 3 Terms of Reference for the Evaluation of SARRAH Annex 4 Evaluation Strategy Report Annex 5 Mid-term Evaluation Report – Assessment of DHM & CEOs Annex 6 Mid-term Evaluation Report – Asset Management Annex 7 Mid-term Evaluation Report – Conditional Grant Management Annex 8 Mid-term Evaluation Report – Financial Management Annex 9 Mid-term Evaluation Report – Human Resources Annex 10 Mid-term Evaluation Report – Organisational redesign of NDOH Annex 11 Mid-term Evaluation Report – NHI Annex 12 Mid-term Evaluation Report – NHIRD Annex 13 Mid-term Evaluation Report – PMTCT Annex 14 Mid-term Evaluation Report – Quality Assurance Annex 15 Mid-term Evaluation Report – SAHPRA Annex 16 Mid-term Evaluation Report – SANAC Annex 17 Mid-term Evaluation Report – STP Annex 18 Mid-term Evaluation Report – TAC Annex 19 List of Key Informant Interviews Conducted Annex 20 Bibliography LIST OF ACRONYMS AGSA Auditor General of South Africa A-Plan PMTCT Accelerated Plan APP Annual Performance Plan ART Antiretroviral Therapy CBO Community Based Organisation CD4 Cluster of Differentiation 4 CEO Chief Executive Officer CFO Chief Financial Officer CG Conditional Grants CSO Civil Society Organisation CSP Comprehensive Service Plan DA Democratic Alliance DBSA Development Bank of Southern Africa DDG Deputy Director General DG Director General DFID Department for International Development (UK) DHIS District Health Information System DHM District Health Manager DORA Division of Revenue Act DPSA Department of Public Service and Administration EDMS Electronic Document Management System FDA Food and Drug Administration FGD Focus Group Discussion GBV Gender-Based Violence HISP Health Information Systems Programme HIV Human Immunodeficiency Virus HLSP HLSP is a trading name of Mott MacDonald Ltd. HR Human Resources HSRC Human Sciences Research Council ICT Information & Communication Technology KPA Key Performance Areas KZN KwaZulu-Natal M&E Monitoring and Evaluation M&OD Management and Organisational Development MCC Medicines Control Council MCWH Maternal Child and Women’s Health MDG Millennium Development Goal MFMA Municipal Financial Management Act MMC Medical Male Circumcision MRA Medical Research Affairs MTCT Mother-to-Child Transmission Coffey International Development December 2013 5 MTE Mid-Term Evaluation MTEF Medium Term Expenditure Framework M&E Monitoring & Evaluation M&OD Management & Organisational Development NCE New Chemical Entities NDOH National Department of Health (South Africa) NGO Non-Governmental Organisation NHC National Health Council NHI National Health Insurance NHIC National Health Information Centre (now known as NHIRD) NHIRD National Health Information Repository and Data Warehouse NSDA Negotiated Service Delivery Agreement NSP National Strategic Plan OSD Occupation Specific Dispensation PEPFAR President’s Emergency Plan for AIDS Relief PFM Public Financial Management PHC Primary Health Care PMTCT Prevention of Mother-to-Child Transmission POC Parliamentary Oversight Committee QA Quality Assurance SAHPRA South African Health Products Regulatory Authority SANAC South African National AIDS Council SARRAH Strengthening South Africa’s Revitalised Response to HIV and Health SCOPA Standing Committee on Public Accounts STI Sexually Transmitted Infection TA Technical Assistance TAC Treatment Action Campaign TB Tuberculosis TOC Theory of Change TOR Terms of Reference UNAIDS Joint United Nations Programme on HIV and AIDS VFM Value for Money Coffey International Development December 2013 6 EXECUTIVE SUMMARY Introduction SARRAH is a five-year DFID funded programme, delivered by HLSP, and which provides technical advice, funding and support to strategic national initiatives that aim to strengthen South Africa’s response to HIV and health. SARRAH’s main focus is to support the National Department of Health (NDOH) in efforts to strengthen and coordinate the national response to HIV and AIDS and make it more accountable to the South African public, and strengthen the health systems overall and their ability to provide quality healthcare to all. It is designed to support the achievement of the targets set out in the government’s Negotiated Service Delivery Agreement (NSDA) for health. SARRAH is based on partnerships with leading national players in HIV and health. Key partners include NDOH and the South African National AIDS Council (SANAC). The programme also supports civil society through the Treatment Action Campaign (TAC) and the work of the Joint Committee on HIV and the Parliamentary Portfolio Committee on Health. SARRAH commenced in January 2010 and is expected to finish delivering its activities in December 2014. This mid-term evaluation covers the period from January 2010 to December 2012. The total value of SARRAH when it commenced in 2010 was £13,500,000. SARRAH was allocated a further £6,852,217 in December 2011, bringing the total programme value at the end of December 2012 to £20,352,217. Evaluation approach and methods The detailed approach and methodology for the evaluation of SARRAH are presented in Section 1.4 and 1.5. SARRAH is structured around a series of initiatives or “work-streams” that are different in scope, scale, timing and objectives. The SARRAH Mid-Term Evaluation (MTE) recognised this diversity and considered the work-streams individually in the first instance, before making an attempt to aggregate and synthesise findings at the programme level. Evidence was collected for each work- stream by different evaluation leads and using different tools and techniques drawing on in a wide array of sources of data. To enable the evaluation team to conduct this assessment rigorously, the following approach was adopted: 1. Individual evaluations of work-streams, or groups of work-streams were conducted using appropriate methodologies, and exploiting potential synergies and complementarities between initiatives; 2. A quality assurance of the evidence collected was undertaken centrally by Coffey to ensure that evidence was of a sufficient standard to assess the overall performance of SARRAH; and 3. A synthesis of the various individual work-stream reports, complemented by a qualitative analysis of circa one hundred interview notes using Atlas.ti, was conducted in order to inform conclusions and answer the key evaluation questions. The evaluation team used the OECD-DAC criteria1 of Relevance, Effectiveness, Efficiency, Impact and Sustainability as the core evaluation framework of the MTE. The inception phase shed light on the nature of the SARRAH programme and made it clear to the evaluation team that the realisation of tangible and sustainable impacts on the South African health system would only be measurable towards the end of the programme in 2014-2015. Therefore the Impact criterion was not assessed to the same extent as the other DAC criteria, but will be the core focus of the final evaluation of SARRAH. 1 OECD (1991), DAC Principles for the Evaluation of Assistance Coffey International Development December 2013 7 Moreover, a numerical scoring tool was developed by the evaluation team as an analytical tool to support the findings emerging from the qualitative research and to enable comparison between different work-streams against the key performance criteria.