Progressio PPA IPR Independent Progress Review Final th 12 October 2012 Jon Barnes & Josephine Tsui Church & Court Barn, Church Lane Tickenham, Bristol, BS21 6SD United Kingdom Tel: +44 1275 811 345 [email protected] www.theIDLgroup.com Progressio PPA IPR Acknowledgements We would like to thank the teams at Progressio in the UK office for being generous with their time and patience. The IPR team admires your dedication and hard work to Progressio’s values. A big thank you also to staff in Santo Domingo supporting work in the Dominican Republic and Haiti, who have been so charitable in hosting and supporting the team. We would like to acknowledge Rachel Percy who has commented on drafts of this report. Progressio PPA IPR Contents EXECUTIVE SUMMARY………………………………………………………………………… (i) 1 INTRODUCTION....................................................................................1 1.1 PURPOSE OF THE EVALUATION ............................................................... 1 1.2 SCOPE OF THE EVALUATION ................................................................... 1 1.3 FOCUS OF THE EVALUATION ................................................................... 1 1.4 ORGANISATION CONTEXT .............................................................................. 1 1.5 LOGIC AND ASSUMPTIONS (I.E. THEORY OF CHANGE) SUPPORTING DFID PPA FUNDED PROJECT AND/OR PROGRAMME ACTIVITIES.................................. 4 1.6 OVERVIEW OF PPA FUNDED ACTIVITIES ............................................................. 5 1.7 RELATIONSHIP OF DFID PPA FUNDED ACTIVITIES TO OTHER PROGRAMME ACTIVITIES ... 6 2 EVALUATION METHODOLOGY .............................................................7 2.1 EVALUATION PLAN ...................................................................................... 7 2.1.1 Evaluation questions ....................................................................... 7 2.1.2 Evaluation design (and rationale for design).................................... 7 2.1.3 Research methodology (and data collection strategy) ..................... 7 2.1.4 Analytical framework ...................................................................... 7 2.1.5 Approach to quality assurance of research ...................................... 8 2.2 RESEARCH PROBLEMS ENCOUNTERED .............................................................. 8 2.3 STRENGTHS AND WEAKNESSES OF SELECTED EVALUATION DESIGN AND RESEARCH METHODS IN RETROSPECT ........................................................................................ 8 3 FINDINGS .............................................................................................9 3.1 RESULTS ................................................................................................. 9 3.1.1 Performance assessment against logframe ..................................... 9 3.1.2 Intended and unintended effects (positive and negative changes) on poor and marginalised groups and civil society ............................................ 17 3.2 RELEVANCE .............................................................................................. 22 3.2.1 Progressio’s Theory of Change and the PPA logframe .................... 22 3.2.2 Relevance of Progressio’s work to UK’s and local partner’s priorities 22 3.2.3 Representativeness ....................................................................... 22 3.2.4 Targeting ...................................................................................... 23 3.3 EFFECTIVENESS ..................................................................................... 25 3.3.1 Learning........................................................................................ 25 3.3.2 Innovation..................................................................................... 26 3.3.3 Partnership working ...................................................................... 29 3.3.4 Sustainability ................................................................................ 30 3.4 EFFICIENCY .............................................................................................. 32 3.4.1 Value for money assessment ......................................................... 32 3.5 IMPACT AND VALUE FOR MONEY OF PPA FUNDING............................................ 34 3.5.1 Attributable impacts of PPA funding on results, relevance, effectiveness and efficiency.......................................................................... 34 3.5.2 Value for money assessment of PPA funding ................................. 35 4 CONCLUSIONS ....................................................................................37 Progressio PPA IPR 4.1 SUMMARY OF ACHIEVEMENTS AGAINST EVALUATION CRITERIA ........... 37 4.2 SUMMARY OF ACHIVEMENTS AGAINST RATIONALE FOR PPA FUNDING . 40 4.3 SUMMARY OF PROBLEMS AND ISSUES ENCOUNTERED ......................... 41 4.4 OVERALL IMPACT AND VALUE FOR MONEY OF PPA FUNDED ACTIVITIES 41 5 UTILITY .............................................................................................. 42 6 LESSONS LEARNT ............................................................................... 43 6.1 POLICY LEVEL ........................................................................................ 43 6.2 SECTOR LEVEL ....................................................................................... 44 6.3 PPA FUND LEVEL ................................................................................... 44 6.4 ORGANISATIONAL LEVEL – MANAGEMENT, DESIGN, IMPLEMENTATION 46 7 RECOMMENDATIONS ........................................................................ 48 Progressio PPA IPR Figures, Tables, and Annexes List of Figures Figure 1-1 Spending allocations in each country for 2011-2012. While this chart illustrates Timor Leste not currently receiving funding, it has more to do with funding cycles than a lack of financial support. .............................................. 4 Figure 1-2 Allocation of Progressio’s PPA budget for 2011-2012 ............................ 5 List of Tables Table 1-1 Where Progressio works, separated to country and focus area .............. 4 Table 3-1 DFID – Progressio PPA Progress Assessment Against the Logframe ...... 10 List of Annexes Annex A: PPA IPR terms of reference.................................................................. 53 Annex B: Evaluation research schedule and timescales ...................................... 63 Annex C: Data collection tools ............................................................................ 64 Annex D: List of people consulted ...................................................................... 69 Annex E: List of data sources .............................................................................. 73 Annex F: Bibliography ........................................................................................ 73 Annex G: Sub-reports of country visits, case studies, and the PPA logframe. ...... 74 Annex G1: IPR field research and case studies on Progressio’s country-level capacity building.......................................................................................... 74 Annex G2: Progressio’s work in International Advocacy...................................... 91 Annex G3: PPA reporting and Progressio’s planning, monitoring and evaluation systems: strengths, weaknesses and recommendations............................... 97 Annex G4: Advocacy on faith and development ............................................... 109 Annex G5: Logframe ......................................................................................... 110 Annex H: Details of the evaluation team .......................................................... 117 Annex I: PPA organisation’s management response to report’s findings (post- submission) ............................................................................................... 118 Annex J: Evaluation Manager’s response to report’s findings (post-submission)121 Progressio PPA IPR Progressio PPA IPR Acronyms and Abbreviations APPG All-Party Parliamentary Group ART Anti-retroviral Therapy BC Business Case CAFOD Catholic Agency for Overseas Development CAT Capacity Assessment of Partners CIIR Catholic Institute for International Relations CIT Comité Inter-Municipal Transfronterizo (Cross-Border Inter- Municipal Committee) CSO Civil Society Organisations CoEH Coordination Europe-Haiti DEFRA Department for Environment, Food and Rural Affairs DFID UK Department for International Development DW Development Worker EU European Union FCO Foreign Commonwealth Office FEDOMU Federación Dominicana de Municipios (Dominican Federation of Municipalities) FONGTIL The NGO Forum for Timor Leste GDP Gross Domestic Product GOLD Governance and Local Development programme GPAF Global Poverty Action Fund HAART Highly active antiretroviral therapy HAPI-UK Haiti Advocacy Platform Ireland-UK HQ Headquarters ICS International Citizen Service IPTCS Integrated Prevention, Treatment, Care and Support Centre IPR Independent Progress Review LIDER (Lian District Ermera) a consortium of member organisations in Ermera District that are members of FONGTIL M&E Monitoring and Evaluation MDG Millennium development goals MIS Management information system MOU Memorandum of Understanding MSATM Ministry of State Administration and Territorial Management MTCT Mother to child transmission NGO Non-governmental organisational PATT Participation and Transparency Tool PLHIV People Living with HIV and AIDS PMTCT Prevention of