A University of Sussex PhD thesis Available online via Sussex Research Online: http://sro.sussex.ac.uk/ This thesis is protected by copyright which belongs to the author. This thesis cannot be reproduced or quoted extensively from without first obtaining permission in writing from the Author The content must not be changed in any way or sold commercially in any format or medium without the formal permission of the Author When referring to this work, full bibliographic details including the author, title, awarding institution and date of the thesis must be given Please visit Sussex Research Online for more information and further details University of Sussex THE EFFECTIVENESS OF POLICY EVALUATION Insights from the health care sector in Mexico and Chile Blanca Odille López Rodríguez A thesis submitted to the Department of Politics of the School of Law, Politics and Sociology of the University of Sussex for the degree of Doctor of Philosophy in Politics Brighton, December 2017 ~ 2 ~ Declaration I certify that the thesis I have presented for examination for the degree of Doctor of Philosophy in Politics of the University of Sussex is solely my own work. The copyright of this thesis rests with the author. Quotation from it is permitted, provided that full acknowledgement is made. This thesis may not be reproduced without my prior written consent. I warrant that this authorisation does not, to the best of my belief, infringe the rights of any third party. I declare that my thesis consists of 87,676 words. ~ 3 ~ Abstract Over time, the predominant tendency of many governments’ agencies has been to evaluate a programme or policy investing large amount of resources in supporting policy evaluation. However, recommendations suggested by policy evaluators are not always taken up. Moreover, there is relatively little evidence of the extent of policy evaluation effectiveness (i.e. the influence of evaluation on the programme evaluated) and the factors which have significant impact on it. This dissertation aims to shed light on this issue by focusing on the Mexican and Chilean experiences of policy evaluation in the health care sector. It provides a detailed analysis of the extent to which evaluations have led to changes in policies and programmes and reveals a rather limited effectiveness of policy evaluation in these countries. I argue that shortcomings in the effectiveness of policy evaluation can be explained by institutional and political factors, primarily the nature of Intra Governmental Relations (IGR), but also the quality of bureaucracy, the level of democracy, the autonomy of policy evaluators, and the type of policy evaluation framework. While all of these factors seem to have some influence, the relationship between the executive and legislature is clearly the key determinant of the take up of recommendations. Thus, the findings of this thesis suggest that strengthening coordination between the different parts of government is needed to enhance the effectiveness of policy evaluation. In addition, the analysis also suggests that policy evaluation is likely to be more effective when it incorporates budgetary incentives. ~ 4 ~ Acknowledgements This research it would not have been possible without the kind support and help of many individuals. I would like to extend my sincere thanks to all of them. I would like to express my gratitude towards the Mexican government’s Consejo Nacional de Ciencia y Tecnología (CONACYT) for the financial support and assistance to pursue this doctoral project. I take this as an opportunity to express my greatest regards to my family for their patience, support and understanding to pursue this dissertation. I also express my gratitude to my friends for their valuable suggestions and constant encouragement throughout my research period. My thanks and appreciations also go to my colleagues and people who have willingly helped me out with their abilities and comprehensive advice in the conduct of this research. To all the respondents who were very cooperative in giving information which facilitate completion of this project. This thesis has indeed helped me to explore more knowledgeable avenues related to my topic and will help me in my future. ~ 5 ~ Table of contents Abstract 3 Acknowledgements 4 Introduction Why is effectiveness important? 9 Contribution of the thesis 9 The central discussion of the thesis 10 Chapter outline 12 Chapter 1. Broadening understandings of policy evaluation and effectiveness 1.1 Introduction 14 1.2 The roots of evaluation 14 1.3 Evaluation in emerging countries and the New Public Management 17 1.4 A re-conceptualisation of the theory of evaluation use 20 1.5 What explains the effectiveness of policy evaluation? 23 1.6 Factors influencing the effectiveness of evaluation 25 1.7 Concluding remarks 33 Chapter 2. Methodological approach 2.1 Introduction 34 2.2 Research strategy or how the research is to be carried out? 34 2.2.1 Dependent variable and operationalisation of evaluation effectiveness 35 2.2.2 Independent variables 39 Intra Governmental and Interagency Relations ‘IGR’ 40 Quality of the bureaucracy 43 Level of democracy 44 Degree of autonomy 45 Policy evaluation framework 46 2.3 The comparative dimension of policy evaluation 47 2.4 Documentary and analysis sources 51 2.4.1 Elite interviews 52 Chapter 3. Policy evaluation in Mexico 3.1 Introduction 55 3.2 Historical perspective 55 3.3 The current state of affairs of policy evaluation 71 Chapter 4. Policy evaluation in Chile 4.1 Introduction 73 4.2 Historical perspective 73 4.3 The current state of affairs of policy evaluation 80 Chapter 5. The effectiveness of policy evaluation in Mexico 5.1 Introduction 82 5.2 Results-based Budgeting ‘PbR/SED’ 82 5.2.1 How is the federal budget made? 82 5.3 Results of the programmes assessed 83 5.4 The effectiveness of evaluation in the health sector 84 ~ 6 ~ 5.5 The effectiveness of evaluation in the Results-based Budgeting (PbR/SED) 87 5.5.1 What explains the effectiveness in policy evaluation? 91 5.6 Intra-Governmental Relations (IGR) 91 5.7 Quality of bureaucracy 94 5.8 Level of democracy 96 5.9 Policy evaluation framework 99 5.10 Degree of autonomy 104 Chapter 6. The effectiveness of policy evaluation in Chile 6.1 Introduction 108 6.2 The budget process “Results-based Budgeting” 109 6.3 The effectiveness of evaluation in the health sector 110 6.4 The effectiveness of evaluation in the Results-based Budgeting 114 6.4.1 What explains the effectiveness in policy evaluation? 116 6.5 Intra-Governmental Relations (IGR) 117 6.6 Quality of bureaucracy 120 6.7 Level of democracy 124 6.8 Policy evaluation framework 125 6.9 Degree of autonomy 128 Chapter 7. A comparative analysis of the institutional factors influencing the effectiveness of evaluation 7.1 Introduction 131 7.2 Categories of recommendations 131 7.3 Evaluation in the Mexican health sector 132 7.3.1 Status quo programmes 132 7.3.2 Amended programmes 134 7.3.3 Cancelled programmes 135 7.4 Evaluation in the Chilean health sector 138 7.4.1 Status quo programmes 138 7.4.2 Amended programmes 139 7.4.3 Cancelled programmes 140 7.5 The influence of institutional factors on the effectiveness of evaluation 142 Chapter 8. Conclusions 158 8.1 Theoretical contributions and policy implications of the thesis 164 8.2 Limitations and recommendations 166 8.3 Final statements 168 List of acronyms 170 List of interviews 171 References 172 List of annex, charts, figures and tables Annex 1. Measurement of independent variables 194 Annex 2. Questionnaire “The effectiveness of policy evaluation” 195 Annex 3. Office letter no. 1761 DIPRES in Chile 197 Annex 4. Evaluation systems and monitoring practices in Mexico 203 Annex 5. Healthcare programmes of Maule 204 Annex 6. Healthcare programmes of Osorno 206 Annex 7. Evaluation systems and monitoring practices in Chile 208 Annex 8. Categories of recommendations across countries 209 ~ 7 ~ Chart 1. Effectiveness of evaluation in the Mexican health sector 2007-2014 85 Chart 2. Resources provided to coordinators of evaluation in Mexico 93 Chart 3. Effectiveness of evaluation in the Chilean health sector 2007-2014 111 Chart 4. Resources provided to evaluations across the Chilean public 119 administration Chart 5. Remunerations (real) in the Chilean economy and public sector 1978- 121 2000 Figure 1. The effectiveness of policy evaluation 21 Figure 2. Elements of the policy evaluation framework 46 Figure 3. Planning and evaluation functions in the SPP and SECOGEF 59 Figure 4. Modernization of the Mexican Federal Public Administration 61 Figure 5. Results-based Budgeting/Performance Evaluation System 69 Figure 6. Annual Evaluation Programme ‘PAE’ 2007 to 2013 70 Figure 7. System of evaluation and management control ‘SECG’ 78 Figure 8. Budget cycle in Mexico 83 Figure 9. Budget cycle in Chile 110 Figure 10. Dimensions of political relationships 144 Table 1. Planning, control, monitoring and evaluation practices in Mexico 217 Table 2. Sources of elite interviews 53 Table 3. The effectiveness of policy evaluation in Mexico 223 Table 4. Allocation of financial resources based on evaluations in Budget (PEF) 88 FY 2011 Table 5. Government effectiveness (quality of bureaucracy) in Mexico 95 Table 6. The effectiveness of policy evaluation in Chile 232 Table 7. Financial resources of the Chilean health sector 115 Table 8. Comparison of resources to perform evaluations 120 Table 9. Evolution of incentives in Chile 1998-2010 through the PMG 122 Table 10. Government effectiveness (quality of bureaucracy) in Chile 122 Table 11. Categories of recommendations of sample of programmes assessed 137 across countries Table 12. Recommendations achieved of sample of programmes assessed across 141 countries ~ 8 ~ Introduction In recent decades, evaluation has become a byword within public management discourse for most governments.