The Institutionalization of Monitoring and Evaluation Systems Within International Organizations: a Mixed-Method Study
Total Page:16
File Type:pdf, Size:1020Kb
The Institutionalization of Monitoring and Evaluation Systems within International Organizations: a mixed-method study by Estelle Raimondo B.A. in Political Science, June 2008, Sciences Po Paris M.I.A in International Affairs, May 2010, Columbia University M.A. in International Economic Policy, June 2010, Sciences Po Paris A Dissertation submitted to The Faculty of The Columbian College of Arts and Sciences of the George Washington University in partial fulfillment of the requirements for the degree of Doctor of Philosophy May 15, 2016 Dissertation directed by Kathryn Newcomer Professor of Public Policy and Public Administration The Columbian College of Arts and Sciences of The George Washington University certifies that Estelle Raimondo has passed the Final Examination for the degree of Doctor of Philosophy as of February 25, 2016. This is the final and approved form of the dissertation. The Institutionalization of Monitoring and Evaluation Systems within International Organizations: a mixed-method study Estelle Raimondo Dissertation Research Committee: Kathryn Newcomer, Professor of Public Policy and Public Administration, Dissertation Director Jennifer Brinkerhoff, Professor of Public Policy and Public Administration, of International Business, and of International Affairs Catherine Weaver, Associate Professor of Public Affairs, The University of Texas at Austin, Committee Member ii © Copyright 2016 by Estelle Raimondo. All rights reserved iii Dedication To my beloved parents. iv Acknowledgements While a dissertation can sometimes be a long and relatively lonely journey, I was fortunate to have a number of key people by my side in this voyage of discovery. I am grateful to my parents for being my "biggest fans" and for having made my "American dream" possible. My mom, a teacher, instilled in me the rigor, dedication, and resilience that are necessary in pursuing studies at the doctoral level. My dad, never doubted of my capacity to succeed, and was always there when I needed a boost of confidence. Without their many sacrifices, both financial and emotional, I would not have made it this far along the academic road. I also owe a big piece of this journey to my twin sister, Julie, who has always encouraged me to pursue my own calling, even if it meant being 6,500km away. Her daily phone calls and cheers have kept me going. I was fortunate to count on a number of scholars who inspired and supported me along the way: Prof. David Lindauer at Wellesley College planted in me the seeds of my passion for international development, and Prof. Kathy Moon whose rigorous and transformative research has long been a source of inspiration. Prof. Maxine Weisgrau and Dr. Jenny McGill at Columbia University gave me the opportunity to conduct my first evaluation research assignment. All of them wrote countless recommendation letters to help me get to where I am today. My adviser, Prof. Kathy Newcomer, naturally played a key role in my journey. Her enthusiasm for evaluation, her unparalleled energy, and her consistently reassuring feedback helped me find the confidence and positive attitude to make steady progress on my research. Her rigorous and pragmatic approach helped me tremendously in making important methodological and conceptual decisions along the way. I am also deeply thankful to the other members of my dissertation committee. Prof. Jennifer Brinkerhoff pushed me to look for the "big picture" and asked fundamental questions, v when I would get lost in the details of the analysis. She also contributed her immense experience of the field. Prof. Kate Weaver very generously accepted to be a key member of my committee after only one phone call and did not hesitate to travel to DC for important milestones in my journey. Her brilliant work on the World Bank's culture was at the core of my conceptual framework and she provided tremendously helpful advice on how to be theoretically sound and empirically grounded. Prof. Lori Brainard's seminar on Public Administration theory inspired me to tackle organizational and institutional issues in my research, she also taught me how to master the art of writing literature reviews, which was invaluable for my dissertation. Finally, Dr. Jos Vaessen has been a great mentor for years, and I am in constant admiration of his superior analytical mind, exceptional evaluation skills, and his capacity to tackle complex topics with nuance and rigor; qualities that I have striven to apply in my research. He has provided tremendously helpful methodological advice and helped me craft my conclusions and policy recommendations. Additionally, I am indebted to Mrs. Caroline Heider, and Dr. Rasmus Heltberg, for including me on an exciting evaluation project to study the self-evaluation system of the World Bank and for his guidance in conducting my own research on the topic. I am also grateful to all the people who participated in my research, and express my admiration for the many individuals who are working tirelessly towards better development results, even when these results are hard to measure. Finally, I could not have completed this journey without my partner Dominique Parris, who was by my side through every landmarks, at high and low points. She cheered for me, put me back together after difficult episodes, slowed me down when needed, time and time again. She also allowed me to be as disconnected from practical realities as I needed to be to complete my coursework, exams and research. Dominique: we did it and I can't thank you enough! vi Abstract of Dissertation The Institutionalization of Monitoring and Evaluation Systems within International Organizations: a mixed-method study Since the late 1990s, Results-Based Monitoring and Evaluation (RBME) systems have seized the development discourse. They are institutionalized, and integrated as a legitimate managerial and governance function in most International Organizations. However, the extent to which RBME systems actually perform as intended , make a difference in organizations' performance, and their roles in shaping actors' behaviors within organizations, are empirical questions that have seldom been investigated. This research takes some steps towards addressing this topic. Drawing on an eclectic set of theoretical strands stemming from Public Administration theory, Evaluation theory and International Organizations theory, this study examines the role and performance of RBME systems in a complex international organization, such as the World Bank. The research design is scaffolded around three empirical layers along the principles of Realist Evaluation: mapping the organizational context in which the RBME is embedded; studying patterns of regularity in the association between the quality of project-level monitoring and evaluation and project outcome, and eliciting the underlying behavioral mechanisms that explain why such patterns of regularity take place, and why they can be contradictory.. The study starts with a thorough description of the World Bank's RBME system's organizational elements, and its evolution over time . I identify the main agent-based driven changes, and the configurations of factors that influenced these changes. Overall, the RBME institutionalization process exhibited key traits of what Institutionalist scholars call "path dependence." The RBME system's development responded to a dual logic of further legitimation and rationalization, all the while maintaining its initial espoused theory of conjointly promoting accountability and learning, despite some evidence of trade-offs. vii The second part of the study uses data from 1,300 World Bank projects evaluated between 2008 and 2014 to investigate the patterns of regularity in the association between the quality of monitoring and evaluation (M&E) and project performance ratings as institutionally measured within the organization and its central evaluation office. The propensity score matching results indicate that the quality of M&E is systematically positively associated with project outcome. Depending on whether the outcome is measured by the central evaluation office or the operational team, the study finds that projects with good quality M&E score between 0.13 and 0.40 points higher—on a six-point outcome scale— than similar projects with poor quality M&E. The study also concludes that the close association between M&E quality and project performance reflects the institutionalization of RBME within the organization and the socialization of actors with the rating procedures. The third part of the inquiry uses a qualitative approach, based on interviews and a few focus groups with operational staff, managers and evaluation specialists to understand the behavioral factors that explain how the system actually works in practice. The study found that, like in other International Organizations, the project-level RBME system was set up to resolve gaps between goals and implementations. Yet, actors within large and complex IOs are facing ambivalent signals from the external stakeholders, that may also conflict with the internal culture of the organization; and organizational processes do not necessarily incentivize RBME. Consequently, the RBME system may elicit patterns of behaviors that can contribute to further decoupling goals and implementations, discourse and actions. viii Table of Contents Dedication.................................................................................................................................. iv Acknowledgements ....................................................................................................................