Exploring Approaches to Effective Evaluation for Public Diplomacy Exchange Programs
Total Page:16
File Type:pdf, Size:1020Kb
SIT Graduate Institute/SIT Study Abroad SIT Digital Collections Capstone Collection SIT Graduate Institute August 2020 Exploring Approaches to Effective Evaluation for Public Diplomacy Exchange Programs Alison Opoku Donyina Follow this and additional works at: https://digitalcollections.sit.edu/capstones Part of the Policy Design, Analysis, and Evaluation Commons, and the Policy History, Theory, and Methods Commons Recommended Citation Opoku Donyina, Alison, "Exploring Approaches to Effective Evaluation for Public Diplomacy Exchange Programs" (2020). Capstone Collection. 3214. https://digitalcollections.sit.edu/capstones/3214 This Thesis (Open Access) is brought to you for free and open access by the SIT Graduate Institute at SIT Digital Collections. It has been accepted for inclusion in Capstone Collection by an authorized administrator of SIT Digital Collections. For more information, please contact [email protected]. Exploring Approaches to Effective Evaluation for Public Diplomacy Exchange Programs Alison (Mason) Opoku Donyina PIM 76 A capstone paper submitted in partial fulfillment of the requirements for a Master of Arts in Intercultural Service, Leadership and Management at SIT Graduate Institute in Brattleboro, Vermont, USA. August 2020 Advisor: Dr. Bruce Dayton EVALUATION FOR PUBLIC DIPLOMACY EXCHANGE PROGRAMS ii CONSENT TO USE OF CAPSTONE I hereby grant permission for World Learning to publish my Capstone on its websites and in any of its digital/electronic collections, and to reproduce and transmit my CAPSTONE ELECTRONICALLY. I understand that World Learning’s websites and digital collections are publicly available via the Internet. I agree that World Learning is NOT responsible for any unauthorized use of my Capstone by any third party who might access it on the Internet or otherwise. Student name: _Alison (Mason) Opoku Donyina__ Date: __07/13/2020___ EVALUATION FOR PUBLIC DIPLOMACY EXCHANGE PROGRAMS iii Abstract This study explores what factors have limited strategic measurement of outcomes for ‘mutual understanding’ across U.S. government-funded public diplomacy exchange programs by conducting a thorough desk review of available materials, completing a meta-review of twelve existing evaluation reports, and coupling this with interviews with eight evaluation practitioners. The sample of programs included were selected based on the criteria that ‘mutual understanding’ was an explicit goal in the program design. Programs included: Fulbright Student Program, Fulbright Scholar Program, International Visitors Leadership Program (IVLP), Youth Exchange and Study Program (YES), Future Leaders Exchange (FLEX) Program, and the Peace Corps. This paper synthesizes definitions of both ‘public diplomacy’ and ‘mutual understanding’ as used across the field of international relations and in government-sponsored people-to-people (P2P) programing. From a review of the literature, this study also captures posited frameworks, approaches and tools that can assist in measuring public diplomacy programming. It also reviews different approaches to evaluation and identifies strategies that have been used previously to measure mutual understanding and identifies pain- points for measurement of ‘mutual understanding’ and provides some possible recommendations moving forward. Key words: mutual understanding, public diplomacy, evaluation, measurement, soft power, Fulbright, IVLP, Peace Corps, Kirkpatrick Model, exchange program, mixed methods EVALUATION FOR PUBLIC DIPLOMACY EXCHANGE PROGRAMS iv Acknowledgements This paper would not have been possible without the opportunity to work alongside the learning-oriented New Knowledge/Knology crew, the wise evaluators of the IIE Evaluation Unit, and the innovative Mandela Washington Fellowship team at IREX. Thank you for your collective wisdom and knowledge on both public diplomacy programming and evaluation. Thank you also to my advisor, Bruce Dayton, for his timely comments and helpful feedback. Dedication This paper is dedicated to my sweet Melody Aseda and midofo Kwaku. I’m so thankful for you both. EVALUATION FOR PUBLIC DIPLOMACY EXCHANGE PROGRAMS v List of Abbreviations ACPD United States Advisory Commission on Public Diplomacy AFS AFS Intercultural Programs America United States of America AMIDEAST America-Mideast Educational and Training Services APDI Advancing Public Diplomacy Impact BBG Broadcasting Board of Governors DOS Department of State ECA Bureau of Educational and Cultural Affairs, Department of State ED Evaluation Division at the Bureau of Educational and Cultural Affairs EMU Department’s Evaluation and Measurement Unit, Department of State FLEX Future Leaders Exchange Program GAO Government Accountability Office G2G Government to Government G2P Government to Publics IIE Institute of International Education IPIC Independent Practitioner Inquiry Capstone IREX International Research & Exchanges Board IVLP International Visitors Leadership Program Meridian Meridian International Center NATO The North Atlantic Treaty Organization OMB Office of Management and Budget OSIRP Office of Strategic Information, Research, and Planning, Peace Corps P2P People to People PART Program Assessment Rating Tool PD Public Diplomacy PD-MAP Public Diplomacy Model for the Assessment of Performance PFP Professional Fellows Program PMI Performance Measurement Initiative RCT Randomized-Control Trial RPCV Returned Peace Corps Volunteer SNA Social Network Analysis U.S. United States of America; United States USAID Agency for International Development USIA United States Information Agency USG United States Government VRF Volunteer Report Form for Peace Corps Volunteers YES Youth Exchange and Study Program YFU Youth for Understanding EVALUATION FOR PUBLIC DIPLOMACY EXCHANGE PROGRAMS vi Table of Contents Abstract ........................................................................................................................................................ iii Acknowledgements ...................................................................................................................................... iv Dedication .................................................................................................................................................... iv List of Abbreviations .................................................................................................................................... v Table of Contents ......................................................................................................................................... vi Introduction ................................................................................................................................................... 1 Learning Objectives .................................................................................................................................. 3 Research Statement ................................................................................................................................... 4 Literature Review .......................................................................................................................................... 5 What is Public Diplomacy? ....................................................................................................................... 5 Traditional Diplomacy & the evolution of contemporary public diplomacy ........................................ 6 The appeal of ‘Soft Power’ in response to anti-Americanism .............................................................. 8 Constructivist theory and influence on understanding outcomes of exchange ................................... 11 Cultural exchange’s role in U.S. Public Diplomacy ............................................................................... 12 The history of sharing America .......................................................................................................... 14 Complexity, context, and the case for evaluation ............................................................................... 17 Measurement approaches to evaluating impacts of public diplomacy .................................................... 21 Frameworks, models, and approaches ................................................................................................ 23 Soft Power 30 Index ................................................................................................................ 23 NATO’s Framework for the Strategic Planning and Evaluation of Public Diplomacy .......... 23 Pathways of Connection framework ....................................................................................... 24 The British Council’s 3 Metric approach ................................................................................ 25 The Kirkpatrick model ............................................................................................................ 26 The dialogic model of public diplomacy ................................................................................ 27 BBG’s Impact Model .............................................................................................................. 28 Public Diplomacy Model for the Assessment of Performance (PD-MAP)............................. 29 Four public diplomacy evaluation models .............................................................................. 29 Elite comparison approach .....................................................................................................