(Prestasi) 2 & 3
Total Page:16
File Type:pdf, Size:1020Kb
PERFORMANCE EVALUATION OF THE USAID/INDONESIA PROGRAM TO EXTEND SCHOLARSHIPS TO ACHIEVE SUSTAINABLE IMPACTS (PRESTASI) 2 & 3 August 3, 2017 This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International. PERFORMANCE EVALUATION OF THE USAID/INDONESIA PROGRAM TO EXTEND SCHOLARSHIPS TO ACHIEVE SUSTAINABLE IMPACTS (PRESTASI) 2 & 3 FINAL REPORT Contracted under AID-497-C-16-00006 Indonesia Monitoring & Evaluation Support Project DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Contents ACRONYMS ....................................................................................................................................... iv EXECUTIVE SUMMARY .................................................................................................................. 1 I. BACKGROUND ...................................................................................................................... 8 II. EVALUATION PURPOSE .................................................................................................... 8 III. EVALUATION QUESTIONS .............................................................................................. 8 IV. THEORY OF CHANGE AND PROGRAM STRUCTURE ............................................ 9 Program Activity ................................................................................................................................................... 10 V. EVALUATION METHODOLOGY ................................................................................... 12 Evaluation Team ................................................................................................................................................... 12 Evaluation Limitations ......................................................................................................................................... 12 VI. EVALUATION QUESTIONS: FINDINGS...................................................................... 13 Evaluation Question 1: Effectiveness of Outreach, Recruitment and Placement ................................. 13 Evaluation Question 2: Impact on Participants Careers and Workplaces ............................................. 21 Evaluation Question 3: Ph.D. Scholar Management ..................................................................................... 27 Evaluation Question 4: Co-funding Mechanism ............................................................................................ 29 Evaluation Question 5: Support to Local Institutions ................................................................................. 32 Evaluation Question 6: Demand for Doctorate Degrees in Indonesia ................................................... 33 Evaluation Question 7: Effectiveness of Short-term Training .................................................................... 34 Evaluation Question 8: Comparison of PRESTASI with Other Programs.............................................. 37 VII. EVALUATION QUESTIONS: CONCLUSIONS & RECOMMENDATIONS ......... 39 EQ1: Effectiveness of Outreach, Recruitment and Placement .................................................................. 39 EQ2: Impact on Participants Careers and Workplaces .............................................................................. 41 EQ3: Ph.D. Scholar Management ..................................................................................................................... 42 EQ4: Co-funding Mechanism ............................................................................................................................. 43 EQ5: Support to Local Institutions .................................................................................................................. 44 EQ6: Ph.D. Supply and Demand in USAID Targeted Sectors ................................................................... 44 EQ7: Effectiveness of Short-term Training .................................................................................................... 44 EQ8: Comparison of PRESTASI with Similar Programs ............................................................................. 46 ii ANNEXES Annex I Evaluation Statement of Work Annex II Program Theory of Change Annex III Analytical Framework: Getting to Answers Matrix for Key Evaluation Questions Annex IV Data Collection Instruments Annex V Sources of Information Annex VI Data and Charts Annex VII PRESTASI Program Application Forms for Employer Cost-Share Annex VIII Supplementary Evaluation Report: PH.D. Supply and Demand in Targeted Sectors in Indonesia iii ACRONYMS ALPHA-I Alumni Association of Scholarship Programs-Indonesia (Asosiasi Alumni Program Beasiswa Amerika Indonesia) AMINEF American Indonesian Exchange Foundation AOR Agreement Officer's Representative BAPPENAS Badan Perencanaan Pembangunan Nasional (Ministry of National Development Planning) CEPA Communicative English Proficiency Assessment COR Contracting Officer’s Representative DO Development Objective EQ Evaluation Question GOI Government of Indonesia GRE Graduate Record Examination IDIQ Indefinite-Delivery, Indefinite-Quantity IIE Institute of International Education IIEF Indonesian International Education Foundation IPB Institut Pertanian Bogor KII Key Informant Interview LPDP Lembaga Pangelola Dana Pendidikan (Institute of Management Education Fund) LTT Long-Term Training M&E Monitoring and Evaluation MESP Monitoring & Evaluation Support Project MOU Memorandum of Understanding PPL USAID’s Policy, Planning and Learning PRESTASI Program to Extend Scholarship and Training to Achieve Sustainable Impact SEADI Support for Economic Analysis and Development in Indonesia (project) SOW Statement of Work STT Short-Term Training TA Technical Assistance TOEFL Test of English as a Foreign Language USAID United States Agency for International Development WSU Washington State University iv EXECUTIVE SUMMARY USAID/Indonesia’s Program to Extend Scholarships and Training to Achieve Sustainable Impacts (PRESTASI) provides U.S. training to Indonesians in the form of long-term degree programs as well as short-term training courses. The program has operated continuously since 2011 through three separate activities: PRESTASI 1, 2 and 3. The implementation period for PRESTASI 2 is 2012-2017, and PRESTASI 3 began in 2014 and will continue through 2019. PRESTASI 2 is implemented by the Indonesian International Education Foundation (IIEF), an Indonesian non-profit organization, with the Institute of International Education (IIE), a US-based non-profit organization, as its subcontractor. PRESTASI 3 is implemented by the same organizations, but with the roles reversed. This evaluation covers PRESTASI 2 and 3 and spans the period of performance from August 2012 to January 2017. The main purpose of this evaluation is to inform USAID’s design of the anticipated PRESTASI 4 program; however, findings may also inform mid-course improvements for PRESTASI 3. The primary audience for this evaluation is USAID/Indonesia. The evaluation results will also be used by PRESTASI implementing partners for program process improvement. The following eight evaluation questions (EQs) are included in the Statement of Work (SOW): 1. Effectiveness of participant outreach, recruitment, and placement. How effective have the approaches implemented under PRESTASI 2 and 3 been in recruiting, placing, and retaining training program applicants in targeted sectors? 2. Impact on participants’ careers and workplaces. What have been the measurable impacts on the careers of returning degree holders under PRESTASI 2 as a result of their participation in training programs and post-training support (e.g., alumni groups, re-entry workshops, etc.)? What have been the measurable impacts on the functioning of returning participants’ workplaces, as a result of the education and skills participants gained during their training? 3. Ph.D. scholar transfer management. What can USAID learn from the experiences during PRESTASI 2 and PRESTASI 3 to make Ph.D. scholars (project) transfer management support between PRESTASI 3 and PRESTASI 4 more successful? 4. Co-funding mechanism. Has the co-funding initiative under PRESTASI 3 resulted in the desired outcomes? 5. Support to local institutions. Based on initial approaches implemented to date, what is the potential value of PRESTASI 3’s technical assistance and partnering with Indonesian scholarship providers? 6. Demand for Ph.D.s in targeted sectors. What is the relevant supply and demand for Ph.D.s across targeted sectors in Indonesia, per issue-specific USAID funding earmarks? 7. Effectiveness of short-term training processes and programs. How effective have the programs and processes for short-term participant training been under PRESTASI 2 and PRESTASI 3 in terms of participants’ learning and improved competencies in the workplace? 8. Comparison of PRESTASI design with other participant training programs. What are the similarities and differences with PRESTASI among other long-term training programs implemented in Indonesia, including the Fulbright Program, LPDP1 Training Program managed by the Ministry of Finance, and Australia Awards Training Program? 1 Lembaga Pengelola Dana Pendidikan (Institute of Management Education Fund). 1 The evaluation used a mixed-methods methodology that combined