USAID LEBANON UNIVERSITY SCHOLARSHIP PROGRAM MID-TERM EVALUATION September 2015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by: Robert Miller, Joanna Khater, Wassim Katerji, Najwa Andraos, Kamil Wanna and Harvey Herr of Social Impact. Cover Photo: A tranquil university setting at one of the three USP implementing partners. Photo by Robert Miller. Acknowledgements: The authors wish to express their appreciation to the USP and university staff at AUB, LAU and Haigazian University who were generous with their time, answered a myriad of questions, facilitated appointments, and supplied data. We also are indebted to the many students who participated enthusiastically in our focus group discussions, interviews and Survey Monkey exercise. A special thanks to the staff of Information International who conducted fieldwork in remote areas, and to Stephanie Deep, who served as reporter for our focus group discussions. Finally, we wish to express our appreciation to staff of USAID Lebanon for providing this opportunity to carry out the evaluation and for providing excellent guidance and assistance in carrying out the task. USAID LEBANON UNIVERSITY SCHOLARSHIP PROGRAM MID-TERM EVALUATION September 2015 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. TABLE OF CONTENTS Acronyms ............................................................................................................................................................ i Executive Summary .......................................................................................................................................... ii Background and Purpose ............................................................................................................................ ii Evaluation Questions ................................................................................................................................... ii Findings and Conclusions .......................................................................................................................... iii Evaluation Purpose & Questions ................................................................................................................... 1 Evaluation Purpose ....................................................................................................................................... 1 Evaluation Questions ................................................................................................................................... 1 Project Background .......................................................................................................................................... 3 Summary of Task .......................................................................................................................................... 4 Evaluation Methods & Limitations ................................................................................................................. 5 Findings, Conclusions, and Recommendations........................................................................................... 8 Annexes ........................................................................................................................................................... 33 Annex 1: The Evaluation Statement of Work ................................................................................... 34 Annex 1A: List And Description of USP Programs Included in Scope of the Evaluation ..... 34 Annex 2: Evaluation Matrix ..................................................................................................................... 35 Annex 3: Data Collection Tools ........................................................................................................... 39 Annex 4: Distribution of USP Enrolled Students by Districts......................................................... 71 Annex 4A: Distribution of USP Enrolled Students by Districts and Sex .................................. 72 Annex 4B: Distribution of USP Enrolled Students by Districts and by USP ............................ 74 Annex 5: Maps ........................................................................................................................................... 75 Annex 6: Distribution of Applications by USP and Geographical Area ........................................ 80 Annex 7: Distribution of Public High Schools by District ................................................................ 82 Annex 8: Distribution of Students by Major and Sex ....................................................................... 90 Annex 9: MEPI and USP Budget Breakdown ....................................................................................... 98 ACRONYMS AO Assistance Objective AOR Agreement Officer Representative AUB American University of Beirut AUC American University of Cairo CAAP Collegiate Assessment of Academic Proficiency CAS Central Administration of Statistics CDCS Country Development Cooperative Strategy COS College Outcome Survey CV Curriculum Vitae DO Development Objective FGD Focus Group Discussion GIS Geographic Information System GPA Grade Point Average GPS Geographic Positioning System HU Haigazian University IM Implementing Mechanism IP Implementing Partner IR Intermediate Result KII Key Informant Interview LAU Lebanese American University M&E Monitoring and Evaluation MENA Middle East and North Africa MEPI Middle East Partnership Initiative OIRA Office of Institutional Research and Assessment PMSPL Performance Management and Strategy Program for Lebanon PPR Annual Performance Plan and Report UNRWA United Nations Relief and Works Agency USP University Scholarship Program USAID United States Agency for International Development University Scholarship Program Formative Evaluation – September 2015 i EXECUTIVE SUMMARY BACKGROUND AND PURPOSE USAID Lebanon initiated the University Scholarship Program (USP) in 2010 at Lebanese American University (LAU) and Haigazian University (HU) and later added American University of Beirut (AUB). The aim of this $63.9 million program is to provide very bright Lebanese public school students who have high financial need with the opportunity to attain a quality higher education in order to maximize their potential to support Lebanon’s democratic and economic development. A total of 496 students were admitted to programs at the three elite universities. The USP scholarship provided each student with full tuition and fees, a book allowance, a living stipend, a computer and all other related education expenses. In 2015, the USAID Agreement Officer’s representative requested a mid-term evaluation of USP (USP I scheduled to end in 2017 to USP V which scheduled to end in 2019) in order to obtain concise, actionable recommendations to inform program implementation for the remainder of the agreements. A list and a description of USP programs that are included in the scope of the evaluation is available in Annex 1.a. EVALUATION QUESTIONS In meetings with the Mission and university staff, six broad questions were provided to the evaluation team to guide the program evaluation, along with numerous sub-questions. These questions covered: national outreach and program coverage; the scholars’ overall performance; programmatic performance; sustainability; appropriateness of the USAID mechanism in use; and a comparison of USP to a similar Middle East Partnership Initiative (MEPI) program. In addition to answering the overall guiding questions which form the main body of this report, the evaluation team reported additional information uncovered in the evaluation that they thought would provide additional understanding of the program’s functioning. METHODS AND LIMITATIONS The evaluation methodology included a desk review of reports, key informant interviews (KIIs) with 28 key informants, five focus group discussions (FGDs) with students, anonymous online survey (215 responses), a review of university records, interviews with five (out of 16) students who were dropped from the program, a field survey with 45 individuals in five remote districts that had fewer students benefiting from the program, and a detailed USAID questionnaire received from 89 newly graduated students conducted in 2014. With this extensive data University Scholarship Program Formative Evaluation – September 2015 ii collected, the evaluation team used a combination of qualitative and quantitative analyses in a process of triangulation to arrive at the most empirically defensible answers to the evaluation questions. Certain limitations of the data and analysis include possible lack of representativeness of respondents, some possible biases among KIIs, and some disparity in the results from different methodologies. FINDINGS AND CONCLUSIONS National Coverage Outreach: The three universities succeeded in achieving broad national outreach. Students were recruited from all of Lebanon’s 26 districts, with select underrepresentation in some programs from specific remote areas (Annex 4.B – Distribution of USP enrolled Students by districts and by USP). Visits by university staff to the districts and high school principals were instrumental in spreading knowledge about USP to the successful applicants. USP student enrollment is gender balanced at AUB, female