SOLAR WATER HEATER INSTALLED BY BALADI/RMF IN MASHMOUSHE. (PHOTO BY PMSPL II)
INTERIM PERFORMANCE EVALUATION Building Alliances for Local Advancement, Development, and Investment (BALADI) René Moawad Foundation (RMF)
This publication was produced by Social Impact, Inc. for the United States Agency for International Development through the Performance Management and Support Program for Lebanon (PMSPL II). It was prepared by John Lis, Christina Abi Haidar, Mayssa Tannir, Joanna Khater, and Najwa Andraos. Note: Limited redactions have been made to this version of the report in accordance with the principled exceptions to the presumption in favor of openness established in OMB Bulletin 12-01, “Guidance on Collection of U.S. Foreign Assistance Data.”
INTERIM PERFORMANCE EVALUATION BUILDING ALLIANCES FOR LOCAL ADVANCEMENT, DEVELOPMENT, AND INVESTMENT (BALADI) RENÉ MOAWAD FOUNDATION (RMF)
Prepared by Social Impact, Inc. for the United States Agency for International Development under USAID Contract/Order No. AID-268-C-15-00001 – Performance Management and Support Program for Lebanon (PMSPL II).
October 2020
This document is not available in print. Documents of this nature are made available to the public through the Development Experience Clearinghouse (DEC) repository at dec.usaid.gov. For additional information, please contact:
Social Impact, Inc. Corporate Office 2300 Clarendon Boulevard Suite 10300 Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 [email protected]
Or
Social Impact, Inc. Arz Street Librex Bldg. Bloc B – 3rd Floor Zalka, Metn, Lebanon Tel: +961-1-879260
DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.
ABSTRACT
This interim performance evaluation analyzed the relevance, effectiveness, and sustainability of USAID/Lebanon’s Building Alliances for Local Advancement, Development, and Investment (BALADI) activity implemented by the René Moawad Foundation (RMF) from 2012 to 2022. The evaluation also examined the involvement of women in the activity and compared the implementation of BALADI by RMF and Caritas Lebanon. The findings show that BALADI/RMF was mostly aligned to support the Country Development Cooperation Strategy (CDCS) and its interventions mostly met the relevant needs of supported communities. The headline achievement included supporting municipalities to provide basic electric, water, education, and health services to residents and to ensure an enabling environment for economic activity. Interventions intended to promote improvements in livelihoods had mixed outcomes as of the date of this report. The Lebanese economic crisis and the COVID-19 pandemic threaten the sustainability of interventions in the agri-business and ecotourism fields. Women’s involvement in the interventions was limited. The BALADI activity sought to increase cooperation between municipalities and local civil society organizations (CSOs) and private businesses; while it strengthened existing partnerships, there was little evidence that new partnerships were established. It supported municipalities in learning how to implement economic development projects and deliver essential services. The evaluation provides several recommendations for USAID and RMF to implement in the final 16 months of BALADI as well as in future USAID programming.
TABLE OF CONTENTS TABLE OF CONTENTS ...... II ACRONYMS ...... III EXECUTIVE SUMMARY ...... IV Introduction ...... iv Evaluation Design and Methods ...... iv Key Findings and Conclusions ...... v Recommendations ...... vi INTRODUCTION ...... 1 Project Description ...... 1 Context ...... 1 Development Hypothesis ...... 3 Critical Assumptions ...... 3 EVALUATION PURPOSE AND QUESTIONS ...... 5 Purpose ...... 5 Evaluation Questions ...... 6 EVALUATION METHODS AND LIMITATIONS ...... 8 FINDINGS AND CONCLUSIONS ...... 9 Question 1 ...... 9 Question 2 ...... 11 Question 3 ...... 19 Question 4 ...... 21 Question 5 ...... 23 RECOMMENDATIONS ...... 26 ANNEXES ...... 27 Annex A: Inception Report and Scope of Work ...... 28 Annex B: Detailed Evaluation Design and Methodology ...... 29 Annex C: Evaluation Schedule and Persons Interviewed ...... 39 Annex D: Documents Reviewed ...... 40 Annex E: Data Collection Instruments ...... 41 Annex F: Conflict of Interest Disclosures ...... 45 Annex G: Evaluation Team Members ...... 46
USAID.GOV ii | BALADI/RMF INTERIM PERFORMANCE EVALUATION
ACRONYMS ADS Automated Directives System AOR Agreement Officer Representative BALADI Building Alliance for Local Advancement, Development, and Investment BALADI-CAP BALADI Capacity Building Program BIAT Business Incubation Association in Tripoli CA Cooperative Agreement CBO Community Based Organization CDCS Country Development Cooperation Strategy CL Caritas Lebanon CLA Collaborating, Learning, and Adapting COVID-19 Coronavirus Disease of 2019 CPC Community Project Committee CSO Civil Society Organization DCOP Deputy Chief of Party DEC Development Experience Clearinghouse DO Development Objective EQ Evaluation Question EQUITM Evaluation Quality Use and Impact ET Evaluation Team FTE Full-Time Equivalent IP Implementing Partner KII Key Informant Interview MEL Monitoring, Evaluation, and Learning NGO Non-Governmental Organization PMP Performance Management Plan PMSPL II Performance Management and Support Program for Lebanon PPP Public Private Partnership QA/QC Quality Assurance/Quality Control RMF Rene Moawad Foundation SOPs Standard Operating Procedures SOW Scope of Work SPD Supplemental Program Description USAID United States Agency for International Development USG United States Government
iii | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
EXECUTIVE SUMMARY
INTRODUCTION This interim performance evaluation assesses the activities of Building Alliances for Local Advancement, Development, and Investment - René Moawad Foundation (BALADI/RMF) for their relevance, effectiveness, efficiency, and sustainability, enabling the United States Agency for International Development (USAID) to implement needed adjustments. The evaluation questions (EQs) were:
1. To what extent is BALADI/RMF contributing to Development Objectives 1 and 2 under USAID’s CDCS for Lebanon? 2. To what extent has BALADI/RMF achieved the activity’s goal and objectives, and what factors have contributed or hindered the achievement (or non-achievement) of the activity’s objectives? 3. At what level were women involved in the supported communities where the activities were implemented? 4. What is the likelihood that the interventions and results BALADI/RMF has achieved are sustainable beyond the life of the activity? 5. USAID does not often have opportunities to compare two activities as similar as BALADI/Caritas Lebanon and BALADI/RMF. How can the two activities be compared and contrasted with regard to the extent to which their objectives and outcomes were achieved? The BALADI project was designed in 2011 and launched in 2012. It was integrated in the Lebanon 2014- 2019 Country Development Cooperative Strategy (CDCS) to support its Development Objective 1 (DO1): Public sector capacity to deliver services improved through the achievement of Intermediate Result 1.3 (IR 1.3), and DO2: Economic opportunity enhanced through the achievement of IR 2.1. The activity was extended until the end of January 2022 and now has a total budget of $28 million. As of the date of this report, RMF is implementing the BALADI activity in 61 municipalities in 16 Lebanese districts. Activities include renewable energy, traditional energy, water infrastructure, socio-cultural1, youth, education, health, eco/rural tourism, and agriculture/agri-business projects. The project’s goal is to achieve long- term collaboration between municipalities or unions, non-governmental organizations (NGOs), community-based organizations (CBOs), and private-sector entities to implement and manage community projects and deliver services. BALADI has two activity purposes: 1. Improve governance of municipalities and improve public service delivery; and 2. Establish sustainable local social capital.
EVALUATION DESIGN AND METHODS The evaluation was conducted from July to October 2020, during the Coronavirus Diseases of 2019 (COVID-19) pandemic period while Lebanon was witnessing a surge in the official number of confirmed cases. Places of business were not completely open, group gatherings were restricted, and practice of social distancing was in place. Also, international travel was not possible, and movements were restricted within the country. In this context, the evaluation team (ET) designed this evaluation using a mixed methods approach, comprising remotely collected qualitative and quantitative data. Qualitative methods included document review and remote key informant interviews (KIIs). That information was
1 RMF lists five interventions under “Socio-Cultural/Youth” (Abra, Douma, Yahchouch, Toula, and Mazraat Shouf) and one as “Education/Socio-Cultural” (Hrar). Douma and Yahchouch include rural tourism elements.
USAID.GOV iv | BALADI/RMF INTERIM PERFORMANCE EVALUATION
supplemented with quantitative data from activity monitoring, but no primary quantitative data were collected. These approaches allowed the ET adequate information to assess the activity’s implementation and results against its objectives, scope of work (SOW), deliverables, work plan, etc.
KEY FINDINGS AND CONCLUSIONS
QUESTION 1: TO WHAT EXTENT IS BALADI/RMF CONTRIBUTING TO DEVELOPMENT OBJECTIVES ONE AND TWO UNDER USAID’S CDCS FOR LEBANON? The Project Goal, Project Purpose 1, and Activity IR 1.1 are aligned to support DO 1 of the CDCS. Activity IR 1.1 and IR 1.3 are aligned to support DO 2, and the Project Goal is in line with Mission IR 2.1. However, Project Purpose 2 and Activity IR 1.2 are not aligned with the CDCS. BALADI/RMF projects addressed the relevant needs in the supported communities at the time that the municipal projects were proposed.
QUESTION 2: TO WHAT EXTENT HAS BALADI/RMF ACHIEVED THE ACTIVITY’S GOAL AND OBJECTIVES? Improved delivery of essential public services such as electricity, water, and health care in 38 municipalities is the headline achievement of BALADI/RMF. The activity had a limited effect on improving municipal governance. USAID and RMF officials said the main contribution of BALADI/RMF to building more capable municipalities was its support in the bidding and project implementation phases. “Our role was not municipal capacity building; it was implementation of awards,” an RMF official said. The activity’s requirement of Community Project Committees (CPCs) to manage interventions did enhance participatory government, but the CPCs will not continue beyond the life of the activity. The ET found that many sampled municipalities regarded CPCs as a box to be ticked to qualify for a project, rather than a vehicle to sustain long-term collaboration.
BALADI/RMF did not create significant sustainable local social capital. Its main contribution to economic growth and private-sector competitiveness is indirect in that provision of essential services improved the business environment, which can enable private-sector economic growth, and it facilitated economic activity in municipalities.
The types of interventions that provided essential services – traditional energy, renewable energy, water, and health care – are the most effective at providing services to communities, and they indirectly improve livelihoods by improving the environment for businesses. Interventions that primarily aimed to improve livelihoods – like agri-business and ecotourism – fell short of job-creation goals. The ET found that only two of seven agri-business projects were operational by March 2020, while ecotourism jobs were part-time and seasonal.
BALADI/RMF strengthened and expanded existing collaborations between municipalities, the private sector and NGOs but it did not establish new, long-term collaboration. In instances where BALADI/RMF municipalities were open to civil-society collaboration, previous collaboration already existed, according to mayors and local CSOs. Conversely, such collaboration was not found in municipalities where it had not previously existed. Collaboration was more suited to projects like agri- business, ecotourism, and community socio-cultural centers.
BALADI/RMF’s communication and outreach efforts generated widespread awareness of the activity, and branding efforts successfully built goodwill toward the U.S. government.
v | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
RMF’s experience and relationships with municipalities and its technical know-how are the main factors that contributed to the achievements. Hindering factors included the Syrian refugee crisis starting in 2011, security risks throughout the duration of the award, the local uprising starting October 2019, the Lebanese economic crisis starting October 2019, and the COVID-19 pandemic starting March 2020.
QUESTION 3: AT WHAT LEVEL WERE WOMEN INVOLVED IN THE SUPPORTED COMMUNITIES WHERE THE ACTIVITIES WERE IMPLEMENTED? Women’s involvement varied considerably by project type. During the identification process, women participated in the proposal writing. They also participated in the implementation process as they took part in the steering committees, but it was men (mayors) who took the main decisions because of the absence of women mayors. Women who contributed to BALADI/RMF interventions had already been active in their communities through similar activities. Therefore, women’s involvement is not attributable to RMF efforts. RMF did not play a significant role in helping communities to identify new fields of intervention for women. The selection of women and their level of their involvement in the project were left to municipalities, and RMF was not involved in identifying CPC members.
QUESTION 4: WHAT IS THE LIKELIHOOD THAT THE INTERVENTIONS AND RESULTS BALADI/RMF HAS ACHIEVED ARE SUSTAINABLE BEYOND THE LIFE OF THE ACTIVITY? Essential services are highly likely to be sustained, both because they meet an important need for communities and because they will be able to generate enough revenue to cover their costs. Sustainability of other intervention types like agri-business is more doubtful. Agri-businesses should become sustainable through the sale of their products, but most examined interventions were not yet operational as of the date of the evaluation. Identifying sustainable operational partners is essential to sustainability. Energy, water, and health projects generated modest numbers of permanent, sustainable jobs. Their primary contribution to employment was the provision of basic services that enabled job creation in the private sector. Other jobs directly created by BALADI/RMF, both permanent and seasonal, are more tenuous.
QUESTION 5: HOW CAN BALADI/CARITAS LEBANON AND BALADI/RMF BE COMPARED AND CONTRASTED WITH REGARD TO THE EXTENT TO WHICH THEIR OBJECTIVES AND OUTCOMES WERE ACHIEVED? Both RMF and Caritas Lebanon (CL) faced different challenges that caused delays to the implementation of the awards. Each successfully addressed those challenges and both organizations learned and adapted the processes to overcome those challenges and reach a relatively successful implementation.
RECOMMENDATIONS 1. USAID should encourage the creation and implementation of long-term local economic development plans as part of its municipal capacity-building efforts. (EQ2) 2. USAID should better coordinate its implementation of municipal infrastructure and capacity- building activities. (EQ2) 3. USAID and RMF should focus municipal energy projects on renewable electricity, with net metering included. (EQs 2 and 4) 4. USAID should increase collaboration between municipalities and civil society through projects that provide wide community benefit, rather than projects that aim to directly create jobs. (EQs 2 and 4)
USAID.GOV vi | BALADI/RMF INTERIM PERFORMANCE EVALUATION
5. RMF should develop standard operating procedures (SOPs) for CPCs to encourage women’s roles. (EQs 2 and 3) 6. When identifying local entities as implementing partners, USAID should look at organizations that have prior experience implementing the types of interventions envisioned in the activity. (EQs 2 and 5) 7. USAID should raise awareness of private-sector partnerships (PPPs) among municipalities and should ensure that municipalities are aware of applicable laws governing PPPs and ways that they can utilize PPPs. (EQs 2 and 4) 8. RMF should include a gender specialist to develop a gender-analysis plan for the whole life of the project. (EQ3)
vii | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
INTRODUCTION
The United States Agency for International Development (USAID) awarded two cooperative agreements to René Moawad Foundation (RMF)—CA No.AID-268-A-12-00004—and Caritas Lebanon (CL)—CA No. AID-268-A-12-00005—to implement the Building Alliance for Local Advancement, Development and Investment (BALADI) Core project.
Initially, both the core BALADI Cooperative Agreements (CA) had a combined value of $27 million over five years; from September 30, 2012 to September 29, 2017. BALADI Core aims at assisting municipalities in overcoming the financial and human resource constraints that get in the way of implementing community development projects. It set out to achieve demonstrated long-term collaboration between municipalities or unions of municipalities, nongovernmental organizations (NGOs), community-based organizations (CBOs), and private-sector entities to implement and manage community projects and deliver services in a gender-balanced manner.
The RMF BALADI activity was extended in May 2020 until the end of January 2022 and now has a total budget of $28,378,088. As of the date of this evaluation, RMF is implementing the BALADI activity in 61 municipalities in the following 16 Lebanese districts: Kesrwan, Jbeil, El- Shouf, Baabda, Aley, Koura, Bcharre, Zgharta, Mennieh-Dennieh, Batroun, Akkar, Jezzine, Saida, Hasbaya, Baalback and Zahle. In March 2020 RMF confirmed that it would begin activities in additional municipalities in Nabatiyeh, Sour, Bint Jbeil and Marjeyoun districts under a recent initiative targeting southern Lebanon. RMF’s activities include renewable energy, traditional energy, water infrastructure, socio-cultural, youth, education, health, ecotourism, and agriculture/agri-business projects.
PROJECT DESCRIPTION Activity Name Building Alliance for Local Advancement, Development, and Investment – René Moawad Foundation Acronym BALADI/RMF Start Date September 30, 2012 End Date January 31, 2022 Budget USD $28,378,088 Awardee René Moawad Foundation (RMF) Cooperative Agreement # AID-268-A-12-00004
CONTEXT The BALADI project was designed in 2011 and launched in 2012 under the USAID Forward initiative. It was integrated in Lebanon’s 2014-2019 Country Development Cooperative Strategy (CDCS) to support Development Objective (DO) 1, through the achievement of Intermediate Result (IR) 1.3 and DO 2 through the achievement of IR 2.1. (See results framework below.) In particular, the project encourages service delivery by local governments by supporting municipalities and municipal unions nationwide to address citizen governance and economic needs.
1 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
The two BALADI implementers, RMF and Caritas Lebanon (CL), solicit applications from municipalities and unions of municipalities for village-community development projects. The selected development projects are implemented directly by the BALADI implementer or with the help of its consortium; therefore, municipalities and unions of municipalities receive only in-kind support to implement their proposed projects. In-kind support is in the form of equipment, small-scale construction, and technical assistance and training related to the supported interventions. The implementing consortium does not provide cash grants to municipalities or unions of municipalities.
The shortlisting and selection of municipal interventions for support is made based on a set of selection criteria that includes the expected social and economic impact, whether the areas have benefitted from past USAID (or other donor) programming or offer new, viable partnership opportunities, and the operational sustainability of the project. BALADI-selected municipal projects are geographically widespread to provide good distribution of USAID support across many districts.
Operation and sustainability plans of each supported intervention are analyzed during the intervention selection and detailed design process; actual operation and sustainability is the responsibility of the community and/or the municipality. All BALADI/RMF-supported interventions have locally owned and managed sustainability plans.
The activity objectives and intended results are listed below in the discussion of the findings to EQ1.
USAID.GOV 2 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
DEVELOPMENT HYPOTHESIS The current structure and function of the central government in Lebanon inhibits complete decentralized decision-making, service provision, and economic development activities. As a result, scarce resources do not address the service needs of communities, resulting in sub-standard public goods. Moreover, pluralistic participation in local decision-making processes is not encouraged, thus limiting the effectiveness of the public services offered by local governments. Municipalities are generally eager to meet the service delivery needs of their constituents, but face a host of challenges, including lack of funds, personnel, and capacity. Municipalities face additional challenges due to the large influx of refugees since 2011 placing heavy burdens on “host communities” with respect to service delivery and supporting local economic development. Municipalities face serious resource strain in supporting service delivery and need better and more effective coordination and support from civil society organizations (CSOs). The slow growth of the Lebanese economy has significantly impacted the local economic growth and opportunities, which are exacerbated by the Syrian crises and refugee influx, placing further stress on local municipalities. Because almost all municipalities in Lebanon are dominated by men, this impedes the access of women to decision-making at the municipal level and their active participation in the development of their own communities.
BALADI/RMF offers an opportunity for proactive, innovative, well-led municipalities, unions of municipalities, and village clusters to address critical resource constraints for service delivery. Through BALADI/RMF, USAID seeks to encourage innovation in service delivery by local governments by sponsoring a grants program managed by local NGOs to fund proposals by municipalities and unions of municipalities that respond to the governance and economic opportunity needs of citizens. This program tests the hypothesis that many municipalities have outstanding leadership but lack the financial capital and gender balance to realize their visions and implement participatory development projects.
CRITICAL ASSUMPTIONS Below are the critical assumptions listed in the BALADI’s monitoring, evaluation, and learning (MEL) plan:
1. USG will continue to partner with the local governments to support the improvement of its local governance capacity and the empowerment of the municipalities and union of municipalities. The existing policies governing engagement with some parties and officials will not be made more restrictive. 2. There is a persistent risk of violence. If conflict flares up, then activities in the affected areas would be temporarily suspended and possibly modified to respond to changing needs. A critical assumption is that Lebanon will remain stable enough to permit access to rural communities in all areas of the country where project implementation is ongoing. 3. Another critical assumption is that the funded projects (activities) proposed by the municipalities will be designed to respond at the same time to citizens needs and to CDCS objectives, and while BALADI Project is to report against a defined set of indicators, not all the selected activities may be related to these same indicators. 4. The activities proposed by the municipalities will be well maintained to ensure sustainability. It is critical that BALADI team deploys enough efforts to support the implementation and operation of the funded projects. 5. The outreach activity will be extended to encourage many local populations/ municipalities/diasporas to participate and to become aware of the project’s scope and benefits. 6. Personal interests of the participants in USG-supported activities may compromise the
3 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
achievement of the project goals and objectives and will make development goals difficult to achieve and sustain. The selection process should ensure equitable screening and selection of projects focusing on transparency of the beneficiaries. Continuing donor and BALADI team supervision and coordination is critical for ensuring that economic growth objectives are achieved.
USAID.GOV 4 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
EVALUATION PURPOSE AND QUESTIONS
PURPOSE With BALADI/RMF reaching its end of term on January 31, 2022, its Agreement Officer Representative (AOR) requested that Performance Management and Support Program for Lebanon (PMSPL II) conduct an interim performance evaluation2 to:
• Assess the activity for relevance, effectiveness, efficiency, and sustainability. • Provide input on the activity’s approach in terms of achieving outputs, outcomes, and stated objectives. • Assess the extent of achievement of BALADI/RMF’s objectives, evaluating its outcomes; documenting its successes, challenges, and lessons learned for future programs; and analyzing interventions’ sustainability for future programming. • Evaluate the activity’s impact on empowering municipalities to implement needed local development activities that improve the social and economic conditions in communities, with special focus on host community needs.
This evaluation also addressed the gender aspect as a crosscutting element in all activities, showing to what extent BALADI/RMF has adhered to USAID’s gender integration requirements.
In the spirit of the Collaboration, Learning, and Adaptation (CLA) approach, the evaluation also provides recommendations for programming for future USAID/Lebanon project designs. Activity stakeholders were included in the review process and asked to participate in making recommendations and reviewing the findings of this evaluation.
The evaluation was conducted with adherence to Social Impact’s Evaluation Quality Use and Impact (EQUI™) approach, processes, and protocols. EQUI processes are designed to achieve a quality evaluation that incorporates learning and utilization. Based on EQUI, this evaluation has a utilization focus. In addition to assessing program performance in achieving planned objectives, the evaluation highlights lessons learned and provides recommendations, taking into consideration future programs in Lebanon. Aligned with this mission, Social Impact will continue to actively engage USAID, implementing partners (IPs), and other stakeholders in the evaluation process.
The evaluation began in July 2020 and was completed in October 2020. A six-day work week is authorized for in-country fieldwork. The evaluation team (ET) included an expatriate Team Leader and two local specialists with expertise in local governance, community development, and public administration. The ET was supported by PMSPL II’s staff in Lebanon and Social Impact’s Headquarters in the US, as needed.
The ET anticipates that the results from the evaluation will be used by USAID/Lebanon during its annual Portfolio Review. The final evaluation report will become publicly available on the Development Experience Clearinghouse (DEC).
2 “A performance evaluation focuses on descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and to answer other questions that are pertinent to future program design, management and operational decision making” (USAID Evaluation Policy, 2011).
5 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
EVALUATION QUESTIONS The ET conducted meetings with USAID and RMF during the evaluation’s kick-off in order to determine priorities and add clarity to the evaluation questions (EQs) based on the Mission’s intended use of the evaluation findings, conclusions, and recommendations.
In accordance with Automated Directives System (ADS) 201.3.5.12, the EQs that were communicated as part of the scope of work (SOW) were organized according to the following areas.
RELEVANCE 1) Main Question: To what extent BALADI/RMF is contributing to DO 1 and 2 under USAID’s CDCS for Lebanon? Sub-questions: a) To what extent are the BALADI/RMF goals, purpose and objectives/results aligned to support the USAID CDCS? b) Did the awarded municipal projects answer and/or address the relevant needs in the supported communities?
EFFECTIVNESS 2) Main Question: To what extent has BALADI/RMF achieved the activity’s goal and objectives, and what factors have contributed or hindered the achievement (or non-achievement) of the activity’s objectives? Sub-questions: a) What are BALADI/RMF’s headline outcomes and achievements, and to what extent were these outcomes in-line with the stated purposes. b) What were the types of interventions chosen by the communities and assisted by BALADI RMF that were most effective in terms of providing services to communities, or promote improvements in their livelihood? c) What are the main factors that influenced positively or negatively BALADI/RMF’s ability to achieve these purposes? d) How successful was BALADI/RMF in creating long term collaboration between sub-national public entities, CSOs, and private sector to implement the awarded projects and deliver services?
e) How satisfied are communities with BALADI/RMF activities, and to what extent did the activities generate goodwill for the municipalities, the USAID Mission, and the U.S. Embassy?
GENDER 3) Main Question: At what level were women involved in the supported communities where the activities were implemented?
USAID.GOV 6 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
Sub-questions: a) Did women actively participate in the identification and the implementation of local development activities? b) What aspects of the program demonstrate women sustainable participation beyond the life of the activity?
SUSTAINABILITY 4) Main Question: What is the likelihood that the interventions and results BALADI/RMF has achieved are sustainable beyond the life of the activity? Sub-questions: a) Which types of interventions and results are or show the most prospect of being sustained and why? b) How sustainable are the permanent and seasonal jobs created by BALADI/RMF?
COMPARISON TO BALADI/CARITAS 5) Main Question: USAID does not often have opportunities to compare two activities as similar as BALADI/Caritas and BALADI/RMF. How can the two activities be compared and contrasted with regard to the extent to which their objectives and outcomes were achieved? Sub-questions: a) What were the internal or external factors that caused significant differences in topline results between BALADI/Caritas and BALADI/RMF? b) What were the successes as well as lessons learned from BALADI/RMF?
Based on the findings and conclusions to the EQs, the ET has provided recommendations on how similar projects could build on the successes of BALADI/RMF and how their designs and implementation could be enhanced based on the lessons learned from the program.
7 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
EVALUATION METHODS AND LIMITATIONS
The evaluation was conducted during a pandemic period and while Lebanon was witnessing a surge in the official number of COVID-19 confirmed cases. Places of business were not completely open, group gatherings were restricted, and a practice of social distancing was in place. Also, international travel was not possible, and movements were restricted within the country. In this context, the ET designed this evaluation using a mixed methods approach, comprising remotely collected qualitative and quantitative data. Qualitative methods included document review and 52 remote key informant interviews (KIIs). This information was supplemented with quantitative data available from activity monitoring data, but no primary quantitative data were collected. This evaluation also relied on the review of activity documentation (e.g. request for application-RFA, work plan, quarterly reports, annual report, technical reports, activity website, etc.). The EQs were mapped against data sources, collection methods, and analysis methods and shown in Annex B.
These approaches allowed the ET adequate information to address the EQs.
A more detailed description of the evaluation design and methodology is found in Annex B. The 52 key informants are divided as follows:
STAKEHOLDER # USAID/Lebanon 3 RMF/Sub-awardees 14 Municipal officials 16 Community stakeholders 17 Other implementers 2 Total 52
USAID.GOV 8 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
FINDINGS AND CONCLUSIONS
QUESTION 1 To what extent is BALADI/RMF contributing to DO 1 and 2 under USAID’s CDCS for Lebanon?
FINDINGS Sub-Question 1A. To what extent are the BALADI/RMF goal, purposes and objectives/results aligned to support the USAID CDCS?
BALADI/RMF goals, purposes, and intermediate results (IRs), are intended to support achievement of the following CDCS development objectives (DOs), IRs and sub-IRs (see above for full results framework):
DO 1: Improved capacity of the public sector in providing transparent and quality services across Lebanon.
IR 1.3: Improved governance in addressing citizens’ needs in public service delivery
• Sub-IR 1.3.1: More capable municipalities working inclusively with citizens to effectively accomplish local development objectives DO 2: Inclusive economic growth enhanced
IR 2.1: Increased private-sector competitiveness
Activity Intended Results Project Goal: Long-term collaboration between municipalities or unions, NGOs, CBOs, and private sector entities to implement and manage community projects and deliver services.
Project purpose: 1. Improve governance of municipalities and improve public service delivery. 2. Establish sustainable local social capital
Activity IR1: More capable municipalities working inclusively with citizens to effectively accomplish local development objectives Activity IR 2: Enhanced participatory decentralized governance to act as catalyst for social capital development Activity IR 3: Broad-based development through support of locally championed plans and reforms
The ET found that the Project Goal and Project Purpose 1 were aligned with DO 1 and Mission IR 1.3, all of which address improving public-sector capacity, public-service delivery, and governance. Furthermore, Activity IR 1 is Mission Sub-IR 1.3.1, verbatim. The Project Goal’s reference to collaboration with private-sector entities tangentially references Mission IR 2.1, and Activity IR 3’s focus on development is in line with DO 2’s call for enhancing economic growth.
The ET found that Project Purpose 2 was not in line with the CDCS. While some economists have argued that increased social capital – i.e., features of social organization such as networks, norms, and
9 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
social trust that facilitate coordination and cooperation for mutual benefit – contributes to economic growth3, the ET found that Purpose 2 had a tenuous link at best to DO 1 or DO 2. Likewise, Activity IR 2 aims to promote both social capital development and decentralized governance. While the latter result may improve governance in a country facing challenges in its national government, decentralization is not an explicit objective of the CDCS results framework.
Sub-Question 1B. Did the awarded municipal projects answer and/or address the relevant needs in the supported communities?
Fourteen mayors interviewed as part of the 16-municipality sample said the BALADI/RMF projects addressed relevant needs in their community. In communities where the projects addressed basic utilities, such as back-up electricity and water, mayors and other officials were definitive in declaring that the projects met the essential needs of their communities. However, one municipal official stated that an agricultural irrigation project was designed so as to benefit prominent officials and landowners of the municipality, rather than the broader community.
Officials from communities with other types of projects – agri-businesses, socio-cultural facilities, or ecotourism activities – were more equivocal. For example, while two officials noted that community centers met a need of their municipalities, their comments were along the lines of “any project can be of good use for the area.” They said that, since the conception of the project, the pressing needs of their municipalities had shifted to more basic services, like electricity, water, and health care. They and other officials said delays in transfer payments from the national government made foreign assistance more important for municipalities to be able to meet essential needs.
This finding was reflected in some ecotourism and agri-business facilities not being operational as of the date of the evaluation. While the cold storage facility in Ehmej meets community needs, according to users and other stakeholders, similar projects in four other municipalities in the 16-municipality sample were not operational as of September 2020. Three of those projects involved partnerships with private- sector entities. The ET found that getting those projects up and running was not a priority for municipal officials and inferred that this was because the projects did not meet the current needs for the community. In some cases, BALADI/RMF projects complemented economic development plans of municipalities; for example, the ecotourism projects in Bcharre and Barouk were part of their broader strategy. Six interviewed mayors said their municipalities conducted a formal needs assessment.
This shift in municipal priorities was evident in the evolution of the projects awarded by BALADI/RMF. Of the nine projects in Round 1, only one focused on water, one on health, and none on electricity. Of the 20 projects in Round 2, six were renewable energy and three were water infrastructure. Of the 31 projects in Round 3, 21 were renewable or traditional energy (including hot-water heaters) and six were health activities.
According to local respondents, the selection of projects to be funded by BALADI/RMF usually came from the municipalities themselves and reflected the priorities of the mayor. Of the 16 municipalities sampled by the ET, there were only four instances where local CSOs proposed the projects, three of them in North Lebanon, and all in Round 2. Stakeholders, including local CSOs, operators and residents,
3 Robert D. Putnam, Making Democracy Work (Westport, Conn.: Greenwood, 1992), p. 167. See Putnam’s footnotes for additional authors supporting this theory.
USAID.GOV 10 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
said their communities were invested in the projects. “There is an ownership among the citizens,” said one local CSO official.
BALADI/RMF has procedures in place to ensure that projects meet the relevant needs of communities. First, RMF requires a statement from the municipality on how the proposed project would meet its needs, including a description of how the community was consulted. RMF staff then engage in ground- truthing the project, ensuring that it is feasible and speaking with residents about the proposal. “We have a questionnaire and meet with people,” one RMF field officer said. “We ask if the project will benefit the village all over.”
While most projects did meet the needs of the communities, there is some evidence that the type of project proposed was influenced by a perception of the type of project that RMF was likely to fund. Based on interviews with RMF field staff, other implementers, and municipal officials, the ET found evidence that funded projects did not always address the primary need of communities and that some communities wrote their proposals based on successful proposals from other communities, rather than the main needs of their own communities. One RMF field staffer said a community proposal to build an auto repair shop was rejected as being outside the scope of BALADI. Similarly, local officials told the ET that their primary needs were solid-waste disposal and sewage, but those are also outside BALADI’s scope. “The BALADI approach is that when communities hear that another municipality won a BALADI project, they change their application to a project that won. That's why you have a lot of solar projects,” said one USAID implementer.
CONCLUSIONS The Project Goal, Project Purpose 1, and Activity IR 1 are aligned to support DO 1 of the CDCS. Activity IR 1 and IR 3 are aligned to support DO 2, and the Project Goal is line with Mission IR 2.1. Project Purpose 2 and Activity IR 2 are not aligned with the CDCS.
BALADI/RMF projects addressed the relevant needs in the supported communities at the time that the municipal projects were proposed. The most pressing needs of Lebanese municipalities have shifted in the first eight years of BALADI/RMF, due to factors like stress on public services from Syrian refugees and poor electricity supply from the central government. As a result, some earlier projects are less relevant today and less of a priority for those municipalities. Essential needs like electricity, water, and health care are the most relevant in 2020.
QUESTION 2 To what extent has BALADI/RMF achieved the activity’s goal and objectives, and what factors have contributed to or hindered the achievement (or non-achievement) of the activity’s objectives?
FINDINGS Sub-Question 2A. What are BALADI/RMF’s headline outcomes and achievements, and to what extent were these outcomes in line with the stated purposes?
The ET found that the primary achievement of BALADI/RMF was to support municipalities to provide essential public services to residents. Of the 60 interventions in the first three rounds, 38 provided renewable energy, traditional energy, water, or health care.
11 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
Public Service Delivery According to RMF reporting data from March 2020, BALADI/RMF had a total of 228,875 direct beneficiaries (110,630 M, 118,245 F). This figure includes 118,823 visitors to facilities, 108,639 beneficiaries of improved services, and 1,413 training recipients. A USAID official said, “The most significant outcome is having municipalities provide services. That's a significant outcome.” An RMF official said, “[The interventions] were a response to increase the services of the municipality. The municipality got the capacity to provide services. It definitely improved services.”
Local stakeholders – municipal officials, local CSOs, operators and beneficiaries – said public service delivery did improve thanks to BALADI/RMF. One mayor whose municipality received solar-water heaters and LED streetlights said, “It helped us as a municipality to address the needs of the people that we couldn’t address or achieve alone.” Another mayor whose village received photovoltaic panels for backup electricity generation said, “The current situation of the public electricity forces the municipality to seek alternatives.” A mayor whose municipality benefitted from a water project said, “Without this implemented project and without this collaboration between the municipality and the water establishment, the village and the citizens will suffer a lot from lack of water supply.”
Inclusively Working with Citizens The ET found less evidence that RMF achieved other goals or results related to inclusion, municipal capacity, economic growth, private-sector competitiveness, and social capital.
The primary vehicle that BALADI/RMF used to promote inclusive municipal governance was the Community Project Committee (CPC), which brings together members of the municipal council and members of the community, including local CSOs and private businesses, to manage the municipality’s project. According to RMF staff, 668 individuals participated in CPCs in the 60 municipalities and union of municipalities, 185 of whom were women (28 percent). RMF had a 20-percent quota for women and youth on each CPC, but this was sometimes ignored by individual municipalities. Of the 16 municipalities in the evaluation sample, one had no women involved.
As a result, the ET found that the degree of inclusion varied by community. For example, a mayor in Akkar said the CPC included a CSO that wrote the proposal and youth representatives. Similarly, a mayor in North Lebanon said the 12-member CPC included women, though she noted that the municipality “did this committee only to respond to the requirement of the project” and there was no further need for it. Conversely, a mayor in the Bekaa Valley said that the CPC had only four members, all from the municipal council, and no women or civil society representatives. The ET found that later interventions generally had less community involvement, most likely because they were focused on infrastructure like generators, streetlights, and water pumps with little need for community involvement or day-to-day management.
More Capable Municipalities USAID and RMF officials said the main contribution of BALADI/RMF to building more capable municipalities was its support in the bidding and project implementation phases. “Our capacity building was what was needed for the success of the project itself. Our role was not municipal capacity building; it was implementation of awards,” an RMF official said.
These officials also said that BALADI/RMF focused less on building the capacity of municipalities than did its sister activities, BALADI Plus and BALADI-Capacity Building Program (BALADI-CAP). BALADI Plus in 2013 and 2014 offered assistance to municipalities that had been selected for awards under the BALADI program, focusing on improving the financial, administrative, and regulatory systems at local municipalities. BALADI-CAP built municipal and CSO capacity from 2013 to 2018.
USAID.GOV 12 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
“BALADI-CAP was supposed to handle capacity building,” a USAID official said. “We should have integrated it more with BALADI/RMF. We segregated the two. BALADI did bricks and mortar; BALADI- CAP did capacity building.”
Economic Growth According to data in the March 2020 progress report, the most recent report made available to the ET, BALADI RMF created 266 jobs (174 men, 92 women) in the 60 municipalities through March 2020, short of the target of 610. Many of those were part-time jobs; the full-time equivalent (FTE) is 55 full- time jobs. BALADI/RMF leveraged private-sector investment of $467,884, according to performance management plan (PMP) data.
Of the 266 jobs created, 223 were classified as short-term and 43 as long-term. The overwhelming majority were created in Mount Lebanon (210). Job creation in other governorates included 32 in North Lebanon, five in Akkar, and 19 in South Lebanon. No jobs were reported to have been created in the Bekaa Valley, where RMF projects focused on service delivery.
The ET found that interventions were not operating yet in four of the 16 municipalities it examined. Three of these were focused on economic growth: two food-processing facilities and one eco-tourism project. The fourth is an educational/socio-cultural facility.
USAID and RMF officials pointed out that job creation was a secondary result of BALADI/RMF, since service delivery is the primary objective. They also noted that BALADI/RMF provided indirect economic benefits such as:
• Cost savings to municipalities from lower costs for utilities, saving funds that can be used for other purposes (one municipality used savings to provide scholarships to low-income students). • Cost savings to individuals benefitting from lower bills, which they can redirect to other purchases. • Creating an enabling environment for economic activity by providing reliable electricity and water for businesses and providing a safe environment for economic activity, e.g., streetlights enabling shops to stay open past dark. • Some indirect economic benefits were reported by officials of municipalities that received ecotourism or recreational facilities, as tourists or visiting sports teams purchased meals or goods in local shops.
Sustainable Local Social Capital The ET found little evidence that BALADI/RMF achieved Project Purpose 2: Establish sustainable local social capital. Because social capital is a broad concept, the ET selected one element of social capital – trust – to use as a proxy when interviewing municipal officials and other stakeholders. Only one of the 16 municipal officials interviewed stated that trust had improved: “It helped in building trust between the municipality and the village community,” said the mayor from North Lebanon. An official in Akkar found the opposite occurred: “At a certain point, and because of the project, the residents felt repulsed by the municipality because the selection of land was done without consultation with them,” he said.
Sub-Question 2B. What were the types of interventions chosen by the communities and assisted by BALADI/RMF that were most effective in terms of providing services to community or promoting improvements in their livelihoods?
13 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
Services As noted above, the ET found that interventions that provided essential services to communities were the most effective. BALADI/RMF’s renewable energy, traditional energy, water, and health care interventions supported municipalities in providing these basic services to residents. In addition to service delivery, USAID and RMF personnel said that the energy and water projects contributed to the enabling environment for economic activity, as described above. RMF officials noted that these projects were not major drivers of employment, with a handful of individuals employed in the construction phase and perhaps one or two maintenance employees. Health-care projects showed similar results, as the main result was service delivery, with one or two technicians hired to operate and maintain equipment.
“What we were looking at is how to get municipalities to create the enabling environment for businesses to create jobs. It's the job of the municipality to create the environment, not to create jobs,” one USAID official said. Another stated, “The projections weren't that accurate. If we wanted to create 10 or 12 jobs, we created three or four. You have to look at the number of beneficiaries. It's not about creating jobs if you benefit the village, directly or indirectly.”
While renewable electricity projects helped municipalities meet the energy needs of their communities, municipal officials said those projects were not as impactful as they could be because they lacked storage batteries. RMF omitted batteries and opted for hybrid systems, including solar farms coupled with generation system, because batteries would have added to costs, both the up-front cost for BALADI/RMF and high, ongoing battery-replacement costs for the municipalities which most municipalities are unable to sustain. Another type of solar energy project – water heaters – brought significant savings to households that received the heaters, local officials said, but only benefitted a small percentage of households in a village; e.g., 65 of 800 (8 percent) households in Kfardlakos, where energy-efficient streetlights were also installed and benefitted the entire village.
Livelihoods BALADI/RMF does not report PMP jobs data by intervention type, so the ET does not have hard data on which types of interventions were most effective at job creation. BALADI/RMF does report jobs data by district, and a single district – Jbeil – accounted for 150 of the 266 jobs created by the project. There are three interventions in the Jbeil district: one cold-storage facility and two ecotourism projects, one of which was operational as of the date of the evaluation. Another 45 jobs were created in the Shouf district, where three ecotourism interventions, one water project and one health facility were established. Fewer than 20 jobs were created in any of the other districts. One RMF manager said, “The expectations of job creation were higher than were realized.”
With regard to direct employment, the ET anecdotally found that ecotourism created the most jobs directly. For example, we found that one ecotourism project created 24 jobs: mostly young people who were trained as guides. However, the ET found that these jobs were part-time and seasonal; whether a guide had work on a given day was a function of how many visitors sought to hire him/her. As noted above, local stakeholders reported indirect economic benefits from both tourism and recreational projects. “The people who were walking the trail would also eat in the village and purchase items, which stimulated economic activity,” one CSO official said.
Agri-business projects were intended to provide direct employment to food-processing workers, as well as to improve the livelihood of farmers in the area by providing a buyer for their produce. However, the ET found that only two of seven such interventions from the first three rounds were operational as of March 2020. Furthermore, the operator of one operating agri-business project reported that the facility needed to expand its operations in order to become profitable and offset the costs of electricity and a backup generator.
USAID.GOV 14 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
Sub-Question 2C. What are the main factors that influenced positively or negatively BALADI/RMF’s ability to achieve these purposes?
Positive Factors USAID and RMF officials cited competent, professional RMF management as a key factor that enabled BALADI/RMF achievements. They noted that RMF had been working with Lebanese municipalities since 1991 and had previously served as a sub-contractor on USAID projects, giving the foundation experience in working on municipal development and in meeting USAID requirements. “They are good at finding solutions,” a USAID official said. Mayors cited “very good” experiences working with RMF. “RMF staff has a good relationship with communities and mayors, based on their whole portfolio of connections across Lebanon,” said a BALADI/RMF sub-grantee.
A BALADI/RMF sub-grantee said RMF’s experience allowed it to negotiate otherwise challenging political environments: “In Lebanon, you need to know the dynamic, what is left unsaid, what is possible. It's difficult to act on the ground in Lebanon. I think RMF has a good understanding. That's one of the reasons for success.” Another sub-grantee said, “They listened to the communities. Their people are connected to the communities. They have people who live where they work and can deliver very quickly.”
A USAID official cited RMF’s “careful ground-truthing and due diligence, including a feasibility study” as another factor in ensuring that the projects met relevant needs of the communities and were feasible. “We changed a lot of projects from the original proposals and rejected some applications after ground- truthing because they were not feasible,” said an RMF official. An RMF field staffer stated, “The better the ground-truthing phase results, the better the project.”
RMF staff also cited the quality assurance/quality control (QA/QC) process as an essential component of successful projects. “The QA/QC brings an added value, especially that it documents everything and every step taken with the contractors. It doesn’t leave it to the contractor’s verbal commitment,” an RMF field staffer said. Similarly, local stakeholders cited the technical support provided by RMF. Said a CSO official, “RMF is knowledgeable in the agricultural sector; they helped refine the proposal and feasibility studies and provided technical expertise and guidance.” Three mayors also cited the quality of interventions: “Everything was implemented with high standards of quality. The quality was satisfying,” one said.
Municipal officials said their communities took ownership of the interventions, which were selected in consultation with municipalities rather than being imposed by an outside donor. “From here this activity is considered essential and efficient to the village,” said one mayor. Another stated, “It helped us as a municipality to address needs of the people that we couldn’t address or achieve alone.” A local CSO official said, “There is an ownership among the citizens.” An RMF official echoed the theme of municipalities taking ownership of projects, saying, “These municipalities feel they own the project; it's not imposed on them. They feel that this is their own baby. They will take care of it.”
Another factor contributing to success was the identification and involvement of the operating partner from the beginning of the process. The ET found that the four non-operational projects in the 16 municipalities did not have an operating partner identified at the beginning of the process. “If you have a strong partner at the beginning of the project – design and application – it's easier to have an operator when the project is in place. It is important to have the operator at the selection phase,” an RMF official said. A colleague added, “If you don't have a local entity able to cooperate with the municipality, it won't work.”
15 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
Negative Factors USAID and RMF officials cited four primary factors that negatively influenced BALADI/RMF’s ability to achieve its objectives:
• The influx of Syrian refugees put pressure on basic services. • The lack of physical security limited movement. • The Lebanese economic crisis resulted in a lack of municipal cash to meet cost-sharing requirements and investments by the private sector or operators, while lower personal incomes reduced spending on tourism. • The COVID-19 pandemic had a particularly harsh effect on tourism due to lockdowns and fear of contagion. In its first three years of operation, BALADI/RMF had subgrants with the Business Incubation Association in Tripoli (BIAT) to provide support and training for PPPs and with Beyond Beirut to connect ecotourism initiatives with tour operators. Those subgrants were allowed to lapse because the subgrantees and RMF found they were not cost-effective, and RMF decided to hire consultants on an as- needed basis.
Sub-Question 2D. How successful was BALADI/RMF in creating long-term collaboration between sub- national public entities, CSOs and the private sector to implement the awarded projects and deliver services?
CSOs The ET found long-term collaboration between the municipalities and CSOs in four of the 16 municipalities that we sampled. In Douma, a local sports club approached the municipality to submit a proposal for a recreation center, which the club operates. In Bcharre, a local scouting organization helped to develop the proposal and is involved in managing the ecotourism center. In Barouk, several NGOs collaborated to develop the proposal and manage the trail, information center, and market. In Ehmej, a development NGO led the proposal, but the municipality later took over operation.
In other municipalities, RMF took advantage of existing collaboration and helped to strengthen it; for example, the mammography machine in Fih was established in an existing clinic. NGOs in Barouk, Bcharre, Tal Abbas, and Ehmej said that their collaboration with the respective municipalities predated BALADI/RMF. “Baladi has strengthened an already existing collaboration,” one mayor said.
“Our goal was to establish sustainable collaboration with the municipalities and CSOs to deliver services,” an RMF official said. “The collaboration was really effective. Sometimes municipalities resorted to a local NGO or individual to help write the proposal, then didn't continue. There was other cooperation between other organizations who were operators of the projects.”
While creating such collaboration is an important component of the Project Goal, the ET found that the focus of later interventions shifted to the delivery of essential public services, particularly in Round 3, where only four of 31 projects were socio-cultural or agri-business initiatives. “A solar farm doesn't have much to do with CSOs or the private sector,” a USAID official said. “If it's not meaningful, why would you do it? It's not a dealbreaker.”
As noted above, the CPCs were the primary vehicle to promote both the inclusion of citizens in the interventions and partnerships with civil society and the private sector; however, the ET found that many CPCs in the sampled municipalities were regarded as a box to be ticked to qualify for a project,
USAID.GOV 16 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
rather than a vehicle to sustain long-term collaboration. One mayor reported that the CPC met monthly in the high season and quarterly in the off season. Conversely, a CSO in another municipality reported that the CPC met once in five years, and another CSO reported that the CPC members had not met since the proposal-writing stage. The ET found that the openness to collaboration with civil society depended on the mayor and the municipality. In instances where BALADI/RMF municipalities were open to civil-society collaboration, the mayors and local CSOs said that there was previous collaboration. Conversely, such collaboration was not found in municipalities where it had not previously existed.
Public-Private Partnerships The ET found that PPPs are provided for in law, but some municipalities are unaware of the legal framework governing these partnerships and lack the know-how and financing that private-sector partners can provide. Municipalities are cash-poor and land-rich, so municipal land and properties could be used to attract capital investment through the PPP model. While BALADI/RMF’s partnership with BIAT helped raise awareness of PPPs, the ET found limited use of the PPP model after that partnership ended in 2015. An RMF official noted a reluctance of businesses to join PPPs, saying, “The private sector does not trust municipalities.”
The ET found that PPPs were considered in four of 16 sampled projects from the first three rounds, plus one of five projects funded under the Arsal supplemental program description (SPD). At the time of fieldwork in September 2020, local officials and stakeholders told the ET that:
• In Arsal, the dry fruit facility intervention has not yet been implemented. • In Qaytouli, the municipality would benefit only from rent income, and the eventual management entity would need to manage the facility because the municipality cannot bear the risk. • In the Union of Jezzine the private-sector partner that was considered to manage the table-olive facility did not continue due to several factors, including internal matters not related to the project. • In Fatri, one of several private-sector partners in the project dropped out of the project (which is not operational yet). • In Ehmej, the municipality was managing the cold storage facility. Municipal officers and other local stakeholders said the publicly funded Ehmej facility was offering the service at a lower price than neighboring facilities that are totally reliant on private-sector funding.
Sub-Question 2E. How satisfied are communities with BALADI/RMF activities, and to what extent did the activities generate goodwill for the municipalities, the USAID Mission, and the U.S. Embassy?
All but one of the municipal officials interviewed expressed satisfaction with BALADI/RMF’s activities. (In that one case, the official said the project only benefitted the wealthiest families in the municipality.) RMF staff confirmed the satisfaction that the ET noted. “Mayors were very happy with our work,” one RMF field staffer said. Another RMF official told us, “It is a very popular project. Everyone wants to be part of BALADI.”
Local stakeholders and RMF staff said local communities were aware that the assistance came from the American people through USAID, the BALADI/RMF project, and their municipality, in part because USAID branding requirements were followed for each project. “When you go to a village, people know that it is BALADI assistance. They connect the project with BALADI and USAID. It shows the active role of the municipality in providing services to the community,” one RMF official said.
17 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
A USAID official said, “People are grateful. They express that to us.” On the same note, an RMF field staffer stated, “Municipality members are very happy for what you've done for them. People are thankful for USAID and the embassy and have good feelings for them.” A local CSO official also said, “The farmers are aware that the project is financed by USAID in collaboration with the municipality.”
Interviews with mayors and other local stakeholders found that they became aware of BALADI/RMF through (i) meetings that RMF held around the country; (ii) learning about projects in neighboring communities, (iii) personal contacts in other municipalities or at RMF, or (iv) local NGOs that encouraged the municipality to apply for a project. Some local NGOs had previous experience with USAID. “Too many municipalities are contacting us; we get many calls from municipalities that are not able to participate. It's a good indication that the project is perceived as successful by the municipalities,” an RMF official said.
CONCLUSIONS Improved delivery of essential public services like electricity, water, and health care in 38 municipalities is the headline achievement of BALADI/RMF. The activity had a limited effect on improving municipal governance; its primary focus was on building project-management capacity, rather than the broader municipal capacity-building activities of BALADI Plus and BALADI-CAP. BALADI/RMF did not help municipalities to create development plans or reforms, which was outside of its scope. The activity’s requirement of CPCs to manage interventions did enhance participatory government, but the CPCs will not continue beyond the life of the activity. BALADI/RMF did not succeed in creating sustainable local social capital.
BALADI/RMF’s main contribution to economic growth and private-sector competitiveness is indirect: the provision of essential services improved the business environment, which can enable private-sector economic growth, and facilitated economic activity in municipalities. While the project did create hundreds of jobs, the number of FTEs is far smaller. However, its economic benefits are indirect and should be regarded in tandem with its achievements in improved service delivery.
The types of interventions that provided essential services – traditional and renewable energy, water, and health care – are the most effective at providing services to communities and indirectly improve livelihoods by improving the environment for businesses. Interventions that primarily aimed to improve livelihoods – like agri-business and ecotourism – fell short of job-creation goals, and many ecotourism jobs are part-time and seasonal.
RMF’s experience with municipalities, technical know-how, and relationships are the main factors for its achievements. Other key factors are its ground-truthing process and its consultations with municipalities, which led municipalities to take ownership of projects. The identification of an operator at the early stages of an intervention is important to ensure that the latter will be operational after it is completed. Factors that negatively influence achievement include the Syrian refugee crisis, security risks, the Lebanese economic crisis, and the COVID-19 pandemic.
BALADI/RMF did not establish new, long-term collaboration with the private sector or NGOs, though it strengthened and expanded existing collaboration. Collaboration was more suited to projects like agri- business, ecotourism, and community socio-cultural centers. Projects seeking to directly create jobs were more effective with private-sector collaboration. Collaboration with NGOs and the private sector was less evident in Round 3 projects that focused on the delivery of essential public services.
BALADI/RMF’s communication and outreach efforts generated widespread awareness of the activity, and the initial projects generated interest in neighboring communities. Branding efforts successfully
USAID.GOV 18 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
communicated that the projects were funded by USAID (which built goodwill toward the U.S. government) and implemented through the municipality.
QUESTION 3 At what level were women involved in the supported communities where the activities were implemented?
FINDINGS A. Did women actively participate in the identification and the implementation of local development activities?
Identification and Design Five out of 15 municipal leaders who responded to the question said that women were involved to some extent in project design or selection. For example, In Douma, the president of the club who co-wrote the project proposal along with the municipality was a woman. In Fatri as well, there were two women from the private sector involved in the identification phase and proposal writing. In the ten other projects, women were not involved at all in the project identification process. Two mayors from the Bekaa Valley who participated in the interview said that women did not get involved because of cultural reasons, noting that “in some areas women are not allowed to intervene, in regard to their culture and the traditions.” All the municipal leaders interviewed were men. No women mayors benefited from the BALADI/RMF activity. According to the UNDP Lebanese Elections Assistance Project report, 663 women were elected to municipal councils in 2016, out of 12,139 seats nationwide (5.5 percent). Figures were unavailable for the number of women elected as mayors in 2016; however, in 2010, there were six women mayors out of 1,038 nationwide (0.58 percent).
Project Implementation Four municipal leaders identified roles held by women, including CPC member, member of management committee, and facility manager. An RMF official said 185 women participated on CPCs (out of a total of 668 members nationwide), including some elected officials. RMF staff ensured the participation of women in CPCs but did not confirm the extent of their influence on the decision-making process. They stated that women were present in steering committees, but decisions were ultimately made by the mayors, who are mainly men.
Three municipal leaders cited examples of women in leadership roles and in charge of operating the completed facilities (Douma, Kousba, and Bcharre), but the most significant step in favor of women’s participation was the establishment of a women’s cooperative, which was in charge of managing an agri- processing facility in Qaa El Rim.
Three RMF respondents mentioned activities such as agricultural products, mammography clinics, echography that were specifically designed to benefit women. Women were mostly active in ecotourism, agri-businesses, and food processing projects. Their participation was reported as less significant in projects on construction, water, energy, and electricity compared to the level of their participation in the identification, design, and management phases.
Out of the 16 municipalities sampled, the ET found that two women benefited from job opportunities in Douma and one woman in Fih. Six women benefited from training on chocolate production in Kaitouly. Two of the completed projects reviewed by the ET are now managed by women, as there are women heads or members of management committees in Douma and Bcharre.
19 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
B. What aspects of the program demonstrate women’s sustainable participation beyond the life of the activity?
Mayors and local stakeholders in four of the 16 sampled municipalities – Douma, Bcharre, Ehmej, and Fih – said women’s participation was sustainable.
RMF conducted a gender evaluation analysis in 2014 at the end of the project design phase for Round 1 and at the very beginning of the implementation phase. The report suggested the following measures to ensure the proper implementation of USAID's Gender Equality and Female Empowerment Policy:
• To have a gender expert on staff capable of providing continuous and consistent assistance to reinforce the gender aspect in each project, throughout the project cycle; if this option is not feasible, to provide additional gender training to Gender Focus Points. • To increase women’s quota from 20 to 33 percent of CPC membership. • To have female members in the Technical Committee, if feasible, in order for women to be involved in the day-to-day activities and management of the project.
Due to budget constraints, RMF wasn’t able to hire a gender expert for the project as they didn’t consider it to be gender-based. As for the other adjustments, RMF followed up with the municipalities, but the results varied based on the project type and activity.
CONCLUSIONS Women’s involvement varied considerably by project type. It had many levels and varied as well throughout the phases (project identification, design, implementation, and operation or management). During the identification process, women participated in the proposal writing. They also participated in the implementation process, as they took part in the steering committees, but it was men (mayors) who took the main decisions because of the absence of women mayors.
Positive examples of women’s involvement in project design and as facility managers show that they were also engaged in the project in non-traditional ways. Women did benefit sustainably from the income generated from food processing, and there was an increase in the number of jobs held by women managing facilities (e.g., sports facility, ecotourism, and food processing).
Women who contributed to BALADI/RMF interventions had already been active in their communities through similar activities. Therefore, women’s involvement is not attributable to RMF efforts. RMF did not play a significant role in helping communities identifying new fields of intervention for women. The absence of a gender expert reduced BALADI/RMF’s capacity to recognize and respond to such gender imbalances.
The results shed light on gender disparities and the discrimination that Lebanese women still face, caused by a variety of factors, such as social and cultural values, religious and sectarian rules and beliefs, and an absence of gender-oriented development strategies. This also highlights the fact that most municipalities still do not give adequate attention to the importance of gender integration in their respective projects.
USAID.GOV 20 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
QUESTION 4 What is the likelihood that the interventions and results BALADI/RMF has achieved are sustainable beyond the life of the activity?
FINDINGS A. Which types of interventions and results are or show the most prospect of being sustained, and why?
Respondents from all groups interviewed agreed that the essential services provided through BALADI/RMF’s interventions are the most likely to be sustained. These include electricity (both renewable energy and conventional diesel generators), water, education, and health care. “For power, irrigation, and infrastructure, sustainability is guaranteed,” an RMF official said.
The key informants cited two reasons for the sustainability of these essential services: 1. Precisely because the municipalities provide essential services, they have great incentives to maintain them in order to ensure continued service provision; and 2. Citizens who use those services are willing to pay for the service, thus providing revenue for operations and maintenance. The same principle carries over to sustainability of solar water heaters since households will need to maintain the heaters if they want to benefit from hot water.
“A small margin of profit is gained by the municipality to sustain and maintain the project, since the main aim is to provide electricity to the village residents,” one mayor said. “The generator is located on the property of the municipality, and the municipality is maintaining and will maintain the project to sustain its efficiency, which is much needed for the village.” An RMF field representative said, “If they need the project, it should be sustained because the municipality needs it.”
One issue with the sustainability of electric power cited by mayors is net metering, i.e., allowing municipalities to sell excess power to the national electricity company to offset the costs of power consumption. Mayors said they could pass along the savings to their residents, further lowering their electric bills.
Ecotourism projects were designed so the revenues from usage would provide operating income to pay tour guides and maintenance, but operations and revenues have been affected by both the travel restrictions of the COVID-19 pandemic and the Lebanese economic crisis, according to local stakeholders and RMF officials. “Once it becomes operational, sufficient income will be generated,” said a representative of an NGO operating an ecotourism project that has yet to open due to the pandemic.
Agri-business interventions were designed to be sustainable by generating revenue through sales that would cover their operating costs. However, as noted above, only two of the seven agri-business projects were operating as of March 2020: a fishermen’s cooperative in Jounieh and a cold-storage facility in Ehmej, which RMF officials said needs to expand its operations to be profitable. In addition to Ehmej, the ET’s sample included a chocolate factory in Kaitouly and an olive-processing facility in Jezzine, the operation of which lapsed. “Agri-food centers are riskier; you need the proper operator and marketing,” an RMF official said.
In the two socio-cultural interventions in its sample, the ET found that the sustainability of both community centers was tenuous, as the NGOs operating them searched for long-term funding for routine maintenance and operations. “It is now up to the club to fix the emerging issues,” one representative said.
21 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
B. How sustainable are the permanent and seasonal jobs created by BALADI/RMF?
As discussed earlier under EQ2, BALADI/RMF created a limited number of jobs. As of March 2020, the activity created 266 full-time and part-time jobs, equal to 55 FTEs. USAID and RMF officials said BALADI/RMF must also be viewed in terms of service delivery and indirect economic growth generated because of improvements to the business environment from BALADI/RMF’s projects.
While essential services like renewable energy, traditional energy, water, and health are the most sustainable interventions, they do not create much direct employment, as per USAID and RMF officials. Generally, only one or two employees are needed to maintain energy and water projects. Health interventions generally took place in existing clinics, employing one or two additional technicians.
Agri-business projects have the potential for sustainable employment, but they must be operational and profitable. Only two of seven such projects were operational as of the date of this evaluation. Agri- businesses could generate employment, but this is dependent on profitability.
Ecotourism projects have potential for significant employment, but the ET found that these jobs were part-time, seasonal, and very sensitive to demand. The severe drop in tourism due to the COVID-19 pandemic and the Lebanese economic crisis imperils those jobs in the near term. Socio-cultural interventions have limited potential for employment; in addition, they face sustainability challenges if a municipality falls short of funds for operations and maintenance.
Nine mayors that the ET interviewed said BALADI/RMF had directly created jobs in their municipalities. Of these, fewer than five jobs were created in four municipalities (three energy, one health); five to 10 jobs in four municipalities (one each socio-cultural, ecotourism, water, and agri-business); and more than ten jobs in one municipality with an ecotourism project. The only project in the sample with a private- sector partner is in the ecotourism field.
CONCLUSIONS Essential services like renewable energy, traditional energy, water, and health are highly likely to be sustained, both because they meet an important need for communities and because they can generate enough revenue to cover their costs. Implementation of net-billing schemes would further improve sustainability and further reduce costs to customers.
The sustainability of other intervention types is volatile given the current domestic situation. Agri- businesses should become sustainable through the sale of their products, but most of these projects are not yet operational, due to delays encountered in 2019 and 2020. Identifying operational partners is essential to the sustainability of projects that aim to provide employment.
The sustainability of socio-cultural interventions is questionable because long-term funding for operations and maintenance has not been identified. Ecotourism interventions are designed to be sustainable, with revenues from tourists being used to pay operations and maintenance costs, but the pandemic and economic crisis have sharply reduced tourism and delayed the opening of such facilities. As a result of the economic crisis, basic services are a priority for citizens, who no longer have disposable income for luxuries like tourism, threatening the sustainability of ecotourism initiatives.
Energy, water, and health projects generated modest numbers of permanent, sustainable jobs. Their primary contribution to employment was the provision of basic services that enabled business development and private-sector job creation. Sustainable job creation is best supported by providing
USAID.GOV 22 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
basic services to businesses. If viewed solely through the lens of direct job creation, BALADI/RMF is not cost-efficient.
Other jobs directly created by BALADI/RMF, both permanent and seasonal, are more tenuous. Agri- business jobs could be sustainable if facilities become operational and profitable. Seasonal ecotourism jobs could be sustained by operating projects, but they are dependent on the larger economy. Jobs in socio-cultural facilities are also dependent on the number of paid events (weddings, birthdays, exhibits, and other events) these facilities host, which is dependent on the larger economy and how the COVID- 19 pandemic evolves.
QUESTION 5 How can BALADI/Caritas Lebanon and BALADI/RMF be compared and contrasted with regard to the extent to which their objectives and outcomes were achieved?
FINDINGS A. What were the internal and external factors that caused significant differences in topline results between BALADI/CL and BALADI/RMF?
External Factors Most external factors that affected the implementation were faced by both RMF and CL, which ended its implementation of BALADI in 2019. Originally, the Mission designed the project before the Syrian crisis had begun, and it was awarded in 2012 when the influx of Syrian refugees was at its beginning. USAID and RMF officials said that as the project was in the process of implementation, the refugee influx began to pose a burden on already scarce resources and changed the municipalities’ priorities.
For instance, the municipal elections that took place in 2016 caused delays for both implementers before the relationship with the newly elected mayors were established, according to an RMF official and the findings of the BALADI/CL evaluation. Both implementers succeeded in establishing good relationships with the newly elected mayors and continuing the implementation of the activities as planned (except for one project under CL).
The country’s current economic and financial crises proved more challenging to RMF’s projects (which are still in the implementation phase) than to CL’s projects (which are operational) and CL’s partner municipalities (which have completed their cost-share). Mayors said the financial crisis, including the depreciation of the Lebanese pound, has made it more difficult to find suitable private-sector partners who are willing to invest to operate the facilities. The crisis will also increase the cost of maintenance and spare parts (which are usually paid in U.S. dollars), thus challenging the sustainability of the municipal projects. The municipalities who have not completed their cost-share with RMF are having difficulty paying in cash, given that the national government has not yet distributed municipal funds for 2018 and later years, two mayors said. One way to ensure cost-sharing responsibilities are met is to ask the municipality to allocate the budget and pay its share at the early stages of the implementation, and the donor’s contribution would then follow.
The 26 municipal projects that municipalities submitted under CL round 1 and 2 had a higher average budget, with an average of $496,000 (on average, $211,000 paid by CL and $285,000 in municipal cost- sharing), according to CL project documents. When compared to 29 RMF municipal projects in rounds 1 and 2, as reflected in the RMF project financial data, the latter are relatively lower cost, with an average of $285,185 (on average, $185,427 paid by RMF and $90,251 in cost-sharing). The discrepancies are due
23 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
to CL having more construction activities and the fact that the municipal cost-share was mainly covered through an in-kind contribution of the land on which the facility was constructed. RMF’s relatively lower budgets resulted in municipal activities that were oriented towards essential services, rather than construction projects.
Internal Factors As stated in the findings of the BALADI/CL Final Performance Evaluation report (PMSPL II – February 2020), CL addressed the challenges it faced with its initial management team by recruiting more experienced staff at all levels. Caritas was then able to re-establish the relationship with the municipalities and build their trust towards the project. RMF faced no issues in this regard because it started with staff that has proven and solid experience working in development projects at the municipal level and enjoyed a strong presence, particularly in North Lebanon. RMF historically had previous experience working with USAID and was therefore more familiar with USAID’s requirements. However, CL historically had more experience in humanitarian assistance, and the USAID development realm was relatively new turf.
CL built sustainable collaboration between NGOs and municipalities. All of its 26 projects actively involved NGOs in both the implementation and operational phases. By the time the final evaluation was done, agri-business centers were all operated by local women’s cooperatives, and tourism projects were operated by local NGOs. RMF’s 65 municipal projects have less involvement of active NGOs in the operational phase. This is due in part to the nature of the projects, which aim at providing essential services like water, energy, and education, thus requiring that an official authority be responsible for the service. RMF is still working to activate operations in five of its seven agri-business projects. .
Finally, both implementers were prime contractors on the awards. The BALADI/CL evaluation showed that one NGO subcontractor under the CL consortium dropped out in the early phases of the project; and since this subcontractor was supposed to address the projects’ gender aspects, CL was left lacking such expertise. The other NGO on the CL consortium offered environmental inspection services through one person; thus, the input provided by this NGO could have been provided by an environmental expert on the CL team. Similarly, the two NGOs in the RMF consortium offered limited services, and their involvement was discontinued in 2015 at RMF’s decision. Their contribution was substituted by a consultant as needed, offering more specialized and cost-effective services.
B. What were the successes as well as lessons learned from BALADI/RMF?
• The complementarity of project-based and capacity-building components like BALADI/RMF, BALADI Plus, and BALADI-CAP ensures that municipalities receiving project support also receive capacity-building support.
• Consulting with communities to ensure projects meet their relevant needs encourages them to take ownership of projects.
• Involvement of unions of municipalities i enables economies of scale.
• The willingness of communities to involve citizens, including women and youth, defines their success in promoting inclusivity.
• The cultural and political dynamics of each community are factors that should be considered, particularly during the ground-truthing phase. This ground-truthing phase should be allotted sufficient time to ensure relevant needs are addressed and projects are refined to be feasible.
USAID.GOV 24 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
• Feasibility studies must include accurate cost estimates and identify operator and implementing mechanism after construction is completed.
• A consortium should include relevant partnerships and a strong prime with previous experience in leading NGOs. It is preferable that USAID use a single implementer on activities, rather than a consortium; when using multiple implementers, one should be the prime and the others sub- contractors. The prime should be strong enough to manage the sub-contractors.
• Avoid overlap among multiple implementers and ensure there is a clear division of responsibilities.
• Cost-sharing is better completed before the project nears its end to ensure that municipalities have an incentive to fund their share.
• Budget provision should be made for professional and technical fees that contribute to the quality of interventions.
CONCLUSIONS Both RMF and CL faced different challenges that caused delays to the implementation of their awards. Both successfully addressed these challenges and adapted their processes to overcome them and ensure a relatively successful implementation.
25 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
RECOMMENDATIONS Recommendation 1: USAID should focus its support to local governments to improve their ability to provide essential services to all residents and businesses. Assistance to individuals should be based on well-tailored criteria. The provision of essential services will improve the enabling environment for private-sector economic growth. As part of its municipal capacity-building efforts, USAID should encourage the creation and implementation of long-term local economic development plans. (EQ2)
Recommendation 2: USAID should ensure that institutional capacity building for municipalities can incorporate municipalities with different levels of capability, so that such capacity-building is provided to all BALADI/RMF municipalities. (EQ2)
Recommendation 3: USAID and RMF should focus municipal energy projects on renewable electricity (where land presence allows it to be possible, as compared to purely traditional means such as generators), with net metering included. (EQs 2 and 4)
Recommendation 4: USAID should increase collaboration between municipalities and civil society through projects that provide wide community benefits, rather than projects that aim to directly create jobs. (EQ2)
Recommendation 5: RMF should develop SOPs for CPCs in a way that enhances the presence and participation of women in these committees through steps like quotas and weighted voting, in order to ensure equal participation in decision making. In addition, RMF should offer women technical training in project operation and management in the early phases of a project and adopt practical measures to ensure women's participation in planning meetings. The location, timing, and structure of meetings may need to be adapted to facilitate women's participation. (EQs 2 and 3)
Recommendation 6: When identifying local entities as implementing partners, USAID should look at those organizations that have prior experience implementing the types of interventions envisioned in the activity. USAID should use a single implementer for activities, rather than a consortium. When using multiple implementers, one should be the prime and the others should be sub-contractors. The prime contractor should be strong enough to manage the sub-contractors. (EQs 2 and 5)
Recommendation 7: USAID should raise awareness of private-sector partnerships among municipalities and ensure that they are aware of applicable laws governing PPPs and how they can apply them and use them in service delivery interventions. USAID should encourage partnerships with the private sector to manage economic activities and should encourage implementers to undertake a mapping exercise to identify a qualified and interested entity to operate interventions after completion. (EQs 2 and 4)
Recommendation 8: RMF should hire a gender specialist to develop a gender-analysis plan for the project, as well as for each municipal project before the project-design phase. This specialist should also be in charge of developing a clear action plan that can be followed throughout project implementation as well as conducting outreach sessions on gender integration and its importance for municipalities and their partner NGOs during a project’s life cycle. RMF should offer gender training to all staff working with all beneficiaries in the field. RMF should map and assist local women institutions and organizations to contribute to each project and should involve women in the needs-assessment and ground-truthing phases. (EQ3)
USAID.GOV 26 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
ANNEXES Annex A: Inception Report and Scope of Work
Annex B: Detailed Evaluation Design and Methodology
Annex C: Evaluation Schedule and Persons Interviewed
Annex D: Documents Reviewed
Annex E: Data Collection Instruments
Annex F: Conflict of Interest Disclosures
Annex G: Evaluation Team Members
27 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
ANNEX A: INCEPTION REPORT AND SCOPE OF WORK [redacted]
USAID.GOV 28 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
ANNEX B: DETAILED EVALUATION DESIGN AND METHODOLOGY
DATA COLLECTION METHODS AND TOOLS
1. Desk Review: The evaluation team did a systematic literature search to identify evaluations, tools and other documents on municipal-focused projects. The evaluation team reviewed existing documents including CDCS, RFA, contract agreement, contract modification, audit reports, initial assessments, periodic reports (annual, quarterly, other) and all other reports produced by USAID, Social Impact, and BALADI/RMF to determine service delivery mechanisms, quality of activity results, compliance with CDCS, fulfillment of development hypothesis, quality of program management and the likelihood of achieving activity targets. The desk review provided the basis toward formulating the final set of questions as well as determining the KII and the sampling process for the FGD if needed.
2. Interviews with BALADI/RMF, USAID key staff members, municipal representatives, and other partners: The team designed an interview matrix, met remotely with the activity staff and conducted 52 key informant interviews (KII) to obtain more detailed, in-depth understanding about specific issues. The remote KIIs provided detailed and in-depth understanding of the key issues associated with the activity. The discussions focused on how questions of relevance, effectiveness, activity management, partnership and sustainability were addressed. Interviews were also carried out with other key stakeholders using a structured interview guide. A list of key informants was developed and agreed upon by partners and it included staff members from BIAT and Beyond Beirut from the 2012-2015 period, as well as other service providers and partners involved in the activity.
All interview questions were asked in a way that took into consideration gender sensitivity and gender mainstreaming.
Due to the COVID-19 pandemic, the ET was not able to conduct focus group discussions or site visits.
SAMPLING STRATEGY The evaluation team interviewed stakeholders from 16 municipalities engaged with BALADI/RMF to conduct remote KIIs. Fourteen municipalities were selected by the ET through a purposive process by which the evaluation team sought to reflect characteristics of the overall activity, and two were added at the request of USAID.
The selected municipalities and interventions are:
29 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
MUNICIPALITY DISTRICT GOVERNORATE ACTIVITY ROUND TYPE OF ACTIVITY Union of Jezzine Jezzine Establishing a table olive processing facility in Roum 1 South Lebanon Agriculture/Agri-Business Municipalities Barouk Shouf Mount Lebanon Promoting Sustainable Tourism in the Barouk-Fraidiss region 1 Ecotourism
Ehmej Jbeil Mount Lebanon Establishing a fruit cold storage, sorting and packing facility 1 Agriculture/Agri-Business
Fatri Jbeil Mount Lebanon Establishment of an eco-tourism area in Fatri 2 Ecotourism Establishing a community events area and equipping a computer Hrar Akkar Akkar 2 Education/Socio-Cultural center in the public school of Hrar Kaitouly Jezzine South Lebanon Establishing a chocolate production workshop 2 Agriculture/Agri-Business Tal Aabas Improvement and Rehabilitation of the Irrigation Water Conveyance Akkar Akkar 2 Water Infrastructure Al Gharbi Network in Tal Aabbas El Gharbi Bcharre Bcharre North Lebanon Improving Ecotourism in Bcharre Region 2 Ecotourism Enhancing Recreational Youth Activities and Developing Rural Douma Batroun North Lebanon 2 Socio-Cultural/Youth Tourism in Douma Install a 315 KVA generator to provide electricity backup during Qousaya Zahleh Bekaa 3 Traditional Energy blackout and LED lights in community football and basketball courts Substitute conventional streetlight lamps with energy efficient LED Fardis Hasbaya Nabatiyeh 3 Renewable Energy lights and procure a 400 KVA generator Equip the village health clinic with a mammography machine and Fih Koura North Lebanon 3 Health accessories Install 75 solar water heaters in the village to reduce electrical bills Kfardlakos Zgharta 3 North Lebanon of underprivileged residents, in addition to energy efficient Renewable Energy
streetlights. Equip the village with photovoltaic panels for electricity backup Jran Batroun North Lebanon 3 Renewable Energy generation to couple community generator. Baalbeck- Equip around 150 kW solar pumping system for the village water Ras Baalbeck Baalbeck 3 Water Infrastructure Hermel wells. Baalback- Arsal Baalback Install 5 electrical transformers to enhance national electrify power 3 Traditional Energy Hermel
USAID.GOV 30 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
MUNICIPALITY DISTRICT GOVERNORATE ACTIVITY ROUND TYPE OF ACTIVITY Baalback- Arsal Baalback Provide agricultural capacity building for famers in Arsal ASPD Agriculture/Agri-Business Hermel Baalback- Arsal Baalback Rehabilitate and equip a fruit drying facility ASPD Agriculture/Agri-Business Hermel Baalback- Expansion of Arsal technical Institute premises to host additional Arsal Baalback ASPD Education Hermel classes Baalback- Equip an estimated 200KWP solar pumping system for the village Arsal Baalback ASPD Renewable Energy Hermel water well Baalback- Install 5 additional electrical transformers to enhance national Arsal Baalback ASPD Traditional Energy Hermel electrify power
31 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
Three municipalities or union of municipalities were selected due to unique characteristics of their activities:
• Arsal (Baalbeck District, Baalbeck-Hermel Governorate) is the subject of a Supplemental Program Description (SPD) and hosts six activities to date, compared to one each in the 60 other municipalities. • Fatri (Jbeil District, Mount Lebanon Governorate) hosts a Round 2 ecotourism activity whose infrastructure is being used for other purposes, according to reports received by SI. • Union of Jezzine Municipalities (Jezzine District, South Lebanon Governorate) is the only union of municipalities in the program. Two municipalities with renewable energy projects were selected at the request of USAID:
• Ras Baalbeck (Baalbeck Disrict, Baalbeck-Hermel Governorate) hosts a solar pumping system for the village water wells. • Jran (Batroun District, North Lebanon Governorate) hosts a solar power array. In selecting the municipalities in the sample, the following characteristics were considered:
• Governorate: Four municipalities from the governorate where BALADI/RMF was most active (North Lebanon); two municipalities from each of the three governorates where BALADI/RMF was relatively more active (Baalbeck-Hermel, Akkar and Mount Lebanon); one municipality from each of the three other governorates (South Lebanon, Bekaa and Nabatiyeh).
• District: Municipalities from the district(s) in each governorate where the program was most active: Akkar, Jezzine, Batroun, Koura, Zgharta, Shouf, Jbeil, Zahleh, Baalbeck and Hasbaya. Two municipalities were selected in Akkar District because it is the only district in Akkar Governorate; two municipalities were selected in Baalbeck District to reflect the emphasis on Arsal; two municipalities were selected in Batroun, which was the most active district in North Lebanon, and one municipality was selected in each of the other districts.
• Type of project: RMF characterizes activities by type. The sample generally includes one activity of each type. The exceptions are renewable energy activities, which are represented four times because they are by far the most frequent intervention (21 of 65 total), and education/socio- cultural/youth activities, which have the widest variation in design. In the latter case, one community education activity and one youth activity were selected.
• Activity round: RMF awarded activities in three rounds, consisting of nine, 20 and 32 activities, respectively. The sample includes three activities from Round 1, six from Round 2 and seven from Round 3. (Activities in Arsal were awarded both in Round 3 and in the Arsal SPD.)
ETHICAL CONSIDERATIONS Evaluators informed key informants that their comments will remain anonymous and that any attribution will be indirect. The ET administered informed consent scripts and asked for respondents’ verbal consent to ensure that respondents understood the purpose and voluntary nature of the KII. All participants were informed about the potential risks and notified that they could withdraw from the evaluation at any time without penalty. KII and FGD participants were informed that there are no right or wrong answers, and there were no negative consequences for sharing candid information about how BALADI/RMF was designed or implemented. The ET addressed questions from the respondents and was prepared to stop FGDs and KIIs should the participant request to do so.
USAID.GOV 32 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
The safety and confidentiality of respondents is of utmost priority for SI. The ET produced reports that aggregate data and omit personal identifiers. No individual data will be shared outside the ET. Data Preparation The privacy of all participants who take part in the data collection was respected throughout the evaluation. All notes from FGDs and KIIs, and any identifying information was stored in password- protected folders that are only accessible to the ET and were not be made publicly available. We used the 7zip software for secured electronic transfer of information. SI will use SharePoint to store all data and reports and it is accessible only to authorized personnel on this evaluation. Strong passwords to protect against unauthorized access were used and were shared via a different mode of communication (e.g. instant messaging applications, separate email). SI uses several data security measures, including the physical security of any hard-copy materials and the secure storage, transfer, destruction, and access of digital materials.
DATA ANALYSIS The data gathered from the KIIs were used as the basis for data analysis. Techniques in data analysis included review of existing documentation obtained, conceptualizing, coding and categorizing of information obtained in the KII and FGD, examining relationships between the different activity components, activities and achievements as revealed by stakeholders, and authenticating conclusions by triangulation of data obtained from different sources. The evaluation team prepared a data source matrix (also called an evaluation matrix) that clarified how the data were collected. It included the evaluation questions, the evaluation tool(s), data source(s) and analysis plan for each question. An illustrative version of the matrix is provided below.
33 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
Area of Main Question Sub-Questions Data sources Data collection Data analysis Evaluation methods, plan sample and tools Relevance • To what extent • To what extent are the activity goals, purpose • BALADI/RMF’s • Key Informant • Qualitative BALADI/RMF is and objectives/results aligned to support the Staff; and technical Interviews (KIIs) data analysis contributing to USAID CDCS? experts with staff, of KIIs Development • To what extent did the awarded municipal • Key stakeholders at technical experts Objectives One and projects answer and/or address the relevant USAID; and USAID Two under USAID’s needs of the supported communities? • Activity Documents stakeholders CDCS for Lebanon? and M&E Plan. • BALADI Program • desk review of Assessment 2014 activity documents
Effectiveness To what extent has • What are BALADI/RMF’s headline outcomes • BALADI/RMF’s • Key Informant • Qualitative BALADI/RMF and achievements, and to what extent were Staff and technical Interviews (KIIs) data analysis achieved the activity’s these outcomes in-line with the stated experts with staff, of the KII goal and objectives, purposes. • BIAT and Beyond technical experts and group and what factors have • What were the types of interventions chosen by Beirut key staff; and USAID interviews contributed or the communities and assisted by BALADI RMF • Key stakeholders at stakeholders • Quantitative hindered the that were most effective in terms of providing USAID; • FGD or group analysis of achievement (or non- services to communities, or promote • Activity Documents interview with activity’s achievement) of the improvements in their livelihood? and M&E Plan. NGOs, reported activity’s objectives? • What are the main factors that influenced • Municipalities committees and data and positively or negatively BALADI/RMF’s ability to where activities are Municipality’s indicators achieve these purposes? implemented staff based on an and • How successful was BALADI/RMF in creating • Community interview guide comparison long term collaboration between sub-national Development or questionnaire. with the public entities, CSOs, and private sector to Committees • Desk review of results from implement the awarded projects and deliver • Beneficiaries activity the field visits services? • Activity documents. documents • • How satisfied are communities with • USAID • Direct Qualitative BALADI/RMF activities, and to what extent did Stakeholders observation analysis and the activities generate goodwill for the guide/ narrative municipalities, the USAID Mission, and the U.S. Checklist for field report of the Embassy? visits results. • Documents collected from the IP.
USAID.GOV 34 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
• Questionnaire or interview guide
Gender • At what level were • Did women actively participate in the • Females • KIIs asking • Qualitative women involved in identification and the implementation of local participating in the awareness analysis and the supported development activities? activity. Women in questions of narrative communities where • What aspects of the program demonstrate the community, women report of the the activities were women sustainable participation beyond the life Women on the participants results. implemented? committees. of the activity? • Representatives of Women NGOs in the supported areas. • Opinion leaders in the supported areas. Sustainability • What is the • Which types of interventions show the most • Representatives of • KII and support • Qualitative likelihood that the prospect of being sustained and why? Municipalities, documents data analysis results BALADI/RMF • How sustainable are the permanent jobs created participating NGOs provided by the of the KII has achieved are by BALADI/RMF? and local key informants. and sustainable beyond stakeholders • Have Quantitative the life of the • Activity documents. stakeholders analysis of activity? • BALADI/RMF’s described the activity’s Staff; and technical sustainability indicators experts scenario • Key stakeholders at (prospects and USAID; how to improve prospects) then use qualitative methods to unpack common themes. • Sustainability Plan submitted by the municipality as part of their proposal.
35 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
Comparison USAID does not often What were the internal or external factors that Document review desk review of Qualitative to have opportunities to caused significant differences in topline results activity documents data analysis BALADI/CL compare two activities between BALADI/CL and BALADI/RMF? and Baladi/CL as similar as evaluation report BALADI/CL Lebanon What were the successes as well as lessons learned and BALADI/RMF. from BALADI/RMF? How can the two activities be compared and contrasted with regard to the extent to which their objectives and outcomes were achieved?
USAID.GOV 36 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
This matrix ensured that multiple data sources were considered and that the team was able to triangulate data to answer each question. All the data collected from different sources was reviewed for reliability and validity and findings were compared using multiple methods, data forms and data sources. Data were organized to answer evaluation questions in the final report. Differentiation between the findings, interpretations, judgments, recommendations, and the specific sources of evidence supporting these, were made clear in the final evaluation report. Gender and Social Analysis Plan Consistent with USAID evaluation policy and recognizing that effects of integration and the success of the activity might vary across gender, SI applied a gender perspective and ensured that research activities include female study participants. The PMPSL Gender Specialist helped the ET ensure compliance and gender sensitivity of the data collection protocols and tools, as well as other contract deliverables.
The KIIs included questions to assess the extent to which gender has been integrated into the program. Participant-level data was disaggregated by sex, when feasible. The ET analyzed how gender issues have been addressed both in the initial design of the activity and in implementation. More specifically, the ET assessed: How women were involved in the supported communities (EQ3); The extent to which women actively participated in the identification and the implementation of the grant-funded community projects; Whether and how successfully women were encouraged to take on leadership roles in the projects; What aspects of the program demonstrate women’s sustainable participation beyond the life of the activity; How well BALADI/RMF achieved its Gender Strategy
LIMITATIONS AND BIASES The evaluation team is aware of several noteworthy risks to its data collection and analysis:
• Recall Bias: Recall bias is a common evaluation problem; however, the ET found that beneficiaries did not blend their experiences into a composite memory. • Response Bias: Response bias is the risk that key informants may be motivated to provide responses that would be considered socially desirable or influential in obtaining donor support. For example, a training participant may provide positive remarks about an activity because s/he would like to attend such trainings in the future. Beneficiaries may believe that negative evaluation findings could result in reduced assistance from USAID/Lebanon. The evaluation team mitigated the risk of response bias by utilizing data triangulation to note discrepancies in responses through different data sources (KIIs, document review) and preparing follow-up questions for informants that encourage dmore specific responses. For example, evaluators asked respondents for specific examples to back up certain responses. • Selection Bias: Selection bias is an inherent risk when implementers facilitate contact with activity beneficiaries. To eliminate selection bias, the evaluation team selected the target municipalities without input from the implementer, thereby ensuring that the sample is not skewed. The ET was aware of the risk of selection bias due to invited interlocutors being unfamiliar with or unable to access technology used for Internet-based meetings but encountered no such issues during interviews. • Remote Data Collection: The COVID-19 pandemic restricted the ability of the evaluation team to meet with individuals for KIIs and FGDs. Because face-to-face contact is not possible, the team conducted remote interviews with key informants. Because the Team Leader could not travel from the United States to Lebanon, he conducted interviews by telephone or online video conferencing with staff from the Mission and implementers. Local team members conducted interviews with local beneficiaries in Arabic and transcribed their notes in English for the team leader’s review.
37 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
DISSEMINATION AND UTILIZATION After the conclusion of the field work, the Evaluation Team presented initial findings and conclusions during a remote out-brief with USAID/Lebanon on September 18, 2020, and a Mission-wide briefing on October 8, 2020.
SI submitted this draft evaluation report to USAID/Lebanon for review... USAID will provide comments on the draft report. The Evaluation Team will revise the draft report to address USAID comments and suggestions, and the final evaluation report will be submitted within three weeks of receiving feedback from USAID on the draft report.
The final evaluation report will be disseminated within USAID/Lebanon. Upon instruction from USAID, SI will submit the final evaluation report (excluding any potentially procurement-sensitive information) to the Development Experience Clearinghouse (DEC) for dissemination among implementing partners, stakeholders, and the general public. The DEC submission must be within three months of USAID’s approval of the final report. Any personally identifying information of evaluation respondents will be appropriately de-identified from data and from the evaluation report per SI guidelines prior to submission to the DEC to ensure respondent confidentiality.
As a part of its post Evaluation Action Plan, and at the discretion of USAID/Lebanon, SI will follow up to facilitate and enhance use of evaluation findings through one or more of these steps: Distribute the approved evaluation report to BALADI/RMF or other USAID implementing partners; facilitate learning events in Lebanon to disseminate findings; Conduct a client satisfaction survey three months after completing the evaluation; Conduct a utilization/tracer survey six months after final report.
USAID.GOV 38 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
ANNEX C: EVALUATION SCHEDULE AND PERSONS INTERVIEWED
[redacted]
39 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
ANNEX D: DOCUMENTS REVIEWED • BALADI/RMF award & modifications • BALADI/RMF Work Plans • BALADI/RMF M&E Plan and quarterly reports • BALADI/RMF technical proposal and design • Country Development Cooperation Strategy (CDCS) of USAID/Lebanon • 2014 Social Impact BALADI assessment • 2018 Social Impact BALADI/CAP evaluation • 2020 Social Impact BALADI/CL evaluation
USAID.GOV 40 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
ANNEX E: DATA COLLECTION INSTRUMENTS
Informed Consent Agreement
Purpose: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called Social Impact, a company that is based in the United States. Our team is conducting a study about the BALADI program that is implemented by the René Moawad Foundation and funded by the United States Agency for International Development, or USAID.
You have been asked to participate today so that we can learn how BALADI/RMF can better support development in Lebanon. We are speaking to a variety of stakeholders in Lebanon: municipal officials, beneficiaries of BALADI/RMF programs, civil society, USAID implementing partners, and USAID officials. We would like your honest impressions, opinions and thoughts about various issues related to foreign assistance in Lebanon. We are independent consultants, not USAID employees, and we do not represent the Government of the United States nor the Government of Lebanon.
Procedures: If you agree to participate, we ask you to discuss your experience and opinion of the BALADI/RMF program. The interview will take about one hour of your time. Our findings will be presented to USAID, but all of your answers will be kept confidential. Nothing you tell us will be attributed to any individual person. Rather the presentation will include only a composite of all of the answers received by all of the individuals we interview. Although we may use quotes, no direct quotes will be attributed to any individual.
Risks/Benefits: There are no significant risks to your participation in this study. You will not receive any direct benefit or compensation for participating in this study. Although this study will not benefit you personally, we hope that our results will help improve potential future USAID programming in Lebanon.
Voluntary Participation: Participation in this interview is completely voluntary. You do not have to agree to be in this study. You are free to end the interview at any time or to decline to answer any question which you do not wish to answer. If you decline to participate in the interview, no one will be informed of this.
Do you have any questions at this time? [Interviewer should answer any questions]
Permission to Proceed I understand the purpose of the interview as outlined above and understand that I can withdraw from the interview at any time and for any reason. I agree to participate in the interview (Evaluator records).
Yes No
Initials of evaluator to indicate receipt of verbal consent: ______
Date ______
41 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
USAID
1. What has been your overall experience with BALADI/RMF? 2. How has BALADI/RMF contributed to improving municipal capabilities? 3. How has BALADI/RMF helped municipalities to work inclusively with citizens to achieve local development objectives? 4. How has BALADI/RMF helped to “establish sustainable local social capital”? 5. How has BALADI/RMF enhanced economic growth? 6. Are all activities designed to enhance economic growth (e.g., LED streetlights, solar water heaters, youth athletic facilities)? 7. How well is BALADI/RMF result framework aligned with the CDCS? 8. Do the activities address the real needs of the supported municipalities? 9. What are the most significant outcomes and achievements of BALADI/RMF, and were they in line with the BALADI/RMF’s objectives? 10. What interventions were most effective in providing services or jobs to residents? 11. What are the main factors that influenced the success of BALADI/RMF activities? 12. How successful was BALADI/RMF in creating and sustaining collaboration between municipalities, civil society and the private sector? 13. How effective were outreach and communication, and how was publicity generated for the Mission and the Embassy? 14. How involved were women in BALADI/RMF implementation, and is their participation sustainable? 15. Did women actively participate in identifying and implementing activities? 16. How sustainable are BALADI/RMF interventions (including seasonal jobs), and what are the main contributors to sustainability? 17. What are the main lessons learned from BALADI/RMF for the design and implementation of similar projects in the future? 18. What recommendations would you offer for BALADI/RMF in its last 18 months and for future projects? 19. Can you compare how RMF implemented its part of BALADI to how Caritas Lebanon implemented its BALADI project? 20. What are the primary factors that caused differences in results between BALADI/RMF and BALADI/Caritas? Implementers
1. What has been your overall experience with BALADI/RMF? 2. How has BALADI/RMF contributed to improving municipal capabilities? 3. How has BALADI/RMF helped municipalities to work inclusively with citizens to achieve local development objectives? 4. How has BALADI/RMF helped to “establish sustainable local social capital”? 5. How has BALADI/RMF enhanced economic growth? 6. How do you select the municipalities where you undertake activities? 7. How do you ensure that the activities address the real needs of the supported municipalities? 8. What are the most significant outcomes and achievements of BALADI/RMF, and were they in line with the BALADI/RMF’s objectives? 9. What interventions were most effective in providing services or jobs to residents? 10. What are the main factors that influenced the success of BALADI/RMF activities? 11. How did BALADI/RMF achieve a successful start? 12. How successful was BALADI/RMF in creating and sustaining collaboration between municipalities, civil society and the private sector?
USAID.GOV 42 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
13. How effective were outreach and communication, and how was publicity generated for the Mission and the Embassy? 14. How involved were women in BALADI/RMF implementation, and is their participation sustainable? 15. Did women actively participate in identifying and implementing activities? 16. How sustainable are BALADI/RMF interventions (including seasonal jobs), and what are the main contributors to sustainability? 17. What are the main lessons learned from BALADI/RMF for the design and implementation of similar projects in the future? 18. What recommendations would you offer for the last 18 months of BALADI/RMF and for future projects? 19. Can you compare how RMF implemented its part of BALADI to how Caritas Lebanon implemented its BALADI project? 20. What are the primary factors that caused differences in results between BALADI/RMF and BALADI/Caritas? Municipal Officials
1. What has been your involvement and overall experience with the BALADI/RMF? 2. How has BALADI/RMF contributed to improving capabilities of your municipality, including your local economic development plan? 3. How has BALADI/RMF helped you to work with citizens to achieve local development objectives, and how could it have done better? 4. How has BALADI/RMF helped to build trust between municipal officials and citizens? 5. How has BALADI/RMF enhanced economic growth in your municipality, and how do you measure that? 6. Does the activity address the real needs of your municipality? 7. Was the activity effective in providing services or jobs to residents? 8. How did BALADI/RMF help you to collaborate with civil society and the private sector? 9. Is this collaboration continuing, and did BALADI RMF contribute to sustainable collaboration? 10. How did you learn about BALADI/RMF? 11. How involved were women in activity implementation, and is their participation sustainable? 12. Did women actively participate in identifying and implementing activities? 13. How will your municipality sustain this activity after BALADI/RMF support ends? 14. Are the jobs created by the activity sustainable after RMF support ends? 15. What recommendations would you offer for the last 18 months of BALADI/RMF? Community Project Committees
1. How did you get involved in the Community Project Committee? 2. What has been your overall experience with BALADI/RMF? 3. How has BALADI/RMF helped citizens to work with municipal officials to achieve local development objectives? 4. How has BALADI/RMF helped to build trust between municipal officials and citizens? 5. How has BALADI/RMF enhanced economic growth in your municipality? 6. Does the activity address the real needs of your municipality? 7. How did you select this activity? 8. Was the activity effective in providing services or jobs to residents? 9. Who are the main users of this activity? 10. How did BALADI/RMF help you to collaborate with civil society and the private sector? 11. How involved were women in the Community Project Committee?
43 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV
12. Did women actively participate in identifying and implementing activities? 13. How will your municipality sustain this activity after BALADI/RMF support ends? 14. Are the jobs created by the activity sustainable after RMF support ends? 15. What recommendations would you offer for the last 18 months of BALADI/RMF? Non-Governmental Organizations
1. What has been your overall experience with BALADI/RMF? 2. How did you learn about BALADI/RMF? 3. What role does your NGO play in the management of the BALADI/RMF, and who from your NGO is involved? 4. Has BALADI helped to build the capacity of your NGO, and if so, how? 5. How has BALADI/RMF helped NGOs to work with municipal officials to achieve local development objectives? 6. How has BALADI/RMF helped to build trust between municipal officials and citizens? 7. How has BALADI/RMF enhanced economic growth in your municipality? 8. Does the activity address the real needs of your municipality? 9. Who are the main users of this activity? 10. Was the activity effective in providing services or jobs to residents? 11. How did BALADI/RMF help you to collaborate with the municipality and the private sector? 12. How involved are women in this activity? 13. How will your municipality sustain this activity after BALADI/RMF support ends? 14. Are the jobs created by the activity sustainable after RMF support ends? 15. What recommendations would you offer for the last 18 months of BALADI/RMF? Facility Managers
1. What has been your overall experience with BALADI/RMF? 2. How many people use this facility? (per day, per week, etc.) 3. Who are the main users of this activity? Woman, men, youth, elderly? Do you have statistics? 4. Does the activity address the real needs of your municipality? 5. What is the main benefit of this activity to the community? 6. How is the activity managed? 7. How has BALADI/RMF helped to build trust between municipal officials and citizens? 8. How has BALADI/RMF enhanced economic growth in your municipality? 9. Was the activity effective in providing services or jobs to residents? How many? 10. How many women are employed? 11. How did BALADI/RMF help you to collaborate with civil society and the private sector? 12. How involved are women in managing this activity? 13. How will your municipality sustain this activity after BALADI/RMF support ends? 14. Are the jobs created by the activity sustainable after RMF support ends? 15. What recommendations would you offer for the last 18 months of BALADI/RMF?
USAID.GOV 44 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
ANNEX F: CONFLICT OF INTEREST DISCLOSURES
John Lis COI.pdf
Maysa El Tannir - Conflict of Interest.p
Christina Abi Haidar - Conflict of i
45 | BALADI/RMF INTERIM PERFORMANCE EVALUATION USAID.GOV Disclosure of Conflict of Interest for USAID Evaluation Team Members
Name John Lis Title Team Leader Organization Social Impact Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) AID-268-C-15-00001 USAID Project(s) Evaluated (Include BALADI_CAP Mid-term Performance Evaluation, project name(s), implementer Counterpart International, AID-268-A-13-00002 name(s) and award number(s), if applicable) I have real or potential conflicts of Yes No interest to disclose. If yes answered above, I disclose the I have worked as a short-term consultant for MSI, the following facts: Real or potential conflicts of interest may include, implementing associate whose project is being evaluated. but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation.
I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature
Date John Lis January 31, 2018
2
Consultant
X
ANNEX G: EVALUATION TEAM MEMBERS John Lis, Team Leader: John Lis is a democracy and governance consultant with two decades of experience in the U.S. Congress and international parliamentary bodies. He led the 2014 evaluation of the PACE program in Lebanon and the 2018 BALADI CAP evaluation, and he has conducted evaluations and assessments of USAID and State Department programs in Iraq, Kyrgyzstan, Bangladesh, Burma, Kenya, Ukraine, and the Middle East and North Africa. From 2003 to 2013, he was a Professional Staff Member for the House Committee on Foreign Affairs, including eight years as Staff Director of the House Democracy Partnership, the peer-to-peer legislative strengthening initiative of the U.S. House of Representatives, which worked in 17 countries including Lebanon. He worked in Brussels from 1999 through 2002 as Director of the Defense and Security Committee of the NATO Parliamentary Assembly. A former journalist, he has worked at the Congressional Budget Office and Senate Foreign Relations Committee. He holds a bachelor’s degree in history from Stanford University and a master’s degree in international affairs from Columbia University.
Christina Abi Haidar, Team Member: Christina Abi Haidar is a lawyer with varied experience in law matters. She has an in-depth knowledge and significant experience with Lebanese municipal and administrative laws and regulations for acting as a legal expert in several projects related to this sector and as a board member of Talia municipality for six consecutive years. She has extensive experience in developing policies on issues related to governance, local empowerment, socio-economic development, youth empowerment, women empowerment, social entrepreneurship, civil society. She facilitated and conducted capacity building interventions involving multi-stakeholders in public, private and non-profit sectors in Lebanon and the Arab region. She has a wide knowledge regarding the public-sector management and development throughout different consultancies and assignments with different ministries specifically in the Ministry of Energy and Water and Ministry of Interior. She worked with the Ministry of Energy and Water in respect to the PPP framework, worked on legal Implementation Mechanism to PPP laws, in addition to Green building code. She also worked on the development and elaboration of a financial mechanism for distributed solar energy technologies, specifically for municipalities. Christina has participated in several USAID project evaluations and was part as a local team member in the Political Economic Assessment. She holds a Bachelor’s degree in Law from the Faculty of Law and Political Science at the Lebanese University and holds an MA in International Law from Notre Dame University and is a Ph.D. candidate at Centre D’Etudes Diplomatique et Strategiques in Paris.
Mayssa Tannir, Team Member: a certified project manager (PMP) and a local development professional with experience in implementing projects on local development in municipalities and unions of municipalities. Throughout her eight years of experience, she worked towards recognizing the role of Lebanese municipalities in local development, and boosted up dialogue between local communities, government agencies and international organizations in aspects related to municipalities. She supported municipalities and unions of municipalities to introduce Municipal Offices for local development into their municipal institutional framework. She organized trainings for elected officials and municipal technicians in Lebanon and elsewhere and on varied topics such as Integrated Solid Waste Management, Transport, Water and Wastewater Treatment, Strategic Planning, Local Development, Citizen Relation Management, Communication Skills, Conflict Management, Surveillance and Reporting data for early detection, emergency preparedness and response. She holds a bachelor degree in Law, Economics & Business from the Conservatoire national des arts et métiers de Paris and a Master degree in travel and tourism management from the Lebanese University.
USAID.GOV 46 | BALADI/RMF INTERIM PERFORMANCE EVALUATION
Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA 22201 Suite 1000 Tel: (703) 465-1884 Fax: (703) 465-1888 www.socialimpact.com