Final Evaluation Report

Final Evaluation Report Evaluation of UNICEF’s response to the Water, Sanitation and Hygiene Needs in Jordan as a result of the Syrian refugee crisis (July 2012 to July 2017)

PREPARED BY INTERNATIONAL SOLUTIONS GROUP An IMC Worldwide Company

PREPARED FOR UNICEF Jordan

16 April 2019

Evaluation Timeframe This evaluation covers UNICEF’s WASH programme activities between July 2012 and July 2017. The evaluation was conducted between November 2017 and September 2018. Data collection activities took place during July and August 2018. Geographic Scope This evaluation covers UNICEF Jordan WASH activities in the four camps of Za’atari, Azraq, Cyber City (closed in September 2016) and King Abdallah Park, as well as Rukban and Hadalat (abandoned in September 2017) at the Syrian/Jordanian border. Although there are Syrian refugees registered in all 12 Governorates of Jordan, the evaluation focused on the refugees registered in Zarqa, Irbid, Mafraq and Amman Governorates, which represent 90 percent of registered refugees. Organizations and Consultants conducting the evaluation This evaluation was conducted and led by International Solutions Group, a division of IMC Worldwide, and its partner, the Interdisciplinary Research Consultants, a research firm based in Amman, Jordan. UNICEF Jordan was the evaluation’s commissioning organization.

Table of Contents THE WASH PROGRAMME AND CONTEXT ...... 6 DESCRIPTION OF THE WASH PROGRAMME ...... 6 PROGRAMME HISTORY ...... 6 WASH PROGRAMME PURPOSE, COMPONENTS, AND MANAGEMENT ...... 8 WASH PROGRAMME LOGIC MODEL AND RESULTS FRAMEWORK ...... 9 PROGRAMME CONTEXT...... 12 MANAGING AN INFLUX OF REFUGEES FROM A RELATIVELY WATER-SUFFICIENT AREA ...... 12 PRE-EXISTING POLITICAL ATTITUDES TOWARD REFUGEES ...... 13 LEGAL AND INSTITUTIONAL FRAMEWORK OF THE WASH SECTOR IN JORDAN ...... 13 ECONOMIC TENSION ...... 13 FUNDING ENVIRONMENT ...... 14 DESCRIPTION OF THE EVALUATION ...... 15 PURPOSE AND OBJECTIVE ...... 15 SCOPE ...... 15 EVALUATION METHODOLOGY...... 15 DESK REVIEW ...... 15 QUANTITATIVE ANALYSIS...... 16 PRIMARY RESEARCH: QUALITATIVE DATA COLLECTION METHODS AND SAMPLING ...... 16 COVERAGE IN INTERVIEWS AND FOCUS GROUPS ...... 16 RAPID ASSESSMENTS AND USER INTERVIEWS ...... 17 WASH MESSAGING AND PRACTICES ...... 17 PARTNER INTERVIEWS ...... 17 DUTY BEARERS AND STAFF ...... 17 STAKEHOLDER INTERVIEWS...... 17 ANALYSIS METHOD...... 17 LIMITATIONS TO THE EVALUATION...... 18 ETHICAL PRINCIPLES IN CONDUCTING THE EVALUATION ...... 18 FINDINGS ...... 19 RELEVANCE ...... 19 EVALUATION QUESTION 1 ...... 19 EVALUATION QUESTION 2 ...... 26 EVALUATION QUESTION 3 ...... 27 EFFECTIVENESS ...... 29 EVALUATION QUESTION 4 ...... 29 EVALUATION QUESTION 5 ...... 53 EFFICIENCY ...... 63 EVALUATION QUESTIONS 6 AND 7 ...... 63 EVALUATION QUESTION 8 ...... 69 EVALUATION QUESTION 9 ...... 70 EVALUATION QUESTION 10 ...... 72 SUSTAINABILITY ...... 72 EVALUATION QUESTION 11 ...... 72 EVALUATION QUESTION 12 ...... 74 EVALUATION QUESTION 13 ...... 74 EVALUATION QUESTION 14 ...... 75 EVALUATION QUESTION 15 ...... 75 EVALUATION QUESTION 16 ...... 76 1

COVERAGE ...... 77 EVALUATION QUESTION 17 ...... 77 EVALUATION QUESTION 18 ...... 79 COORDINATION ...... 79 EVALUATION QUESTION 19 ...... 79 EVALUATION QUESTION 20 ...... 80 CONCLUSIONS ...... 80 LESSONS LEARNED AND RECOMMENDATIONS ...... 84 LESSONS ...... 84 RECOMMENDATIONS ...... 86 WORKS CITED ...... 89 ANNEX A: EVALUATION TERMS OF REFERENCE...... 92 ANNEX B: ANONYMIZED LIST OF INTERVIEWS ...... 104 ANNEX C: LIST OF SITES VISITED ...... 105 ANNEX D: DOCUMENTS CONSULTED ...... 106 ANNEX E: DATA COLLECTION TOOLS ...... 112 ANNEX F: TEAM DESCRIPTION...... 119 ANNEX G: EVALUATION MATRIX ...... 120 ANNEX H: UNICEF WASH PROGRAMME RESULTS FRAMEWORK...... 149 ANNEX I: UNICEF PROVIDED FINANCIAL ESTIMATES ...... 155 ANNEX J: ADDITIONAL FINANCIAL INFORMATION ...... 162

Table of Figures Figure 1 WASH Programme Implied Logic Model ...... 11 Figure 2 Funds budgeted vs mobilised by year (USD in millions)...... 14 Figure 3 Public Latrine at Azraq camp ...... 23 Figure 4 The stand-by power generation increases the ability of the water supply and distribution system to function...... 24 Figure 5: Za'atari Camp 2012 (Source:Times of Israel, 29 August 2012) ...... 33 Figure 6 Very clean household latrine observed in Za'atari camp...... 54 Figure 7 Rubbish Bin in Azraq ...... 55 Figure 8 Power generators and fuel storage at Rukban water system. Extra power generation and fuel storage capacity minimizes the potential for water system down time...... 58 Figure 9 Unimproved pit latrine cover at an ITS near Amman ...... 61 Figure 10 Cost of providing a cubic meter of water by year based on partner budgets ...... 65 Figure 11 Water provision cost savings in Za'atari (Source: UNICEF WASH Programme) .. 66 Figure 12 Membrane bioreactor wastewater treatment system functioning properly at Za'atari Camp...... 67 Figure 13 Division of labor among UNICEF Partner organizations. (Source: WaSH in Za'atari Snapshop, Jan-Feb 2014) ...... 69 Figure 14 Percent of UNICEF Total Expenditure by Beneficiary Type (August 2012 – July 2017) ...... 71 Figure 15 WASH Programme expenditure by activity category (August 2012 – July 2017) . 71

Acronyms and Abbreviations ACF Action Contre la Faim ACTED Agency for Technical Cooperation and Development BPRM The Bureau of Population, Refugees, and Migration CEDAW Convention on the Elimination of All Forms of Discrimination against Women CRC Convention on the Rights of the Child DAC Development Assistance Committee FGD Focus Group Discussion GIS Geographic Information System GTZ Gesellschaft für Internationale Zusammenarbeit IEC Information, Education, and Communication IMF International Monetary Fund INGO International Non-Governmental Organization ISG International Solutions Group ITS Informal Tented Settlements JEN Government of Japan supported development agency JHCO Jordanian Hashemite Charity Organization JRP Jordan Response Plan JVA Jordan Valley Authority KAP Knowledge, Attitudes, and Practices

KII Key Informant Interview MBBR Moving Bed Biofilm Reactor MOE Ministry of Education MWI Ministry of Water and Irrigation NGO Non-Governmental Organization NRC Norwegian Refugee Council NRW Non-Revenue Water OECD Organization for Economic Cooperation and Development RAM Results Assessment Module TDS Total Dissolved Solids TOR Terms of Reference UNHCR Office of the United Nations High Commissioner for Refugees UNOPS United Nations Office for Project Services UNRWA United Nations Relief and Works Agency USAID US Agency for International Development VIP Ventilated Improved Pit WAJ Water Authority of Jordan WASH Water, Sanitation, and Hygiene WIS WASH in Schools

Executive Summary In March 2012, when a relatively small number of Syrians were fleeing the civil war into Jordan, UNICEF responded by setting up Water, Sanitation, and Health (WASH) services and facilities at border crossings, called transit centers. The initial numbers of refugees were in the thousands rather than the much higher numbers that UNCHR was predicting, and some members of the international press described the international community’s response and predictions of a massive refugee influx alarmist and misleading (Seely, 2012). By August UNICEF’s and UNHCR’s preparations paid off. One thousand refugees per week crossing the border quickly became over 10,000 per week (UNHCR, 2012). Some refugees found sponsorship and entered host communities, where they had access to household and government WASH services. Others, unable to return home and without other options, entered the Za’atari refugee camp that opened on 30 July 2012. By December, over 66,000 refugees were living in the camp (UNICEF, December 2012). In July 2012, UNICEF was the only organization in Jordan that had the resources, capacity, and institutional commitment to take leadership of the WASH response related to the Syrian refugee crisis. UNICEF accepted responsibility for providing WASH services in a country that was exhausted by previous refugee crises, water scarcity, a complex political environment, and funding uncertainty. From July 2012 through July 2017, UNICEF provided life-saving water and sanitation resources under these difficult conditions for approximately 400,0001. The programme that UNICEF developed as its WASH response to the crisis was to become one of the largest programmes in UNICEF Jordan’s portfolio. The scope of the programme was initially a mandate to provide services to refugees in camps. It expanded to include services for water-deprived Jordanian host communities that were resource constrained before the crisis and were further burdened by the crisis. The WASH programme’s scope also grew to provide urgent WASH services for refugees living in the border settlements at Hadalat and Rukban. The emergency response required quick and flexible decision making and efficient utilization of scarce resources, which UNICEF successfully performed. The WASH programme was not without missteps or shortcoming, which the evaluation team discusses in the full report. Overall however, the programme provided an effective response in line with its core mission, providing life-saving WASH services in response to the Syrian refugee crisis. It has also increased efficiency and worked to ensure coverage of vulnerable populations. Two difficult achievements in a dynamic and complex environment. By 2017, The WASH programme had evolved in response to its changing environment, adopting a framework that worked at four levels: 1. Provision of equitable WASH services and the dissemination of WASH messages in the four camps (Za’atari, Azraq, King Abdullah Park and Cyber City) and two settlements in Rukban and Hadalat, to the most vulnerable people namely children and women.

1 Evaluation Terms of Reference, page 4.

2. Provision of gender equitable support to the most vulnerable people in host communities (in all 12 Governorates) namely to children and women. 3. Support to the WASH sector through technical support to the Ministry of Water and Irrigation. 4. Coordination of the sector at camps and national level. The WASH programme’s objective, its highest-level target, is that2 “Quality WASH facilities are sustained and utilised, and hygienic behaviours practiced, by the most vulnerable while ensuring the protection of the environment.” The programme measures the achievement of this objective through four outcome indicators: Indicator 1: Existence of strategic planning capacity in MoWI and sector and other relevant institutions in Emergency preparedness and response. Indicator 2: Proportion of population at camps and host communities including schools using improved drinking water as per Jordan standards. Indicator 3: Proportion of population at camps and host communities including schools using improved sanitation facilities. Indicator 4: Proportion of population practicing hygiene promotion at targeted camps, host communities and schools. Background and Context The WASH programme has faced complex external social, environmental, political, and institutional challenges from its inception in 2012. Among the first was managing expectations among the Syrian and Jordanian populations, particularly regarding the provision of water to refugees in view of a population that had to pay for the resource. Other challenges included Jordan’s economic crisis in 2012, managing perception of the refugee population, and making do with financial resources that fell short compared to need throughout the programme. UNICEF dealt with these obstacles while implementing a complex WASH programme and serving as a conscientious steward of its donors’ resources and interests. Purpose of the Evaluation The purpose of this evaluation was to assess the degree to which UNICEF’s WASH programme had achieved its intended results from its beginning in July 2012 through July 2017. The audience for the evaluation is current programme stakeholders, and future WASH programme implementers that may benefit from its lessons learned. The evaluation’s objective was to independently assess the degree to which the WASH programme’s design and implementation was relevant, effective, efficient, and will lead to some definition of programme sustainability. The evaluation also sought to discover if the programme had covered all relevant populations, especially the most vulnerable, and the degree to which the programme coordinated well with other similar initiatives and government strategy. UNICEF excluded the impact criterion from the evaluation’s scope, stating that “due to the relatively short time period since the start of the interventions.3 This evaluation covers WASH programme activities from July 2012 to July 2017. Geographically, it covers programming in four camps (Za’atari, Azraq, Cyber City, and King

2 Results frameworks from 2013 – 2017 are included as Annex H 3 Evaluation Terms of Reference – page 4. 2

Abdullah Park), host communities, informal tented settlements, and Rukban and Hadalat, the settlements on the Syrian/Jordanian border. It also covers refugees and Jordanians living in host communities with high numbers of refugees. Methodology ISG conducted a summative evaluation of UNICEF’s WASH programme. As described in the evaluation’s terms of reference, the team used a mixed method approach that included the review of UNICEF’s data and documentation, and the collection of qualitative data through interviews and Focus Group Discussions (FGD). In addition, the team reviewed WASH programme initiatives through on-site assessments. The document review established the programme’s implied logic model and delineated its goals and objectives. The review was also used to create interview and focus group guides. As described in the evaluation’s ToR, quantitative analysis relied on a review of the programme’s documents and other secondary sources. Primary, qualitative data collection was carried out through interviews and focus groups with stakeholders, beneficiaries, programme partners, UNICEF staff, donors, government officials, and other key informants. The evaluation team used a rapid assessment methodology to evaluate specific WASH programme interventions intended to improve water and sanitation for targeted populations. The evaluation team interviewed UNICEF partner organizations that were involved in the construction or rehabilitation of WASH facilities, the provision of WASH services, and promotion of hygiene initiatives. The purpose of these interviews was to understand partner perspectives on the degree to which interventions met evaluation criteria and to understand ways in which management of the programme excelled and would benefit from improvement4. Findings The full evaluation report organizes findings according to the Organization for Economic Cooperation and Development – Development Assistance Committee (OECD-DAC) criteria included in the evaluation’s ToR, which include relevance, efficiency, effectiveness, and sustainability as well as relevant humanitarian criteria. Findings are extensive as they cover five years of programme operations across a variety of geographic contexts and the range of programme activities. Here we provide a summary of the important findings. Relevance: In camps and settlements interventions were relevant and appropriate in terms of meeting rapidly changing WASH needs of targeted populations under difficult circumstances. The programme’s context was particularly difficult in its early phase. UNICEF maintained programme relevance during the winter storms, outbreaks of illness, and other obstacles to smooth management (such as trucker strikes) that arose throughout the 2012-2017 period. During 2012 and 2013, UNICEF determined needs by estimating the quantity of water, sanitation services capacity, and numbers of facilities that people displaced by the Syrian civil war required at transit centers and in Za’atari, King Abdullah Park and Cyber City. In this period UNICEF designed and implemented water and sanitation service options and level of services based on its analysis of the dynamic emergency response context, opinions of partners and other stakeholders, and the resources it had available. Beginning in 2014, UNICEF used an assessment process to maintain the relevance of its interventions. Examples of these assessments include a multi sector assessment of ITSs conducted by REACH in 2014, and assessments in the Za’atari and Azraq camps in 2015 and 2017. Other means of

4 A list of partners that the team interviewed is included in Annex B. 3

maintaining relevance were implemented throughout the programme and are discussed in the full report. UNICEF has made crucial improvements to water and wastewater infrastructure in host communities and informal tented settlements. However, it is difficult to judge the relevance of these interventions as many other international organizations provide services in these communities and the degree to which the UNICEF programme addresses needs generated by the refugee crisis is unclear. Effectiveness: In the provision of equitable WASH services and dissemination of WASH messages in the four camps, the WASH programme achieved high levels of effectiveness for the targeted population across all years. The programme also very effectively supported the WASH sector through technical support to the MoWI, as well as National WASH in School Standards. Government officials at the MoWI and WAJ reported satisfaction with the WASH programme’s contribution to planning documents, strategies, and policies. UNICEF’s achievement included successful coordination of the WASH sector from the camp to the national level. The programme met its targets under each of these activity areas. The WASH Programme struggled to meet its targets in its activities focused on supporting host communities, including WASH in schools. Efficiency: The WASH programme worked continuously to improve the efficiency of its operations. As an example, the evaluation team estimates that between 2013 and 2016, the cost of delivering water to refugees in the camps dropped from 3.34 Jod/m3 to 0.24 Jod/m3.5 The programme made similar efficiency gains in sanitation services and solid waste collection. Other Innovations led to more efficient coverage and management of services, such as introducing a voucher system for contractors performing desludging that made them accountable to camp residents and reduced potential corruption. Sustainability: UNICEF has done everything possible to ensure that its interventions are long-lasting and that mechanisms are in place for their operation. Examples include organizing WASH committees to oversee water access and hygiene promotion, constructing systems out of durable materials, and implementing messaging campaigns for hygiene promotion and water conservation. However, the ultimate sustainability of the programme is a complex issue. It includes determining if and when UNICEF will hand WASH management off to another entity, managing the transition of refugees either back to their place of origin or a permanent settlement elsewhere, the dismantling or integration of camp WASH infrastructure into Jordan’s municipal systems, and dealing with Jordan’s medium- term water scarcity issues. Coverage: The degree to which the programme is focused on the most vulnerable people is influenced by the WASH programme initial mandate, which is to maintain WASH service levels and standards in the camps. UNICEF maintains initiatives that monitor coverage in the camps and works with partners to ensure vulnerable populations receive necessary services. These initiatives work specifically to monitor services to people with disabilities, identify people who meet UNICEF’s vulnerability criteria, and ensure that interventions target vulnerable people and households. While people entering the camps may have been the most vulnerable at the programme’s outset, the evaluation team believes that in later years occupants of ITSs were more vulnerable and received less attention from the WASH programme. However, the necessity

5 Figures based on estimates for the cost of water delivery included in partner organization budgets from 2012 – 2017. 4

of maintaining service in the camps combined with budgetary shortfalls has prevented UNICEF from identifying and reaching that population extensively. Coordination: UNICEF greatly minimized duplication of services and service gaps in camps. Its role as WASH sector lead allowed it to assign sectors of Za’atari to the responsibility of partner organizations and to set standards among those organizations. It created similar standards and management arrangements in other camps. As sector lead, UNICEF implemented mechanisms for coordination in the face of emergencies, such as winter storms, and established third party monitoring systems that allow for quality and security incident reporting. UNICEF partners and GoJ officials that the evaluation team interviewed stated satisfaction with UNICEF as sector lead. One partner said that, “UNICEF … always (has) staff in Azraq.” A donor said that they were very happy with UNICEF’s coordination and partnership. A staff member of one partner said, “UNICEF does a very good job of controlling the work.” Lessons Learned and Recommendations The full report includes sixteen lessons for the reader’s consideration. Most important among them are the following: Quick decision making, and decisive action saves lives. The decisions and actions UNICEF took in the early days of the programme were crucial for initial inhabitants of Za’atari. In the settlements, UNICEF provided water and sanitation services to respond to the settlements’ exigencies. UNICEF moved quickly and effectively, with clear goals, such as ensuring that each person had at least 15 liters of water per day. The WASH programme also recognized that its intervention would have an impact on the larger Ruwaished host community. To address that impact, UNICEF rehabilitated the community’s water and sanitation facilities and provided support to identified vulnerable HHs. Installing quality water treatment and delivery equipment is crucial for the long-term sustainability and use. UNICEF’s installed higher cost and higher quality water infrastructure at Za’atari, Rukban and Azraq, including the treatment plant for Zaatari. In each case, a robust design was followed by very good installation and excellent post construction operation and maintenance. This was borne out through the evaluation team’s on-site investigations. In a key-informant interview (KII) with the contractor in Rukban, the operators noted that UNICEF always purchases high quality equipment. The result is operational efficiency, equipment that is long-lasting in the harsh environments, and reduced maintenance costs, downtime, and interruptions to supply. When direct consultation is not possible, other means of assessment can ensure programme relevance. In addition to direct consultation with beneficiaries, UNICEF put a system of assessments and Knowledge, Attitudes, and Practices (KAP) surveys in place. These studies kept UNICEF staff and sector participants informed of changes in the camps and other locations and assisted the programme in staying relevant and responding quickly to eventualities. Based on the documentation that was shared with the evaluation team as well as its interviews with UNICEF staff, partners, and other stakeholder, the programme appears to lack analytical rigor for determining the programme’s objectives, indicators or resource allocation, which has led it to overcommit to initiatives for which it did not have resources or capacity, primarily in host communities. The WASH programme should implement a monitoring and evaluation system, and comply with the strategic programme development protocols and reviews that are detailed in the Jordan Country Programme 2013-2017.

The WASH Programme and Context The object of this evaluation is UNICEF’s response to the Water, Sanitation and Hygiene (WASH) needs in Jordan as a result of the Syrian refugee crisis (July 2012 to July 2017).6 The response took the form of a programme of activities that began as an effort to provide WASH facilities and services to refugees entering Jordan as a result of the Syrian civil war. At its peak, the programme served refugees in six camps and settlements, host communities and informal tented settlements (ITS) across Jordan, and national government ministries. The programme’s main activities included: • Provision of water, sanitation, and hygiene facilities and services in the camps and settlements the programme serves. • Promulgation of hygiene and water conservation messages to refugees and host community members. • Provision of WASH infrastructure support in host communities to benefit host community members, refugees, and inhabitants of ITSs. • Promotion of Wash in School (WIS) improvement, including infrastructure support, a nationwide assessment, message promulgation, and capacity building for government ministries to better support WIS. • Delivery of technical support to the Ministry of Water and Irrigation (MoWI) to develop strategies and accompanying documents, and leading sector planning. • Coordination of the WASH sector in camps and at the national level. Description of the WASH programme Programme History UNICEF Jordan launched its WASH programme in Jordan in response to the influx of Syrian refugees during the first half of 20127. Prior to the programme’s launch in July 2012, UNICEF provided WASH services at Jordanian transit centers as an activity under UNICEF’s health programme8. The centers, which acted as Syrian refugees’ entry point into Jordan, predated the refugee camps UNICEF served as part of the WASH programme. Syrian refuges at transit centers that could prove that they had a sponsor in a Jordanian host community stayed at a transit center for one week, and then were permitted into Jordan. UNICEF’s WASH activities at transit centers included providing toilets, water tanks, water trucking to fill the tanks, and sanitation services to these centers9. As the number of refugees at transit centers increased, the Government of Jordan, the United Nations High Commissioner for Refugees (UNHCR), and the Jordanian Hashemite Charity

6 Evaluation Terms of Reference, page 2. 7 UNICEF did not participate in the WaSH sector before the refugee crisis (Source: Evaluation Terms of Reference page 4.) 8 Evaluation Terms of Reference, page 5 9 Source: Interview with UNICEF Staff- 26 July 2018

Organization (JHCO) obtained a tract of land owned by Jordanian’s armed forces to design and build the Za’atari refugee camp (Ledwith, 2014). UNHCR and its partner agencies cooperated to determine how to delegate the work for managing the crisis and the camp10. In the delegation of responsibilities, UNHCR asked UNICEF Jordan to take responsibility for leading the WASH sector. UNICEF was quick to rally expertise and funding for WASH programme management11. UNICEF’s primary role as WASH sector lead in July 2012, the start date of the scope of this evaluation, was to coordinate WASH activities for the Za’atari refugee camp as well as the Cyber City and King Abdullah Park camps, which by January 2013 housed approximately 1,500 people (UNICEF Jordan, 2013). Coordinating WASH during this phase was an expensive endeavor, further complicated by the Government of Jordan’s moratorium on building permanent structures in Za’atari. The response in this phase was expensive because UNICEF had to rely on contractors to provide water and mobile sanitation facilities quickly during the fast-moving crisis. UNICEF’s WASH team had five days to prepare for the opening of Za’atari camp12. The WASH team’s first objectives were to provide contingency supplies, supply water, acquire mobile toilets, and engage a contractor to provide desludging services.13 The WASH team also sought to engage partner NGOs to undertake WASH infrastructure construction, operations, and hygiene promotion in the camp. In Jordan, two organizations were available and had the necessary resources on hand to serve as partner organizations. UNICEF signed an agreement with one of the partners. However, that partner backed out of the agreement two days before the camp opened, forcing UNICEF to request that the remaining partner, the German Federal Agency for Technical Relief (THW) mobilize quickly. THW mobilized in 24 hours and was ready to serve the camp when it opened14. By September of 2012, the pace of new arrivals slowed from 100 per day to 50 per day15. UNICEF advocated with the Water Authority of Jordan to transition to more efficient means of serving the refugee population living in Za’atari. This advocacy resulted in UNICEF building WASH blocks to serve refugees living in Za’atari. WASH block construction started in the part of the camp where the first arrivals lived and moved to the newer areas. UNICEF also transitioned sanitary facilities away from mobile toilets to sealed pits and steel tanks made for sewage collection, commissioned Mercy Corps to drill and install two boreholes in the camp, and partnered with Japan Emergency NGO (JEN) and Relief International (RI) to provide hygiene promotion. In late 2012, UNICEF began serving host communities, starting with water system construction in Ramtha and Mafraq (UNICEF Jordan, 2012).

10 The evaluation team did not have access to programmatic documentation, such as proposals from UNICEF or agreements with donor agencies, that would have described why UNICEF was awarded responsibility for WaSH activities, or what UNICEF proposed. The presented information comes from evaluation interviews. 11 Source: UNICEF Staff Interview – 26 July 2018 12 Ibid. 13 Ibid 14 ibid 15 ibid

As it became apparent at the end of 2012 and beginning of 2013 that the refugee crisis would not soon subside, UNICEF Jordan developed a results framework that included output level WASH results for its 2013 and 2014 commitments. These included ensuring that relevant core commitments on WASH were met for Syrian refugees and affected Jordanians in all humanitarian settings (Output 1.3)16. The core commitments provided for: • Establishing a working group to coordinate the WASH sector, • Providing safe access to water for drinking and domestic use, • Improving sanitation facilities, • Providing hygiene items for those that needed them, • Promoting hygiene messages, • Providing for WASH services in schools, and, • Working with the MoWI and sector actors to address the impact of the influx of refugees on the sector’s longer-term chronic needs. UNICEF’s targeted WASH results changed between 2014 and 2015 as the programme aligned its objectives with government policies and sought to run its WASH operations more efficiently. In Za’atari UNICEF found that camp inhabitants preferred private WASH facilities to community WASH blocks and converted WASH infrastructure to meet that demand, resulting in the phasing out of 400 WASH blocks and working with the community to set standards for private facilities17. In 2014, a new camp, Azraq, was completed and UNICEF led its WASH activities as well. By 2016, UNICEF’s targeted results included working with the national government to finalize and disseminate the new policies, conduct research on water conservation, create standards for WASH in schools, promote water conservation and hygiene among the most vulnerable populations in host communities, camps, ITSs, and to provide for the WASH needs of vulnerable women and children through contracted tankering and also water networks in the camps and host communities. WASH programme Purpose, Components, and Management The WASH programme’s purpose evolved as the programme’s context changed from emergency response to longer-term operations. From 2012 to early 2014, the programme’s purpose was to provide for the immediate WASH needs of refugees arriving from Syria, and to respond to contingencies, such as the tensions that developed in host communities, as they arose18. In 2014 UNICEF began the process of planning for more efficient, longer-term operations in refugee camps. An example of an initiative developed through this process is the construction of a water network in Za’atari camp, which reduced the cost of water provision, improve the sustainability of the system, and make access to water more equitable19. In 2014 UNICEF also contributed to the Jordan Response Plan (JRP) 2015, which sought to “bridge the divide between resilience and humanitarian systems,” (Ministry of Planning and International Cooperation, 2014). UNICEF’s WASH programme coordinated with the JRP by formally planning for host community interventions, leading on the refugee response and supporting the government on the resilience component. The UNICEF WASH programme also aligned itself with the GoJ National Water Strategy 2016 - 2025, assisting

16 UNICEF’s WaSH results framework is included as Annex H. 17 Source: UNICEF Staff Interview – 26 July 2018 18 Source: UNICEF Staff Interview – 6 August 2018 19 Additional examples are discussed in the Efficiency section of the Findings chapter of this report. 8

the government to meet its objectives regarding efficiency and the National WASH in Schools Assessment (Ministry of Water and Irrigation, 2016). By 2017, the WASH programme divided its activities into four activity components: 1. Provision of equitable WASH services and the dissemination of WASH messages in the four camps (Za’atari, Azraq, King Abdullah Park and Cyber City) and two settlements in Rukban and Hadalat, to the most vulnerable people, namely children and women. 2. Provision of gender equitable support to the most vulnerable people in host communities (in all 12 Governorates) namely to children and women and including ITSs. 3. Support to the WASH sector through technical support to the Ministry of Water and Irrigation. 4. Coordination of the sector at camps and national level. As July 2017, the end date for the scope of this evaluation, UNICEF continued to implement the WASH programme along the four components described above. The WASH programme’s budget was approximately $355 million USD from 2013 to 2017. It averaged approximately $71 million USD per year. WASH Programme Logic Model and Results Framework The WASH programme has operated without an explicit strategy document or logic model to guide its implementation. The evaluation team also found no evidence of a programme document that would have detailed donor expectations. However, each phase of the programme demonstrated an implicit logic model that has evolved20. In the first phase of the programme, July through December 2012, the evaluation team found no evidence of a planned structure to the programme, such as a results framework or workplan21. The first results framework set objectives, outcomes, and output indicators for the years 2013 and 2014. Outcome level indicators in this framework were not entirely focused on WASH goals. The WASH programme still used a general UNICEF outcome framework that included indicators such as “existence of a comprehensive national monitoring system for school readiness to inform policy decisions on child disparities,” and “existence of a child and neonatal information system in support of child and maternal death audits.” The 2013/2014 output indicators seek to measure results from general activities and do not disaggregate achievement by targeted population. For example, they measure the cumulative “number of people provided with safe access to sufficient water for drinking and domestic use.” An output indicator from the 2013/2014 framework states “Relevant Core Commitments on WASH are met in all humanitarian settings.” In 2015, the WASH programme set the objective level indicators that it used at least through December 2017. In this framework, outcome indicators disaggregated by target population are specific to the WASH programme, such as “Proportion of population at camps and host communities including schools using improved drinking water as per Jordan standards.” Output targets and indicators are also adapted to the WASH sector. Examples include, “children, parents and community members are aware of and practice hygiene and water conservation,” and, “Number refugees provided with safe access to sufficient water for drinking and domestic use through water trucking.” While those indicators are not

20 The WaSH programme’s results frameworks from years 2012 to 2017 are attached as Annex H. 21 After the conclusion of the evaluation, UNICEF provided a workplan for the 2016/2017 period. 9

disaggregated by target population, the baseline data collected in support of them was, as was the end of year reporting that the WASH programme completed. In 2016, the results framework kept the same outcome level indicators, and evolved to provide an indicator for each population that the programme targeted. Examples include “Number of people in camps supplied with water (tankering or network),” and, “Number of people with access to improved WASH facilities in institutions.” By the most recent results framework that falls within the scope of this evaluation, which covers the period from 2015 to 2017, an implicit logic model had evolved. The WASH programme’s documentation implies a logic model that seeks to provide an effective response to the refugee crisis while protecting Jordan’s peace, prosperity, and resources. The programme has sought to achieve that objective by attempting to either implement or coordinate WASH activities in four areas: • Building capacity at the national level to equitably provide WASH services, • Promoting hygiene and water conservation practices in camps, host communities, schools, and ITSs, • Providing WASH services directly for vulnerable people, and, • Building the capacity of institutions, utilities, and other organizations to provide equitable access to water and sanitation services. The WASH programme states its objective, its highest-level target, as follows22: “Quality WASH facilities are sustained and utilised, and hygienic behaviours practiced, by the most vulnerable while ensuring the protection of the environment.” The programme measures the achievement of this objective through four outcome indicators. • Indicator 1: Existence of strategic planning capacity in MoWI and sector and other relevant institutions in Emergency preparedness and response. • Indicator 2: Proportion of population at camps and host communities including schools using improved drinking water as per Jordan standards. • Indicator 3: Proportion of population at camps and host communities including schools using improved sanitation facilities. • Indicator 4: Proportion of population practicing hygiene promotion at targeted camps, host communities and schools. Each outcome indicator’s achievement is in turn defined by output-level indicators. The implied logic model is illustrated in figure 1.

22 Results frameworks from 2013 – 2017 are included as Annex H 10

Figure 1 WASH Programme Implied Logic Model

The programme’s outcome statement and output statements, as articulated in the above logic model, come from UNICEF Jordan’s 2016 annual report (UNICEF Jordan, 2016)23. The activities reflect the reported activities and accomplishments of the WASH programme. The results framework reflects a broad measurement of achievement under three of the four programme components; briefly, 1) support to the camps and settlements,24 2) support to host communities and ITSs, and 3) support to the WASH sector through technical support to the MoWI. The results framework does not include a direct measurement of the fourth component - Coordination of the humanitarian WASH sector at a camp and national level. However, the degree of the programme’s achievement under the coordination component is closely related to the degree of achievement under the other three components. Programme Context The WASH programme faced difficult social, political, and institutional challenges from its inception in 2012. Managing an influx of refugees from a relatively water-sufficient area There is a large disparity in access to water resources among countries in the Middle East. Regionally, the average per capita total renewable water resources is 645 m3, however, the median country, Palestine, has 179.3 m3 for each of its residents25. Jordan and Syria tend toward opposite ends of the water access spectrum. Jordan has access to 123.4 m3 per capita, ranking the country below the median and well below the average for the region. Syria has greater than average access to renewable water resources, amounting to 908 m3 per capita26. The difference in access to resources sets the scene for the challenges that UNICEF would face in supplying a population used to ample water resources who had relocated to a region with far less water (UNICEF, 2012). Jordan’s largest refugee camp is the Za’atari camp located in Mafraq governorate. In 2013, 90 percent of the population of the Za’atari refugee originated in Daraa Syria. Daraa is a well-irrigated agricultural region, sourcing water from groundwater (wells), or the ample (but steadily declining) lake Muzayrib (Leestma, 2017), (Salman & Mualla, 2008) People from Daraa fled the Syrian civil war and began relocating to Za’atari camp in the Mafraq governorate in 2012. UNICEF’s challenging task was to meet the daily water needs of the refugee population, which fell short of the quantity of water refugees perceived that they needed, and balancing that against local Jordanian’s perspective that their scarce resources were being extracted and given away (REACH, 2014). Za’atari is located in the Mafraq governorate and has grown to be the 4th largest urban settlement in Jordan. In Mafraq, like much of Jordan, nearly all households are covered by a piped water network. However, water is only supplied one or two days a week into household storage containers. While the frequency of delivery is common for urban areas in the region, water is delivered less frequently to Mafraq than Za’atari. When that water is consumed, residents must purchase water privately. Jordanian’s living in Mafraq had stretched declining resources before the new influx of refugees increased water demand by 21 percent

23 The annual report can be found at the link below. Note that while the report lists 5 output statements, the activities under output 1 and output 4 are nearly identical, and the output statements are quite similar: https://www.unicef.org/about/annualreport/files/Jordan_2016_COAR.pdf 24 Settlements such as Rukban are not explicitly mentioned in the results framework. 25 FAO Aquastat: http://www.fao.org/nr/water/aquastat/data/query/results.html 26 Ibid.

(International Monetary Fund, 2017). By mid 2013, tensions over those resources would lead to local animosity and protest (Wildman, 2013). Pre-existing political attitudes toward refugees Within two years of Jordan’s independence in 1946, the Kingdom had to dedicate resources to host the approximately 100,000 refugees fleeing the 1948 war that established Israel as a nation. In 1967, war broke out again and 140,000 additional people fled into Jordan (Bocco, 2010). Over the subsequent decades, further inflows of refugees included Iraqi refugees in the 1970s, 80s, and 90s; 300,000 Jordanian expatriates returning unexpectedly in the 90s, and with the second war in Iraq in the early 2000’s approximately 750,000 returnees and refugees (Al Wazani, 2014). As a result of previous experiences hosting refugees, the GoJ has made decisions that influence the current refugee crisis and particularly the WASH programme. These decisions are discussed throughout this document. Examples of decisions that influenced the program were the GoJ’s initial decision to not allow the construction of permanent structures in Za’atari refugee camp, and the GoJ’s steadfast decision that they will not take ownership over the refugee camps, including the water network and supply of WASH materials and services. Legal and Institutional Framework of the WASH sector in Jordan One benefit of having low water usage per capita is that Jordan has achieved nearly universal coverage in water services (International Monetary Fund, 2017). Achieving strategic results such as universal coverage is a result of the water sector’s institutional and policy framework. The central institution responsible for the water management in Jordan is the Ministry of Water and Irrigation (MoWI). Under the MoWI, there are 15 units that manage administrative and technical functions. Included among these units are two agencies that deal with water issues and policies in Jordan and report directly to the Minister; The Water Authority of Jordan (WAJ) and the Jordan Valley Authority (JVA). The WAJ manages municipal water supply and wastewater services. It also manages operations, maintenance, planning and construction. Municipal water utilities fall under WAJ and are responsible for managing subscription issues and their own facilities. The JVA is responsible for the “social and economic development of the Jordan River Valley” including the development and protection of water resources (Kis, 2016). Several other institutions are important to Jordan’s water management. There are three water companies, The Yarmouk Water Company, The Aqaba Water Company, and the Miyahuna Water Company. Other ministries that play a role in water management are the Ministry of Agriculture and the Ministry of the Environment. The Ministry of Health regulates drinking water quality in Jordan. It monitors sources for water quality and inspects potential sources of pollution. The Ministry of Health issues permits for all produced or imported potable water. Economic Tension In 2012, as Syrian refugees were entering into Jordan and the newly constructed Za’atari camp. initially, patient Jordanians watched as free water and other resources, carried by a large number of trucks flowed into the camp (UNICEF, 2012). Meanwhile, Jordan was confronting an economic crisis. In March of that year, the International Monetary Fund (IMF) determined that the country’s debt levels were dangerously high given its economic turndown as a result of the 2008 global financial crisis (International Monetary Fund, 2017). The IMF recommended a series of measures meant to

stabilize the economy, including cutting subsidies for fuel.27 By November, protests erupted in Amman and Irbid.28 The image of Syrians receiving daily water deliveries extracted from depleted Jordanian resources while poor Jordanians were made even poorer exacerbated tensions in camps and in communities (Luck, 2013). Funding Environment In 2018, international donors have only provided approximately 43 percent (Financial Tracking Service, 2018) of what is required to respond to the regional Syria Refugee Crisis, and about 34 percent of the cost of providing a WASH sector response (Financial Tracking Service, 2018). In 2017 donors funded 52 percent of the requirements of the regional response plan for the Syrian Crisis (ReliefWeb, 2017). The international community’s difficulty in coming up with sufficient funds to sustain an effective response to the crisis has created a heavy burden for UNICEF’s WASH programme in Jordan. The lack of funding has particularly reduced UNICEF’s ability to serve host communities and ITSs (UNICEF Jordan, 2013)29. Figure 2 below shows the amount UNICEF budgeted for interventions across programs for the Jordan Response Plan against the amount that was raised and spent by year30.

Figure 2 Funds budgeted vs mobilised by year (USD in millions)31

300

250

200

150

100

0 2013 2014 2015 2016 2017

Budget Mobilised

27 https://www.imf.org/external/pubs/ft/scr/2012/cr12119.pdf 28 https://uk.reuters.com/article/uk-jordan-protests-idUKBRE8AD00F20121114 29 Lack of funding was also the reason given for missing host community targets even in 2016, when UNICEF Jordan exceeded its fund raising targets. 30 UNICEF Jordan notes that the table includes rolled over funding and earmarked amounts totaling $30 million from German donors for Za’atari. 31 The 2015 Annual report included funds mobilized through November. The chart adds an additional 1/12 to funds mobilized to estimate funds utilized in December.

Description of the Evaluation Purpose and Objective The purpose of this evaluation was to assess the degree to which UNICEF’s WASH programme had achieved its intended results from its beginning in July 2012 through July 2017. UNICEF intends to use the lessons that the evaluation process generates to inform future WASH programmes that occur in a similar context. The Evaluation’s objective was to independently assess the degree to which the WASH programme’s design and implementation were relevant, effective, efficient, and will lead to some definition of programme sustainability. The evaluation also sought to discover if the programme had covered all relevant populations, especially the most vulnerable, and the degree to which the programme coordinated well with other similar initiatives and government strategy. UNICEF excluded the impact criterion from the evaluation’s scope, “due to the relatively short time period since the start of the interventions.32” The evaluation questions are included in the findings section of this report, and in Annex A: Evaluation Terms of Reference. Scope This evaluation covers UNICEF’s WASH programme from July 2012 to July 2017. This evaluation assessed WASH programme activities, including water provision, sanitation facilities and services, hygiene promotion, solid waste management, desludging, WASH in Schools initiatives, WASH sector coordination, and support to the Government of Jordan through the Ministry of Water and Irrigation (MoWI). Geographically, it covers programming in four camps (Za’atari, Azraq, Cyber City, and King Abdullah Park (KAP), host communities, informal tented settlements, and Rukban and Hadalat, the settlements on the Syrian/Jordanian border. It also covers refugees and Jordanians living in host communities with high numbers of refugees. Evaluation Methodology ISG conducted a summative evaluation of UNICEF’s WASH programme. The team used a mixed method approach that included the review of UNICEF’s qualitative and quantitative data, and the collection of qualitative data through interviews and Focus Group Discussions (FGD). In addition, the team reviewed WASH programme initiatives through on-site assessments. The Evaluation Matrix, included as Annex G, provides the evaluation questions, detailed answers to those questions, the key judgement criteria that was used in constructing the evaluation team’s response to evaluation questions. It also includes the sources of information that provided evidence for the response to each question. The matrix is organized according to the evaluation questions in the evaluation’s ToR and by evaluation factors, namely; relevance, effectiveness, efficiency, sustainability, coverage, and coordination. Desk Review The desk review established the programme’s implied logic model and delineated its goals and objectives. UNICEF provided the evaluation team with an initial set of documents at the start of the evaluation. The evaluation team reviewed these documents to understand how the WASH programme evolved as the refugee crisis developed, estimate the WASH

32 Evaluation Terms of Reference – page 4.

programme’s achievements related to goals and objectives, understand how programme management determined strategies, set targets, designed work, and the degree to which this planning led to successful outcomes. For the inception phase of the evaluation, the desk review was also used to create interview and focus group guides. The evaluation team also noted where information was incomplete or missing and provided requests to UNICEF for additional documentation to fill in information gaps over the course of the evaluation. Document collection and review occurred throughout all phases of the evaluation. The list of documents that the evaluation team reviewed is included in Annex D, with additional documents listed in the Works Cited section. Quantitative Analysis As described in the evaluation’s ToR, quantitative analysis relied on a review of the programme’s documents and other secondary sources. The review identified the programme’s goals, objectives, and output targets for each year of implementation and sought to understand the degree to which targets were met. The indicator framework that informed this analysis is included as Annex H to this report33. The evaluation team also used expenditure reports to attempt to understand the programme’s financial management and generate analysis for the questions related to efficiency in the evaluation’s TOR. UNICEF’s financial information provided to the evaluation team was incomplete and not well documented. UNICEF does not track expenditure by year, programme component, or result. It also does not track indirect costs associated with the programme’s implementation. These limitations prevented a detailed analysis of efficiency. The evaluation team estimated costs from an incomplete list of the WASH programme’s expenditures and estimates included in partner organization’s budgets. Documents that included quantitative data are also included in Annex D. Primary Research: Qualitative Data Collection Methods and Sampling Qualitative data collection was carried out through interviews with stakeholders, beneficiaries, programme partners, UNICEF staff, donors, government officials, and other key informants. The evaluation team also conducted FGDs among programme beneficiaries. The goal of interviews and FGDs was to capture information representing the range of experiences among each targeted population group, with a focus on the most vulnerable. Thus, for interviews, the evaluation team utilized a purposive sampling strategy in camps, host communities and ITSs. The number of interviews for each targeted population sought to fully answer each research question to the point that each source of information is saturated. The number of interviews location and type is included in Annex B34. Interview and focus- group guides are included in Annex E. Coverage in Interviews and Focus Groups The evaluation team sought to ensure that the views of all different groups covered by the WASH programme were represented through interviews and FGDs. Interviews and focus groups included women and men, people living near resources and more distant, people new

33 Annex H details information extracted from UNICEF’s annual workplan and End of Year Reports. 34 The number of interviews assumes that the team is able to access each camp and host community and that appropriate respondents are available and willing to participate in interviews.

to their areas of inhabitation and people that have lived in the area for a long time. However, the numbers of people interviewed does not represent all groups equally across topic areas. Rapid Assessments and User Interviews The WASH programme carried out rehabilitation and new construction of WASH infrastructure, and organized WASH services to improve water access and sanitation. The evaluation team used a rapid assessment methodology to evaluate specific projects intended to improve water and sanitation. Infrastructure evaluations included facility inspections and interviews with key informants about the infrastructure, including how it was and is being utilized. Through this approach the evaluation team developed a clear picture of the construction quality and operational effectiveness of the infrastructure. The team evaluated UNICEF interventions in two parts. The first part was the observation component. In this component, the evaluator inspected the intervention and determined where critical control points or bottlenecks are apparent, while at the same time observing how people interact with the intervention. The second component comprised qualitative data collection. During this component the evaluator interviewed people who had varying levels of access to the intervention. The evaluator sought out people that should have had access to the intervention but don’t to understand why access is limited. WASH Messaging and Practices The evaluation assessed the WASH programme’s messaging and behaviour change activities according to the evaluation criteria. The evaluation explored the degree to which messages were absorbed and converted into new practices by reviewing the data provided for desk review and analysing the information gathered in stakeholder interviews. Partner Interviews The evaluation team interviewed UNICEF partner organizations that were involved in the construction or rehabilitation of WASH facilities, the provision of WASH Services, and promotion of hygiene initiatives. The purpose of these interviews was to understand partner perspectives on the degree to which interventions met evaluation criteria and to understand ways in which management of the programme excelled and would benefit from improvement. A list of partners that the team interviewed is included in Annex B. Duty Bearers and Staff The evaluation team coordinated with UNICEF Jordan to interview appropriate duty bearers in government ministries, utilities companies, municipalities, and communities. These interviews included, the MoWI, the Yarmouk Water Company, and the Water Authority of Jordan. These duty bearers have assisted UNICEF in program delivery and/or benefited from UNICEF’s capacity building and sector coordination activities. The evaluation team also interviewed UNICEF administrative, programmatic, and senior staff. Stakeholder Interviews Where applicable, the evaluation team interviewed stakeholders that had an overview of the WASH programme’s implementation in their community. These stakeholders include WASH committee members, camp or host community duty bearers, and other stakeholders that UNICEF staff thought would offer insight into the programme. Analysis Method Durng the desk review and after field visits, the ISG Team coded qualitative data to ensure that themes matched evaluation criteria and that issues of gender, vulnerability, and protection were highlighted. Coding the data facilitated identifying clear themes across primary and secondary research.

Much of UNICEF’s quantitative data is maintained in unsearchable .pdf documents. To extract as much of the data as possible, the evaluation team used optical character recognition software to recover as much quantitative information as possible. This data was particularly useful in answering questions related to efficiency and effectiveness. Limitations to the Evaluation There are several factors that may limit the evaluation’s analysis. These are listed below • Quality of data provided by UNICEF – The evaluation team’s ability to verify indicator targets and select appropriate respondents for qualitative data collection depends on the quality and organization of the WASH programme’s data. Populations that have participated in the programme, but for which no records were kept, or records are incomplete, may have received insufficient treatment by the evaluation. These populations include host communities benefiting from infrastructure improvements and residents of camps that have closed. Records for the early years of the programme may have been incomplete and/or poorly organized. • Financial and Procurement Data Provided by UNICEF – Efficiency and value for money calculations depend on the reliability of the financial and procurement data that UNICEF provides. UNICEF does not track financial information by result, activity, programme component, or year. UNICEF also does not track indirect costs allocable to a specific programme. This evaluation provides an analysis of the data available to the evaluation team, but cannot audit its accuracy or reliability. Evaluation findings regarding efficiency and value for money are estimated. • Sample Selection – The qualitative data collection employed by this evaluation relied on purposeful sampling rather than utilizing random sampling. While the qualitative data collection creates a narrative around the evaluation’s criteria, it also suffer from particular biases, including UNICEF’s ability to provide information about appropriate project sites, time restrictions in data collection, and access to small numbers of beneficiaries. Additionally, the evaluation was conducted during summer months, when most schools were closed, which limited the evaluation team’s collection of data related to Wash in Schools. • Availability of Respondents – Many of the WASH programme’s activities were carried out four or more years ago. Many interventions may have been carried out without the knowledge of direct beneficiaries, especially those that involved general infrastructure or were implemented through partners or contractors. The lack availability of respondents limited the number of interviews and focus groups the evaluation team carried out. • Reliance UNICEF for guidance – The evaluation team relied on UNICEF staff for access to stakeholders and guidance for examples of project successes and failures. Even with the best of intentions this type of reliance may create biases in the final report, which the evaluation team will identify and describe. Ethical Principles in Conducting the Evaluation ISG’s work is based on four core principles: 1. Participatory evaluation design and implementation: To the extent possible, all people who have a stake in the outcome of a study or evaluation must have a chance to identify risks in conducting the study and opportunities to suggest ways to reduce those risks. Following this principal, we solicit comments and input from our clients on inception and design documents, as well as data collection and research tools. When clients have created terms of reference, we review the terms for input from stakeholders and ask for a representative stakeholder review when possible.

2. Respect the rights, privacy and dignity of evaluation stakeholders: ISG aims to minimize risk in evaluation management and outcomes. The primary aim of our work is to benefit the people who are most affected by its outcomes. We put the safety, dignity, and privacy of those that participate in our projects above the rewards that we hope to achieve for ourselves or our firm. As such, we ensure that survey, interview, and focus group participants are fully informed of the nature and purpose of the research that we are conducting, obtain their consent before asking any questions or engaging them in any other research, and allow them the opportunity to deny or remove consent at any point in the process. We do not use names or identifying information in reports, except in specific circumstances and then only if the participant is fully informed and in agreement. We minimize risk to participants, including carefully designing questions that may recreate traumatic or harmful feelings. Finally, ISG believes that participants in our work have the right to benefit from it. We work with our clients to produce multiple versions of documents and materials to facilitate the distribution of results. 3. Informed and reasonable judgements: The work that ISG conducts often influences the distribution of resources and activities in vulnerable communities. We consult with our clients to ensure that conclusions are drawn from rigorously vetted evidence, and that following actions are based in reliable findings. ISG’s evaluators detail the strengths and weaknesses of our methodology and the limitations of the study given available resources and contextual barriers. 4. Secure Data Storage Protocol: ISG utilizes secure data protocols to ensure that respondents’ information is not used in any way beyond that which they have provided permission. At the end of each research study, evaluation, or project that ISG concludes, project team members submit hard copies of records, including but not limited to research notes, photographs, or portable recordings to ISG’s designated project manager. The project manager ensures that notes, recordings, and all other records are moved onto ISG’s secure servers and deleted off of laptops and other portable storage devices. ISG stores our clients’ data, as well as data collected through primary research on secure servers through which only the project team has access. If requested, ISG destroys all data related to a project, including paper records, or records kept through all other means. If not request is made, ISG’s director of programs oversees the maintenance of data on the secure server in perpetuity, or for the time period each client or agreement mandates.

Findings This section details the evaluation team’s findings. The section is generally organized by evaluation criteria, question, and then geographic intervention target. The sections discuss findings by programme year, where possible. A summary of the judgement criteria, indicators, sources of documentary information, and data collection tools used to derive each finding is included in Annex G: The Evaluation Matrix. Relevance Evaluation questions 1 through 3 related to the programme’s relevance. Evaluation Question 1 • To what extent were affected people consulted on their preferences with regards to water and sanitation service options and level of services? • Were interventions appropriate in terms of meeting their basic needs?

• Was there a feedback loop and monitoring system in place for reflecting the learning and evidence to improve programming, especially for the humanitarian response? The emergency response period (2012 – 2013) UNICEF and its partners were assigned responsibility for providing WASH services in the newly opened Za’atari refugee camp in July 2012. Construction of the camp began on July 20th and the camp was opened on July 30th (Wilkes, 2012). During those ten days, the WASH programme had to create and implement a response plan for the 10,000 refugees that would shortly arrive, and the approximately 100,0000 more that would come over the following two years. UNICEF and partner organization staff that the evaluation team interviewed recalled the urgency of operations in that time period. One staff member recalled that once he was notified, “we had five days to respond. All we had was some contingency supplies and orders from Copenhagen.” A UNICEF partner team member explained. “At the beginning, it was an emergency response. We were not worrying about vulnerable populations. I don’t think there was a lot of consultation at the time35.” UNICEF’s determined how to best provide for refugees’ ‘WASH needs through talks with partners such as WAJ, Mercy Corps, ACTED, Oxfam, and Relief International, and based on the assessments and information that they’d gathered. The tactics that they employed took into consideration the Convention of the Rights of the Child (CRC), UNICEF’s Core Commitments to Children in Emergencies, and the Convention on the Elimination of All Forms of Discrimination against Women (CEDAW). Through these means, UNICEF created estimates of the best way to provide WASH services for the newly arriving refugees. UNICEF also consulted humanitarianresponse.info for guidance on issues such as accountability and feedback mechanisms (WaSH Sector Working Group, 2012). UNICEF utilized institutional resources in this phase. A regional WASH Specialist was mobilised and a Regional WASH Advisor from South Asia was sent to Amman to coordinate the response. Also, staff reported that experts from other offices assisted the effort as did UNICEF headquarters. The evaluation team did not find documentary evidence of support during this phase, and it is unclear which internal resources UNICEF consulted for guidance on WASH response in an emergency situation. For example, UNICEF may have consulted the Global WASH Cluster, an organization that UNICEF leads, published the “WASH Cluster Coordination Handbook” in 2009. The handbook includes specific guidance and tools for managing WASH in emergency situations, including a schedule of activity and methodology for conducting assessments. UNICEF’s first concern was providing a steady flow of water and adequate sanitation facilities. Once minimum water and sanitation standards were met, UNICEF began to focus on providing facilities for women and people with disabilities. A UNICEF staff member who worked in Za’atari recalled that “At the beginning, water supply was the main focus. There was no electricity, no toilet access for night, especially for women. There were no proper toilets for people with disabilities at the beginning. Some NGOs tried to have a chair and a small pit for people with disabilities36.”

35 World Vision Staff Interview – 30 July 2018 36 UNICEF Staff interview – 26 July 2018

To determine the needs of vulnerable people, including people with disabilities, UNICEF relied on the advice of other organizations operating in Jordan and Sphere standards. For example, in November of 2012, Handicap International provided the WASH sector with a document titled. “Accessible WASH Facilities to promote inclusion of persons with disabilities, injuries, and other vulnerabilities; Za’atari Refugee Camp.” The document details standards for communication, entrances, fixtures, toilets, and showers. Facilities that met these criteria were eventually installed, although it is unclear from the documentation when. Other documents from the 2012-2013 period illustrate the criteria the WASH programme used to determine service options and levels. The Jordan RRP5 Update in August 2013 notes that WASH services were to be designed and delivered in camps with “households living in temporary settlements, female headed households and the needs of children, disabled, and elderly (The United Nations, 2013).” The next priority listed is “host communities with the largest concentration of refugees." The degree to which these priorities were addressed in a timely fashion is unclear from the documentation. However, the referenced documentation from 2013, the earliest to which the evaluation team had access, shows that providing water supply and basic sanitation was the first priority, followed by ensuring that it covered vulnerable populations. In the emergency response phase of the programme, hygiene was poor and waterborne diseases were common. In 2013 World Vision, a WASH programme partner, estimated that there were 110 cases of waterborne disease for every 10,000 people; a very high rate (World Vision Programme Cooperation Agreement 19-13 ). UNICEF and its partners would address this issue in the next phase of the programme. Achievements and Results (2013 – 2015) In the emergency response phase, the WASH programme had some remarkable achievements. Principal among them, the programme supplied water and sanitation for all inhabitants of the camp including those that arrived newly each day. The 2015 KAP survey for Za’atari revealed UNICEF’s responsiveness to refugees needs regarding sanitation. During the planning phase for the waste water network, 94 percent of residents were aware of the planned network and 96 percent of those felt it would improve life in the camp (Za'atari Hygiene Promotion Working Grouip, 2016). A clear indication that the WASH programme was working in concert with Za’atari’s residents. These achievements are discussed in detail in the effectiveness section of this report. However, the unavoidable process of quick decision making and implementation that occurred in 2012/2013 had some consequences. Locating Za’atari over one of Jordan’s major aquifers led to expensive measures to protect wastewater leakage. The materials of the initial septic tanks were not adequate to protect the groundwater, and so they had to be replaced. The WASH blocks in Za’atari were difficult to maintain because they were prone to vandalism and theft, and unpopular with the camp residents. As a consequence, camp residents improvised private sanitation facilities, leading to pools of greywater throughout the camp37. Working with local partners, UNICEF developed a plan to drain the greywater and provide a sufficient drainage solution, including desludging on a daily basis. Eventually,

37 WaSH Sector Coordination; Minutes of weekly Za’atari camp WaSH sector coordination group. Sunday 24 November 2013

UNICEF followed the lead of camp residents and installed household sanitation facilities for vulnerable households that did not already have facilities, leading to another round of removing and installing environmentally effective septic tanks. The WASH blocks would be decommissioned. Though recycling happened where possible, it was an expensive process38. Another consequence of the need for quick decision making was that, in the initial emergency response phase, the camp’s effect on Jordanians living near the camp and in other host communities with high concentrations of refugees was not taken into consideration adequately. As an official at the MoWI said, “The more high-quality services you provide inside the camp, the more people outside the camp will care.” As described in the context section of this report, Mafraq, where Za’atari camp is located is a poor area and per capita receives less water than the refugees were accustomed to in their home of origin. In addition, Mafraq residents pay for inconsistently available water, while refugees inside Za’atari received regular shipments of free water. In October 2012, tensions between refugees and Jordanians living in Mafraq were erupting (Seely, 2012). This led to engagement in host community projects, discussed in the next section. Stability and Efficiency (2014 – 2017) In April 2014, the Azraq refugee camp was opened. The WASH programme used a number of lessons learned from Za’atari in designing WASH services for Azraq. For example, water supply and sanitation facilities were less centralized and located closer to smaller clusters of households, household clusters were composed of people that came from similar areas of Syria, cleaning was organized by cluster with no incentives for cleanliness provided, community mobilization and hygiene promotion was initiated as soon as refugees moved into the camp, refugees were more involved in the day to day operation of WASH facilities, and WASH committees were set up earlier than in Za’atari camp (World Vision Programme Cooperation Agreement 19-13 ). However, the public toilets and showers in Azraq remained unpopular as in Za’atari. Focus group discussions that the evaluation team held in Azraq indicated that, as in Za’atari camp, women resist using public facilities due to lack of privacy.39 However, In 2017, a KAP study conducted by Action Against Hunger indicated that 100 percent of survey respondents said that they used WASH blocks in Azraq and felt safe (ACF, 2017). In 2014, UNICEF began seeking other means to understand needs and preferences of refugees living inside and outside of the refugee camps, and of Jordanians living in communities that were hosting large numbers of refugees. In January 2014, a WASH Sector Gender Analysis was published, to guide future programme implementation. Broad assessments were also carried out to understand targeted populations’ needs and preferences, with a focus on the most vulnerable populations. The first such assessment, undertaken in August 2014, was a multi-sector assessment by REACH, an international initiative to develop information tools. This assessment covered 125 ITSs and included a household survey as well as semi-structured ad hoc interviews and focus group discussions. The assessment was used to demonstrate how the most vulnerable in the ITSs could be engaged in defining their

38 UNICEF Staff Interview – 26 July 2018 39 Focus Group Discussion number 1– Azraq

needs (REACH, 2014). In addition, UNICEF conducted a number of Comprehensive Child Focused Assessments in Azraq and Zaatari camps in June 2015. The process was repeated in Azraq in February 2017. These assessments were also carried out in partnership with REACH. During this evaluation’s focus groups and interviews, residents of Za’atari and Azraq indicated that they are asked for their suggestions and opinions during awareness raising sessions. Residents in Azraq said that at the beginning of their residency, they participated in the process of improving the camp40. After the emergency response phase, UNICEF worked with its partners to adapt WASH facilities and services to the needs and behavior of camp inhabitants. These needs and behaviors were identified through observation and through feedback and requests from community members. For example, in Za’atari UNICEF observed residents creating private facilities inside their homes rather than using the public WASH blocks. Private bathrooms were provided in some homes, while in may people obtained equipment and installed their own bathrooms. In response, UNICEF adapted its intervention to provide communal tanks for households. Each tank served up to five households. This adjustment better suited the customs and privacy needs of the refugees. It also responded to the realization that the camp’s residents would live in Za’atari for an undetermined period rather than the initially planned for six-month period41. In Azraq, UNICEF increased the number of water tap stands and reduced the distance from housing units. As of July 2017, water was Figure 3 Public Latrine at Azraq camp distributed from multiple points within the camp (tap stands) in two shifts, from 7am to 11 am and again 2pm to 6 pm. It was then carried to homes via containers and then stored and used over 24 hours or more. Usually these tap stands were provided with 4 faucets, where each faucet served 5 to 6 neighborhoods. Each neighborhood has 12 households with an average 5 individuals. Participants in this evaluation’s focus groups stated that the increase in number water tap stands was an important improvement. They said that when the number of tap stations was limited, access to water was difficult and water was scarce. The process of getting water was not well controlled as people had to arrange themselves in queues, which made the waiting time longer and resulted in conflicts. These issues have been reduced since the number of taps has increased. Interview respondents said that walking home with the heavy weight was burdensome, though not as much as previously42. Focus groups in Azraq also indicated lack of satisfaction with sanitation facilities. The number of toilets is limited and insufficient43. They expressed their concern regarding the

40 Azraq FDGs 1 - 4 41 Source: Interviews and on-site assessments in Za’atari camp. 42 Source: Interviews and on-site assessments in Azraq camp. 43 UNICEF notes that there is 1 male toilet and shower and 1 female toilet and shower per 12 households. The facilities are on opposite ends of a 30-meter plot.

ratio of users to toilets, uncleanliness of toilets, and cultural taboos regarding women using public showers. They also explained that the public showers are not used by women or girls. UNICEF staff explained that in Za’atari, most households installed their own toilets. There are fewer building and financial resources in Azraq, so households are less able to construct their own toilets44. Along with adopting facilities to the requirements of camp inhabitants, the WASH programme focused on efficiency, such as reducing the cost of water provision through the development of boreholes, pumping stations, pipelines, and a wastewater network. The programme also sought to control the supply and standards of services, and to depend less on contractors incase of an increase in price, a reduction of quality, or a strike. Settlements In Rukban and Hadalat, as in the camps, UNICEF served a population with dire needs. Time pressure and the desperation of the Settlements’ inhabitants made consulting them about their preferences impractical. Hygiene services for the people in the settlements were not managed as the population rose rapidly. In January 2016 the population reached 17,000 in the settlements, with 1,400 in Hadalat and 15,600 in Rukban. By March 2017, the population had grown to over 77,500 in habitants (Jordan and the Berm Rukban and Hadalat 2017-2018, 2017). The WASH Programme also assisted with transitioning 28,000 people from the settlements to Azraq camp. The urgency of the situation is detailed in UNICEF’s six-month emergency plan for these settlements. The emergency plan notes that there were a “disproportionate number of pregnant women” among the residents of the settlements, that the risk of disease because of unsanitary conditions was growing, that the difficult winter weather posed risks for children, and that there were ongoing protection concerns for vulnerable populations. Between January 2015 June 2016, the WASH programme installed mobile sanitation trailers and distributed general hygiene kits , jerrycans, and buckets. UNICEF also provided training of health staff and community workers. By December of 2016, The WASH programme was operating more stable and cost-effective solutions to provide water and sanitation. The evaluation team’s assessment reviewed Rukban’s borehole, pumping stations, and treatment system. The relevance of the water Figure 4 The stand-by power generation increases the ability system was demonstrated by its strong design, of the water supply and distribution system to function needed in harsh and resource scarce areas such as Rukban. The water system has capacity for multiple contingencies including leaks, power outages, or effluent quality issues (figure 4). There is 800 cubic meters of storage at the borehole site with tanks with the ability to operate independently in case of leakage. The system appears easy to operate as well. Gama Engineering and contracting Corporation is the

44 UNICEF’s explanation was provided as a comment to the first draft of this report.

operator of the system. They maintain staff on-site 24 – 7. Staff performs daily O&M including: i) pumping from the borehole to the tanks, ii) checking the chemical concentrations every 2-3 hours, iii) routine cleaning of the sand filter daily, and iv) repair and replacement of parts as needed45. UNICEF provided water and sanitation services to respond to the settlements’ exigencies. They moved quickly and effectively, with clear goals, such as ensuring that each person had at least 15 liters of water per day. The WASH programme also recognized that its intervention would have an impact on the larger Ruwaished host community, so they undertook to rehabilitate sanitation facilities there. UNICEF also implemented projects in a hospital, schools, and vulnerable households chosen through a vulnerability assessment. The WASH programme has maintained weekly updates since the initiation of work. The updates monitor operations and needs. The programme’s monitoring includes regular surveys and assessments. These tools ensure that WASH programme activities remain relevant and effective. Host Communities and ITSs It is difficult to judge the degree to which the WASH programme is relevant to host communities and ITS needs. The WASH programme’s purpose was to respond to “Water, Sanitation, and Hygiene needs in Jordan as a result of the Syrian refugee crisis.” Eighty percent of the WASH programme’s funds and resources have gone into the management and construction of the refugee camps, which host 20 percent of the refugees in Jordan, while 80 percent of Syrian refugees living in Jordan are in host communities46. Many other international organizations and agencies provide services in these communities. In the emergency response phase of the programme, the WASH programme focused on the camps because the programme was the only source of water sanitation for that population, whereas refugees in host communities and ITSs presumably were availing themselves of other sources. UNICEF’s improvement of the water and wastewater infrastructure in host communities is significantly important. However, the degree to which it addresses the needs generated by the refugee crisis and the needs of the most vulnerable is uncertain. UNICEF’s work in ITSs is guided by vulnerability maps which indicate areas of greatest need and targets vulnerable households. UNICEF also focused on areas where government has indicated a specific need. Even with UNICEF’s improvements, some communities continue to only get water every 7 to 9 days and just for few hours, as was the case prior to UNICEF’s involvement47. Also, Non-Revenue Water (NRW) in many communities is higher than 50%. More relevant and strategic initiatives may have involved promoting wide scale NRW reduction, rainwater water harvesting, water conservation, and groundwater recharge programs as real strategic interventions that would yield long standing results.

45 Onsite assessment and interview with Gama Engineering 46 UNICEF notes that donors earmarked much of the funds for specific infrastructure interventions. 47 Host community interviews in Irbid

UNICEF reports that they have served 114 ITSs, but the total number of ITSs is unknown, and the level of that service is unclear. Monitoring System for Learning The WASH programme has had a monitoring system set up in the camps since the beginning of the program. A contractor initially managed the monitoring system.. However, the contractor’s performance was found to be unsatisfactory, so the programme decided to directly manage the system. As a demonstration of the improvement in services, logged points of contact increased from 7 per week at the beginning of the program to 20 per week shortly after UNICEF took over management. At its peak, the programme managed 60 points of contact per day48. The evaluation team could not find evidence that the WASH programme is structured as a learning organization. It does not appear to have taken guidance from UNICEF’s experience in WASH programming at the outset of the programme. While it has certainly made improvements to its activities in terms of efficiency and responsiveness to people’s needs, it did not build monitoring networks or feedback loops that fed into a system that would allow for learning or informing strategic management. The evaluation team found no documentary evidence that UNICEF’s WASH programme adhered to its Monitoring, Evaluation and Programme Management policy or annual reviews, a mid-term review in 2015, and specific programme performance assessments (UNICEF Jordan, 2012). It also did not use its experience or other information to create programme strategies. A 2016 report, “UNICEF WASH Actions in Humanitarian Situations: Synthesis of Evaluations 2010 – 2016,” The report found that there is a “need for a more consistent process of learning, disseminating, and applying knowledge and good practices generated through experience (Hūls, 2017).” This evaluation finds that this conclusion also applies to the UNICEF Jordan WASH programme. This theme is further developed in the efficiency, effectiveness, and sustainability sections of this report. Evaluation Question 2 • To what extent did the interventions consider the needs and the priorities of the MWI and were they coherent with global references and the regional and national response to the Syria Crisis? UNICEF’s work in WASH is in line with the priorities of the National Water Strategy, for which the key reference document is the National Water Strategy 2016 – 2025. UNICEF is focused on the sustainable operation of boreholes to supply water and create resilience. UNICEF also recognizes the government’s concern that Jordan’s northern governorates have not benefited from the government’s efforts to increase water supply (Ministry of Water and Irrigation, 2016). In response, UNICEF has focused much of its host community interventions on improving WASH in the northern governorates. Those efforts are based UNICEF’s vulnerability mapping results.

48 Interview with UNOPS Staff – 1 August 2018

Both the MoWI and WAJ reported in interviews that the GoJ-UNICEF partnership is successful. The WAJ praised UNICEF’s success at sector coordination despite budget cuts49. The MoWI appreciated UNICEF’s role in coordinating the sector50. Interviews at MoWI and WAJ demonstrate the Government of Jordan’s struggles with its strategy and UNICEF’s response. Senior staff at the MoWI praised UNICEF’s responsiveness inside the camps but sees that project’s host communities are more dependent on available funding51. MoWI said that when the WASH programme first took on host community projects, there was a miscommunication regarding the meaning of the protocol UNICEF established with the MoWI. The protocol included a mandate and budget for UNICEF to improve infrastructure in governorates. The ministry didn’t realize that the budget was prospective and moved those items off of its budget for the year. When fundraising fell short, those governorates were underserved52. An interview with senior staff at MoWI indicated two concerns. The GoJ has made it clear that the government will not take over responsibility for WASH services in the camps in any form. The MoWI has prevented studies of potential government involvement. The MoWI wonders what UNICEF’s plan is for sustainability or handover in the camps. Secondly, the MoWI wondered about UNICEF’s medium to long-term role in the sector and was desirous of closer collaboration if UNICEF were seeking to look at medium to long term investments53. Evaluation Question 3 • How responsive was the UNICEF WASH programme over time to changes in the external environment? UNICEF has responded to changes in the external environment, while maintaining standards in its programs. In the initial phase of the programme, the GoJ and INGOs in Jordan prepared for a temporary camp. This preparation included procuring tents, portable sanitary facilities, and trucked in water supplies. While these interventions were effective, their expense made them unsustainable. When it became apparent that the need to house refugees was more than temporary, UNICEF devised solutions that required investment, but would be cheaper and more sustainable to operate in the medium to long-term. These interventions included WASH Blocks, borehole construction, water network construction and operations, establishment of a treatment plant, and established hygiene campaigns. UNICEF also made these adjustments in the settlements, establishing water infrastructure that was both resilient enough to withstand difficult conditions and use, but also reduced operating funding needed for maintenance54.

49 WAJ Senior Staff Interview – 30 July 2018 50 MoWI Senior Staff Interview – 30 July 2018 51 UNICEF notes that in 2018, MWI asked UNICEF inflate budgetary figures so that they include all host community projects. 52 MoWI Senior Staff Interview – 30 July 2018. 53 Ibid. 54 UNICEF Staff Interviews – 10 October 2017

As senior UNICEF staff noted, “Transitioning from humanitarian response to systems and infrastructure…is incredibly challenging, particularly in environments like the berm (Rukban).55” One type of transition in which the WASH programme struggled was in implementing host community projects. As an interview respondent at the MoWI suggested, UNICEF did not originally plan on implementing host community projects. UNICEF incorporated host community projects into the programme when faced with external pressure from municipalities where refugees were putting stress on local water resources56. The WASH programme boldly took up the challenge and set high goals for itself, such as increasing the percentage of the population in Jordan that has access to adequate sanitation facilities (2014), or increasing the number of inhabitants with safe water and storage facilities by 1.4 million (2015)57. While UNICEF missed these targets in each year, it continued to set high targets in this component. As mentioned in the Funding Environment section of this report, UNICEF blamed shortfalls in fundraising for missing host community targets in each year of the programme. However, the evaluation team did not find evidence that the WASH programme considered its funding outlook when planning these targets, or, given the regular shortfall in funding, how it might better spend these funds. We discuss this topic further in the Effectiveness and Efficiency sections. In this evaluation’s focus groups, camp residents expressed what they felt were the most important improvements that had taken place in WASH services, and which issues had not yet been addressed. In Za’atari, focus group participants said that the installation of private toilets was the most important improvement. They said that public toilets had cleanliness and security issues, and the ad hoc pit latrines that many households dug next to their housing units were unsanitary and dangerous. Private toilets were an enormous improvement. As far as issues that have gone unaddressed in Za’atari, participants mentioned that they have observed water supply and sewage disposal pipes overlapping in the same trench, and they fear sanitation threats. They also mentioned households receiving the same quantity of water regardless of the number of household members as an issue that requires a solution58. In Azraq, participants said that the increase in tap stands was an important improvement. Participants in Azraq had a number of complaints, mainly focused on lack of private facilities, high chlorine content in water59, wasted water through misuse of some residents, and lack of hygiene products such as detergent and soap60. In King Abdullah Park, focus group respondents felt that improvements were particularly made in the hygiene of the latrines and solid waste collection. They credited prompt and frequent maintenance61. In the current phase, the WASH programme is moving to address longer-term, nationwide issues of sustainability. These include, climate change, depleting water resources, and assisting communities that are becoming water scarce62.

55 UNICEF Senior Staff Interview – 6 August 2018 56 MoWI Senior Staff Interview – 30 July 2018 57 See results framework, Annex H. 58 Source: Focus Group Discussions – Za’atari Camp 59 UNICEF Jordan reports that chlorine content is within Jordanian standards. 60 Source: Focus Group Discussions – Azraq Camp 61 Source: Focus Group Discussions, KAP Camp 62 Source: Interviews with UNICEF senior staff – 6 August 2018

Effectiveness Evaluation Question 4 • To what extent did UNICEF’s response achieve its intended outcomes? • To what extent did UNICEF achieve equity results, especially for children and women specific interventions? The WASH programme’s achievements were tracked against different outcome statements, output targets, and indicators in three phases; 2013/2014, 2015, and 2016/201763. In 2012, when the programme launched, it operated without a results framework64. The programme’s main objective was to respond to refugee needs in a quickly changing, resource constrained context.65 In 2013 the programme drafted indicators and partially reported achievement compared to them. Details were provided for of IR 1.3 indicators, but not IR 1.6. indicators66. The full results framework for each year is included as Annex H. The next section provides the programme’s objective and/or intermediate results and output statements under each programme phase67. Following that, the report provides an overall comment on the evaluations measure of effectiveness, and then the report presents evaluation’s findings against indicators under each of the programme’s four components for each year. Objective and Output Statements 2012 UNICEF and the WASH programme were operating as an emergency response programme in 2012. In the second half of 2012, the WASH programme’s main focus was serving refugees in the newly opened Za’atari refugee camp and at transit centers. Later in the year, UNICEF began work in host communities, KAP and Cyber City. The evaluation team used available documentation to assess achievements under each component. Much of the information in this section came from Results Assessment Module (RAM) reports, which UNICEF uses to capture achievement against indicators. The WASH programme did not provide a workplan, grant agreement from donors, or RAM report from 2012. The programme’s documentation of its activities and achievements begins on October 4,, 2012 with the publication of UNICEF’s weekly “Syrian Refugees Response Overview (UNICEF Jordan, 2012) .” Below are the objective and output statements for 2013 – 2017:

63 See results framework, Annex H 64 The evaluation team found no evidence of a results framework from this year. 65 Source: UNICEF, World Vision, and Mercy Corps staff interviews 66 IR 1.3 was Relevant Core Commitments on WASH are met in all humanitarian settings. IR 1.6. was Support the Ministry of Water and Irrigation and Sector partners to address the impact of refugee influx as well as chronic sector needs 67 UNICEF’s indicator frameworks used different language to describe result levels in 2013 and 2014.

2013/2014 National Institutions provide improved health and neonatal care Objective services and quality early childhood care with a focus on disadvantaged groups. Intermediate Result Relevant core commitments on WASH are met in all 1.3 humanitarian settings. Support the Ministry of Water and Irrigation and Sector partners Intermediate Result to address the impact of refugee influx as well as chronic sector 1.6 needs.

2015 Quality WASH facilities are sustained and utilised, and hygienic Objective behaviours practiced, by the most vulnerable while ensuring the protection of the environment. Government and WASH sector partners plan and optimally utilize Output 4.1 WASH resources and services for equitable access. Children, parents and community members are aware of and Output 4.2 practice hygiene and water conservation. Urgent needs of vulnerable people especially children are met Output 4.3 through quality and lifesaving WASH services.

2016/201768 Quality WASH facilities are sustained and utilised, and hygienic Objective behaviours practiced, by the most vulnerable while ensuring the protection of the environment. Institutions, utilities and organisations provide equitable access to Output 4.2 sustainable water and sanitation services for the most vulnerable. The most vulnerable women and children are aware of and Output 4.3 practice key hygiene behaviours and practice water conservation. The urgent WASH needs of the vulnerable women and children Output 4.4 are met through safe, predictable and sustainable water and sanitation services and facilities.

68 The evaluation team did not have access to the 2017 RAM report. We assume targets were the same for 2016 and 2017. Results are based on UNICEF’s public reports and any internal records that the WaSH Programme provided to the evaluators.

Overall comment on effectiveness This evaluation’s ToR asks that the evaluation team judge the WASH programme’s effectiveness, in part, by the degree to which the programme achieved its intended outcomes69. This question is difficult to answer. First, in the emergency response phase of the programme the intended outcome was singular, provide sufficient WASH services to the refugees who so direly needed them70. In subsequent phases, the evaluation team found no evidence that the WASH programme conducted strategy development or work planning processes, which would have produced specific, measurable targets that performance could be judged against. The WASH programme has filed annual RAM reports since 2013. These reports include UNICEF’s estimate of its achievements against a set of outcomes. However, the intended outcomes are often vague in that they do not describe a means of addressing specific population’s need that has been identified as strategically important. An example of an intended outcome as described by RAM for 2017 is: Outcome 4 Sustainable and equitable WASH system and key behaviours practiced. Output 4.2 Institutions, utilities and organizations provide equitable access to sustainable water and sanitation services for the most vulnerable. Indicator 2 3.2 million people with access to improved municipal water services.

While those targets are certainly worthwhile, they don’t indicate the specific behaviours, institutions, organizations, and people that the programme has identified as important to responsiveness to the Syrian refugee crisis. Also unclear is whether the 3.2 million people is the annual target or the cumulative programme target. These objectives do not meet the requirements for specificity, measurability, relevance, or time-boundedness usually required in results frameworks71. Secondly, the WASH programme maintains limited monitoring data. Through UNOPS, the programme monitors activity in the camps to ensure coverage and equity. The programme collects reports from its partner organizations that include accomplishments against specific targets defined in work plans, and budgets disaggregated by activity and result. The WASH programme also collects water monitoring reports, and regularly completes its own reports that include an accounting of the programme’s outputs and outcomes. However, UNICEF does not aggregate the information in a format that would allow for its analysis. Partner reported data is maintained in a format such as Microsoft Word, or an unsearchable .pdf document, making it impractical for the evaluation team to extract and use the data as a source of support for the claims made in the RAM and other reports.

69 Evaluation Terms of Reference, page 6. 70 Source: Multiple interviews with UNICEF and partner staff. 71 See, for example, The WaSH Cluster Coordination Handbook, pg 258; or UNHCR’s Practical Guide to the Systematic Use of Standards and Indicators, pg. 57.

Lastly, the WASH programme’s donors do not require that UNICEF report results or financial information disaggregated by year, achievement, activity, or other programme category72. The lack of requirement on the part of donors provides a disincentive for UNICEF to implement a standard financial reporting or monitoring and evaluation system.

72 Source: UNICEF Senior Staff and PMU Interviews

Component 1: Provision of equitable WASH services and dissemination of WASH messages in the four camps Achievement of Intended Outcomes Overall Findings73:

Under this component, the WASH programme achieved high levels of effectiveness for the targeted population across all years. Details 2012 Component 1 results: • Provision of between 1 and 1.3 million liters of Figure 5: Za'atari Camp 2012 (Source:Times of water per day to Za’atari through water Israel, 29 August 2012) trucking. • The completion of two boreholes to provide a lower cost and more secure source of water for Za’atari camp. • The completion of 122 WASH Blocks completed with construction of another 160 in progress. The WASH blocks provided 662 latrines and 546 showers. The WASH units are equitably provided for women and men, and facilities are included for people with disabilities. • 151 mobile latrines and 68 mobile showers installed. • 9,931 hygiene promotion sessions held by UNICEF’s partner organization, ACTED. • Employment of 262 beneficiaries in a cash-for-work program in Za’atari, 38 in KAP, and 20 in Cyber City. • Removal of between 400 and 650 m3 per day of solid waste from Za’atari. • Partners ACTED and THW monitor water quality arriving in Za’atari through residual chlorine tests. • Removal of 300 to 500 m3 per day of waste water from Za’atari. • Maintenance of water and sanitation facilities at transit centers. 2013 In 2013 the WASH programme successfully achieved its targets under this component. 100 percent of refugees in Za’atari camp provided with drinking water. (IR1.3, Indicator 2) • 100 percent of refugees in Za’atari provided with soap and benefited from hygiene promotion activities. (IR 1.3, indicator 3 and Indicator 5) • 100 percent of refugees in Za’atari have access to and use secure sanitation facilities. • 2,340 toilets made available in Za’atari. (IR 1.3, indicator 4) • 2,000 toilets and showers made available to serve the refugees in Azraq camp. (IR 1.3, indicator 4) • 2,379 septic tanks installed in Azraq, or one tank for every 12 residents. (IR 1.3, indicator 4)

73 Results in this section come from UNICEF SitRep reports, Annual Reports, and RAM Reports

In addition to achieving its targets, the WASH programme contributed to the design and construction of Jordan’s second largest refugee camp, Azraq. The programme incorporated lessons learned from Za’atari and made improvements to its activities and infrastructure in the camps that increased overall effectiveness. UNICEF made these strides under difficult circumstances. In 2013, Jordan experienced extreme winter weather conditions, including its heaviest rainfall in 20 years. The WASH programme struggled with wastewater removal services provided by a contractor, and drilled a new borehole for Azraq that failed to meet national water standards. Meanwhile, the inflow of refugees nearly doubled over the previous year. 2013 Component 1 results not measured in UNICEF’s indicator framework include: • Winterization of 72 WASH blocks completed by the end of March. • Za’atari services increased to providing a daily average of 3,322,000 litres of water, or up to 25 liters per inhabitant. • Removing 1,250 m3 of waste water and 1,400 m3 of solid waste by the end of the March. • Water storage facilities/tanks are built in Za’atari, which are operational at the end of May. 2014 In 2014 the WASH programme reported that it successfully achieved most of its targets under this component74. It met its targets to provide water and sanitation services to residents of the refugee camps. The WASH programme failed to achieve its targets under the distribution of hygiene items, providing items to 107,000 of the 525,000 targeted beneficiaries75. It is unclear what proportion of hygiene items were distributed to people in camps versus other locations. In 2014, Azraq camp opened, a waste water treatment plant was constructed, and new hore holes at Za’atari and Azraq were drilled. Code Indicator Target76 Result Output Number of people provided with 100 percent Target met including 98,000 1.3, Ind safe access to sufficient water for of refugees refugees in four camps provided 2 drinking and domestic use in camps with at least 35 liters per person (cumulative beneficiaries per day. reached) Output Number of persons benefiting 100 percent Target met including 98,000 1.3, Ind from improved sanitation of refugees refugees in four camps 3 (cumulative beneficiaries in camps benefiting from desludging of reached) communal, household and institutional latrines

74 2013 & 2014 RAM Report 75 UNICEF reports that one reason for underdistributing items was that the market in Za’atari had become more efficient at meeting demand than distribution activities. 76 The target figure represents the target for the population specified under each component when possible. For example, some indicators report provision of a service for all refugees, whereas the component describes provision to those in camps, host communities, or elsewhere. In each component the target indicator is reported for the beneficiary group described under that component.

Output Number of people provided with 100 percent Target met including UNICEF 1.3, Ind hygiene items of refugees reports 107,000 of the 525,000 4 in camps targeted beneficiaries were reached. It is unclear what portion were in camps and which were in other locations. Output Number of people involved in 100 percent Target met including 98,000 1.3, Ind hygiene promotion activities of refugees refugees in four camps reached 5 in camps through direct messaging. Output Number of people provided with 100 percent Target met including 98,000 1.3, Ind access to solid waste of refugees refugees in four camps benefited 8 management (cumulative in camps from of 800 m3/ day on average beneficiaries reached) of solid waste removal

2015 In 2015, the WASH programme reported that it met its targets for water provision and sanitation in the camps. A key achievement in 2015 was a general improvement in health in Za’atari camp. In both the 2013 and 2014 KAP surveys in Za’atari, 58 percent rated their health as good or excellent. The percentage increased to 69 percent in 2014 and 81 percent in 2015 (Za'atari Hygiene Promotion Working Grouip, 2015). While the WASH programme reported 100 percent water coverage in Zaatari, the 2015 KAP survey showed that only 61 percent of residents accessed water from the public tanks. Support for the piped water network was high with 63 percent rating this project as “very positive”. The 2016 Azraq KAP Survey reported that 95 percent of people felt the water supply was adequate or very adequate77. Access was good with 46 percent collecting water within 15 minutes, well within the 30-minute Sphere standard (The Sphere Project, 2011). Ninetly-six percent accessed water for free (from public tapstands and not private vendors). Ninety percent were happy with the water quality (ACTED, World Vision, ACT, RI, and Save the Children, 2016). The third indicator under this component, “proportion of population practicing hygiene promotion (sic)”, went unmeasured by UNICEF in 2015. The programme reported that monitors had observed increasing rates of key hygiene practices, such as handwashing with soap. UNICEF reported that 100 percent of refugees in camps were provided with safe solid waste management. In 2015, the quality of that management may have not been sufficient. In 2014 88 percent said their area was generally clean but this dropped to 71 percent in 2015. No reason was given for the drop. Accomplishments in 2015 include Za’atari’s waste water treatment plant becoming operational, work on Za’atari’s water network starting, and Azraq’s borehole becoming operational. Comprehensive child focused assessments were carried out in Azraq and Za’atari, and KAP surveys were conducted. UNICEF also managed to provide services

77 The 2016 Azraq Kap survey was completed in January, representing 2015 results.

during a five-day truck driver strike in Za’atari and implemented activities that prevented any further major strikes. Note that for the output indicators in the table below, UNICEF estimated that 140,000 refugees would reside in the camps. However, the population never grew that large, so the number of refugees reached represents a higher proportion that represented using the 140,000 figure. Code Indicator Target Result Outcome Proportion of population at camps 100% of 100% of refugees in camps 4, Ind 2 using improved drinking water as refugees in were provided with safe water per Jordan standards. camps (with the tankered quantities in excess of the minimum standard of 35l/p/d)78.

Outcome Proportion of population at camps 100% of 100% of refugees in camps used 4, Ind 3 using improved sanitation refugees in improved sanitation facilities in facilities. camps either communal WASH blocks or their own constructed facilities in their shelters.

Outcome Proportion of population 100% of No result reported for camps 4, Ind 4 practicing hygiene promotion at refugees in camps. camps Output Number refugee in camps through 140,000 100,000 reached, 4.2, Ind Hygiene messages. 1 Output Number refugees provided with 140,000 122,055 provided with 35 litres 4.3, Ind safe access to sufficient water for per day 1 drinking and domestic use through water trucking. Output Number of refugees provided 140,000 122,055 provided with 4.3, Ind with safe waste water disposal wastewater disposal 2 through dislodging of septic tanks. Output Number of refugees provided 140,000 No result reported. The 4.3, Ind with safe waste water disposal wastewater network was not yet 3 through provision of Waste Water complete. Network in Zaatari Camp. Output Number of refugees provided 140,000 122,055 provided with solid 4.3, Ind with safe solid waste management waste management. 4 facilities.

78 Statement appears exactly as written in UNICEF’s 2015 RAM report

Output Number of refugees provided 140,000 122,055 provided with WASH 4.3, Ind with safe access to sanitation blocks in four camps. 5 facilities by operating and maintaining the existing WASH Blocks through WASH Committees and/or community involvement Output Number of refugee Boys and Girls 45,000 50,691 benefited from WASH 4.3, Ind in camps provided with WASH in in schools. 6 schools

2016 In 2016, the WASH programme reported that it met or exceeded its targets under component 1. Definitive measurements were not available for Outcome 4, Indicator 4 related to hygiene practices. However, a KAP study conducted in Za’atari and the output figures for hygiene activities in settlements imply that the target was met (UNICEF Jordan, 2016). Other WASH programme achievements in 2016 include managing the large population increase in the Rukban and Hadalat settlements, managing the transfer of refugees to Azraq, and continuing management of Azraq and Za’atari water networks.

Code Indicator Target Result Outcome Proportion of population at camps 100% of 100% of refugees in camps 4, Ind 2 using improved drinking water as refugees in were provided with safe water per Jordan standards. camps and (with the tankered quantities in settlements excess of the minimum standard of 35l/p/d).

In the camps and at the border, 244,052 people were provided with water over the course of 2016

Outcome Proportion of population at camps 100% of 100% of refugees in camps and 4, Ind 3 using improved sanitation refugees in settlements used improved facilities camps sanitation facilities in either communal WASH blocks or their own constructed facilities in their shelters.

Outcome Proportion of population 100% of Za’atari 2015 KAP indicated 4, Ind 4 practicing hygiene promotion at refugees in between 80-96% of females targeted camps, host communities camps with childcare responsibility and schools washed their child’s hands at key times and 92% said that they did so using soap and water. Hygiene products and services delivered to the settlements.

2017 In 2017, achievements included high levels of satisfaction in Azraq. In the 2017 Azraq KAP survey, 68 percent of the participants replied the water was very adequate compared to 44 percent in the 2016 survey. In 2016, water storage was a concern, but residents felt it had been resolved. 88 percent of respondents in the 2017 survey said they need less than 15 mins to collect water compared to 46 percent in 2016. Satisfaction with water quality dropped in 2016 to 75 from 90 percent in 2016. The reported reason was poor taste, smell, and colour (UNICEF, ACF, 2017). In 2017, the programme completed the waste water system at Rukban. The programme also ceased tankering, at Azraq realizing cost savings. Code Indicator Target Result Outcome Proportion of population at camps 100% of 100% of refugees in camps 4, Ind 2 using improved drinking water as refugees in were provided with safe water per Jordan standards. camps and (with the tankered quantities in settlements excess of the minimum standard of 35l/p/d).

In the camps and at the border, 19.7 litres per person per day were provided to refugees in settlements.

they did so using soap and water. Hygiene products and services delivered to the settlements.

Component 2: Provision of support in the host communities including WASH in schools Overall Findings: As detailed with citations below, the WASH programme consistently struggled to achieve targets under this component. Two issues created difficulties: funding and lack of strategic planning As described in the Funding Context section of this report, UNICEF Jordan struggled with funding issues. Maintaining services in the camps necessarily received first priority for funds at the expense of achieving targets in host communities. That said, after several years of missing targets, programme management documentation indicates a lack of adjustment to strategy to make the programme more effective or realistic79. For example, from 2013 to 2017 the programme included an indicator intended to set goals for provision of WASH services to students80. The target was missed each year of the programme by a wide margin (table 1). Other indicators following this example include those regarding hygiene promotion, are illustrated below.

Table 1 Students benefiting from improved WASH facilities in the learning environment, Target vs Achievement Year Target Achieved Percent Achieved 2013 230,633 72,985 31.65% 2014 188,000 100,000 53.19% 2015 200,000 91,757 45.88% 2016 60,000 37,693 62.82% 2017 114,000 7,980 7.00%

Additionally, the indicators aren’t sufficiently specific under this component, so it’s difficult to know what the programme is trying to achieve. Using the above example again, in 2016, the indicator was stated as “Number of people with access to improved WASH facilities in institutions,” and the year’s achievement was stated as “37,693 students benefited from provision of support to 45 schools.” It is unclear if the initial intention was that “people” in institutions should mean students, or if the achievement was retroactively fitted to the indicator. UNICEF’s Outcome 4, Indicator 2 target is that 1.5 million host community

79 UNICEF reports that the outcome level was fixed in 2014 for the period of 2015 – 2017. The target was increased in 2016/2017 because of an optimistic fundraising outlook in 2015, which did not materialize. 80 The indicator was stated differently over the course of the programme: 2013/2014 – IR 1.3, Ind 6: “Number of children (girls and boys) provided with access to safe water, sanitation and hygiene facilities in their learning environment and in child friendly spaces.”, 2015 - Output 4.1, Ind 3, “Number of students provided by (sic) WASH services.”, 2016/2017 (Target and baseline were general, but achievement reported in terms of students) – “Number of people with access to improved WASH facilities in institutions.”

residents use improved drinking water as per Jordanian standards. In its 2015 RAM report, the WASH programme stated that “In Host Communities, 41% (634,858) of the target population of 1,550,500 benefitted from improved access to water facilities and systems as a result of rehabilitation of water infrastructure, as well as support to vulnerable households.” In its 2016 RAM report, for the same indicator the WASH programme reported, “In Host Communities in four governorates (Mafraq, Madaba, Balqa and Irbid), 237,360 additional people were reached in 2016. Since 2013, an estimated 1,482,402 people have improved access to water services …which equates to 96 percent of the Country programme target.” It is unclear if the target is that UNICEF serve 1.5 million per year under this outcome as the 2015 report implies, or 1.5 million cumulative people as the 2016 report implies. It is also important to consider the improvement. For example, in Irbid UNICEF made improvements to the water distribution system. This should not imply that everyone served by the system received better water services as a result of the intervention. Water services are still very limited, and there are still serious problems with the systems distribution network, water safety, and availability. Achievement of Intended outcomes 2012 2012 results: • The first work on host community water systems, in Ramtha and Mafraq, was completed. • Relief International hygiene promoters complete house-to-house visits in March with hand-washing messaging and soap distribution to 2,000 households in Mafraq and Ramtha. Other evidence of component effectiveness in 2012: Za’atari opened in Mafraq governorate in July, 2012. Tensions began to rise both within the camp and between the camp and communities outside of the camp. In response, the UN identified serving vulnerable Jordanian’s as an important component of the Syrian crisis response (Seely, 2012). The WASH programme expanded its scope to serve community needs as well as camp needs. 2013 In 2013, the WASH programme met its commitments to refugees, but underachieved in terms of the targets it set for serving Jordanians, as the results against targets below demonstrate:

Code Indicator Target Result IR 1.3, Number of emergency-affected Approximately 150,000 refugees in HC’s Ind 2 population (male and female) 215,00081 non- benefit from rehabilitation of provided with access to drinking camp refugees existing boreholes82. and domestic water and 180,000 Projects in host communities Jordanians benefit 400,000 people (UNICEF Jordan, 2013)

IR 1.3, Indicator 3: Number of Approximately 201,920 refugees provided with Ind 3 Emergency affected population 215,000 non- soap and hygiene kits83. (male and female) provided with camp refugees access to soap and other hygiene and 180,000 items Jordanians IR 1.3, Number of Emergency affected 180,000 No results outside of camps Ind 4 population (male and female) Jordanians reported achieved. with access to appropriately designed toilets and sanitation services IR 1.3, Number of population (male and Approximately 201,920 women, men, boys and Ind 5 female) covered through face to 215,000 non- girls have been participating in face (including HH and Group camp refugees face to face hygiene promotion sessions) hygiene promotion and 180,000 activities including focus group messages Jordanians sessions and house to house visits IR 1.3, Number of children (girls and 230,633 72,985 benefit from 91 Ind 6 boys) provided with access to rehabilitated WASH facilities safe water, sanitation and hygiene in schools. facilities in their learning environment and in child friendly spaces WASH facilities were rehabilitated in over 100 Jordanian schools benefiting 23,000 Jordanians and 8,000 Syrian children.84

81 Target is estimated. UNICEF described its target as “345,000 refugees (in camps and off camp) and 180,000 affected Jordanians.” The evaluation assumes the most conservative estimate of refugees in camps and subtracts that from the total number. 82 2013 RAM Report 83 2013 RAM report. 84 Ibid.

IR 1.3, Number of children (girls and 100% No measurement reported. The Ind 7 boys) not practicing open programme reported, “host defecation in host communities. communities (have) improved including hygiene awareness messages on risks of open defecation.

2014 The WASH programme struggled to meet its targets under this component in 2014, as the table below demonstrates. The exception was in the area of WASH in Schools (indicators 6 and 7). The programme significantly exceeded its target promoting hygiene messages to students. Code Indicator Target Result Output Number of people provided with Approximately approximately 7,000 people in 1.3, safe access to sufficient water for 255,000 non- vulnerable households benefited Ind 2 drinking and domestic use camp refugees from household water (cumulative beneficiaries reached) and 180,000 improvements. Jordanians an estimated 450,000 people (refugees and host community) benefitted from increased access to water as a result of rehabilitation of existing water systems. Output Number of persons benefiting Approximately 10,000 people benefited from 1.3, from improved sanitation 255,000 non- removal of sewage network Ind 3 (cumulative beneficiaries reached) camp refugees blockage. and 180,000 7,000 people in vulnerable Jordanians households benefited from new or rehabilitated facilities Output Number of people provided with Approximately No results reported for Non- 1.3, hygiene items 255,000 non- camp refugees or Jordanians. Ind 4 camp refugees and 180,000 Jordanians Output Number of people involved in Approximately 10,000 people were reached 1.3, hygiene promotion activities 255,000 non- through mobilisation activities Ind 5 camp refugees on key hygiene messages and 180,000 Jordanians

Output Number of children benefitting 100,000 188,000 WASH facilities in 164 1.3, from improved water, sanitation schools85 Ind 6 facilities in their learning environment (cumulative beneficiaries reached) Output Number of children benefitting 100,000 116,788 students reached in 1.3, from improved water, sanitation host communities Ind 7 facilities in their learning environment (cumulative beneficiaries reached) Output Percent increase of targeted host 5% increased No result reported86 1.6, population accessing sufficient specifically in Ind 3. using appropriate sanitation the Northern facilities Governorate

2015 The WASH programme underperformed against most of its targets under this component in 2015. The closest target the programme came to meeting was Output 4.1, Indicator 4, Number of population at host communities benefiting from sewage systems improvement, where the programme improved sewerage for about 83 percent of its target population. Note that in the table below, student targets were set before UNICEF conducted the Nationwide WASH in schools assessment. UNICEF reports that it anticipated a higher need for interventions before the assessment. Code Indicator Target Result Outcome Proportion of population in host 1,550,000 634,858 HC residents benefited 4, Ind 2 communities using improved host from improved water drinking water as per Jordan community, infrastructure. standards. 200,000 91,757 students benefited from students school WASH facility rehabilitation and construction

Outcome Proportion of population at host 545,000 host 390,065 benefited from 4, Ind 3 communities and schools using community, improved access to sanitation improved sanitation facilities 200,000 systems and facilities. students;

85 This figure, reported by UNICEF, implies 1,150 students per school, which is possible considering that large classes sizes and double shifts common at many Jordanian schools (Middle East and North Africa Out-Of- School Children Initiative, 2014)) 86 In the 2014 RAM report, the WaSH Programme reported that 10,000 people benefit from removed blockages in the sewage network. However, this result does not apply to this indicator, and was already reported as the result of Output 1.3, Indicator 3 in that year.

91,757 students benefited from school WASH sanitation facility rehabilitation and construction

Outcome Proportion of population 150,000 host 100,000 people reached through 4, Ind 4 practicing hygiene promotion at community, mobilization sessions. targeted host communities and 200,000 73,268 students received schools students hygiene kids. Output Number of population benefiting 150,000 619,450 people benefitted from 4.1, Ind from safe water and storage refugees and improvements to the municipal 2 facilities improvement. 1,400,000 water network residents in 10,021 people benefitted from HCs provision of key infrastructure items 5,387 people in Vulnerable Households were supported with improvements to household wash facilities Output Number of students provided by 200,000 91,757 students served by 4.1, Ind (sic) WASH services students in WASH facilities 3 HC Output Number of population at host 150,000 331,872 people benefitted from 4.1, Ind communities benefiting from Syrians in support to improvements to 4 sewage systems improvement HC, 395,000 sewerage systems . in HC 8,255 people benefited from mobile latrines. 52,806 people benefitted from provision of key infrastructure items. Output Number refugee in host 150,000 at 100,000 people reached through 4.2, Ind communities and students reached host com mobilization sessions. 1 through hygiene messages and 200,000 73,268 students received students hygiene kids.

2016 In 2016 the programme did not meet targets under this component, falling short more drastically than in previous years. The evaluation team believes that at this point it becomes apparent that the issue was more of a lack of strategic design process than underperformance given programme resources and objectives.

Code Indicator Target Result Outcome Proportion of population in host 1,550,000 237,360 reached 4, Ind 2 communities and schools using host improved drinking water as per community, Jordan standards. 200,000 37,593 students benefitted from students improved WASH facilities Outcome Proportion of population in host 545,000 host 35,000 people in Irbid have 4, Ind 3 communities and schools using community, improved sanitation access as a improved sanitation facilities 200,000 result of the rehabilitation of the students; sanitation system. 37,593 students

Outcome Proportion of population 150,000 host No result reported 4, Ind 4 practicing hygiene promotion at community, targeted camps, host communities 200,000 and schools students Output Number of people living in 18,000 No direct result reported 4.2, Ind vulnerable Households with Assessment carried out. 1 access to improved household WASH facilities and services Output Number of people with access to 600,000 237,260 benefited from 4.2, Ind improved municipal water improvements to water 2 services infrastructure. Output Number of people with access to 400,000 35,000 people gained access to 4.2, Ind improved municipal sewerage improved municipal sewerage 3 services services. Output Number of people with access to 60,000 37,593 students benefitted from 4.2, Ind improved WASH facilities in the provision of support to 45 4 institutions schools. Output Number of people in the host 18,000 No direct result reported. 4.3, Ind community reached through Assessment carried out in 3 social/community mobilisation on preparation. key WASH messages and water conservation. Output Number of people in institutions 60,000 37,593 students 4.3, Ind in the host community reached 4 through social/community mobilisation on key WASH messages and water conservation Output Number of people in informal 15,000 7,298 vulnerable people in ITSs 4.3, Ind settlements reached through received key WASH messages. 5 social/community mobilisation on key WASH messages and water conservation

2017 It is difficult to tell how UNICEF performed against WASH targets from January – July of 2017. The difficulty arises because the WASH programme does not maintain a monitoring database that tracks progress against indicators on a regular basis. Below we use the 2017 Annual Report, where possible, to show the WASH programme achievement over the course of the year. Though much information is missing, UNICEF reports that the WASH programme underperformed against targets in this component. One important accomplishment in 2017 was that the national WASH in Schools Standards were released. Code Indicator Target Result Outcome Proportion of population in host 1,550,000 258,000 people in host 4, Ind 2 communities and schools using host communities (UNICEF Jordan, improved drinking water as per community, 2017) Jordan standards. 200,000 students 14,000 students Outcome Proportion of population in host 545,000 host 258,000 people in host 4, Ind 3 communities and schools using community, communities (UNICEF Jordan, improved sanitation facilities 200,000 2017) students; 14,000 students Outcome Proportion of population 150,000 host No measurement provided 4, Ind 4 practicing hygiene promotion at community, targeted camps, host communities 200,000 and schools students Output Number of people living in 44,000 No measurement provided87 4.2, Ind vulnerable Households with access 1 to improved household WASH facilities and services Output Number of people with access to 3.2 million 258,000 4.2, Ind improved municipal water services 2 Output Number of people with access to 2.1 million No measurement provided. 4.2, Ind improved municipal sewerage 3 services

87 In the 2017 Annual Report, UNICEF states, “Little progress was made toward the target for sanitation. This was due to sectoral lack of funds, as well as prioritization of water projects for the Government, due to the enormous pressure to deliver water. UNICEF continued to work with the Government to advocate for appropriate allocation of resources to meet sanitation needs. “

Output Number of people with access to 114,000 7,98088 4.2, Ind improved WASH facilities in 4 institutions Output Number of people in the host 44,000 No measurement provided. 4.3, Ind community reached through 3 social/community mobilisation on key WASH messages and water conservation. Output Number of people in institutions 114,000 No measurement provided. 4.3, Ind in the host community reached 4 through social/community mobilisation on key WASH messages and water conservation Output Number of people in informal 15,000 No measurement provided. 4.3, Ind settlements reached through 5 social/community mobilisation on key WASH messages and water conservation

Component 3: Support to the WASH sector through technical support to the MoWI, as well as National WASH in School Standards Overall Findings: Government officials at the MoWI and WAJ reported satisfaction with the WASH programme contribution to planning documents, strategies, and policies89. The MoWI said that, as head of the WASH Taskforce, UNICEF always has a good plan, and staff that serve together on technical committees have good partnerships with UNICEF. The Ministry also praised the work done by the consultant that UNICEF seconded to Ministry, and wished he could have stayed longer. Achievement of Intended outcomes 2013 In 2013 the WASH programme had one target under this component; IR 1.6, Indicator 5: “Increased institutional capacity in the MWI for undertaking planning and coordination leadership role in various Sector Coordination platforms (WG/TF). Development of Sector M&E sytem/MIS. Evidence based Advocacy to increase funding flow to the sector. The target was the development and effective dissemination of hygiene messages. “ It is unclear whether this target was achieved in 2013. However, the WASH programme did arrange for a seconded staff member to work with the ministry in the following year.

88 In the 2017 Annual Report, UNICEF states, “The achievements for WaSH in Schools were also significantly lower than planned, at 7 per cent of the target. This was primarily due to reduced funding and a focus on smaller scale and higher impact interventions through the connection of schools to the municipal water network, with vulnerable households connected en route.

89 Source: Interviews at MoWI and WAJ

2014 In 2014, WASH programme support together government included: • Conducting a water vulnerability assessment, including data on per capita water usage and refugee distribution. • Ensured that the sector prioritized areas of greatest need for investment. • Secondment of a senior advisor to lead in the development of the National Water Strategy. 2015 The WASH programme main target under this component in 2015 was to “build strategic planning capacity in MoWI and sector and other relevant institutions in Emergency preparedness and response (Outcome 4, indicator 1). In 2015, the WASH programme achieved this target by supporting the development of several strategies, regulations, and policies. They were: • The Jordan Response Plan (2016 – 2018) • The National Water Strategy (2016 to 2025) The WASH programme also developed assessments and tools to assist the government, including: • Water and Sanitation Vulnerability Maps, which were upgraded using internal WASH capacity and more complex criteria. • An assessment of the Geographic Information System (GIS) structure and capacity of the government. • A project monitoring database. • A Nationwide WASH in schools assessment. 2016 In 2016, the WASH programme worked with the government to achieve the following results: • The National Water Strategy (2016 - 2025) was finalized, with direct support from UNICEF, and released in April. • The Water and Sanitation Vulnerability maps were updated in September using data generated from the 2015 census. • The Jordan Response Plan was updated to cover the period 2017 to 2019. • A number of result-based tools were developed which have facilitated the generation of systematic data, including the Comprehensive Vulnerability Assessment. • The Water Authority of Jordan coordinates amongst the different water companies to ensure they have sufficiently planned for emergency responses and has Long Term Arrangements with borehole owners and suppliers which can be activated, as needed.

2017 UNICEF supported the MoWI to review and finalize the baseline values for access to safely managed water and sanitation services under the SDGs. This included UNICEF working with the Ministry to strengthen the decision-making framework that ensures accurate baseline measurements. The transition from MDGs to SDGs for water entails a small reduction of 5.3 per cent. The change for sanitation is significantly higher, at 24.6 per cent. The dramatic reduction in Jordan’s sanitation rating is expected to redress the imbalance of the sector on

water, and to scale-up sanitation interventions, particularly for the most vulnerable. To increase the percentage of the population using safely managed sanitation services, key activities were included in UNICEF’s new country programme, and will be progressively scaled up over the programme period. In 2017, UNICEF also led the publication of National WASH in Schools Standards. Component 4: Coordination of the sector at camps and national level UNICEF began leading as sector coordinator in 2014. Coordination happens at three levels; national level, camp level, and working group level. Achievement of Intended outcomes Overall Findings: Stakeholders and UNICEF partners felt that UNICEF was effective as WASH sector lead. One contractor that serves Rukban said “UNICEF staff are super cooperative and available anytime of the day or night”.90 The evaluation team also spoke with two of the WASH programme’s donors, The Bureau of Population, Refugees, and Migration (BPRM) at the United States State Department, and the German owned development bank Kreditanstalt für Wiederaufbau (KfW). BPRM commented that UNICEF was effective in leading the sector, and excellent to work with from a donor perspective. BPRM felt that they had benefited from UNICEF’s guidance in setting their own strategy for addressing the crisis. KfW appreciated the guidance UNICEF could provide in constructing infrastructure in the camps and assistance in host communities. As sector lead, UNICEF has led WASH activity coordination in the Camps, including coordinating responses to the major storms and flooding that occur in the camps, particularly Za’atari each year. UNICEF leads contingency planning for these storms and the emergency reporting structure that allows for tracking storm impact. UNICEF also manages other emergencies such as the outbreak of Hepatitis A in Azraq that occurred in 2015 and 2016. As part of camp coordination, UNICEF leads a bi-weekly Za’atari coordination meeting that has been effective in setting standards and avoiding duplication of effort. UNICEF also improved the MoWI’s use of data to create water management strategies, and led the sector’s WASH in schools initiatives. One area in which UNICEF could improve its coordination is in strategic planning and communication with partner organizations. Several partner organizations, particularly INGOs, mentioned that they were surprised by UNICEF’s decisions in the camps, often requiring that they make major adjustments on short notice, or conduct work for which they hadn’t planned or budgeted91. Examples included sudden decisions to hand operations over to other organizations or contractors with short notice, without guidance on the handover process. Other partners mentioned contracting processes that took a long time only to have a requirement that initiatives start days after a contract is signed. Decisions such as these, that do not appear as part of a strategically designed plan, may force risk and costs on to partner organizations. It may also create risks for UNICEF that its monitoring and accounting systems cannot bear. These risks could also be mitigated by an annual strategic planning

90 Source: Interview with Gama Engineering 91 Interviews with Mercy Corps, World Vision, ACF, and Oxfam Staff

process that would set expectations and guidelines based on clear analysis and reduce last minute decision making92. 2014 • Met output 1.3, indicator 1 target: established a WASH working group that guided the sector response. • Met output 1.6, indicator 1 target: Created a comprehensive WASH master plan aligned to the National Plan (2022). • Led the coordination of WASH activities with meetings conducted on a monthly basis at a national level, in addition to weekly meetings at camp levels. Key achievements of the coordination are summarised as: - Incorporation of key WASH issues in the Regional Refugee and Resilience Plan (3RP) and the Jordan Response Plan (JRP). - standardisation of hygiene promotion messages and materials. • Five Task Forces were established; - WASH in Schools; - Hygiene Promotion; - Water distribution network in Zaatari; - Wastewater in Zaatari; and, - Development of Minimum Standards. • Provided support to the development of the Strategic Framework to implement the National Resilience Plan (2014 to 2016). • Led partners in an exercise to jointly identify the WASH sector’s needs. • Facilitated sector coordination that enabled partners to jointly identify sectoral needs. • Provided technical assistance for a water vulnerability assessment, which identified areas in need of support to improve water access. The assessment criteria were agreed to with the Government, which ensured the use of the vulnerability map by the Government to plan and prioritise areas of intervention. 2015 - 2017 During 2015 - 2017, UNICEF continued its coordination of the sector in the camps, including the development of contingency plans for a number of scenarios and emergency possibilities. UNICEF also coordinated the response to the Hepatitis A outbreak in Azraq and undertook hygiene campaigns on important topics, such as effective handwashing. UNICEF also coordinated interagency response at the Berm, coordinated task forces on Solid Waste and Community mobilization, oversaw the transition of 28,000 refugees to Azraq camp, oversaw the response to the May 2015 contractors strike, and coordinated the Azraq water system redesign. Extent that UNICEF Achieved Equity Results The UNICEF WASH programme operates in a rapidly changing dynamic environment, particularly in its first two years. Achieving a permanent state of equitable delivery of services is a very difficult task. The UNICEF WASH programme continuously strove to

92 The UNICEF Jordan Country Programme Document 2013-2017 mentions a two year rolling workplan which is measured through annual component reviews and a mid-term review in 2015. The evaluation team did not see evidence that the WaSH Programme had complied with this mandate.

provide equitable results. In response to the quickly changing environment, the programme established comprehensive feedback systems to rapidly respond to problems and complaints as quickly as possible. Infrastructure was rapidly assessed, and when deemed appropriate, implemented to get as near equity as possible. Providing WASH services equitably is an enormous challenge and a moving target. Initially, all people entering the camps were by definition the most vulnerable and UNICEF’s challenge was to provide as many as possible a minimum acceptable level of water per day and sanitary facilities93. As the situation stabilized, UNICEF has more resources to focus on equitable access and focusing on women and children. In WASH sector coordination meetings from 2013, the issues of gender, children, and equity are not discussed. The first documentation of a focus on gender is a brief WASH Sector Gender analysis dated January 2014. The document focuses on safety for women and girls, as well as providing for their hygiene and sanitation needs. By March, 2014 the WASH working group began to focus on the safety of women and girls in relation to WASH services.94 That meeting also reported on an assessment done of access for people with disabilities. In 2014 and 2015, UNICEF and its partners made sure equitable results were factored into the design of its activities in the camps. ACTED recruited female Hygiene Promoters and Social Mobilisers and Cash For Work personnel to promote equitable allocation of the resources, incorporate women’s feedback. ACTED also tailored its hygiene promotion activities in camps. It developed approaches that targeted young children and mothers in Za’atari and Azraq (ACTED Programme Cooperative Agreement 23-13). UNICEF supported vulnerable households with WASH facilities and ensured they met standards. A household sanitation assessment was used to guide the household level improvements. Action Contre la Faim (ACF) implemented a women-centered model for community mobilization in Azraq called the Lead Mother Model. The model involves training a lead mother, who is responsible for disseminating messages on hygiene, sanitation, and water for 12 households. The lead mother works with the other 12 mothers in her area to promote hygiene, sanitation, and water conservation in their area. As of this evaluation, the initiative had worked with 732 mothers95. In 2016, Oxfam paid particular attention to ensuring WASH services for women, teenage girls, and boys and girls under the age of 12. They noted that large percentages of these groups felt threatened when seeking WASH services in the camps in previous years. They indicated that the water and wastewater network, when completed, could solve this problem. In the meantime, they recruited female staff and held weekly gender segregated feedback sessions with the community. They also employed a dedicated gender advisor and worked with UNICEF to provide protection staff. Improvements in camp infrastructure improved results for women and children. Completion of the water network in Azraq included a reduction in the distance from households to tap stands, which made access less burdensome and safer. Access to private sanitary in facilities

93 Source: Interview with UNICEF Staff – 6 August 2018 94 WaSH Working Group Meeting # 3, Minutes of Meeting: March 2014 95 Source: Interview with ACF - 1 August 2018

has improved safety in Za’atari, though women in Azraq still struggle with using public showers due to cultural norms around privacy and immodesty. The WASH programme has struggled with equity in water distribution throughout the programme and has made a continuous effort to improve equitable access. ACTED was responsible for water delivery in Za’atari, and developed a computer based application to monitor distribution in the camp. In 2015, ACTED highlighted that the network of tanks around Za’atari provided an opportunity for corruption of the water system. Drivers of water trucks were pressured to deliver water first to some tanks, leading to other tanks not getting replenished as often as required96. Oxfam noted that water drivers incompletely filled public water tanks, diverting water to wealthier households’ private tanks for pay. Oxfam also noted in 2016 that wastewater drivers illegally charged households to desludge pits97. Hotlines were established and there were follow-ups to complaints. Both organizations hoped that the construction of the water network would solve this problem. However, the water network also struggles with equitable delivery. In interviews and focus groups conducted for this evaluation, residents of Za’atari noted that each housing unit is delivered the same amount of water, regardless of the number of people that live in the unit. Household size in Za’atari ranges from one member to more than eight members.98 Thus, smaller households receive more water per person than larger households. Also, some residents game the system by removing floats that trigger a shut off mechanism from their tanks or using suction to increase the flow of water to their units. Both mechanisms result in a shortage of waters to others, further along the network. One focus group participant in Za’atari stated that he needs to call the hot line daily and complain of water shortage so that he can get enough water for his 30+ family members. It is difficult to point to ways that UNICEF has promoted equitable results in host communities. While its host community projects are important, and help municipalities with severe water and sanitation issues, the projects tend to cover a wide area and don’t particularly target women and children99. Improvements to household facilities did focus on vulnerable households, but they were relatively small in number. Regarding hygiene promotion, in the Jordan Response Plan for the Syria Crisis 2017 – 2019, the evaluation team notes that the overall budget for Result 6.1, to Improve Hygiene Practices in Zaatari, Azraq and KAP camps, is nearly the same as the budget for Result 3, improving hygiene in host communities, despite 80 percent of refugees living in the host communities.100

96 Source: Interview with ACTED staff 97 Source: Interview with Oxfam staff 98 WaSH Infrastructure and Services Assessment in Za’atari; March 2017 99 Source: Site inspections of host community projects in Irbid. 100 Jordan Response Plan pg 124.

Evaluation Question 5 • What were the strengths and weaknesses of the interventions in achieving results in the camps/settlements, ITSs and host communities on the following: o Water (Litre/day, supply, quality, consistency, reliability, accessibility, and equity)? o Sanitation (adequacy, appropriateness, accessibility)? o Hygiene promotion (appropriateness of the messages, timeliness, behaviour change)? o WASH in schools? o National level support? Camps Water In 2012, such as 67 percent of KAP survey respondents in Za’atari saying that they did not have enough water (Za'atari Hygiene Promotion Working Groups). However, this high percentage is likely the result of refugees moving from a relatively high-water access area in Syria to a resource constrained environment. The shortage may also have been due to restrictions on water delivery put in place by the Government of Jordan. By the 2013 iteration of the KAP survey, 80 percent said that they had enough water, a clear improvement in perception as well as reality. Service levels were strained when the population numbers rose to a peak in 2013/2014 and have since improved as population numbers declined and steadied. An evaluation focus group participant in Za’atari commented that “before there were complaints about the quantity of water supply. Up to 30 people would share water from one tank101. There are fewer complaints now except from those not connected yet to the network or those who suffer from low pressure.” For both Za’atari and Azraq, the interventions were effective. Residents of both camps received at least 35 liters of water per day, well in excess of the 15 liters per day that the Sphere handbook defines as sufficient (The Sphere Project, 2011). The evaluation team inspected water tanks outside a shelter in Za’atari and found the tanks to be. easily accessible, directly outside the home, and water was present in the tank. Azraq’s two boreholes (2 and 3) combined produce 120 cubic meters per hour. The water is treated, disinfected and sent to the network. The treated water is pumped to storage/pumping structures and then to the network tap stands. The evaluation team believes that this structure represents an appropriate level of service given available funds. The system is a lower cost option for delivering safe water as close to households as possible. UNICEF reports that some households use hoses from the tap stands directly to their residence. In Za’atari, the evaluation team’s opinion is that the installation of the water network is both needed and appropriate. Za’atari is the size of a city and could exist for some time to come. Water improvements such as network water connections to dwellings reduce NRW costs and improve customer satisfaction and willingness to pay for the service. It makes water and sanitation services more accessible and equitable to all of the camp residents.

101 One tank holds 1,000 litres.

Sanitation While camp sanitation systems may not be ideal in Figure 6 Very clean household latrine observed in terms of cost efficiency, given political realities and Za'atari camp. physical limitations, evaluation team site inspections revealed that they are appropriate and adequate. The exception is the wastewater management in Azraq camp, where the treatment plant is non-operational and septage is still being hauled vast distances ant a huge expense. The system in Za’atari, described below, is an innovative approach that works well and is cheaper to install than a traditional sewer system. The boreholes and water systems at both Zaatari and Azraq produced high quality, and are well designed systems are being run by high quality, experienced Jordanian contractors with the assistance of beneficiaries working on a cash-for-work basis. These arrangements contribute to sustainability. In Za’atari, UNICEF initially constructed WASH blocks that were primarily used in the early days upon refugee’s arrival to the camp. Za’atari’s residents were uncomfortable with public facilities, which were necessary to accommodate the large numbers of arrivals. As the camp evolved, residents progressively took responsibility for constructing their own WASH facilities inside their shelters. Leading to over thousands of open pits throughout the camp. Residents that the evaluation team interviewed said that, before connection to the sewerage network they had access to septic tanks which they could have emptied upon request102. Although, residents reported that sometimes it was difficult to get a response103. UNICEF addressed the shortcoming in infrastructure by constructing communal tanks and connected them, which doubled the volume of wastewater collected in the camp. The evaluation team observed a household toilet, which was a pour flush latrine within a household compound. The toilet was fully functioning and connected to the sewage network. Household members stated that the toilet had functioned this way for 6 months. Currently, UNICEF is connecting the communal household tanks to a constructed treatment plant via the wastewater network. An issue that came up frequently in focus groups and household interviews was the quantity of dust that is produced by water system construction. Interview respondents and focus group participants stated that the dust results in health issues such as triggered allergies and respiratory infections. For Za’atari, the evaluation team believes that the common septic tank and small diameter sewer is an appropriate wastewater collection model. It has enabled the camp to reduce its septage trucking from a fleet of 39 trucks to 5 trucks, which will no longer be needed once

102 Source: Interviews with ACTED staff 103 Household interview in Za’atari.

the network is completed. The gravity flow from common septic tanks to the sewer trunk line, and then to the lift stations where it is pumped to the treatment plant is a major infrastructure improvement. These are both appropriate and necessary for the number of people served. The treatment system composed of both membrane bio reactors and trickling filters is effective. The system functions well and provides the level of treatment required by national law, which is a key requirement. Obtaining labor for cash from camp residents also helps reduce costs and add value for money104. It also contributes to sustainability as workers have a sense of pride and ownership. The biogas plant, a project implemented by the Food and Agriculture Organization, is also an appropriate technology that should yield positive benefits to the energy balance. Mixing food waste with septage biosolids is a recognized method of maximizing the volume of biogas produced from anaerobic digestion. In Azraq, the WASH programme has implemented gradual improvements of basic service. In 2015 shelters in Azraq were not connected to a greywater network, meaning greywater and mud tends to accumulate in the ditches surrounding the shelters. UNICEF with partners constructed a plot-level grey water network, and provided income generating opportunities within the camp contributing to camp cleaning and sustainable waste management practices. Communal toilets remain unpopular in Azraq. Women that the evaluation team interviewed in Azraq reported that they did not like having to fully dress and cover their hair every time they go to a toilet; children get dirty on their way to and from bathrooms, and pit toilets have holes that are too big for young children. As a solution, refugees often construct bathroom inside their kitchen space, but this leads to problems related to odors and hygiene. Also, there is no space for drying laundry, and it is culturally inappropriate to dry underwear in a place where it can be seen by others. Respondents also commented that the latrines for women and men, though separate, are very close to each other and perceived as not safe105. That said, the 2017 Azraq KAP survey indicated that 100 percent or respondents said that they used the WASH blocks and that they felt safe, compared to 96 percent in 2016. For Azraq, the installation of the Moving Bed Biofilm Reactor (MBBR) system, which provided by a donor turned out to be a poor choice, (although the rationale for selecting it was at the time thought to be valid) and did not meet the needs of the people served or the program in general. Wastewater strength was higher than anticipated and after lots of money spent, the system was eventually abandoned. Wastewater is now extracted from tanks and transported also at great expense. Until there is some centralized wastewater treatment option, septage will require hauling some 90 KM away106. The evaluation team believes it is surprising that the Azraq camp would have more difficulty in establishing a wastewater system as Figure 7 Rubbish Bin in Azraq

104 UNICEF reports that it has supported skills training initiatives in Za’atari to upgrade from low skilled to higher skilled labor. 105 Comments from interviews and focus groups in Azraq 106 Source: Azraq site assessment

it is neatly laid out, compared to Za’atari, which has developed an elegant and lower cost solution and has evolved into something that could become sustainable over the long term as economic development in the camp increases. Solid waste collection in the camps is adequate. The evaluation team observed rubbish bins, which appeared substantial in terms of their volume and appropriateness. Large bins on wheels that help minimize vectors such as flies and rats. Camp residents reported that rubbish is collected once per day. Focus groups that the evaluation team held in Za’atari reported that solid waste collection had improved over time.107 Hygiene Promotion The absence of assessment data with respect to the early stages of the response made it difficult to determine what the key public health issues were in relation to hygiene, and the behaviours that needed to be targeted, at least in the programme’s initial stages. The first Azraq camp KAP was conducted 21 months after the camp opened, making it difficult to establish an initial baseline. This contrasts to the initial Za’atari KAP survey conducted only 4 months after opening. The Za’atari baseline showed considerably lower perceived health status levels. A general observation by the evaluation team was the lack of a consistent connection to the health status of the population and the infectious disease risks prevailing at various times of the year. It is acknowledged that the health sector would house such data also but the connection to health status and infectious disease risk was not always present in the KAP surveys. For example in the 2013 KAP survey for Za’atari, reference is made to frequency of respiratory problems, and a suggestion is made to conduct messaging in 2014 (at least 18 months after the camp opened) on methods of preventing respiratory infections. Despite this suggestion, the only disease related questions in the 2014 KAP survey relate to diarrhoea. A notable exception was the Hepatitis A outbreak that afflicted Azraq. A notable feature for Za’atari and Azraq was the strong commitment to repeating KAP surveys in both camps on an annual basis enabling one to measure change over time. The evaluation team did not have access to hygiene data with respect to Host Communities, those in the ITSs, or the settlements (Hadalat and Rukban). However, the evaluation team understands that given the circumstances it was not feasible to gather hygiene data in those areas. In Za’atari, the baseline KAP survey (November 2012) indicated that hygiene knowledge was low. Only 50 percent named handwashing before eating as a diarrhoea prevention mechanism, and 36 percent named handwashing after using the latrines. This result was explained by UNICEF staff, who indicated that the residents of Za’atari at this time were coming from rural areas of Syria where levels of education were low. The percentage of people measured through observed practice though was high (an unusual result) with 55 percent of adults observed handwashing. As a point of comparison, an estimated global average for the number of people who wash their hands with soap after going to the toilet is just 19 percent (Freeman & al, 2014). Over time the KAP survey results for Za’atari showed an improvement. In terms of knowledge for example the baseline indicated that only 50% knew that handwashing before eating helped to prevent diarrhoea which rose to 76% in the 2015 survey. In 2013 for

107 FGD 1, 5 August 2018.

example the KAP survey reported an improvement in handwashing, with 98 percent claiming to wash hands with soap and water - up from 68 percent in 2012. This high level was supported/triangulated by observed data where 95 percent of assessors reported seeing evidence of soap in the house. This particularly high reported result should be viewed with caution and suggests potential bias in the results, as the results are exceptionally good. Nevertheless they do show an upward trend. (Za'atari Hygiene Promotion Working Groups). Some direct observations by the evaluation team also demonstrated high hygiene standards in areas such as water storage, cleanliness of toilets and general cleanliness of the household. The exception to continuous improvement was in 2015 when there was a drop in General Cleanliness from the 2014 results from 88 percent reporting their area was generally clean to 71 percent reporting in 2015. The KAP survey reports don’t provide much in the way of analysis to explain results but in 2015 this might have been explained by the increase in the camp’s population which was at its height in 2015 at a reported 120,000 (Za'atari Hygiene Promotion Working Grouip, 2015). In the area of hygiene supplies, interview respondents in Za’atari indicated that they did receive hygiene items, but not enough of certain items. For example in the 2015 KAP survey 88 percent reported receiving soap in the distribution but 88 percent also reported it was not enough. Other items commonly felt lacking in 2015 were laundry detergent and shampoo in addition to soap. From a gender perspective only 21 percent of respondents reported receiving sanitary pads in the 2015 survey, an indication that perhaps the contents of hygiene kits were not gender proofed. Again the results of the KAP survey do not offer a rationale behind such a low result. Note that of the 88 percent who answered that soap received was not enough, 86 percent still managed to access extra soap in the market. UNICEF indicated that the people of Za’atari had more means to access hygiene items in the market and therefore did not need “free” distributions of such items. This result was supported by the evaluation team’s interviews and focus groups. During the evaluation, respondents provided feedback on hygiene related items that indicated that people did receive the items they needed but sometimes not enough of certain items like soap powder and sometimes at irregular intervals. Hygiene was communicated in a number of different ways including through community level hygiene promoters, children’s hygiene clubs, campaign days linked to key global days such as global handwashing day and use of the mass media. A disappointing aspect for the evaluation team was the paucity of Information, Education, and Communication (IEC) materials provided for review108. In Zaatari, one of the useful indicators to highlight effectiveness of the WASH programme overall was reported general health. In the 2013 survey 58 percent answered their health was good or excellent which rose to 81 percent by 2015. Azraq opened in April 2014 quite some time after Zaatari. The first survey reviewed was from January 2016, 21 months after the camp opened which is a significant time lag before undertaking a KAP survey . The fact that the survey was undertaken in winter indicates some results, such as satisfaction with the quantity of water supplies could be very different if undertaken in the summer months when the demand for water is much higher. The baseline

108 The UNICEF WASH Programme provided materials used for hygiene promotion after the evaluation period. These included flyers, posters, and other materials promoting practices including dental hygiene, water conservation, sunstroke prevention, maintaining good health in winter, and other key practices.

survey in 2016 does indicate a high level of satisfaction with the water supply, with 95 percent expressing the supply was adequate or very adequate. Access was good, with 46 percent collecting water within 15 minutes. Quality also scored highly with 90 percent responding it “satisfies” them. Results in 2017 compare favourably, with a higher percentage, up from 44 percent in 2016 to 68 percent in 2017 satisfied. Satisfaction with quality dropped from a high of 90 percent in 2016 to 75 percent in 2017 with the issue of bad taste and/or bad smell cited as the reason. No explanation is given for this result. The Border Settlements (Rukban and Hadalat) Water The intended results at the settlements were to Figure 8 Power generators and fuel storage at Rukban provide water to the Rukban and Hadalat water system. Extra power generation and fuel storage capacity minimizes the potential for water system down Refugee Camps as effectively as possible while time. achieving value for money. The evaluation team inspected the equipment and operations of the water system at Rukban. The borehole and related equipment are of high quality, the design is appropriate, the construction is of high quality, and the operations and maintenance activities are performed properly and on schedule. Staff working for the contractor that, UNICEF hired to supply and operate the system at Rukban reported in an interview that UNICEF always purchases high quality equipment. The borehole in Rukban was drilled in December 2016 with an estimated yield of 80 m3/hour. The system contains: i) the production well (borehole), ii) chlorination, iii) sand filtration, iii) de chlorination, iv) reverse osmosis water filtration, and v) post processing chlorination. At the time of the inspection, chlorine residual was being maintained at between .6 and .8 mg/l, which meets or exceeds international standards. The throughput of the system is estimated at 56 m3/hour post-treatment. This water treatment plant has a capacity of 800 cubic meters of post-treatment water storage. In addition to the borehole at Rukban, UNICEF has constructed two pumping stations; the old pumping station (with two sets of storage tanks at the old station) and the new pumping station (with a storage capacity of 760 m3). “The old pumping station is located approximately 3km west of the Rukban borehole/reverse osmosis plant and the new pumping station is located 10km further west of the old pumping station and is approximately 13km west of the Rukban borehole/reverse osmosis plant. The two pumping stations pump across No Man’s Land to water distribution systems along the Syrian border and into the camp (Terms of Reference for Service Contracting - Rukban).” The new borehole came on line in September 2017. It has a capacity of 85 cubic meters per hour, which is greater than the expected 70 cubic meters per hour, and is 572 meters deep. Today the new borehole and treatment unit are connected with the old infrastructure by pipeline as the best way to secure the most water possible for the refugees. This treatment system is sophisticated and costly, but warranted and necessary due to the concentration of certain constituents of the raw water, and the fact that the system has to work properly and consistently with no down time. Water quality of the groundwater is poor with TDS at 3,230 us/cm and high levels of fluoride and iron. In consideration of the very high cost of the trucking operation, full cost recovery will be achieved from the savings alone within a matter of months. So far, the investment has saved almost 1 year of trucking, which cost 18.5 JOD per cubic meter. At 600 cubic meters per day, that is a cost of 4 million JOD

per year that is not being spent on trucking. The UNICEF team reports that the system implementation is 85% less expensive than trucking operation and delivers 30% more water. Other important reasons for installing the water system include: • increased volume of water resulting from no longer needing to comply with Army restrictions; • eased management resulting from reduced tension with the host community in Ruwayshed; • improved water quality and more consistent chlorination; and, • improved consistency of water delivery resulting from removal of truck and trucker management. Therefore, the value for money from this investment is high. The design of the system is strong with a capacity for multiple contingencies including leaks, power outages, or effluent quality issues (figure 2). There is 800 cubic meters of storage at the borehole site with tanks with the ability to operate independently in case of leakage. The system appears easy to operate as well. The UNICEF hired contract that manages the system maintains staff on-site 24 – 7. Staff performs daily O&M including: i) pumping from the borehole to the tanks, ii) checking the chemical concentrations every 2-3 hours, iii) routine cleaning of the sand filter weekly, and iv) repair and replacement of parts as needed. The evaluation team held one focus group at Rukban. Participants claimed that water has high levels of chlorine that is solely detected by smelling the water itself. One participant elaborated that she believed the chlorine caused intestinal inflammation and diarrhea in two of her family members. This could refer to enteritis (an intestinal inflammation that is usually accompanied by diarrhea, or perhaps other medical cases that are related to gastrointestinal diseases. All participants agreed that they were not willing to wait for the chlorine to fully evaporate before using their water. The team observed a chlorine test of the water coming out of the tank that goes through the pipe that is directly connected to the one of the tap stands. The indicator showed chlorine levels are at 0.5-0.7 mg/l. Participants in the FGDs indicated that they also have experience multiple water cutoffs and sometimes water was supplied 2 hours a day, sometimes more, sometimes less. They believe that cut-offs take place when there is a high demand on the tap stands. While this may have been the case during the earliest days of the water operations, it does not appear to be the case now, or at least since April of 2016 when UNICEF staff created a log book for operational issues, which indicates very minimal disruptions. UNICEF invested close to $1 million for the water system at Hadalat but it was only operational for 6 months. At the time when the investments were made, Hadalat was served by trucking operations that originated at Ruwayshed. The trucking was very expensive and unreliable with truck drivers going on strike demanding more money. The decision to improve the borehole at Hadalat and develop the water treatment system was sound, and actually saved 6 months of trucking expenses. It is also noted that water from Hadalat was used to supplement the water in Rukban to avoid increasing the load on Ruwayshed. It was impossible to tell at the time that the camp would be abandoned, so it is difficult to criticize the decision to invest in new infrastructure. UNICEF reports that the cost for the RO unit and the new pumps/gensets were recovered by the operation of the system, as well as supplementing water to Rukban. The system remains operational should a viable water supply be required for renewed trucking operations or should the camp become populated

again. UNICEF also reports that the system was handed over to WAJ in November 2018. The system remains operational should a viable water supply be required for renewed trucking operations or should the camp become populated again. Sanitation The evaluation team could not enter the camp to visibly inspect. However, FGDs on these points offered the following: • Numbers of toilets, privacy, adequacy and functionality is currently acceptable. • Vandalism of public and communal toilets early on was an issue that helped drive the movement towards individual toilets, one per family which is mostly the current case • Solid waste management is adequate • Desludging is being conducted on a regular basis. Some odors are observed but overflows are rare. While the FGDs were interesting, it should be noted that they were conducted under the observation of government minders, which may have influenced the comments of the participants. For example, while the FGD participants say they are receiving desludging services which are just fine, UNICEF notes that in actuality, no formal desludging has been accomplished since June of 2016. Hygiene Promotion Hygiene promotion in the settlements seems to be limited to the distribution of hygiene kits on a periodic basis. Initially, the plan was to distribute once a month but focus group discussions revealed that gaps in distributions may have lasted as long as four months. The dynamic situation and insecure environment limits, and often prevents, UNICEF’s ability to access the area, which explains gaps in distribution. Contents of the kits included collapsible jerrycans, detergent, sanitary pads, solar lights, soap, toothpaste, shampoo, and diapers. In addition, UNICEF have delivered training to 20 – 25 Community Health Volunteers for delivery of face to face hygiene promotion in Rukban. There appeared to be a lack of Information, Education and Communication(IEC) materials provided to the settlement population. IEC materials could have been included in the hygiene kits distributed periodically. UNICEF reports that hygiene promotion efforts were carried out through ACF until the June 2016 attack, when activities were suspended. Following the attack, UNICEF distributed flyers and soap through its clinic, as well as carrying out training to health workers. Activities were limited and difficult to arrange due to extreme security measures imposed by the military. ITSs and Host Communities It is difficult to gauge the WASH programme’s success in ITSs and host communities. The evaluation team had no access to numbers and locations of refugees living in host communities and the percentage of those served by the programme. It is also difficult to know if conducting projects in host communities serves UNICEF’s WASH mission in Jordan. Large donors like USAID and GIZ work with the MoWI to do large infrastructure projects. UNICEF’s projects are small and general. The projects that the WASH programme implements in host communities and ITS are effective. For example, in Irbid, where the evaluation team observed WASH programme initiatives, the needs of the host community are extreme regarding water and a key focus of UNICEFs efforts. While just a fraction of the amount spent on camps is going to the host community, interventions that UNICEF does implement are well planned and executed. In

Irbid, the interventions were sewer line improvement projects in city center, the rehabilitation of two boreholes, and a pumping station improvement project. The interventions were appropriate given the level of funding. Most of Irbid receives water every 5 to 8 days, so it is extremely scarce. However, the water utility has a NRW of almost 50 percent. In addition, more boreholes appear to be needed. It is assumed that the aquifer is well mapped in Irbid, however, operators report that static water levels are down almost 24 meters in one year. Operators don’t know if this is aquifer wide or a localized phenomenon because proper monitoring is not conducted. Modeling the aquifer through drawdown testing would be extremely useful. It would indicate if drilling a new borehole and connecting it to the system would be useful. The evaluation team visited two ITSs; camp 702 and camp 709. Life in camp 709 is harsh but basic services of shelter, toilets, water and sanitation appear to be met. The people that live there are extremely vulnerable and UNICEF aid is reaching them. One-hundred and twenty people live in the camp and work for low wages in agricultural endeavors. They have one latrine for every 20 people. There is a school with a UNICEF trained teacher on site. People pay 3 JOD for potable water per cubic meter from a vendor, or .9 JOD per cubic meter for piped water meant for agricultural use. Figure 9 Unimproved pit latrine cover at an ITS near Respondents to evaluation interviews Amman reported that they find the water storage volume and number of toilets lacking. They would prefer to have one of each per family instead of having to share with other families. It was reported that there is a fear that toilets and water tanks will be stolen if too many are present. However, the family head said he would take personal responsibility for these if UNICEF would just provide more. There is no indication of waterborne disease, and the health of the children seems good according to the residents. When asked if UNICEF related services have been improving over the years, the residents said yes, very much. UNICEF reports that it would like to do more in ITSs, and is encouraged by the benefits of innovations such as the modular collapsible toilet, which UNICEF developed. However, lack of donor interest, and earmarking of funding for WASH in ITSs, limits their ability to make progress in the area. Camp 702 is a cluster of 12 houses with 72 people. It is a nomadic agricultural camp similar to camp 709. Residents move seasonally between this site and a similar site in the Jordan Valley. Residents have refugee status, but it is still considered illegal to live in the ITS. People fear they will sent be to the formal camps, even though they have employment and livelihood opportunities here. Sanitation In the ITSs, UNICEF considered the need for on-site wastewater systems and determined that pour flush toilets and offset pit latrines would be most effective in terms of cost and function. This is the sanitation scheme in camps 702 and 709, where such latrines are shared by 3 families. For the most part, the pits were in poor shape, not having fly-proof or safe covers. Venting was also absent, so while the need for sanitation was considered, at least in this regard the implementation was poor. Upgrading latrines from single pit systems to proper Ventilated Improved Pit (VIP) systems would be a good next step.

WASH In Schools Between December 2014 and April 2015, the WASH programme assessed 3,681 schools, identifying needs and prioritizing schools for receiving WASH services. Of this number 400 have been reached. There were significant gaps to correct for all schools to reach the National WIS standards, which were only finalized and released in December 2017. For example, 35 percent of schools (1,297) failed to meet the standard for student to latrine seat ratio. It is important to note that the Ministry of Education requires all school interventions, including new classrooms and WASH facilities, to be delivered as a package. This adds to the challenge of delivering WASH in schools as it must be combined with non-WASH components. The evaluation team observed the WIS project implemented by WASH programme partner, the Norweigan Refugee Council (NRC). The quality of the project was clear from NRC’s report dated November 2017. The programme had a clear plan to both address infrastructure deficiencies (WASH blocks built in 3 of the 5 schools NRC targeted), and also build capacity (169 teachers trained on classroom activities on cleaning and hygiene. NRC produced a Manual with guidance for schools on how to carry out the “soft” part of the WASH in Schools programme). The evaluation team noted that NRC’s KAP survey was very high quality and its report was objective and informative. For WASH in schools, there is a complex regulatory environment when it comes to connecting new WASH infrastructure to existing sewer systems, as was found out in Irbid, where UNICEF invested in WASH blocks, only to have them still not connected to a sewer after more than a year. The lesson learned is to secure the permits prior to beginning construction. Coverage with respect to WIS in schools would appear to be low at roughly 10 percent of the total assessed (3,681). This finding should be put into context for a number of reasons. Firstly, UNICEF are not the only agency working on WIS. Secondly, not all schools were below standard and therefore did not require targeting. As mentioned above 35 percent did not meet the standard for toilet seat to student ratio. Lastly, in the schools targeted water supply was extended into the community adding to the per student cost and therefore limiting the coverage possible with limited funding. UNICEF did oversee several successful WIS pilots, such as the NRC project mentioned above. However, the successful results often didn’t receive follow-up for a variety of reasons, including the lack of suitable partners, lack of capacity within UNICEF to conduct community mobilization109, and lack of donor support. Partners also reported that the coordination of WASH in schools was a sub-group under the WASH Sector and only “ran sporadically.110” NRC also said that coordination engaged WASH in Schools Implementing Partners but not the Ministry of Education itself, who it is reported were invited but did not attend. NRC records indicate that UNICEF did not promote or try to expand results, possibly indicating a lack of focus on the WIS portion of the WASH programme. UNICEF decided to carry out the WASH in Schools KAP survey in December 2016 and repeated again in May 2017. This was too short a time frame to adequately assess changes in knowledge and behavior.

109 Statement made in UNICEF staff interviews with those that managed WIS efforts. 110 Source: Interview with NRC Staff – 29 July 2018

National Level Support UNICEF’s approach to national level support is solid at a basic level. Despite the highly politicized nature of the WASH sector in Jordan, UNICEF has provided assistance and guidance that the government appreciates. Examples include assisting with the development of the National Water Strategy and then disseminating it to promote its implementation. UNICEF also commissioned and developed the National WASH in Schools Standards in cooperation with the government and facilitated the distribution of the standards. UNICEF also conducted national vulnerability mapping, developed project databases, and developed GIS hubs that will serve the Government of Jordan for many years. UNICEF’s challenge will come in future phases as it works toward sustainable solutions for the camps that the GoJ might resist. Efficiency Evaluation Questions 6 and 7 • To what extent did the programme use available resources in an economic manner to provide WASH services and facilities in camps/settlements and host communities, to ensure child rights, equity and gender equality? • To what extent were the costs to deliver water and wastewater services, as well as solid waste management, rationalized and optimized in the camps and settlements to ensure Value for Money? How do these compare to similar situations at the local or regional level? Could more cost effective operations/interventions have been undertaken at an earlier stage? The evaluation ToR asks that the evaluation team assess the WASH programme’s economic use of resources and the extent to which costs to deliver water and services optimized. Assessing the programme’s use of resources is difficult for several reasons. First, UNICEF Jordan’s financial systems are organized to manage the organization’s risk, manage compliance issues, ensure operational solvency, and protect the organization against fraud111. Neither UNICEF Jordan nor that WASH programme track expenditures for management purposes112. The evaluation team could not obtain documentation that demonstrated expenditure by year, activity, programme component, or beneficiary group. Also, the programme does not track its indirect costs or general and administrative expense rates related to the programme, making it difficult to know the resources required to manage and execute the programme or to compare that to other similar programmes. UNICEF Jordan reports that UNICEF deducts 8 percent of all raised funds for HQ support and limits international partners to 7 percent allocation to partners headquarter costs. Additionally, a maximum of 7 percent is used to pay for Salaries of WASH staff and 5 percent is taken for cross sectoral support costs. The evaluation team did not see year by year information on these costs however, which would have indicated whether efficiencies were realized as the programme progressed from year to year and as new innovations were implemented.

111 UNICEF Finance staff interviews. 112 After the evaluation period, the WASH Programme provided estimates of its efficiency gains and cost savings. Those estimates are attached to the document as Annex I.

The WASH programme was able to provide some documentation regarding contract expenditure and Programme Cooperative Agreements (PCA)113. The evaluation team was provided information that accounted for $268,614,178114 of the approximately $355 million that programme managed between 2012 and 2017. However, the evaluation team notes that some significant agreements were missing from the documentation, such as PCAs with Mercy Corps, which financed Za’atari’s boreholes among other activites, and the PCA with the Norwegian Refugee Council, which implemented WASH in Schools projects on behalf of UNICEF115. Extent to which the programme used available resources in an economic manner to provide WASH services and facilities in camps/settlements and host communities, to ensure child rights, equity and gender equality The evaluation team understands this question to ask for evidence of the extent to which the programme used its resources in an economic manner, and the extent to which child rights, equity, and gender equity were part of the resource use equation. Broadly, the WASH programme’s resource allocation has five drivers in order of importance116: 1. WASH service and facility coverage for people living in the camps and settlements. 2. Improvements that increase efficiency in coverage in camps and settlements, as well as continuously seeking to ensure equity of service coverage and quality of services. 3. Improving water systems in host communities and ITSs. 4. Wash in Schools 5. Sector support for the MoWI. This decision-making hierarchy developed as a consequence of the programme’s evolution, the earmarked nature of its funding, allocations of areas of responsibility in response to its programme management structure. Camps and settlements were allocated resources first because, in the early days of the crisis, they were seen as having had no other options for WASH services other than UNICEF, as opposed to refugees who could prove sponsorship and had government or private options in host communities. After UNICEF was certain basic needs in camps and settlements were covered, resources were invested in efficiency improvements in the camps and settlements. Budget that was available after the effort in the camps and settlements was allocated to host communities, ITSs and Schools117. Expenditure and Efficiency - Camps In July 2012, Syrian refugees had two legal options. Refugees that could provide proof of sponsorship could live in a host community. Refugees that did not have sponsorship accepted a tent in Za’atari. At this stage, the international community mobilized and assigned degrees of urgency to the populations that needed assistance. The population that was under the direst circumstances was clearly the population moving into the Za’atari camp. Believing that this

113 PCA’s are the form of agreement UNICEF uses with non-profit partner organizations. 114 This figure includes the amount that UNICEF reported on each contract, and the value of the PCAs’ budgets that the evaluation team were provided. The evaluation team could not verify the degree to which partner organizations had spent or exceeded the budgeted amount included in each PCA. 115 UNICEF Jordan shared these documents after the evaluation period (29 November 2018) 116 This hierarchy is evident in UNICEF Jordan Annual Reports and it its allocation of resources. 117 Source: UNICEF Staff Interview – 6 August 2018

population had the most urgent needs, and that the crisis would not last long, UNICEF and other international actors in Jordan quickly mobilized to provide WASH services and facilities however they could be obtained. There is no indication that the cost of providing water services in the early stages of the intervention was rationalized. UNICEF was operating in an emergency situation and cost was secondary to providing lifesaving water, which involved expensive trucking operations, and renting mobile sanitation facilities until more permanent arrangements could be made. Design mistakes were made that would later prove costly, such as the WASH Blocks constructed in Za’atari that would eventually have to be dismantled because they didn’t meet Syrian cultural standards for cleanliness or privacy, and the location of Za’atari over one of Jordan’s largest aquifers, which provides 37 percent of Jordan’s water.118 UNICEF took steps to bring down the unit cost of providing water to refugees in the camps. As figure 10 below shows, the evaluation team estimates that between 2013 and 2016, the cost of delivering water to refugees in the camps dropped from 3.34 Jod/m3 to 0.24 Jod/m3.119 Figure 11 presents UNICEF’s estimate of cost savings realized in Za’atari. The drastic drop between 2014 and 2015 was a result of UNICEF reducing Za’atari’s dependence upon external water trucking by over 50 percent through new storage facilities and the operationalization of a third borehole.120 Another contributing factor was the operalization of a borehole at Azraq camp which led to external tankering. The drop in cost between 2015 and 2016 represents phase 1 of the water network coming into operation.121

The expense and inefficiency of the Figure 10 Cost of providing a cubic meter of water by year emergency phase of the response was a based on partner budgets natural result of the urgency, limited capacity, limited options, and limited resources that were available. As soon as the camps were stabilized, the WASH programme began taking steps to reduce costs and improve delivery. It also mitigated risks by reducing vulnerability and inconsistency in water access caused by trucker strikes. In Zaatari and Azraq, the investments in borehole construction and upgrading, water treatment and pumping equipment, and eventually network development, were all made with the specific goals of improving quantity, quality, access and the efficiency of the entire water service delivery operation, while reducing the high costs of

118 Source: UNICEF Jordan WaSH Specialist interview 119 Financial data estimated from UNICEF partner budgets included in PCAs, RAM reports, Annual Reports, and other sources of demographic data. Costs of delivering water come from partner organization budgets and may not accurately reflect actual expenses. 120 https://www.unicef.org/appeals/files/UNICEF_Syria_Regional_Crisis_SitRep_Sept_2015.pdf 121 UNICEF Annual Report 2016 - Jordan

trucking. To a great extent, UNICEF accomplished this goal through these investments. The investments were optimized by: i) utilizing high quality equipment and components, ii) training local operators, and iii) utilizing local labor for network construction. Figure 11 Water provision cost savings in Za'atari (Source: UNICEF WASH Programme)

There is some indication that trucking water into Za’atari camp went on longer than it should have, draining the programme’s resources. Construction of the network began in 2014 and was not completed until 2017. The delay was related to securing funding for the pipeline, and a long process to approve the design of the network. However, in Azraq, UNICEF transitioned there in there in May/June 2017 and it appeared to be very smooth with no subsequent issues or need to supplement with external water. There, the system is run by gravity, while simultaneously reducing the walking distance for the camp residents. UNICEF didn’t have the funds for these improvements, but instead advocated across the sector successfully for others to mobilize, which they did. This is an important indicator of success of UNICEF's leadership role, not only in addressing the immediate crisis, but the broad array of stakeholders as well.122 Sanitation (WASH facilities, solid waste management and desludging) Initially wastewater management was inefficient. Issues such as contractors claiming to remove up to 40 percent more sludge than they actually removed burdened the programme. Drivers claimed trucks were capable of holding more volume than they actually did and sometimes carrying waste in to the camp from outside, which was subsequently charged to the UNICEF operation123. Also, on some occasions, it was learned that drivers claimed for the official licensed volume of the trucks, but they had been modified to carry less. This inefficiency was fixed by UNICEF introducing a comprehensive 3rd party monitoring system whereby trucks were physically inspected entering and leaving the camp, and the dispatch point of the trucks was controlled through the voucher system. UNICEF developed the Waxi app to ensure that wastewater was collected on a priority basis and to reduce the risk of full tanks (people put blankets and other materials into the network and tanks) – this allowed the efficient and optimal collection of wastewater across the camp and ensured a near continuous flow of wastewater into the tanks at the Za’atari Wastewater Treatment Plant, which ultimately kept the drivers happy.

122 Source: Interviews with UNICEF senior staff. 123 Source: Interview with UNOPS

The wastewater system in Za’atari Figure 12 Membrane bioreactor wastewater treatment system evolved from unimproved pit latrines to a functioning properly at Za'atari Camp. system of shared septic tanks, a wastewater collection network, and eventually a wastewater treatment and reuse system. The program realized value for money by: i) utilizing a lower cost and innovative wastewater collection scheme, ii) utilizing cash for work in network construction, and iii) utilizing appropriate wastewater treatment and reuse technologies (MBBR and Trickling filter) which appear to be currently working reasonably well. The network and the internal system allowed UNICEF to have leverage with the desludging contractor. UNICEF also rebid services and infrastructure regularly to reduce the ultimate cost. In Azraq, the wastewater treatment plant is not functioning and has been abandoned. The anecdotal evidence from stakeholder interviews indicates that this was a MBBR system (moving bed biological reactor) that was a donation and sized based on wastewater characteristics from Zatari, and before Azraq was even inhabited. The wastewater strength turned out to be very strong, and also toxic, which killed the media regularly. UNICEF was assured that the treatment facility would be appropriate and it was a donation and so they did not pay for it (they paid for the installation and configuration) and it was a better alternative than expensive long distance tankering. UNICEF is currently looking at local alternatives In the area of Solid Waste Management, there were initially 180 waste pickers employed in Za’atari. This was rationalized down to 40, who only picked waste in neighborhoods once per week in an attempt to improve attitudes and practices with regard to waste among residents. The reduction is staff improved efficiency124. Unicef was the driver of this, but experienced resistance from the other NGOs, something that ultimately reduced the efficiency and effectiveness of the activity. Another innovation that UNICEF developed was the “Waste Taxi”, which served as an on- demand removal system. It was a predictive model designed to address with the various different volumes of wastewater tanks that were introduced when people began constructing their own private washing and toilet facilities in Za’atari. This system predicted when tanks were full or close to full and ensured tanks were emptied at just the right time thereby saving money125. Rukban and/or Hadalat Water UNICEF was paying for the water tankering, and it was causing tension with the host community. In Rukban, and to a lesser extent in Hadalat, the issue was not just the cost, but the consistency of the service, strikes, and water quality. Providing water to Rukban

124 Ibid. 125 Source: Intervuew with UNOPS Staff Member – 24 August 2018

required: i) initial emergency water hauling, ii) improvements to the water sources at Ruyashed and Hadalat, iii) conducting the hydrogeologic study, installing the new borehole, treatment and pumping equipment, plus a 13km pipeline and two parallel water distribution systems. Delivering water by truck across the desert 90 KM one way was very expensive but necessary for the function of the camp during the initial stages of the intervention. Investments to improve the water source at Ruyashed were necessary to ensure water quality met the applicable standards and was a necessary expenditure. The hydrogeologic study and subsequent new source development represented value for money as it defrayed the cost of trucking. UNICEF estimates that the solution was 85 percent less expensive than trucking. The rationalization was that these investments directly reduced the cost of the trucking, and were therefore appropriate, which in retrospect was sound. Water trucking to Rukban (from Ruwaished) used to cost 380,000 JOD per month. It cost 18.5JOD as it included 15.5 JOD plus 3JOD/m3 as a tariff for the Govt, for both) per cubic metre) and went on for 15 months. After an investment of 500,000 JOD (for borehole) UNICEF paid for the RO / sand filtration unit, and generators, fuel and O&M costs at Rukban, resulting in the supply cost being reduced to 100,000 JOD per month – a reduction of 74 percent of the original cost for 30 percent additional water.126. Costs associated with the Hadalat water system, were rationalized at the time. However, the sudden evacuation of the camp significantly reduced the return on investment. The system still has value as a viable but unused water source in an arid region that is short of water. The situation at Hadalat, as well as the timing for making the new source investments at Rukban, illustrate the difficulty of making decisions on infrastructure expenditures in a climate of unknowns. The evaluation team believes that that UNICEF invested appropriately for water infrastructure, was timely in their investments, and received value for money related to the new source in Rukban and Hadalat. Host Communities and ITSs In the beginning of the emergency response phase of the programme, Jordanians were supportive of efforts to make arrangements for Syrian Refugees. Tensions quickly developed however between Jordanians that lived close to Za’atari in Mafraq governorate, and the refugees that lived in Za’atari. These tensions developed largely over the demands that refugees put on local resources (REACH, 2014) The WASH programme has allocated funds for host community projects based on availability and procured contractors or INGOs to carry out the projects through UNICEF’s procurement system and in consultation with the government. An example is HC projects carried out in Irbid. There was little funding available for water improvements in the host city of Irbid. Consultations with the water utility, city government, and ministry were conducted as well as an evaluation of the water network. Settling on investments to improve the existing boreholes and pump station significantly improved the efficiency of the water source network. The project represents great value for money as these were sorely needed interventions that in the end did not cost that much compared to the added volume and efficiency of the water system that resulted.

126 Source: Interview with UNICEF WaSH Officer

Evaluation Question 8 • To what extent did UNICEF try to minimize duplication/gaps in WASH interventions in camps? And how early did UNICEF move to more cost-efficient interventions?

In this section we describe how UNICEF minimized duplication in camps. Please see the evaluation team’s answer to Evaluation Question 7 regarding UNICEF’s evolution to cost efficient interventions. UNICEF minimized duplication or service gaps in camps by assigning different partner organizations responsibility for WASH services in each sector127. UNICEF’s role as sector lead allowed it to unify service standards among the different partners.

From July through December Figure 13 Division of labor among UNICEF Partner organizations. (Source: of 2012, UNICEF worked WaSH in Za'atari Snapshop, Jan-Feb 2014) with Mercy Corps128 to establish and manage WASH services in Za’atari, and also partnered with ACTED, who conducted hygiene promotion. In 2012, UNICEF also began working with Oxfam followed by ACTED to provide WASH services Cyber City, KAP, and “the stadium,” three transit centers as they were known at that time129. Between January and June of 2013, the refugee population in Za’atari grew from 32,000 to over 116,000 (OXFAM PCA 32-13 Executive Summary). To address the task of managing WASH services for the huge population growth and to avoid duplication of services in the camp, UNICEF worked with UNHCR’s division of Za’atari into twelve districts and three sectors. UNICEF partner organizations each managed one sector and coordinated activities at weekly Za’atari Coordination meetings. ACTED managed districts 1,2,9,10,11, and 12; JEN managed 3,4 and 5; and Oxfam managed 6, 7, and 8. In Azraq, a similar coordination structure was followed as Za’atari. ACTED, World Vision, and THW provided WASH services with Relief International providing hygiene promotion and WASH in schools. However, coordination among the three entities appears to have been difficult. There is far less documentation on coordinated efforts in Azraq. UNICEF reports that there were weekly coordination meetings in Azraq. UNICEF also comments that partner organizations often brought in activities funded from other donors, which added to the challenge of coordination in Azraq.

127 Some UNICEF partners were assigned responsibility for camp-wide services, such as solid waste collection. 128 WaSH Sector Coordination meeting minutes indicate that Mercy Corps worked with UNICEF through 2016. The evaluation team had no documentation of Mercy Corps’ involvement in the WaSH Programme. However, the team was able to interview two Mercy Corps staff who participated in WaSH activities. 129 The source of this information is the PCA’s signed with each organization.

In June 2015, UNICEF decided to simplify operations in Azraq by partnering with one organization to handle WASH services. It asked partners to submit proposals and awarded WASH operations to ACF. With the assistance of World Vision, ACF took over WASH operations in October 2015. While these transitions were handled abruptly, causing some discomfort for UNICEF’s partner organization, ACF reports that they reduced operations and management costs bringing efficiency to the camp. Recently, UNICEF has transitioned programme management in camps again from ACF to a private contractor. Moving from an International Non-Governmental Organization (INGO) to a local business is a new direction for the programme and carries some benefits, such as reduced indirect costs. The new approach also carries some new risks, as a private contractor may not have the same incentives to manage depreciation of equipment and infrastructure or project cost overruns as an INGO, and UNICEF does not currently have systems capable of monitoring and accounting for those items. Likewise, in Zaatari, the WASH programme has reduced costs and realized efficiency by reducing the numbers of partners that they are managing. In Za’atari, UNICEF has reduced the number of partners to ACTED and Oxfam. Evaluation Question 9 • To what extent were resources (financial and human resources) allocated for the range of intervention types, with particular attention paid to the respective allocations for humanitarian response and resilience, to achieve the intended results? The WASH programme has four components, briefly; WASH in the camps, WASH in host communities and schools, support to the national government, and coordination of the humanitarian sector. During the period covered by this evaluation, the great majority of the WASH programme’s resources went to humanitarian response as represented by WASH management in the refugee camps, which was their direct responsibility. UNICEF’s responsibility for financing the WASH services in the camps constrains its ability to address more strategically important issues, such as the approximately 80 percent of refugees that live in outside of the camps, or the high levels of NRW in Jordan. UNICEF Staff estimated that approximately 5 percent of UNICEF’s budget goes to host community interventions. The evaluation team’s estimate, which as mentioned is based on incomplete information130, is that from January 2013 - -July 2017, about 63 percent of project funds went to camp operations, about 15 percent is spent on host communities, and another 15 percent is spent in settlements, Rukban and Hadalat (figure 13).

130 The financial numbers used in this report were derived from UNICEF’s estimates of expenditure on contracts and PCA agreements provided by UNICEF, which were often in a format that was difficult to read. PCA numbers were budgeted amounts, not actual expenditure. These figures also do not include any indirect costs that may not have been allocated proportionally by beneficiary group.

Figure 14 Percent of UNICEF Total Expenditure by Beneficiary Type (August 2012 – July 2017)

Figure 14 represents 97 percent of expenditure over the life of the programme, according the data the evaluation team was able to access. In terms of types of intervention, and water infrastructure construction and sanitation infrastructure construction are the largest expenditure category (WatCon and WatProv), together describing about 47 percent of the programme’s resources since its launch. Next is water provision (WatProv), which accounts for another 20 percent. The remainder is Sanitation Mobilization (SanMo), Desludging, Solid Waste Collection (SWCol), distributed kits, and funds that went to activities to support MoWI. Figure 15 WASH Programme expenditure by activity category (August 2012 – July 2017)

Note that following the evaluation period, the WASH Programme did provide some additional financial information regarding the breakdown of expenditure by activity categories. That information is summarized in Annex J.

As regards human resources, the evaluation team could not find documentation of how human resources were utilized, such as organization charts or job descriptions. Evaluation Question 10 • To what extent did UNICEF advocate with the Ministries to mobilise their available resources to address the identified needs? And how efficient was UNICEF’s support to ministries (i.e. MWI and MOE) to reaching most vulnerable people? Did the WASH programme maximise the potential collaboration with other UNICEF programme sections and partners? Over the time frame that this evaluation covers, UNICEF worked with the MWI to develop and update vulnerability maps, design the Jordan Response Plan, and provide financial support to GoJ priorities. UNICEF’s engagement with the WI included a consultant UNICEF hired to assist the government in developing strategies and response plans. UNICEF also interacted with the government to support host communities. The evaluation team saw no documentation of WASH programme advocacy with the MoE during the period described by the scope of the evaluation. UNICEF reports that it proposed the WASH in Schools Standards to the MoE and led their development, as well as coordinating the review. UNICEF Jordan has education programmes that may have had primary engagement with that government ministry. Still, a UNICEF partner reported that there is more demand for education services than the MoE can handle. UNICEF could work more to support the capacity of the MoE and should consider building those activities into its future strategy in this area. Sustainability Evaluation Question 11 • Was the infrastructure in the camps constructed in such a manner to ensure long term functionality, and will continue to service the most vulnerable people even when UNICEF is no longer directly supporting? This should be investigated in terms of Water and Wastewater infrastructure. Following the emergency response phase of the programme, the infrastructure in the camps was constructed to ensure long-term, sustained functionality. The GoJ prohibited the construction of permanent structures at the beginning of the crisis, forcing UNICEF to make use of mobile sanitation and trucked in water. When it became apparent that the refugee crisis would not abate quickly, the government recognized the need for longer-term solutions and UNICEF provided lasting, sustainable solutions. UNICEF was provided with highly mechanized treatment systems for both Za’atari (trickling filter and MBR) and Azraq (the MBBR system, which eventually failed). These systems are expensive, but the decision to use these systems was sound and valid. The national laws of Jordan specify stringent effluent quality from municipal wastewater sources. Even though camps are surrounded by vast unused parcels of government land, wastewater systems must comply with environmental standards. Also, the 4-month rainy season that makes it more difficult for passive wastewater treatment systems to produce quality effluent year-round. As written in the Efficiency section of this report, UNICEF has worked to maximize sustainability in Za’atari with innovative wastewater collection models, elimination of trucking, and the use of local labor. UNICEF has also installed dataloggers in boreholes to measure water levels and monitor water level changes. UNICEF also points to the location of

Za’atari and that the GoJ’s involvement in Za’atari’s WASH service design will facilitate turning the system over to the government if/when the refugees leave the camp. For Azraq, though the MBBR system failed, the rationale for selecting it was valid. UNICEF’s plan for sustainability in Azraq is difficult to discern. As UNICEF reports, “It is very unsustainable to build a massive wastewater network across (an area) the size of Azraq, in the middle of the desert, with no alternative use for it, and that the Syrian situation suggests that peope in Az would not stay.” For Azraq, the installation of the MBBR system, translocated from Afghanistan and provided by, turned out to be a poor choice, although the rationale for selecting it was valid. Unfortunately, the system did not meet the needs of Azraq’s inhabitants or the program in general. Wastewater strength was higher than anticipated. Following an expensive attempt to make it work, the system was eventually abandoned. Wastewater is now pumped and transported at great expense. There is a greywater management scheme that has yet to come on line fully, and until there is some centralized wastewater treatment option, septage will require hauling some 90 KM away. It is surprising that the Azraq camp would have more difficulty in establishing a wastewater system as it is neatly laid out, compared to Za’atari, which has developed an elegant and lower cost solution, which has evolved into something that could become sustainable over the long term as economic development in the camp increases. The water systems in the camps are designed, installed, operated and maintained to provide long term functionality, even if UNICEF were to leave, which is unlikely. The quality of the equipment is high, which adds to the longevity of the systems. The design provides maximum operational flexibility, with multiple redundancies. The system is relatively easy to operate, which also adds to the long-term sustainability. In Azraq, local camp residents are performing the operations tasks with oversight provided by a local company, which is a strong indication of sustainability. However, actual operational costs are high. The high costs are largely due to the energy it takes to run the systems, but also the cost of consumables, spare parts, water fee payments to the WAJ and generator costs in Azraq. UNICEF also reports that it prefers paying more for high quality contractors, as lower capacity contractors become much more expensive if system disruptions require a return to tankering. If money is not available for these, the systems will not achieve long term sustainability. The wastewater system at Za’atari is functioning well and should continue to do so given the required operation and maintenance. Wastewater systems have design lives of 20 years or more. This system is very robust and should provide long term and compliant service, again, given proper O&M. The system itself, MBR and Trickling filter technology is relatively easy to operate, which adds to its long-term sustainability. The collection system utilizing common septic tanks for between 5 and 8 families requires a camp-wide organized desludging program. This program is an integral component of the collection system that will require long term funding. The use of cash for labor on the installation of the sewer network built a sense of ownership of the infrastructure for some camp residents and also adds to long term sustainability. The water system at Rukban is constructed of high quality components that are well designed and relatively easy to operate. A local engineering firm has been contracted to operate the system and they are doing so effectively. As long as funding is available for O&M, the system should function sustainably for many years. During KIIs of the water operators, it was learned that UNICEF tends to purchase high quality equipment and replacement parts, which minimizes down time and should add to long term functionality.

The upgrades to the boreholes and pump station in the host community of Irbid are well designed and constructed and should continue to operate give the required O&M for many years. While the UNICEF interventions are sustainable, the overall water situation in Irbid however, is not. The extreme water scarcity, depleting aquifer, population growth, and high NRW rates are all causes for concern. They suggest areas for next steps should donor funding be available. Evaluation Question 12 • To what extent did UNICEF take sustainability into account at the early stage for water, wastewater and solid waste management services (e.g. transition to local government institutions and local capacity)? The evaluation team has no evidence that UNICEF has a transition plan to government or any other institution in place. The GoJ at all levels has made it clear that they will not take ownership or spend public funds on the crisis. The government understands that it is providing an international public service in using internal resources such as water to sustain the refugees and expects the international community to provide for the rest of their needs131.

UNICEF’s approach to sustainability has three aspects132: • Build high quality infrastructure: Building systems and infrastructure that are consistent with national standards and have long term potential to serve Jordanians and the national government. • Leverage other, non-governmental resources to contribute to ongoing operations of the camps: This approach includes building the capacity of refugees to manage their own WASH issues and partnering with national entities to engage in management. • Become as efficient as possible: UNICEF continues to find ways to bring down costs so that resources are used as efficiently as possible, and available as long as refugees need WASH services. A question remains of whether running a municipal water utility, which is essentially UNICEF’s role in the camps, is an appropriate one for the organization. Strategically, UNICEF will have to decide whether it wishes to orient itself toward permanently providing WASH services in the camps or should develop a strategy to hand it over to an organization that may be better suited for those types of long-term administrative activities. A lack of alternative organizations and guaranteeing payment for a replacement organization are complicating factors. UNICEF reports that they investigated options for transferring solid waste and desludging management to municipalities but lacked the resources and interest on the part of municipalities required to make the transition. Evaluation Question 13 • To what extent did UNICEF’s interventions increase the resilience of the Government and respective target populations? To what extent will UNICEF-supported projects (schools, ITSs, water and wastewater infrastructure, mobilisation) continue to be operational after the end of UNICEF’s financial support

131 Source: Interviews at the MoWI, WAJ, and UNICEF. 132 Source: Interviews with UNICEF executive staff.

This evaluation question is difficult to answer because it appears that target populations will require UNICEF’s financial support well into the future. UNICEF increased the resilience of supported populations and the GoJ in the following ways. • Designed projects in accordance with the JRP, linking refugee assistance in Za’atari and Azraq with resilience-oriented projects that rehabilitate or extend systems in urban areas with large refugee populations. • Highlighting need through vulnerability maps and mobilizing for funds for resilience projects described in the JRP to assist the GoJ. • Instituted consistent standards and policies around water access in camps to promote equitable access and reduce conflict. • Worked with the MWI to design the National Water Strategy. • Ensured that infrastructure was built using high quality materials and that construction was carried out according to national standards. • Built capacity of stakeholders in camps to manage water, sanitation and hygiene within their communities. • Created mechanisms for hygiene and water conservation promotion that operate under the leadership of local stakeholders, such as WASH committees and the Lead Mother model in Azraq. • Improved water and sanitation infrastructure and services in some host communities with high refugee populations to reduce tensions. • Worked with the government to create Wash in School Standards to set a bar for water, hygiene and sanitation in Jordan’s often overburdened schools. The standards included a nation-wide assessment. • Raised awareness regarding water conservation and promoted a leakage detection app. • Transferred a process for creating response and water strategies to the GoJ that they can replicate in the future. • Promoted the use of data for strategy development at the MoWI. Evaluation Question 14 • Does the MWI have the necessary mechanisms in place and capacity to ensure the water and sanitation needs of the most vulnerable people will be met as a result of UNICEF’s support and advocacy, if another large scale humanitarian crisis occurs in the future? The evaluation team did not find evidence that MWI has the necessary mechanisms in place for a future, similar crisis. One reason for this is that the government does not want to own these crises or been seen as a refuge for at-risk populations. Additionally, UNICEF reports that public support has dropped considerably compared to the situation in 2012. Evaluation Question 15 • To what extent did UNICEF document its WASH interventions and make available to the sector learning on water, wastewater, solid waste management and mobilisation? Documentation of interventions for the purpose of sector learning was limited. Lessons learned were shared at coordination meetings. The evaluation team did not find evidence that the WASH programme documented its experience for the purpose of sector or institutional learning. This evaluation is the first attempt to document UNICEF’s experience implementing the WASH programme in Jordan. UNICEF reports that it created a new position in January 2018 to document sectors learning and the position is working well.

Evaluation Question 16 • To what extent were environmental protection and sustainability taken into account in the design and implementation of programmatic interventions? Early in the programme, environmental protection was not taken into account sufficiently. Lack of focus on environmental issues led to Za’atari’s construction over an aquifer that accounts for 34 percent of Jordan’s water supply without proper facilities for managing wastewater and preventing it from leaching into the ground. As described in earlier sections of this report, the problem was exacerbated when open pits filled with graywater developed as camp residents began to create private sanitation facilities in their housing units. UNICEF corrected this issue in Za’atari by installing wastewater collection tanks, reaching 2,000 m3133 of collected wastewater per day, utilizing backwash from borehole water filters for construction and cleaning purposes, and constructing a water treatment plant. UNICEF also backfilled over 11,000 open pits. Water usage was controlled through UNICEF’s enforcement of regulations and daily volume collection and distribution. Seasonal water allocations are used to ensure sufficient levels, and in Azraq, the community has been engaged to prevent wastage. UNICEF applied the lesson it learned in opening a site location and water system for Azraq. Prior to the construction of Azraq, UNICEF worked with the MoWI to conduct an environmental impact assessment. The assessment identified a high risk of underground water contamination. Also, the GoJ was concerned about increased abstraction from the well field near Azraq. UNICEF incorporated protections in its design of Azraq’s WASH services. It supported installation and operation of containerized wastewater units. As seen earlier in this report, the treatment plant ultimately did not work out, but the evaluation team recognizes that the decision to include it was appropriate at the time. The septage receival and dewatering units would have permitted the system to receive wastewater from trucks and bring down the cost of desludging operations. In 2016, a pond was also constructed at Azraq camp to store wastewater that could be used for agriculture. UNICEF also assisted the government in improving environmental protections as regards WASH services. For example, UNICEF built the capacity of the MoWI to conduct groundwater monitoring more effectively. Other initiatives that UNICEF implemented that took environmental protection and sustainability into account include: • Working with the Royal Scientific Society to study the most environmentally sensitive way to provide heating for camps. Providing heating with gas was chosen. • Promoting water conservation in camps and host communities through implementation partners. • Celebrating world events, such as World Environment Day. • Promoting recycling where feasible. • Leading the construction of household connections in Azraq to reduce potential contamination.

133 UNICEF reports this represents a doubling of wastewater volume collected.

Coverage Evaluation Question 17 • Did UNICEF’s WASH response identify and reach the most vulnerable people? To what extent did specific interventions strike an equitable balance between interventions in camps/settlements and in host communities, to reach the most vulnerable people? Identification and outreach to the most vulnerable people It is difficult to know if the response identified and reached the most vulnerable people over the course of 2012 – 2017. At the initiation of the project it was assumed that refugees housed in Za’atari camp were the most vulnerable, as refugees taking up residence elsewhere had other options for WASH services, such as private sources or the government.134 The evaluation team did not find documentary evidence of an attempt to identify the most vulnerable people and reach out to them following the emergency response phase of the project through July 2017. For example, while UNICEF has served inhabitants of some ITSs, there is no comprehensive understanding of the number of ITSs or their population. That said, UNICEF described its efforts to the evaluation team. These included: • Sanitation assessments in Za’atari to identify households that needed the most assistance. These households were supported by partner organizations. • Identification of households that included people with a disability, and a subsequent construction of ramps to meet their needs. • Vulnerability assessments conducted in Ruwayshed followed by a prioritization of the households which met vulnerability criteria for assistance. • Identification of households in extreme in partnership with ACF, and conducting interventions for these households. • Continued work with especially vulnerable schools, ITSs, and host communities.

Following the emergency response phase, the WASH programme continued to devote most of its resources to camps throughout the life of the programme. The programme did not methodologically study the number and status of refugees outside the camps, and the proportion of programmatic resources that went to people outside of the camps was small. Though approximately 80 percent of the Syrian refugees in Jordan live outside of the camps, only between 5 and 15 percent of the WASH programme budget went to host communities. Finding resources to serve host communities more equitably would have meant reducing standards in the camps. Within the camps, UNICEF’s WASH programme has had a focus on ensuring access for the most vulnerable. This includes, women, children, and people with disabilities. Please see the section titled “Extent that UNICEF Achieved Equity Results” under evaluation question 4 of this report. Equitable balance between interventions in camps and host communities The degree to which the programme is focused on the most vulnerable people is influenced by the WASH programme first mandate, which is to maintain WASH service levels and standards in the camps. While people entering the camps may have been the most vulnerable

134 Source: Interviews with UNICEF senior staff.

at the programme’s outset, the evaluation team believes that in later years occupants of ITSs were more vulnerable and received less attention from the WASH programme. However, the necessity of maintaining service in the camps combined with budgetary shortfalls has prevented UNICEF from identifying and reaching that population. Until July 2012, UNICEF Jordan did not have a WASH programme. It had WASH activities within its health unit, but not a stand-alone programme. When the Syrian refugee crisis began, the Jordanian offices of UNHCR and UNICEF came to an agreement that UNICEF would provide WASH services in the camps, and that UNICEF would lead the WASH sector. Thus UNICEF became tied to managing WASH in the camps as a first priority. As executive staff at UNICEF Jordan put it “The camps had to be a starting point for funding allocation because there was no other option for those people. 80,000 in Za’atari had no other way to survive. Others had government and other options. The starting point is how do we maintain services in the camps.” As the number of refugees quickly increased, tensions between host communities and refugee populations grew. Three months after Za’atari opened, Mercy Corps conducted a study to document the growing tensions between Jordanians and the refugees. UNICEF began allocating some resources to projects in the host communities, particularly Mafraq and Ramtha, but its focused remained, appropriately, on the camps, which provided resources to people with no other alternative. In the 2012 UNICEF Jordan Annual Report, UNICEF Acknowledged, “here has been an additional challenge with the identification of the most vulnerable children within the host communities,” That challenge continued through 2013 and 2014 as the population of the camps continued to grow. UNICEF implemented projects in host communities as well, but they were for the purpose of quelling tension rather than serving the most vulnerable. REACH’s June 2014 report, “Access to Water and Tensions in Jordanian Communities Hosting Syrian Refugees,” illustrates why UNICEF chose the projects it did in host communities, and why reducing tensions was a worthwhile goal. A host community project in Irbid serves as a good example. In Irbid, the WASH programme improved sewer lines and rehabilitated two boreholes. The improvements were hugely important in Irbid, a governorate with a high refugee population and degraded water infrastructure. The evaluation team’s inspection found that the interventions added volume and efficiency to the water system. As early as 2014 the WASH programme began to target indicators improving water access for Jordan as a whole in its indicator framework. By 2016, though it was clear that the large majority of refugees were outside the camps and that those living in ITSs were existing under dire circumstances, still only 4 percent135 of the WASH programme’s budget went to host communities. Serving the most vulnerable in the camps Within the context of the camps, the WASH programme has been conscientious of serving vulnerable inhabitants. This evaluation report details initiatives to serve vulnerable population in the camps in response to Evaluation Question 4.

135 Based on incomplete information submitted to the evaluation team.

Evaluation Question 18 • Did UNICEF as Sector Lead ensure that the needs of those in most need were addressed in a coherent manner, particularly in comparison to alternative existing systems at the local or regional level (i.e. UNRWA camps)? If not, how? As described elsewhere in this report, the evaluation team found that UNICEF ensured the needs of those in camps and at the settlements were addressed in a coherent manner. UNICEF organized sector coordination efforts and coordination efforts at the camp and settlement levels to design activities, assign responsibilities to avoid duplication, and set standards to ensure equity of results as far as that was possible. Following that, UNICEF worked in host communities and ITSs to the extent resources allowed. The evaluation team does not have evidence that UNICEF used its position as sector lead to make sure those in need were identified and that their needs were addressed. UNICEF reports that it shared vulnerability mapping widely across the sector and encouraged partners to raise funds to serve vulnerable people. Coordination Evaluation Question 19 • To what extent did UNICEF role as WASH sector lead enhance response processes and results to avoid duplication and gaps in interventions?

Please see the evaluation’s response to question evaluation question 8 regarding avoiding duplication.

Regarding enhancing response processes, a sample of WASH sector meetings from May 2016 through April 2018 show 13 organizations regularly attending. Action points and focal points are thoroughly covered, and cover issues from the National Water Strategy to Schools Assessments and issues in the camps.

UNICEF also established a number of reporting mechanisms through third party monitors including water quality reporting, and security incident reporting. UNICEF established the use of vulnerability Maps for Water and wastewater. The maps revealed Azraq governorate as the most vulnerable place in both water and sewage. UNICEF has also worked with partners to submit proposals for sources of funding to address specific issues.

UNICEF has also put tremendous effort toward preparation for winter storms including stockpiling supplies, coordinating partner roles, and negotiating contingency plans. UNICEF also provides ongoing monitoring during storms as well as clear communication with partners. UNICEF partners that the evaluation team interviewed revealed partner satisfaction with UNICEF as sector lead. One partner said that, “UNICEF are very good coordinators. They always have staff in Azraq.” A donor said that they were very happy with UNICEF’s coordination and partnership. A staff member of one partner said, “UNICEF does a very good job of controlling the work. 95 percent approval rating for UNICEF.”

Evaluation Question 20 • Were partnerships effective and leveraged to the maximum extent to assist the most vulnerable population in camps/settlements, ITSs and in host communities?

The evaluation team believes that partnerships were effective and leveraged to the maximum extent in the camps and settlements. As described in this report, UNICEF utilized its partners to ensure full coverage in the camps and compliance with standards. Based on the limited financial information that the evaluation team was provided, the team estimates that UNICEF also leverages over $27, 250,000 in in-kind contributions from the partners it worked with136.

UNICEF also effectively worked with partners in host communities. Partners were relied on to identify and manage key interventions. UNICEF had clear communication with partners to identify ITS and school needs and to mobilize appropriate funds. WASH in schools programs were also managed effectively through partners. One aspect of working with partners that UNICEF could have improved was communicating strategic decisions and medium-term vision for the project. The UNICEF partners that the evaluation team interviewed reported that they often heard about drastic changes to plans or operations with little notice or time to prepare. One partner noted “UNICEF take quick decisions – the change from Mercy Corps to ACF in Azraq could have been done in a better way. It was not a transparent process.”

Conclusions In July 2012, UNICEF was the only organization in Jordan that had the resources, capacity, and institutional commitment to take leadership of the WASH response related to the Syrian refugee crisis. UNICEF accepted responsibility for providing WASH services in a country that was exhausted by previous refugee crises, water scarcity, a complex political environment, and funding uncertainty. From July 2012 through July 2017, UNICEF provided life-saving water and sanitation resources under these difficult conditions for the approximately 400,000 people that benefited from WASH services137. The programme that UNICEF developed as its WASH response to the crisis was to become one of the largest programmes in UNICEF Jordan’s portfolio. The scope of the programme was initially a mandate to provide services to refugees in camps. It quickly expanded to include services for the water-deprived Jordanian host communities. These communities were resource constrained before the crisis and were further burdened by the crisis. The WASH programme’s scope also grew to include providing urgent WASH services for refugees living in the border settlements at Hadalat and Rukban. The emergency response required quick and flexible decision making and efficient utilization of scarce resources, which UNICEF successfully performed. The quick acceptance of responsibility in July 2012 and rapid action that followed led to several missteps. The evaluation team could not find evidence that UNICEF WASH staff utilized UNICEF’s internal resources that provide guidance in managing WASH during the first phase of an emergency response. For example, UNICEF has led the Global WASH Cluster for over a decade. In 2009 the organization published a handbook that described how

136 Estimate based on a review of PCA’s during the life of the WaSH Programme 137 Evaluation Terms of Reference, page 4.

to WASH services in an emergency like the Syrian Refugee Crisis (redrUK, 2009). Utilization of the handbook might have assisted the WASH programme team in contributing to the decision regarding the location of Za’atari camp, assessing refugee needs and preferences, conducting ongoing monitoring, and allocating responsibilities to various partners were made. The UNICEF WASH programme was a conscientious steward of its donors’ contributions. In the emergency response phase of the programme, UNICEF staff worked under difficult conditions to bring WASH services to refugees on short notice. After the situation stabilized, UNICEF was innovative in its analysis of WASH operations. This analysis led to consistent reductions in operating costs, which allowed financial resources to serve many more people at a higher standard than was otherwise possible. Examples include committing to build new infrastructure that could be operated at a lower cost than emergency interventions, and the “waste taxi” predictive model, which allowed WASH managers to anticipate desludging needs and overflowing tanks. Regarding hygiene, the evaluation notes an absence of reference to key infectious diseases such as diarrhea and respiratory infections in the WASH programme’s documentation, with the exception of the Hepatitis A outbreak in Azraq and the background information provided in ACF’s PCAs. A notable feature of Za’atari and Azraq’s management was the strong commitment to regularly implementing KAP surveys in both camps on an annual basis, enabling the measurement of change over time. However, the evaluation team did not find hygiene data with respect to host communities or those in the ITSs. The information that the evaluation team collected through interviews and focus groups on hygiene related items indicates that people received the items they needed. Sometimes, however, quantities of certain items like soap powder were insufficient and at times arrived in irregular intervals. Hygiene was communicated in a number of different ways including through community level hygiene promoters, children’s hygiene clubs, campaign days linked to key global days such as global handwashing day and use of the mass media. The evaluation found that consultation regarding preferences for WASH service options was not always possible, and that meeting preferences was also not always possible. Nonetheless, the WASH programme’s interventions remained highly relevant. The WASH programme set standards early and used assessments to identify the best way to address needs given available options and resource constraints. When the situation stabilized, the WASH programme consulted more closely with camp residents to discover their preferences and identify ways that operations could improve. The positive press that Za’atari has received, (Jaafar, 2017) and Azraq camp residents’ reports of consistent improvement in services speaks to the relevancy and effectiveness of the WASH programme’s main operations. UNICEF’s work in host communities has reduced tensions and brought relief to hundreds of thousands of people. Its work in host communities was well coordinated and of high quality. One improvement UNICEF could make is rethinking its approach to serving host communities and ITSs. In each year of the project, UNICEF estimated the number of people they would need to serve to effectively achieve the WASH programme’s goal in host communities, and every year the programme fell well short of the target. The reasons for falling short certainly include lack of funding, as UNICEF stated in its annual reports. This is certainly true as the international community has underfunded the humanitarian response regionally, and much of UNICEF’s funding was earmarked for provision of services in camps and settlements. However, missing the target several years in a row should have triggered a strategy review and a reorienting of this programme component. In fact, though it

falls outside of the scope of this evaluation, senior staff interviews indicated that this strategic reorientation may come in 2019, as UNICEF applies its innovative capacity and technical knowledge to assist other donors that are more capable of funding large, municipal systems. Serving refugees in ITSs should be an immediate concern for UNICEF. While in the early days of the response, the most vulnerable people were in the camps, by December 2013 it was clear that people living in ITSs had dire needs that were going unaddressed (REACH, 2013). Despite the vulnerability of people, especially children, living in ITSs, they have remained remarkably underserved. UNICEF reports that it has worked with partners to raise resources for ITSs and Schools. The evaluation team could find no recent information documenting the number, locations, or status of ITSs in Jordan. The ITSs that the evaluation team visited, which UNICEF serves, suffered from unsanitary and dangerous conditions, such as unsatisfactorily covered latrine pits138. One can imagine that ITSs not benefitting from UNICEF’s services are far worse. Equity of coverage remains a challenging issue that the WASH programme consistently seeks to address. The programme struggled early on with equity as public facilities in Za’atari camp were difficult for women and children to securely access, and water trucking and sanitation services were corruptible leading to some households receiving more water and better desludging service than others. Building the water and wastewater networks in the camps, and their management with innovative apps and third-party monitoring, solved the initial problems, and improved the situation greatly. However, other new issues arose, such as households in Za’atari receiving equal quantities of water regardless of the number of household members. The water network in Za’atari was also misused as some households rigged their water tanks to receive more of their share of water. UNICEF remains vigilant in the camps and will continue to work to address these challenging issues. Equity of coverage for women and children in the camps is also challenging. UNICEF manages this challenge through ensuring that partners that provide camp services have sufficient numbers of female hygiene promoters and social mobilizers on staff. Partners also come up with innovative, woman centered mechanisms for programme management, such as the Lead Mother model implemented by ACF in Azraq. UNICEF’s network expansion to address graywater was also implemented to address inequity. The WASH programme’s work at the national level has greatly assisted the government in forming and implementing policy. National level capacity building has also assisted the government in using data more effectively. The GoJ values UNICEF’s partnership and sector coordination. UNICEF could further enhance its role in consulting with the government. Areas of focus could include reducing NRW, advocating for an improved legal framework, improving the efficiency of operations, and assisting with strategy implementation. The evaluation’s interviews with senior UNICEF staff indicate that UNICEF may already have taken steps in that direction. The WASH programme completed a comprehensive nationwide assessment of WASH in schools and created a plan for bringing those schools up to standard. However, achievement against that plan was limited, and strong pilot projects carried out were not followed up on. The limited achievement in the WIS area, as in host communities in general, reflects a lack of strategic consideration when committing to objectives and setting targets.

138 See figure 9.

UNICEF’s stakeholders agree that that it has done a solid job in coordinating the sector. UNICEF has established systems for avoiding duplications or service gaps in the camps. The government appreciated UNICEF’s leadership in contributing to sector efficiency and making it more data driven. Partner organizations that the evaluation team interviewed said that UNICEF was always available to trouble shoot issues or provide useful technical input. The WASH programme should improve its strategic management. Strategic management includes the setting of objectives based on the consideration of the specific needs of a well- defined population compared to an organization’s internal assets, internal shortcomings, external opportunities, and external challenges. Strategic management during the course of the evaluation period suffered because the programme’s rapid start-up meant that the WASH programme team forwent standard assessments, strategic analysis, and programme design activities. Still, the evaluation team was surprised that over the course of the programme, monitoring and evaluation protocols and a monitoring data system were not established. The WASH programme team noted that they do not document their activities adequately139. Included in strategic management is the implementation a standard monitoring and evaluation system, which would track results at all levels. Implementing an adequately managed monitoring system is especially important given the regular, planned turnover among UNICEF staff. Good record keeping prevents important knowledge from disappearing with staff as they move to new assignments. It also allows for catching trends in data that may not otherwise be apparent. Also importantly, UNICEF’s WASH programme lacks an activity based accounting system, which would allow it to track spending by year and programme component. A review of the proposals and reports that UNICEF’s NGO partners demonstrate the standards required to monitor and financially account for programmatic activities. Improving accounting standards will increase in importance as UNICEF moves away from working through partner organizations to directly managing contractors. Contractors will have a greater profit incentive than NGOs and may recommend replacing equipment more frequently than is necessary or other such profit generating activities. UNICEF will have to track depreciation of assets, closely monitor costs, and have the ability to assign indirect costs to programme activities to properly manage resources. UNICEF has done everything possible to ensure that its interventions are long-lasting, and that mechanisms are in place for their operation. Examples include organizing WASH committees to oversee water access and hygiene promotion, constructing well designed systems out of durable materials which meet the national standards, and implementing messaging campaigns for hygiene promotion and water conservation. However, the ultimate sustainability of the programme is a complex issue. It includes determining if and when UNICEF will hand WASH management off to another entity, managing the transition of refugees either back to their place of origin or a permanent settlement elsewhere, the dismantling or integrating camp WASH infrastructure into Jordan’s municipal systems, and dealing with Jordan’s medium-term water scarcity issues. Addressing these sustainability issues will require deeper engagement from multilateral organizations and the agreement of the Government of Jordan. UNICEF’s WASH programme accomplishments are impressive and future challenges complex. The programme’s experience offers much that future programme’s could learn

139 Source: WaSH programme staff interview – 10 October 2017

from. This report presents these lessons and recommendations for future programmes in the following section.

Lessons learned and Recommendations This section presents lessons learned, followed by recommendations. At the end of the section we include further recommendations based on the evaluation’s findings. Lessons Lesson 1. Quick decision making, and decisive action saves lives. The decisions and actions UNICEF took in the early days of the programme were crucial for initial inhabitants of Za’atari. In the settlements, UNICEF provided water and sanitation services to respond to the settlements’ exigencies. UNICEF moved quickly and effectively, with clear goals, such as ensuring that each person had at least 15 liters of water per day. The WASH programme also recognized that its intervention would have an impact on the larger Ruwaished host community. To address that impact, UNICEF rehabilitated the community’s water and sanitation facilities and provided support to identified vulnerable households. Lesson 2. Installing quality water treatment and delivery equipment is crucial for the long-term sustainability and use. UNICEF’s installed higher cost and higher quality water infrastructure at Za’atari, Rukban and Azraq, including the treatment plant for Zaatari. In each case, a robust design was followed by very good installation and excellent post construction operation and maintenance. This was borne out through the evaluation team’s on-site investigations. In a KII with the contractor in Rukban, the operators noted that UNICEF always purchases high quality equipment. The result is operational efficiency, equipment that is long-lasting in the harsh environments, and reduced maintenance costs, downtime, and interruptions to supply. Lesson 3. Innovative approaches to inefficiencies are scalable: Initially wastewater management was hugely inefficient. For example, some contractors claimed to remove up to 40 percent more sludge than they actually removed. This stemmed from issues such as carrying waste in to the camp from outside, which was subsequently charged to the UNICEF operation. This inefficiency was fixed through the introduction of a voucher system where communities in receipt of services had to sign for them. Another innovative approach was the development of a third-party monitoring system, which paid off in improved service, greater coverage and increased efficiency. Lesson 4. When direct consultation is not possible, other means of assessment can ensure programme relevance. In addition to direct consultation with beneficiaries, UNICEF put a system of assessments and KAP surveys in place. These studies kept UNICEF staff and sector participants informed of changes in the camps and other locations and assisted the programme in staying relevant and responding quickly to eventualities. Lesson 5. The division of labor system, which UNHCR determined and UNICEF followed, in the camps worked well to determine roles, prevent duplication of services, ensure that standards were set and enforced, and protect against gaps in coverage. The system also assisted in understanding cost drivers and making improvements to efficiency. Lesson 6. Setting strict standards for private sector engagement and holding contractors to those standards improves efficiency. Emergency situations require engagement with the private sector, but also present opportunities for gaming the system. Evidence of this dilemma in the Jordan WASH programme includes some contractors water delivery and desludging services giving preference to some people’s needs over others and claiming more

volume was moved than was accurate. It may also have occurred in ways that weren’t spotted. Lesson 7. Documents, tools, and methodologies are available to quickly assist UNICEF staff in designing and managing WASH programmes in emergency contexts. There are many lessons learned from the early days of the project regarding issues such as camp site selection, means of conducting assessments, and methods of determining beneficiaries’ preferences. Most of these have been recorded and addressed in UNICEF guidebooks and lessons learned documents. Many of these predate the WASH programme's initiation. Lesson 8. A complaint hotline should not be run by the same organization that is responsible for the delivery of services. Early in the programme, ACTED managed service delivery and the complaint hotline. This arrangement was problematic. Complaints were better addressed when delivered to the implementing partner via UNICEF’s third-party complaint response mechanism. Lesson 9. The programme lacks analytical rigor for determining the programme’s objectives, indicators, or resource allocation, which has led it to overcommit to initiatives for which it did not have resources or capacity, primarily in host communities. Examples include the programme’s overestimation of the degree to which it could serve host communities, and its funding of successful WIS pilots that did not receive follow up or resources to scale. The WASH programme has operated without strategic planning, and operating theory of change, or specific, measurable, timebound indicators. The lack of strategic programme design protocols may have prevented the programme from leading strategically more appropriate initiatives. These initiatives may have involved promoting wide scale NRW reduction, rainwater water harvesting, water conservation, and groundwater recharge programs as real strategic interventions that would yield long standing results. Lesson 10. One of the greatest challenges faced by the programme was managing the expectations of refugees and host community members. Refugees came from a relatively water sufficient region that valued privacy and cleanliness. Host community members lived in a water scare region and were concerned about newly arriving refugees depleting their water supply, as well as receiving a free and higher standard of service than locals were receiving. Creating an information campaign to keep both of these groups aware of the work that UNICEF was doing, where sacrifices were required, and what they could expect in the future might have prevented unrest and other difficulties Lesson 11. The clearest path to sustainability for a programme like the UNICEF Jordan WASH programme is to transfer activities to the government. However, in the Syria Refugee Crisis, handing the programme to the government of Jordan was never an option. In taking lead in the sector, UNICEF has indefinitely taken ownership over the equivalent of a municipal utility in the form of water and sanitation service provision to Za’atari and Azraq. Lesson 12. When it initially established protocols with the GoJ, particularly the MoWI, UNICEF presented a budget and estimation of contributions to infrastructure for which it hadn’t yet secured funding. The consequence was a misunderstanding with the government that led to UNICEF not covering project costs because it didn’t meet fundraising targets, and the GoJ taking infrastructure projects out of its portfolio because it thought they were covered. Lesson 13. International NGOs have different risk factors, management structures, and commitments than UNICEF and other UN agencies. UNICEF often asked NGOs to make rapid changes, perform out of scope work, or quickly transition in and out of camps. These requests often felt like last minute decisions to UNICEF’s partners, and didn’t allow them to prepare adequately in terms of notifying staff, changing commitments, or transitioning assets.

Lesson 14. The emergency response phase of the programme required quick decision making and action. As the report mentioned, UNICEF staff said that there was five days between UNICEF taking WASH sector lead and the opening of Za’atari. As a result, UNICEF staff were not aware of UNICEF’s institutional resources that would have helped response staff plan and manage the organization’s activities in the early days of the programme. Lesson 15. Decisions in implementing WASH programmes require trade-offs. For example, deciding to locate a camp close to an urban population means easier access to municipal infrastructure, but more tension between the camp and community; whereas locating a camp in a distant location means less civil tension, but more expensive services like desludging. Camp residents and community members see the effect of these trade-offs and speculate as to why they were made. Lesson 16. Beginning in 2012, UNICEF clearly desired to implement a comprehensive programme that responded to all refugees needs. As the programme evolved, it has provided excellent infrastructure and services for people in camps, and struggled to serve refugees outside of the camps. Thus, commitments to the GoJ and host community and WIS fell short. This struggle is partially due to a very difficult funding environment, but also a result of a missing strategic development process. Recommendations

Reference Lesson Recommendation

Lesson 6 - Private UNICEF should develop a system for contracting WASH services and sector engagement monitoring those services. The system should include regular report and contractors objective measures back to the contractor as a means of increasing efficiency.

Lesson 7 – UNICEF should have a global or regional WASH manager that is Utilizing responsible for ensuring that emergency responders have the guidance UNICEF’s tools and tools they need at the beginning of a response. These could include and methodologies tools like the WASH Cluster Coordination Handbook.

Lesson 8 – Hotline Ensure that future programmes set up a mechanism for beneficiaries to management report issues/complaints directly to UNICEF, not through an implementing partner. If UNICEF is the direct implementer, an accountability mechanism unconnected to the implementation of the programme should be established.

Lesson 9 – The WASH programme should implement and monitoring and Improving evaluation system and comply with the strategic programme analytical rigor development protocols and reviews that are detailed in the Jordan and planning Country Programme 2013-2017 document. The WASH programme should hold itself to the same standard it holds its partners. UNICEF’s PCA agreements require that partner organizations provide detailed analysis of the reasoning behind proposed interventions and directly

connect activities to that analysis. It also requires that partners set specific, carefully designed objectives that are strategically coherent. UNICEF’s partners put monitoring systems in place to track progress toward these indicators. Monitoring systems are also useful for making decisions during programme implementation, as they allow managers to detect initiatives that go astray.

Lesson 10 – Future WASH emergency response efforts would benefit by Managing understanding how refugees’ standards in camps differ from what stakeholder they're accustomed to. When camps are near urban populations, expectations programmes should seek to understand how host community members perceive WASH operations. Information campaigns and rallying resources to support both populations could head off future tensions.

Lesson 11 – UNICEF should decide if it has the capacity to serve as the manager of Sustainability WASH services to Za’atari and Azraq indefinitely, and if that is a proper role for the organization. If it determines that it is not a proper role, it must explore options for transfer. If it decides to stay in the role for as long as required, it should upgrade its management and accounting systems appropriately.

Lesson 12 – In the future, UNICEF should make sure that it communicates clearly Managing with government partners the likelihood of protocol commitments government occurring. This issue is also tied to shortfalls in strategy and planning. expectations Lesson 13 – UNICEF should seek to better understand their partners business Managing NGO models and capacities. Forming an understanding would smooth partners management of projects implemented through these partners. Lesson 14 – UNICEF should pre-load a computer with the software and manuals Managing that emergency response staff need in the early days of a project. The institutional preloaded information should include up-to-date WASH emergency resources response manuals or tools, a pre-developed results indicator framework and logic model, and other strategic tools that will assist in decision making and goal setting. It should also include management tools. Such as a pre-set up version of QuickBooks or similar software with classes and accounts need to track expenditures, and any contract templates, human resources documents, financial information, or other required management tools. If UNICEF has these resources already available, a system for ensuring that deployed staff have downloaded the materials would be helpful. The staff should also have a conversation with someone in the emergency so that they are prepared, and that there is HQ follow up support for staff once deployed staff arrive at their post.

Lesson 15 – Clear communication about why one decision was made over another Negotiating trade- would help diffuse tensions in camps and communities. Where offs among possible, UNICEF should seek to explain why decisions were made, beneficiaries such as the decision to put public latrines in Azraq, but private latrines in Za’atari, so that people feel informed and their expectations are managed.

Lesson 16 – UNICEF should formulate a realistic strategy for the services it can Achieving some provide given the needs of the population it serves and the resources it programme targets has access to. An important part of any organization’s strategy is while missing others. knowing what it cannot provide. Once that is identified, UNICEF could serve in an important leadership role to assist the international community in rallying resources and capacity to address the uncovered and vulnerable populations.

Other Recommendations • UNICEF should address the disparity in water allocation in Za’atari. The water network in Za’atari provides the same quantity of water to each household regardless of the number of people in the household. Households can have from one person to more than eight people. If UNICEF strives for 35 liters per person per day on average, the wide variation in household size means that some people are getting very little, while others receive much more than they are entitled. • The trend toward cutting funds for hygiene promotion is disconcerting. UNICEF should avoid cutting those funds if possible, or if they must, plan for the contingencies that could result. • Partner interviews revealed that UNICEF’s role in the Wash in Schools initiatives may not be clear. UNICEF should clarify its role in this category of activity and coordinate a stakeholder strategy.

Works Cited 1. ACF. (2017). Azraq Camp Knowledge Attitudes and Practices Survey Report 2017. Amman: ACF. 2. (n.d.). ACTED Programme Cooperative Agreement 23-13. 3. ACTED, World Vision, ACT, RI, and Save the Children. (2016). Azraq Camp Knowledge Attitudes and Practices Survey Report 2016. 4. Al Wazani, K. W. (2014). The Socio-Economic Implications of Syrian Refugees on Jordan. Amman: Konrad Adenauer Stiftung. 5. Bocco, R. (2010). UNRWA And The Palestinian Refugees: A History Within A History. Oxford Journals. 6. Financial Tracking Service. (2018, October). Appeals and response plans 2018. Retrieved from https://fts.unocha.org/appeals/overview/2018/sectors?f%5B0%5D=destinationPlanIdName%3 A%22629%3ASyria%20Humanitarian%20Response%20Plan%202018%22&f%5B1%5D=de stinationPlanIdName%3A%22658%3ASyria%20regional%20refugee%20and%20resilience% 20plan%20%283RP%29%2020 7. Financial Tracking Service. (2018, October). Financial Tracking Service. Retrieved from Syrian Arab Republic - Civil Unrest (from 2012) 2018: https://fts.unocha.org/emergencies/600/summary/2018 8. Freeman, M., & al, S. M. (2014). Hygiene and health: Systematic review of handwashing practices worldwide and update of health effects. 9. Hūls, V. (2017). UNICEF WASH Action in Humanitarian Situations; Synthesis of Evaluations 201-2016. New York: UNICEF. 10. International Monetary Fund. (2017). Jordan, Selected Issues. Washington, DC: The International Monetary Fund. 11. Jaafar, S. (2017, April 5). This refugee camp in Jordan has turned into a frontier town for Syrians escaping war. Retrieved from PRI: https://www.pri.org/stories/2017-04-05/refugee- camp-jordan-has-turned-frontier-town-syrians-escaping-war 12. (n.d.). JEN Programme Cooperative Agreement 25-14. 13. Kis, A. e. (2016). Water demand management in the context of water services: Jordan. Regional Centre for Energy Policy Research. 14. Ledwith, A. (2014). Zaatari: The instant City. Boston: The Afforable Housing Institute. 15. Leestma, D. (2017, October 1). Draining a lake in Daraa: How years of war caused Muzayrib to dry up. Retrieved from Middle East Eye: https://www.middleeasteye.net/news/draining-lake-daraa-how-years-war-caused-lake- muzayrib-largely-disappear-1202640340 16. Luck, T. (2013, April 21). In Jordan, tensions rise between Syrian refugees and host community. The Washington Post. 17. Middle East and North Africa Out-Of-School Children Initiative. (2014). Jordan Country Report on Out-Of-School Children. UNICEF. 18. Ministry of Planning and International Cooperation. (2014). Jordan Response Plan 2015 for The Syria Crisis. Amman: The Hashemite Kingdom of Jordan.

19. Ministry of Water and Irrigation. (2016). National Water Strategy 2016 - 2025. Amman: Hashemite Kingdom of Jordan. 20. OXFAM PCA 32-13 Executive Summary. (n.d.). 21. REACH. (2013). Informal Tented Settlements in Jordan: A Multi-Sector, Baseline Assessment". UNICEF. 22. REACH. (2014). Access to Water and Tensions in Jordanian Communities Hosting Syrian REfugees. Amman: REACH. 23. REACH. (2014). Syrian Refugees Staying in Informal Tented Settlements in Jordan. Amman: REACh. 24. redrUK. (2009, January). Water, Sanitation, and Hygiene (WASH) Cluster Coordination handbook. 25. ReliefWeb. (2017). Humanitarian Funding Update; December 2017. ReliefWeb. 26. Salman, M., & Mualla, W. (2008). Water demand management in Syria: centralized and decentralized views. Water Policy, 549-562. 27. Seely, N. (2012, March 1). Jordan’s “open door” policy for Syrian refugees. Foreign Policy. 28. Seely, N. (2012, October 1). Most Jordanians say no to more Syrian refugees. The Christian Science Monitor. 29. Terms of Reference for Service Contracting - Rukban. (n.d.). https://www.unicef.org/jordan/RUKBAN_ToR.pdf. 30. The Sphere Project. (2018). Water supply standard 1: Access and water quantity. Retrieved from Sphere Handbook: http://www.spherehandbook.org/en/water-supply-standard-1-access- and-water-quantity/ 31. The United Nations. (2013). Syria Regional Response Plan; January to December 2013. The United Nations. 32. UNHCR. (2012, August 28). Syria: Doubling of refugees fleeing to Jordan. Retrieved from UNHCR USA: http://www.unhcr.org/en-us/news/briefing/2012/8/503ca1c99/syria-doubling- refugees-fleeing-jordan.html 33. UNICEF. (2012, October). In Jordan, huge water delivery and testing operation meets the life-saving water and sanitation needs of Syrian refugees. Retrieved from UNICEF Jordan: https://www.unicef.org/wash/jordan_66157.html 34. UNICEF. (December 2012). Syria Crisis Bi-Weekly Humanitarian Situation Report. UNICEF. 35. UNICEF Jordan. (2012). Jordan Country programme document; 2013 - 2017, pg 11. Amman: UNICEF JOrdan. 36. UNICEF Jordan. (2012). SitRep 20 December 2012. Amman: UNICEF. 37. UNICEF Jordan. (2012). Syrian Refugee Response Overview. Amman : UNICEF. 38. UNICEF Jordan. (2013). SitRep 25 January 2013. Amman: UNICEF. 39. UNICEF Jordan. (2013). UNICEF Annual Report 2013 - Jordan. Amman: UNICEF. 40. UNICEF Jordan. (2016). UNICEF Jordan Annual Report 2016. Amman: UNICEF. 41. UNICEF Jordan. (2017). UNICEF Jordan Annual Report 2017. Amman: UNICEF Jordan. 42. UNICEF, ACF. (2017). Azraq Camp Knowledge and Attitudes and Practicies Survey Report 2017. Amman.

43. WASH Sector Working Group. (2012). Water, Sanitation and Hygiene (WASH) Sector Working Group Strategy for the Syrian Refugees Response in Jordan; December 2012 - June 2013. Amman. 44. Wildman, T. (2013). Water Market System in Balqa, Zarqa, & Informal Settlements of Amman & the Jordan Valley - Jordan August - September 2013. Oxfam. 45. Wilkes, S. (2012). Jordan opens new camp for Syrian refugees amid funding gaps. Retrieved from UNHCR: Jordan opens new camp for Syrian refugees amid funding gaps 46. World Vision Programme Cooperation Agreement 19-13 . (n.d.). 47. Za'atari Hygiene Promotion Working Grouip. (2015). Knowledge Attitudes and Practices Survey; Za'atari Refugee Camp 2015. Amman. 48. Za'atari Hygiene Promotion Working Grouip. (2016). Knowledge, Attitudes and Practices Survey Za’atari Refugee Camp 2015. Amman. 49. Za'atari Hygiene Promotion Working Groups. (n.d.). 2012 Baseline KAP Survey, Za'atari.

Annex A: Evaluation Terms of Reference STATEMENT OF WORK/TOR Terms of Reference for the Evaluation of UNICEF#s response to the Water, Sanitation and Hygiene needs in Jordan as a result of the Syrian refugee crisis (July 2012 to July 2017)

Table of Contents 1. Background and context 1 2. Evaluation purpose and objective 2 3. Evaluation scope 3 3.1 Coverage and level of results 4 3.2 Geographical coverage 4 3.3 Time-period of the evaluation 4 4. Key evaluation questions 4 4.1 Relevance 4 4.2 Effectiveness 5 4.3 Efficiency 5 4.4 Sustainability 5 4.5 Coverage 6 4.6 Coordination 6 5. Evaluation Stakeholders 6 6. Methodology 6 6.1 Desk Review 7 6.2 Meetings with WASH section 7 6.3 Visits 7 6.4 Key informant interviews (Kll) 8 6.5 Focus Group Discussions (FGD) 8 6.6 Participation in at least 2 coordination meetings in Za#atari, Azraq and the border, and the sector 8 7. Work assignment 8 7.1 Phase 1. Desk review 8 7.2 Phase 2. Prepare an inception report with project delivery plan 8 7.3 Phase 3. Data Collection 9 7.4 Phase 4. Draft evaluation report 9 7.5 Phase 5. Finalize the Final Evaluation Report 9 7.6 Phase 6. Prepare and present two PowerPoint presentations 9 8. Management of the Evaluation 9 9. Expected deliverables & delivery dates 10 10. Official travel involved 10 11. Desired qualifications 10 11.1 Team leader 11 11.2 Two WASH Experts 11 12. Estimated duration of the contract and payment schedules 11 13. Roles and responsibilities in the evaluation process 11 13.1 The Evaluation Manager 11 13.2 Evaluation Reference Group (ERG) 12 13.3 The Evaluation Team 12 14. Performance indicators for evaluation of results 12 15. Frequency of performance reviews 12 18. Equity, gender, human rights, including child rights 15 19. Ethical principles and premises of the evaluation 15 20. UNICEF recourse in case of unsatisfactory performance 15

1. Background and context

Jordan, is described as being an arid to semi-arid country with average annual rainfall of 200 mm/year over the country. On this basis, it is one of the most water scarce countries in the world, and has been facing chronic challenges to ensure the provision of clean water and sanitation services to its population for a number of years. The chronic water situation has been compounded by successive waves of refugees from neighbouring areas, the most recent being the Syrian crisis. Since the onset of the Syrian crisis in 2011, more than 659,000 Syrian refugees have crossed the border into Jordan to seek assistance and refuge from the atrocities underway in Syria. Since then, the population in some areas has risen dramatically, particularly those in the

north of the country (by more than 60% particularly in the Northern Governorates e g. Irbid). The 2015 Census reported that the total population of Jordan as 9.5 million, showing a considerable increase over the recent years, and 30.6% of the population is non-Jordanian, including Syrian refugees. While an estimated 15% of the registered Syrian refugees live in refugee camps, the remaining 85% are living in host communities, which has caused tension between Jordanians and Syrians, due to a perception that the Syrian refugees are absorbing the limited resources. This tension has resulted in conflicts and disputes, particularly in the northern governorates, where the additional pressure on resources has been most acutely felt.

According to the Joint Monitoring programme (JMP) 2015, access rates to a piped water network in urban areas is estimated at 93%, and 80% in rural areas. However, in reality, most areas have access to water once a week (in the main cities) reducing dramatically in rural areas, with some areas reported to receive water once a month. Due to the limited resources, and associated limited access, average household consumption is 71 l/p/d (National Water Strategy 2016 # 2025). As a result, most households have significant amounts of water storage facilities and when necessary, supplement these with private water tankering services. For wastewater systems, the situation is even more critical. Prior to the Syrian crisis, an estimated 62% of the population was connected to a sewerage system nationally, however this figure is much lower in rural areas. In areas where there has been a significant recent increase in the population, the existing infrastructure has been unable to support the increase in the volume of wastewater which has led to overflows in urban areas, causing significant environmental health concerns.

As a result of the influx, in addition to the growing expansion of the Jordanian economy, as well as natural population growth, the demand for water and wastewater services has increased significantly, which has contributed to over expansion of groundwater from aquifers. The Ministry of Water and Irrigation (MOWI) estimates that average rates of over-pumping exceed 200 Mm3/year resulting in seven of the twelve groundwater basins being over-pumped at between 135 to 225% of the estimated safe yields of the aquifers. The critical situation is further compounded by high levels of leakage, estimated at more than 60% in some of the northern governorates, in the existing systems as well as non-revenue water.

In July 2012, UNICEF commenced its Water, Sanitation and Hygiene (WASH) programme, initially focusing on refugee response and providing WASH services in Za#atari camp, and expanding to a total of six settlements (currently five are active) over time. The initial focus on refugee response, developed in line with the Syria Regional Response Plan (RRP), has gradually expanded to cover resilience in line with the National Resilience Plan which was developed in 2013 to cover the period of 2014-2016 . Also since 2012, UNICEF has led the WASH Sector for the humanitarian response. UNICEF continues to be a key player in providing services to refugees living in the four main camps of Za#atari, Azraq, King Abdullah Park and Cyber City; two settlements in Rukban and Hadalat; as well as in Informal Tented Settlements (ITSs) and host communities. Currently, the population of the five refugee camps/settlements is estimated to be about 170,000 people. It is understood that the camp population peaked to 120,000 in Za#atari camp in late 2013 and the population in Za#atari declined to approximately 80,000 thereafter, with Azraq opening in April 2014. The two settlements at the border, Rukban and Hadalat, expanded rapidly in 2016, to a current estimated combined population of 55,000. As the movement to/from the camps has been very fluid, there is no actual figure of the number of people who have benefitted from WASH services since 2012, however, this is estimated to be more than 400,000.

Although UNICEF has had an active presence in Jordan since 1952, UNICEF was not active in the WASH sector prior to the refugee crisis. As a result, the UNICEF Jordan Country programme of Cooperation for 2013-2017, when originally developed in 2012, did not include WASH programme. Seeing the rapid scaling-up of the refugee response in 2013, UNICEF Jordan revised its results framework for 2013-2017 to include an output-level result focusing on Core Commitments for Children in Humanitarian Action. This was further reviewed in 2014 to reflect resilience more prominently, at the time of the Strategic Moment of Review and Reflection and in light of the National Resilience Plan. The rapid onset and scale-up of the crisis has meant revisions in the UNICEF programme Results Framework.

UNICEF WASH programme has evolved to work at four different levels as follows: - Provision of water, sanitation and hygiene services, as well as the dissemination of key messages to refugees on a daily basis in four camps and two settlements; - Provision of support in the host communities (infrastructural support, support to

vulnerable households, informal tented settlements) including WASH in schools; - Support to the WASH sector through technical support to the Ministry of Water and Irrigation in the development of key planning documents (National Resilience Plan 2014, Jordan Response Plan 2015, 2016-2018, 2017-2019), strategies (National Water Strategy 2016 to 2025) and policies, as well as the nationwide WASH in Schools Assessment (2014/5) and the National WASH in School Standards; - Coordination of the humanitarian WASH sector at a camp and national level.

In terms of UNICEF Country programme in Jordan, the WASH programme has been one of the largest in terms of scale and scope, constituting about 53% (USD 78 million) of the total budget in 2013, 54% (USD 100 million) in 2014, 31% (USD 56 million) in 2015, 27% (USD 61 million) in 2016 and 30% (USD 60 million) in 2017. The costs and logistics required, as well as the monitoring associated with the provision of services in the camps, have been enormous. To address this, and to improve the consistency, reliability, quality and equity of services in the camps, UNICEF has been implementing a transitioning strategy through the construction and operation of five boreholes and two wastewater treatment plants, with water and wastewater networks currently under construction since 2015.

2. Evaluation purpose and objective

The purpose of the evaluation is to assess whether UNICEF WASH programme met its intended results, and to generate lessons learnt which can inform WASH programmes in similar context.

The overall objective is to conduct an independent and critical assessment to the WASH programme and its support to the sector and government institutions, using evaluation criteria of relevance, effectiveness, efficiency and sustainability as well as coverage and coordination in the target communities (camps, ITSs and host communities). The Impact criterion is excluded due to the relatively short time period since the start of the interventions. Additionally, the evaluation is anticipated to collect evidence and lessons learnt from

UNICEF#s WASH programme implementation to inform future programming, and to develop recommendations to further strengthen delivery and planning of WASH services in similar protracted humanitarian crisis in view of building resilience and sustainability. More specifically, this evaluation will aim for the following key specific objectives: a) To assess the relevance of the WASH programme (emergency response and resilience components) in the context of Syrian Crisis in Jordan and its national priorities. To assess the relevance of WASH programme design to achieve the expected results, taking into account the appropriateness of the interventions to targeted communities. b) To assess the effectiveness of the WASH programme and to measure to what extent the programme has achieved its set results in an equitable manner. c) To assess the efficiency of the WASH programme and to what extent the programme has used resources (human, financial and others) and coordinated in an efficient manner over time. To assess the cost of the response per unit of aggregation as compared to the cost being incurred by similar WASH programme implemented in other countries. d) To assess the sustainability of the WASH programme and its results, taking into account the institutionalization and capacity building of national counterparts as well as a potential exit strategy. e) To assess the coverage of the most vulnerable population by the WASH programme and provision of quality services in a coherent manner. f) To assess the coordination of UNICEF as a sector lead to avoid duplication and gap and enable effective partnerships to reach the most vulnerable population. g) To document lessons learnt and good practices that will inform future programming, replication in other countries in the region, suggesting different options for UNICEF and partner agencies.

It is anticipated that the evaluation, and the expected Lessons learned and good practices, will also benefit the Ministry of Water and Irrigation, implementing partners in addition to donors of Jordan, as well as future UNICEF WASH programming in the region and globally.

3. Evaluation scope

This evaluation will be summative in nature. It should focus mainly on UNICEF#s WASH programme as it evolved from refugee response to include resilience building from July 2012 to July 2017. This includes the initial response which aimed to meet the urgent needs of Syrian refugees, primarily residing in refugee camps, as well as resilience programming to support host

communities and government bodies to respond to the impact of the crisis. This evaluation will assess UNICEF#s WASH programme which includes water, sanitation (WASH facilities, solid waste management and desludging) and hygiene promotion implemented by UNICEF#s partners and contractors. While UNICEF started providing life-saving WASH services at refugee transit centres in March 2012, this evaluation will cover UNICEF#s WASH programme from the opening of the Za#atari refugee camp in July 2012, and will look at programming in the four camps of Za#atari, Azraq, Cyber City (currently closed) and King Abdallah Park, in addition to host communities (and informal tented settlements), as well as Rukban and Hadalat at the Syrian/Jordanian border. Although there are Syrian refugees registered in all 12 Governorates of Jordan, 90% of the refugees are registered in Zarqa, Irbid, Mafraq and Amman Governorates, with 99% of the refugees registered in ten Governorates.

In addition, this evaluation will assess support to the sector through technical support provided to the Government, and UNICEF role in leading the sector at the camps and at the national level.

Stakeholder participation is important for this evaluation, and their views and inputs should be reflected through means such as interviews and focus group discussions. Special emphasis will have to be made on Ministry of Water and Irrigation (MWI), Ministry of Education (MoE), UNHCR, WASH sector, local community leaders and International Partners (IPs) who were directly involved in the UNICEF WASH response since its early stages. In addition, the voices of caretakers, parents (both mothers and fathers) and children who benefited from UNICEF#s WASH intervention should be captured during stakeholders consultations. Representatives from the Ministry of Water and Irrigation will comprise a key component of the Evaluation Reference Group.

3.1 Coverage and level of results The evaluation will focus on four components of the UNICEF#s WASH programme in Jordan and will include: a) Provision of equitable WASH services and the dissemination of WASH messages in the four camps (Za#atari, Azraq, King Abdullah Park and Cyber City) and two settlements in Rukban and Hadalat, to the most vulnerable people namely children and women b) Provision of gender equitable support to the most vulnerable people in host communities (in all 12 Governorates) namely to children and women c) Support to the WASH sector through technical support to the Ministry of Water and Irrigation d) Coordination of the sector at camps and national level

3.2 Geographical coverage The geographical scope of the evaluation will cover the following: e) The four major camps of Za#atari, Azraq, King Abdullah Park and Cyber City; and two settlements of Rukban and Hadalat. In addition, the evaluation will cover a number of Informal Tented Settlements (ITSs) (in 2017, 111 ITSs are provided with WASH interventions) and will be based on discussions with the consultant; f) Host community interventions in all 12 Governorates. 3.3 Time-period of the evaluation The period July 2012 to July 2017 will be considered as the timeframe for this evaluation.

4. Key evaluation questions

The evaluation will apply Organization for Economic Cooperation and Development Assistance Committee (OECD-DAC) criteria of relevance, efficiency, effectiveness, and sustainability as well as relevant humanitarian criteria, taking into consideration the purpose, objectives and scope of the evaluation. The Evaluation team should answer the following key questions, but not limited to. They are requested to unpack them and should propose more detailed ones as part of the request for proposal and inception report. The humanitarian criteria of coverage and coordination are directly addressed and should be investigated throughout the evaluation process.

4.1 Relevance a) To what extent were affected people consulted on their preferences with regards to water and sanitation service options and level of services? Were interventions appropriate in terms of meeting their basic needs? Was there a feedback loop and monitoring system in place for reflecting the learning and evidence to improve programming, especially for the humanitarian response? b) To what extent did the interventions consider the needs and the priorities of the MWI and were they coherent with global references and the regional and national response to the Syria

Crisis? c) How responsive was the UNICEF WASH programme over time to changes in the external environment?

4.2 Effectiveness a) To what extent did UNICEF#s response achieve its intended outcomes (as stated in UNICEF#s Country programme Outcome and Outputs as given in Annex A)? To what extent did UNICEF achieve equity results, especially for children and women specific interventions? b) What were the strengths and weaknesses of the interventions in achieving results in the camps/settlements, ITSs and host communities on the following: - Water (Litre/day, supply, quality, consistency, reliability, accessibility, and equity)? - Sanitation (adequacy, appropriateness, accessibility)? - Hygiene promotion (appropriateness of the messages, timeliness, behaviour change)? - WASH in schools? - National level support?

4.3 Efficiency a) To what extent did the programme use available resources in an economic manner to provide WASH services and facilities in camps/settlements and host communities, to ensure child rights, equity and gender equality? b) To what extent were the costs to deliver water and wastewater services, as well as solid waste management, rationalized and optimized in the camps and settlements to ensure Value for Money? How do these compare to similar situations at the local or regional level? Could more cost effective operations/interventions have been undertaken at an earlier stage? c) To what extent did UNICEF try to minimize duplication/gap in WASH interventions in camps? And how early did UNICEF move to more cost-efficient interventions? d) To what extent were resources (financial and human resources) allocated for the range of intervention types, with particular attention paid to the respective allocations for humanitarian response and resilience, to achieve the intended results? e) To what extent did UNICEF advocate with the Ministries to mobilise their available resources to address the identified needs? And how efficient was UNICEF#s support to ministries (i.e. MWI and MOE) to reaching most vulnerable people? Did the WASH programme maximise the potential collaboration with other UNICEF programme sections and partners?

4.4 Sustainability a) Was the infrastructure in the camps constructed in such a manner to ensure long term functionality, and will continue to service the most vulnerable people even when UNICEF is no longer directly supporting? This should be investigated in terms of Water and Wastewater infrastructure. b) To what extent did UNICEF take sustainability into account at the early stage for water, wastewater and solid waste management services (e.g. transition to local government institutions and local capacity)? c) To what extent did UNICEF#s interventions increase the resilience of the Government and respective target populations? To what extent will UNICEF-supported projects (schools, ITSs, water and wastewater infrastructure, mobilisation) continue to be operational after the end of UNICEF#s financial support d) Does the MWI have the necessary mechanisms in place and capacity to ensure the water and sanitation needs of the most vulnerable people will be met as a result of UNICEF#s support and advocacy, if another large scale humanitarian crisis occurs in the future? e) To what extent did UNICEF document its WASH interventions and make available to the sector learning on water, wastewater, solid waste management and mobilisation? f) To what extent were environmental protection and sustainability taken into account in the design and implementation of programmatic interventions?

4.5 Coverage a) Did UNICEF#s WASH response identify and reach the most vulnerable people? To what extent did specific interventions strike an equitable balance between interventions in camps/settlements and in host communities, to reach the most vulnerable people? b) Did UNICEF as Sector Lead ensure that the needs of those in most need were addressed in a coherent manner, particularly in comparison to alternative existing systems at the local or regional level (i.e. UNRWA camps)? If not, how?

4.6 Coordination a) To what extent did UNICEF role as WASH sector lead enhance response processes and results to avoid duplication and gap in interventions? b) Were partnerships effective and leveraged to the maximum extent to assist the most vulnerable population in camps/settlements, ITSs and in host communities?

5. Evaluation Stakeholders

The stakeholders that were found relevant to this evaluation by the WASH programme are listed in the below table 1. They were divided into 2 categories in terms of interest and influence.

6. Methodology The evaluation methodology will mainly utilise qualitative methods, with a combination of tools for data collection and analysis. The participatory approach is suggested to ensure participation of various social groups including the most marginalized. Quantitative analysis will be carried out through the desk review of the following major data sources, but not limited to: Camps: - Monthly, quarterly and final reports from partners - Third party monitoring data (UNOPS) on beneficiary interviews, wastewater volumes, water delivery and water quality - Knowledge Attitude and Practice surveys (Za#atari 2013 & 2014/2015; and Azraq 2016 & 2017) - Comprehensive Child Focussed Assessment (Za#atari 2015; Azraq 2015 and 2017) - Partnership agreements and budgets - Field visit reports - Monthly Donor updates and Situation Reports Table 1. Potential stakeholders Interest Influence - MWI - MoE - Donors - UNICEF - UNHCR - Sector - NGOs - Women - Children (girls and boys) - Local Communities - WASH Committees - Schools (including parents, teachers and students) - Assessment reports - Needs assessments -Transitioning Strategies - Contracts - Terms of Reference for the Groundwater Assessment for Za#atari and Azraq - Water level and quality monitoring data

Settlements (border): - Monthly Border updates and Situation Reports - Partnership agreements - Partner reports - Contracts - Third party monitoring data - Security incident reports - Distribution data - Field visit reports

Host Communities: - Nationwide WASH in Schools assessment - Partnership agreements

- Partner reports - Field visit reports - Contracts - Situation of Syrian children in host communities in Jordan (Frequent monitoring)

ITS: - Partnership agreements - Partner reports - Field visit reports -Transitioning Strategy

National-level support: - Vulnerability assessments for water and wastewater (2014/2015; and 2016) - Jordan Response Plan (2014; 2015; 2016/2017; 2017/2019) - Regional Response Plans - National Water Strategy (2016 # 2025)

The evaluation team is expected to propose a relevant set of tools and methods, as appropriate to answer the evaluation questions. This should be detailed in the form of an evaluation matrix, showing methods against proposed evaluation questions. Methodologies proposed by the consultant will be subject to consultation with the UNICEF WASH and Planning Monitoring and Evaluation (PME) sections, and WASH Working Group as appropriate. As part of methodology, it is essential to articulate how ethical considerations will be taken into account in the design of the evaluation and throughout the evaluation process. Methodology should include, but not limited to the following:

6.1 Desk Review Undertake a Desk Review: review of similar evaluations such as the recent Regional assessment of UNICEF#s response, review of WASH Annual Reports, partner programme Cooperation Agreements and programme Documents, contracts, funding tables, KAP surveys, third party reports, UNOPS and partner reports, minutes and resources of Sector meetings, consultation of UNHCR information platforms, review of Outputs (Water Strategy, vulnerability maps) and other documents relevant to the evaluation.

6.2 Meetings with WASH section At the start of the evaluation, there will be a telephone call for a briefing with key WASH and PME sections personnel for an overview of the activities undertaken and the timeline of events. Arrangements for additional meetings with the WASH section will be arranged as needed, throughout the evaluation process.

6.3 Visits Conduct visits to some of the main interventions in the camps and host communities: a) Visits to the six camps/settlements of Za#atari, Azraq, King Abdullah Park, Cyber City, Rukban and Hadalat; b) Visits to the host communities.

6.4 Key informant interviews (Kll) Conduct interviews with relevant stakeholders and beneficiaries as follow: a) Duty bearers: This would include WASH section personnel, UNICEF Management, UNICEF WASH Regional team for the Middle East and North Africa Regional Office, Ministries, NGO partners (main partners in camps, WASH in Schools and ITSs) and UNHCR representatives. b) Right holders: Camps, settlements, host community and ITS representatives

6.5 Focus Group Discussions (FGD

Conduct Focus Group Discussions with: Beneficiaries in the five major active camps of Za#atari, Azraq, King Abdullah Park, Rukban and Hadalat and the host communities, ITSs and schools. The FGDs should be disaggregated by age and gender (women, men, children and youth);

6.6 Participation in at least 2 coordination meetings in Za#atari, Azraq and the border, and the sector

7. Work assignment

The key tasks to be undertaken under this evaluation are as follows (see summary in Table 2):

Table 2: Key tasks of the evaluation Phase Activity Phase 1 Desk review Phase 2 Prepare an Inception Report Phase 3 Data collection

Phase 4 Draft evaluation report Phase 5 Final Evaluation Report Phase 6 PowerPoint presentations

7.1 Phase 1. Desk review Undertake a desk review of all relevant documents (e.g. the recent Regional assessment of UNICEF#s response), review of WASH Annual Reports, programme Cooperation Agreements, funding tables, consultation of UNHCR information platforms, review of Outputs (Draft Water Strategy) etc.

7.2 Phase 2. Prepare an inception report with project delivery plan The Inception Report should be developed in line with UNICEF global standards and should include: (1) purpose and scope, (2) evaluation criteria and questions, (3) evaluation methodology, (4) evaluation matrix, (5) a detailed evaluation delivery plan within the designated timeframe (seven months), and (6) outline of the final report. Methodology should also include the evaluation approach, and theory of change. The outline of the evaluation report should be in line with the UNICEF-Adapted UNEG Evaluation Reports Standards, and UNICEF Global Evaluation Report Oversight System. The inception report will be used as an initial point of agreement and understanding between the evaluation team and the evaluation managers and should be approved by the evaluation Reference Group. Field visits and data collection can#t start before finalization of the inception report.

7.3 Phase 3. Data Collection Conduct the necessary interviews, field visits, consultations as agreed in the Inception Report, which are expected to include: a) Meetings with WASH section; b) Conduct Field Visits to the six camps/settlements, randomly selected schools and ITSs, in addition to host communities; c) Kll with current and historic duty bearers (including ex-UNICEF and partner staff); d) Kll with right holders; e) FGD with rights holders and WASH partners; f) Attend Coordination meetings in the camps.

7.4 Phase 4. Draft evaluation report Perform the analysis and draft the evaluation report. It must be compliant with UNICEF-adapted UNEG evaluation report standard (http://www.uneval.org/). The Evaluation report should systematically answer the key evaluation questions included in the objectives sections of this TOR. It should fairly and clearly represent the views of the different actors/stakeholders. It should clearly give the findings, conclusions and recommendations in a way that is substantiated by evidence. All recommendations included in UNICEF#s

evaluation require management response. It is pertinent that all recommendations are clear and specific to what UNICEF Jordan could do or influence. Validation process of the inception report will be as follow:

a) Disseminate the Evaluation Report to the WASH and Planning Monitoring and Evaluation (PME) Sections; b) Collate feedback and integrate comments where agreed;

c) Finalize the draft evaluation report and submit to WASH and Planning Monitoring and

Evaluation (PME) Sections.

7.5 Phase 5. Finalize the Final Evaluation Report After review of the draft report, the

consultant will: a) Collate final feedback made on the draft report and integrate all comments; b) Finalize the final evaluation report and submit to WASH and Planning Monitoring and Evaluation (PME) Sections. The final report will be in English and Arabic of 60 to 80 pages in length (excluding annexes). It will also contain an executive summary of no more than 4 pages that includes a brief description of the programme, its context and current situation, the purpose of the evaluation, its methodology and its major findings classified as per the evaluation objectives, conclusions and recommendations.

7.6 Phase 6. Prepare and present two PowerPoint presentations Prepare two PowerPoint presentations detailing the findings of the evaluation. They will be presented during: a) An open session for the Jordan Country Office; b) And a second one for key sector partners.

8. Management of the Evaluation

To ensure independence, this evaluation will be co-managed by WASH Specialist and Monitoring & Evaluation Officer at UNICEF Jordan Country Office. A reference group will be formed, engaging subject matter experts from inside and outside of UNICEF, including the UNICEF Regional Office for Middle East and North Africa. Members of the reference group will be engaged and consulted at key milestones of the evaluation process such as review of the TORs, inception report, and draft reports.

Selection of the evaluation team will be made through an open and competitive bidding process. Review of technical proposal will be done by at least three members.

WASH Specialist will be responsible for technical oversight in relation to the WASH programing, support to the evaluation process by providing necessary background information, data, contact information, as well as management of contractual issues once the bidding process completes. Monitoring & Evaluation Officer will oversee and quality assure the evaluation, ensuring that it complies with the UNICEF global standards and norms.

The evaluation team will report to WASH Specialist and Monitoring & Evaluation Officer at UNICEF Jordan Country Office who will serve as key contact points.

9. Expected deliverables & delivery dates The assessment team will be expected to submit the following deliverables as per the start date of 02.OCTOBER.2017 (table 3):

Table 3: Deliverables and timeframe # Deliverables Type and language Delivery Date 1 Inception Report with Project Delivery Plan Soft

(within 3 calendar weeks of start date, or 21 working days) 2 Draft Evaluation Report Soft copy (in English) 04.MARCH.2017 (within 5 calendar months of the start date, or 126 working days) 3 Final Evaluation Report accepted Soft copy (in English and Arabic) 02.MAY.2018 (within 7 calendar months of the start date, or 180 working days) 4 Two PowerPoint presentations developed and presented at two sessions Soft copy (in English and Arabic) 09.MAY.2018

10. Official travel involved Travel to Amman and field locations in Jordan should be anticipated as per the evaluation methodology and deliverables. All travel costs should be planned properly in financial proposal. UNICEF will not provide transport support or field travel arrangements, thus the estimated cost of travel should be included in the financial proposal. Please note that if selected, the contract can be a supporting document to obtain entry visa (if necessary). UNICEF will be unable to secure travel visas. Any applicable per diems should be included as part of the lump sum price proposal.

11. Desired qualifications Given the broad scope, it is expected that the evaluation will be carried out by an institution or a team of technical specialists, overseen by the Evaluation Team Leader. The evaluation team will consist of at least one international expert as the Evaluation Team Leader, as well as two WASH experts preferably with Arabic speaking skills. Any other capacities, as deemed required for the evaluation such as an expert on Value for Money should be proposed by the evaluation team. It is optional to include more than one expert to ensure that all aspects of water, sanitation and hygiene (WASH) are adequately covered.

11.1 Team leader

100

- Desirable 15 years of experience in conducting evaluation with proven experience in leading evaluations of similar scope, such as WASH evaluations - Demonstrated experience undertaking comprehensive programme evaluations in emergency contexts; - Understanding of Cluster/Sector Coordination. Experience in supporting or working with clusters and/or sectors an asset; - Experience in working in refugee/IDP camps; - Proven experience in projects related to gender equality and human rights, including child rights; - Proven experience in institutional and policy reviews in the field of water and sanitation; - Excellent writing skills; - Fluency in English; - Preferable to have experience working in the Middle East and North African region; - Knowledge of Arabic a considered advantage.

11.2 Two WASH Experts - At least five years of experience implementing water supply interventions in development or emergency contexts (at least one expert should have this level of experience); - At least five years of experience implementing sanitation interventions in development or emergency contexts (at least one expert should have this level of experience); - At least two years of experience implementing hygiene promotion/community mobilisation interventions in development or emergency contexts (at least one expert should have this level of experience);

- Experiences in evaluating WASH programmes an advantage; - Preferable to have experience working in the Middle East and North African region; - Fluency in Arabic would be a considered advantage If the team members do not speak Arabic, then arrangements should be made by the successful bidder to ensure translation/interpretation services are provided, and are included in the costs in the financial proposal

12. Estimated duration of the contract and payment schedules

The contract is expected to commence on 02.OCTOBER.2017 and will be undertaken over a maximum of seven months, and should be completed by 09.MAY.2018.

Payment is contingent upon approval by the UNICEF Evaluation Manager and will be made in four instalments. The following payment schedule is proposed: - 30% of the total contract value will be paid upon acceptance of the Inception Report with Project Delivery Plan; - 40% of the total contract value will be paid upon acceptance of the Draft Evaluation Report; - 30% of the total contract value will be paid upon finalisation and acceptance of the final evaluation report and two PowerPoint presentations (one to the Jordan Country Office and a second one for key sector partners) # the presentations will take place in the UNICEF Jordan Country Office.

13. Roles and responsibilities in the evaluation process

13.1 The Evaluation Manager - Manage the evaluation process throughout the evaluation (design, implementation and dissemination and coordination of its follow up); - Convene the evaluation reference group meetings; - Facilitate the participation of those involved in the evaluation design; - Coordinate the selection and recruitment of the evaluation team; - Safeguard the independence of the exercise and ensure the evaluation products meet quality standards; - Connect the evaluation team with the wider programme unit, senior management and key

101

evaluation stakeholders, and ensure a fully inclusive and transparent approach to the evaluation; - Facilitating the evaluation team#s access to all information and documentation relevant to the intervention, as well as to key actors and informants who should participate in interviews, focus groups or other information-gathering methods; - Provide the evaluators with overall guidance as well as with administrative support; - Oversee progress and conduct of the evaluation, the quality of the process and the products; - Approve the deliverables and evaluate the consultant#s/team#s work in consultation with Evaluation reference group and will process the payments after submission of the deliverables that respond to the quality standards; -Take responsibility for disseminating and learning across evaluations on the various programme areas as well as the liaison with the National Steering Committee; - Disseminate the results of the evaluation.

13.2 Evaluation Reference Group (ERG) The ERG will comprise the representatives of the major stakeholders including Evaluation Manager, Chief of the WASH Section, Planning, Monitoring and Evaluation Officer, and a representative from the Ministry of Water and Irrigation. The Regional WASH Advisor and the Regional Monitoring and Evaluation Advisor will also provide a quality assurance of ToR, inception report, draft and final evaluation report and participate at ERG meetings as necessary. The ERG will: - Provide clear specific advice and support to the evaluation manager and the evaluation team throughout the whole evaluation process; - Review the ToR, inception report and draft evaluation report and ensure final draft meets the UNICEF-Adapted UNEG Evaluatio

102

Reports Standards; - Review and provide comments and feedback on the quality of the evaluation process as well as on the evaluation products (comments and suggestions on the TOR, draft reports, final report of the evaluation).

13.3 The Evaluation Team The evaluation team will report to Evaluation Manager and conduct the evaluation by fulfilling the contractual arrangements in line with the TOR, UNEG/OECD norms and standards and ethical guidelines. This includes developing of an evaluation plan as part of the inception report, drafting and finalising the final report and other deliverables, and briefing the commissioner on the progress and key findings and recommendations, as needed.

14. Performance indicators for evaluation of results Proposed timelines for completion of activities are met and deliverables submitted on time with good quality and as per the standards described in the TORs as well as UNICEF/UNEG global standards. Overall performance at the end of the contract will be evaluated against the following criteria: timeliness, responsibility, initiative, communication, and quality of the products delivered.

15. Frequency of performance reviews Performance review will be conducted after completion of each deliverable. Bi-weekly meeting or meeting at a frequency mutually agreed will be held between the evaluation team and managers.

Equity, gender, human rights, including child rights The evaluation process will need to follow -but not limited to- guidelines and procedures on equity, gender and human/child rights made by UNICEF and the United Nations found in the Zip File "Annex 1: Equity, gender, human rights, key DOCs". Main Instruments or policies on human rights, including child rights and gender equality, that should guide the evaluation process are: - Child Rights and International Legal Framework, https://www.unicef.org/tdad/index_56373.html - Convention on the Rights of the Child (CRC), convention and subsequent comments, guidelines, declarations: https://www.unicef.org/crc/ - Sustainable Development Goal (SDG), namely SDG 5 "Achieve Gender Equality and Empower All Women and Girls", https://sustainabledevelopment.un.org/post2015/transformingourworld, and http://www.unwomen.org/en/news/in-focus/women-and-the-sdgs/sdg-5-gender-equality - Transforming our world: the 2030 Agenda for Sustainable Development, http://www.un.org/ga/search/view_doc.asp?symbol=A/RES/70/1&Lang=E - UNICEF strategic 2014-2017; https://www.uniceforg/strategicplan/index_68123.html - CEDAW, http://www.un.org/womenwatch/daw/cedaw/ - UNICEF Gender Action Plan 2014-2017 - Security Council Resolution 2122 on Sexual Violence in Armed Conflict (2013), http://unscr.com/en/resolutions/doc/2122 - International Conference on Population and Development (1994), http://www.unfpa.org/icpd - Security Council Resolution 1325 on Women and Peace and Security (2000), http://www.un.org/womenwatch/osagi/wps/

Ethical principles and premises of the evaluation The evaluation process will need to adhere to the United Nations evaluation norms and standards (https://www.unicef.org/supply/files/ATTACHMENT_IV- UNICEF_Procedure_for_Ethical_Standards.PDF), and the UNICEF ethical standards found in the Zip File "Annex 2: UNICEF Ethical Standards". Inception report, data collection tools, and final report will be reviewed on ethics by Institutional Review Board and need to be cleared.

103

Annex B: Anonymized List of Interviews Code GENDER ORGANISATION Role Date 01 Male Unicef Jordan Management July 24 02 Female Unicef Jordan Technical Specialist July 24 03 Female Unicef Jordan Technical Specialist July 25 04 Male Unicef Jordan Technical Officer July 25 Information 05 Male Unicef Jordan July 25 Management Officer 06 Male Mercy Corps Technical Specialist July 25 07 Male Mercy Corps Technical Specialist July 25 08 Male UNICEF Jordan Technical Specialist July 26 Unicef Jordan up to 09 Male Technical Specialist July 26 Sept. 2017 10 Male Oxfam Jordan Management July 26 11 Male Oxfam Jordan Project Leader July 26 Norwegian Refugee 12 Female Education Specialist July 29 Council Norwegian Refugee 13 Male Project Manager July 29 Council 14 Female ACTED Management July 29 Monitoring and 15 Male UNICEF July 30 evaluation 16 Female UNICEF Management July 30

Ministry of Water and 17 Male Official July 30 Irrigation

18 Water Authority of Male Official July 30 Jordan 19 Male World Vision Project Manager July 30 20 Male UNICEF Quality Assurance July 31 21 Female ACF Management July 31 22 Male ACF Technical Officer July 31

Unicef Ruwaished, 23 Male Teachnical Officer July 31 Jordan

104

Contractor – four 24 - 27 4 Males July 31 participants 28 Male UNICEF Management July 31 29 Male Kfw Management Aug 1 Unicef Regional 30 Male Technical Advisor Aug 1 Office, Amman US State 31 Male Official August 1 Department/BPRM Management 2 males Future Pioneers (local 32 - 34 and 1 Aug 2 NGO) Monitoring and female evaluation officer Technical Specialist 35 Female Unicef Officer Aug 5 36 Female Unicef Technical Officer Aug 5 37 Male Unicef Technical Officer Aug 5 2 males 38 - 43 and 4 Oxfam Project Staff Aug 5 females 44 - 46 3 males Oxfam Project Staff Aug 5 47 Male UNHCR Management Aug 6 48 Male Unicef Technical Specialist Aug 6 49 Male UNICEF Management August 6 50 Male UNOPS Management Aug 24 51 Female ACTED Management

52 Male Mercy Corps Management Sept 4

Annex C: List of Sites Visited

1. Irbid Governorate Host Community programme 2. Ramtha 3. ITS site 702 4. ITS site 709 5. Ruyaished Host Community 6. Rukban Settlement 7. Za’atari Camp

105

8. Azraq Camp 9. King Abdullah Park Camp 10. WIS Schools Visit in Irbid

Annex D: Documents Consulted

Camp Assessments UNICEF_REACH_Comprehensive Child Focused Assessment Azraq Refugee Camp UNICEF_REACH_Comprehensive Child Focused Assessment Za'atari Refugee Camp

106

Camp Contracts 6.2.2 Aref & Mohamed Lafi Contract for repair septic tanks f6.2.3 Bab El-Amood Contracts 6.2.7 IMDAD 6.2.8 Irbid District Electricity Company contract 43141693 t6.2.9 Open Hands - Monitoring of Refugee Camp wastewater Disposal Contract 6.2.10 Team Connect - Contract # 43143909 6.2.12 Safe Gaz PO # 43133530 6.2.13 Royal Scientific Society contract #43132050 6.2.18 GIS Contract - Ghazi Gsouss 6.2.21 EEC 6.2.25 Site Group 6.2.34 Loyality 6.2.35 New Day 6.2.37 Engicon 6.2.38 ACE 6.2.39 GITEC 6.2.40 ITC 6.2.41 Aquatreat 6.2.42 Health Bureau 6.2.44 Nimer al lawzi 6.2.47 Desluding Zaatari 2015-16 6.2.51 Specalized company for trade service - Hygien Kits - LLTA 42407750 6.2.53Bakhos 6.2.59 JEA 6.2.60 Al Mehwar 6.2.61 GAMA 6.2.64 Al Faris 6.2.65 Al Manaseer

Field Trip Reports ACF Report - March2015.docx programme visit and trip report - ACF-Dec 2016-0&M. programme visit and trip report - ACF-Dec 2016-soft programme visit and trip report - ACF-dec-O&M.docx

107

programme visit and trip report - ACF-July 2017-0&M programme visit and trip report - ACF-June 2017-0&M programme visit and trip report - ACF-June-V2. programme visit and trip report - ACF-Nov-GTD programme visit and trip report - ACF-Oct-GHWD Trip report - Q4 - ACF1. Trip report - Q4 - ACF2 Trip report - Q4 - ACF3 15 06 ACTED Prog Visit.pdf 16 01 20 Za programme visit and trip report.docx 16 06 08 Prog Review visit to Azraq.pdf 16 07 27 programme visit and trip report-ACTED-28.07.2016 Prog Review visit to Azraq 17 0411 Prog Review visit to Zaatari 17 06 20 Prog Review visit to Azraq.pdf 83 ACTED Report April 2017 Jen Reports Q1 2015 Q1 2015 Q1 2016 Q1 2017 Q2 2015 Q2 2015 Q2 2016 Q2 201 Q3 2014 Q3 2014 Q3 2015 Q3 2015 Q3 2016 Q4 2014 Q4 2014 Q4 2015 Q4 2016 Oxfam Reports programme visit_trip report-0xfam20140917

108

programme visit_trip-report-Oxfam20141231 OXFAM-Q1-Prog-Visit_Trip-Report-20150323 Notes from programme Visit Meeting_Oxfam_290615 OXFAM-Q2-Prog-visit-trip-report-20150629 Notes from programme Visit Meeting.Oxfam 300915 OXFAM-Q3-Prog-visit-trip-report-20150930 OXFAM-Q4-Prog-visit-trip-report-20151208 World Vision Prog-Visit_Trip-Report-World_Vision-20141109 programme visit and trip report - World.Vision and IMC-November -SAL programme visit and trip report - World.Vision-December -SAL programme visit and trip report - World_Vision-Feb-SAL programme visit and trip report - World.Vision-January -SAL programme visit and trip report - World_Vision-Mar-SAL programme visit and trip report -WVI-july-SAL programme visit and trip report -WVI-June-SAL programme visit and trip report -WVI-sep Trip report - Q2 - WVI Trip report - Q3 – WVI

Program Cooperative Agreements WV PCA19-13 Prog visit report Azraq - 9 Nov 2014 WV PCA 19-13 Prog visit report Azraq - 9 Nov 2014 WV PCA 19-13 Prog visit report Azraq -11 Sep 2014 WV spot check 2014 0119.pdf WVI PCA 19-13 Prog visit report Azraq -19 Feb 2015 WVI PCA 19-13 Prog visit report Azraq -19 Feb 2015 lWVI PCA 19-13 Prog visit report Azraq - 25 Mar 2015 WVI PCA 19-13 Prog visit report Azraq - 25 Mar 2015 WVI PCA 19-13 Prog visit report Azraq V5 & J stations - 25 Jan 2015 WVI PCA 19-13 Prog visit report Azraq V5 & J stations - 25 Jan 2015 l§J WVI PCA 19-13 Prog visit report Azraq-23 June 2015 L WVI PCA 19-13 Prog visit report Azraq-23 June 2015 KAP Surveys 2017 AZRAQ Camp Knowledge Attitudes and Practices Survey Report Final.

109

KAP 2013 Final Report KAP 2014 Final Report 15.03.15 KAP 2015 Final Report 21.03.16 KAP 2015 Executive Summary 21.03.16 ACF ACF PCA14-PD 1-2015-Azraq ACF PCA 26-14-Houshold-(Ajloun-lrbid) ACF-PCA-14-PD 02 Irbid ACTED ACTED PCA 02 PD 02 2017 ACTED PCA 02-PD 01-2015 ACTED PCA 23-13 JEN JEN PCA 01-PD 01-2015 JEN PCA 01-PD 03-2015 (amend to PCA 25-14) JEN PCA 01-PD 04 - 2017 Za'tari and Mafraq JEN PCA 9-13 JEN PCA 18-15 JEN PCA 25-14 JEN PCA 34-13 Oxfam OXFAM PCA 13-PD 1-2015 OXFAM PCA 13-PD 2-2016 OXFAM PCA 13-PD 3-2017 OXFAM PCA 32-13 SRIPCA 33-13 World Vision (WV) May 2013 Factsheets on Syrian Refugees October 2012 - December 2017 Monthly, Quarterly, and Annual Reports from Partners Partnership Agreements, Budgets, and Ammendments, 2012 - 2017 Other Documents Sitrep Reports December 2012 - August 2018 Third Party desludging, water and solid waste Monitoring data (UNOPS) Water monitoring data, Yarmouk; 2015-2017 Jordan Nationwide Assessment in Public Schools for Strategic Planning

110

Running on Empty: The situation of Syrian Children in Host Communities in Jordan ITS Field Visit Reports Host Community Field Visit Reports AZRAQ Camp Knowledge Attitudes and Practices Survey Report 2016_HJ_HP unicef comment KAP Report 2017(non-public) REACH_JOR_factsheet_CCFA_Feb2017_WASH REACH_JOR_UNICEF Zaatari Camp Assessments List_Oct 2017 REACH_UNICEFJTS_MS_AUGUST2014_FINAL UNICEF_REACH_Comprehensive Child Focused Assessment Azraq Refugee Camp_J.._.pdf p UNICEF_REACH_Comprehensive Child Focused Assessment Za'atari Refugee Camp UNICEF_REACH_Factsheet_Zaatari_CCFA_WASH 2017 UNICEF Jordan Audit Access to water and tensions 2014 ACTED Gender Task Force UNICEF Jordan Annual Reports 2012 - 2017 Controling Groundwater Thesis.html L' Coordination Study.pdf & Cyber City Map EVALUATION_OF_THE_WASH_PROGRAMME_WITHIN_THE_UNICEF COUNTRY_PROGRAMME_IN_LEBANON_2013-2015_2017-00 Manual for maintenance cleaning and hygiene promotion in schools - English Explanation of Gama Contract Groundwater Jordan Handicap International Advice Market Systems 2013 History of Za'atari History of the Palestinian Refugees in Jordan IT Humanitarian Funding Update_GHO_31DEC2017 IMF Jordan Report 2017 Syrian refugees in Jordan, A Protection overview January 2018 C Jordan, 2012 Article IV Consultation; IMF Country Report 12_119; March 23, 2012 Jordan_Water_Strategy_2008-2022 Jordanian attitudes toward refugees 2012 MoWI Annual Report 2016 National Water Strategy( 2016-2025)-25.2.2016 Equity, scalability and sustainability in UNICEF WASH programming: A thematic meta- analysis of UNICEF’s WASH evaluations 2007-2015 Water, Sanitation, and Hygiene (WASH) Cluster Coordination Handbook; January 2009

111

WASH Sector Coordination; Minutes of weekly Za’atari camp WASH sector coordination group.

Annex E: Data Collection Tools

Key Informant and Key Stakeholder Interview Guide Introduction for the Interviewer ISG is conducting an evaluation of a programme managed by UNICEF. The objective of the research is to understand how UNICEF has helped people get better access to water and sanitary facilities, and the results of their hygiene activities where you live. ISG will conduct its research via key informant interviews, key stakeholder interviews, and focus group discussions. The interviews will include male and female stakeholders to ensure differences in uses and needs are captured. This interview guide serves as a manual for data collectors conducting the key informant interview portion of the research. ISG’s researchers will conduct face to face interviews WASH stakeholders. While this guide is intended to assist the interviewer in setting key areas for research and orienting the discussions, the interview should happen as a conversation, allowing the interviewer to ask follow up questions that may lead to discovering new and important information. INSTRUCTIONS FOR THE INTERVIEWER

• Follow the informed consent process below.

• You should ensure that you cover all the major issues noted in this guide. However, some questions may elicit long discussions and others short answers depending on the relevance of individual topics for discussion to individual respondents. Exploring the most relevant areas (and others that may be important to the respondent, but not necessarily covered in this guide) is more important than covering every question in the guide.

• Ensure that women participate in the interview, especially if the interview is taking place in a group.

• The WASH programme’s activities operate in a dynamic and complex environment. It is likely that there are key issues that the author of this guide did not anticipate. The interviewer should feel free to ask questions that the interviewer feels are fruitful whether or not they are included in the guide.

• Please take notes in a separate document.

112

• Make sure to record your notes at the end of each day. Too much time between the interview and recording notes can lead to loss of salient information that will have a negative impact on the rest of the research. Informed Consent Process This section outlines the consent process for the UNICEF Jordan WASH programme Evaluation. While this document provides illustrative language, it is important to tailor both language and presentation to each stakeholder to ensure understanding of the study and his/her involvement. The people that the evaluation team will interview have the right to know that they are participating in a study, that the evaluation team is collecting their opinions as part of a research project, and to be told about the purpose of the research and the possible risks and benefits of the project and their participation. The interviewer will introduce himself/herself and explain the purpose of the interview as detailed in the Informed Consent Form below. Following the explanation, the researcher will ask the respondent to sign the Informed Consent Form:

INTERVIEWER INTRODUCTION Hello! Thank you for participating in this interview. My name is ______and I am working on a project for International Solutions Group, a research organization based in the United States. ISG is conducting research on behalf of UNICEF. The purpose of our research is to assess the effectiveness of UNICEF’s Water, Sanitation, and Hygiene related activities where you live. You may not know about UNICEF’s specific activities. However, UNICEF is interested in understanding if the program has achieved its intended goals, which include improved access to water and sanitation and improved hygiene for people who live in your area. I will ask you questions about your life and experience in relation to UNICEF’s water, sanitation, and hygiene promotion activities. Although you may not directly benefit from participating in this study, we hope that the findings of this study will contribute to the improving UNICEF’s Water, Sanitation, and Hygiene programme’s for people like you. The interview today will last about an hour. During the interview, I’ll take notes so I can remember what you said. I want to mention a few important points before we start.

• There are no ‘right’ or ‘wrong’ answers to any of the questions. We are interested in hearing your opinions, whatever they are.

• Your participation in this interview is voluntary. If you don’t want to answer any question, you don’t have to. The responses that you provide are confidential: we won’t mention your name in any report we write and we will never connect your name with anything you say. Your responses will be combined with responses from other people. Do you have any questions before we begin? Now that I’ve told you about the topics we’ll discuss and answered your questions, are you comfortable proceeding with the interview? 1. IF NO: ASK IF YOU COULD PROVIDE ANY OTHER INFORMATION THAT WOULD MAKE THE RESPONDENT MORE COMFORTABLE. IF RESPONDENT PREFERS NOT TO CONTINUE, TERMINATE INTERVIEW. 2. IF YES: CONTINUE TO THE NEXT SECTION.

113

Household Level Questionnaire Basic Information [NOTE TO INTERVIEWER: Fill out before the interview begins.] 1) Consent form number: 2) Date of Interview: 3) Gender (if apparent): 4) Primary and Secondary location of interview: 5) Interview Number:

Water Supply 6) Is water supply sufficient for your needs? [If no Goto 6a] a) Which needs are unmet [for example: drinking, cleaning, latrine, other] b) How much more water you would need? 7) Where do you most often access water [if outside the home, Goto 7a] a) How often do you fetch water? b) How far do you go? c) How long does it take? d) Are there alternative sources from where you fetch or buy water? 8) Do you buy additional water from other providers? [If yes, Goto 8a. If no, Goto 9] c) Why do you buy additional water? Is it expensive? d) Does the water have good quality? 9) Have you ever experienced problems accessing water? [if yes, Goto 9a] a) What caused the problem? b) How was the problem solved? c) Has it reoccurred? Water Quality 10) Is the quality of the water sufficient for all of your needs [for example: drinking, cleaning, latrine, other]? [if no, Goto 10a] a) For which activities is in insufficient (for example: drinking, cleaning, latrine, other)? 11) Have you ever become sick because of the water? [If yes, Goto 11a. If no, Goto 12] a) What was the illness? b) Was it just one person or many people in your family? c) How often did it happen?

114

d) How was it treated? Facilities 12) Does your family have access to a latrine and bathing facility? a) Is the bathing facility and latrine private or communal? b) Is it satisfactory? c) Are there any improvements you would make to it (if communal, is distance reasonable, is it safe, is it lighted sufficiently, does it lock)? 13) Is there enough privacy for women and children? a) Are sanitation and bathing facilities segregated by gender? b) Is there enough distance between female and male WASH facilities? 14) [if latrines are inside in the house Goto 14a; if outside of the house Goto 15] a) Does the household have access to latrines/septic tanks or mainline sewerage? [if septic tanks, go to 14b] b) Is there any service provider that comes to empty the latrines/septic tanks? Is the service timely and reliable? e) Have you ever paid for the maintenance of your WASH facilities? If so, whom did you pay? Is it the money worth the service? 15) Have you ever experienced problems accessing latrines or bathing facilities? [if yes, Goto 15a] a) Have they ever been closed or not usable for short or long periods of time? If so, why? Was the problem solved? How often did it happen? 16) If people generally have trouble with water or sanitary facility access, or the facilities don’t work, what do they do? 17) Have you or any other member of your community experienced any threat while using the WASH facilities? [if yes, Goto 17a] a) Was it at day or night? b) What do you think can be done to avoid similar situations? c) Has security improved since the event? 18) Does the camp/community have latrines and bathing facilities accessible to people with limited mobility? Where are they? (follow observation) a) Are they sufficient? If not, what could be done differently? 19) Is there any area of this camp/community that has a harder time accessing WASH facilities? a) If yes, which areas? Why? 20) Are there any vulnerable people that have a more difficult time accessing WASH facilities? a) If no, which areas? Why? Operations and Maintenance 21) Who cleans the WASH facilities that you use? Have you ever cleaned them? Is there a system in place to maintain the facilities clean? 22) Is there any committee in charge for the maintenance of the water or sanitation facilities or infrastructure? 23) Is there anyone in your community or immediate circle able to fix minor issues with the WASH facilities? [if yes, Goto 23a] a) If so, is this person paid? b) Does he/she do a good job? c) Is he/she reliable? 24) Do you know if this person was trained by UNICEF or by other organizations? Which organization? Final Questions 25) Were you consulted about your preferences regarding water and sanitation services? If so, how? a) Were the different options clearly explained to you?

115

b) Did you have the opportunity to provide feedback on the services provided? If so, how? Did the feedback you provided have any outcome? (Were the services adjusted/corrected/improved according to your feedback?) 26) Have you noticed major changes related to access to water in your camp since you have been here? [If yes, Goto 26a] a) If so, what kind of changes? b) How did they happen? c) Who made the changes? d) Did the changes improve your situation? 27) Is there anything else that you would like us to know about Water, Sanitation, or Hygiene where you live?

Focus Group Discussion Guide Introduction for the Focus Group Leader ISG is conducting research on behalf of UNICEF. The objective of the research is to understand the impact of the UNICEF’s Water, Sanitation, and Hygiene programme. Particularly, we want to know how the program has worked to meet you expectations, where it has succeeded, and where it could improve. ISG’s research will be conducted via a survey, focus group discussions, and key informant, key stakeholder interviews. This focus group guide serves as a manual for the focus group leader and note taker conducting focus groups. ISG will conduct focus groups with residence of camps, ITSs, host communities and stakeholders in schools: 1. The degree to which project activities served the needs of beneficiaries. 2. The attitude of community members toward ownership and maintenance of infrastructure and asset building projects 3. The effectiveness of program in achieving water, sanitation, and hygiene results for all stakeholders, including vulnerable groups. 4. The sustainability of the programme’s activities and facilities.

The purpose of these focus groups is to present broad issues and to let participants generate ideas and build on each other’s ideas. The focus group leader’s goal is to encourage participation and ask questions that build on participants thoughts. This guide provides follow up questions that the leader may use. The leader may ask specific participants to compare their opinion or thoughts to those of other participants through prods such as “Participant B, how does your experience compare to the experience of participant A that we just heard?” or “Participant C, what would you add to participant B’s comments?” The interaction that the focus group leader encourages will provide depth to the information we are gathering through the survey and interviews. Preparation

• Make sure that informed consent forms are available for each participatng.

• Make sure the room is set up with flip charts or whatever materials you need.

116

• Set up refreshments if appropriate.

• If available, set up small gifts to thank the focus group participants. Agenda Introduction, Ground Rules, Logistics and Sign-in Sheet

[FOCUS GROUP LEADER: Introduce yourself and the note taker.] Thank you all for participating in this focus group. My name is ______and I will lead the group, and this is ______who will take notes. Our discussion will last no more than an hour and a half. Before we get started, let me mention that there are drinks and refreshments available (point them out), and the bathroom is (indicate location of bathroom). Please feel free to get up and move around or help yourself to refreshments during the discussion. The purpose of this focus group is to understand the impact of the UNICEF Water, Sanitation, and Hygiene programme in your community (or school). Particularly, we’d like to understand: • How project has served your community’s needs. • Your opinion of the project’s facilities and services. • What you like best about the project. • What you would most like to improve about the programme

Infront of you is an informed consent form. This form let’s you know about the purpose of the research and the types of questions we will ask. Please read the form and let me know if you have any questions. If you agree to participate in this focus group, please sign the form at the bottom where indicated.

I wanted to mention a few important points before we start. • The purpose is for us to learn from you, both your positive and negative experiences and opinions.

• We’re gathering information, not trying to achieve consensus. Please contribute your experience and opinion. It’s ok for there to be disagreement or different ideas. We have a few ground rules that we hope you’ll agree with: • Please participate. Your contribution is valuable and will improve the program’s activities in the future.

• Please agree to keep the information provided in the focus group confidential. We won’t discuss any sensitive topics. Still, people may share ideas that they’d rather not be discussed privately.

• Stay with the group and please don’t have side conversations.

• Turn off cell phones if possible. Are there any questions before we get started? FOCUS GROUP LEADER: Go around the table and ask people to introduce themselves.

117

Relevance 1. Do you think the interventions responded to your basic needs related to water, sanitation and personal hygiene? a. What’s the most important way your community has improved in terms of water access, sanitation, or hygiene practices? b. What needs does the community have that aren’t being addressed? Why do you think they haven’t been addressed? Effectiveness 2. If you could improve your access to water or sanitation facilities, what would you improve? 3. Is there any group in your community that have not been reached or that has not benefitted from UNICEF like the others? 4. What do people do if they have trouble accessing water, or the sanitary facilities don’t work? a. What authorities are responsible? Are they responsive? WATER 5. How has access to water improved for this community recently? What is the most important improvement? 6. What changes would you make to the community’s ability to access water? SANITATION 7. Do you use toilets or latrines? a. Are they public or private? b. How are they kept clean? c. Do they work all the time? d. Are they safe? 8. What changes would you make to sanitary facilities? HYGIENE 9. Has the hygiene knowledge or practices changed in the community in the last five years? How? a. Do people ever get SUSTAINABILITY 10. What would happen if UNICEF would stop its activities tomorrow? Do you think your community/school would be able to maintain the water and sanitation infrastructures as they are? 11. Is the wastewater and solid waste handled properly? Do you have any concern for the environment?

118

Annex F: Team Description

Mr. Jason Wares; Team Leader Role: Mr. Wares led the team l, organized the completion of deliverables, conducted data collection activities, and arranged team logistics. He also interfaced with the UNICEF at all stages of the assignment. Mr. Luca Palazotto, Mr. David Robbins, and Mr. Niall Roche; Co-Evaluators Role: The Co-Evaluators will assist the Team Leader during the tools development, sampling, desk review, data collection, and reporting aspects. Dr. Bassam will lead in the translation of tools and reporting, as he will serve as the Country Expert for this assignment. interdisciplinary research consultants (id:rc) Role: id:rc organized logistics for the evaluation in Jordan and provided local WASH and monitoring and evaluation consultants. Id:rc participated in data collection and analysis, and ensured that the evaluation was grounded in the Jordanian context.

119

Annex G: Evaluation Matrix

EQ1: To what extent were affected people consulted on their preferences with regards to water and sanitation service options and level of services? Were interventions appropriate in terms of meeting their basic needs? Was there a feedback loop and monitoring system in place for reflecting the learning and evidence to improve programming, especially for the humanitarian response?

Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection JC 1.1: The needs of the - Evidence of an process to - Starting at WASH Working - Specific data analysis population, in particular identify needs prior to the Group paper - World Vision Staff Interview those of vulnerable groups, design and implementation - WASH Cluster Coordination 30 July 2018 were well considered during of interventions. Handbook - UNICEF Staff interview – 26 the programming process - The extent to which chosen - Jordan Regional Response July 2018 and adjusted during interventions were consisted Plan 5 - Focus Gropu Discussion programme implementation. with the needs that - 2015 KAP survey for number 1 through 4– Azraq assessments identified. Za’atari - Official Interview – MoWI - The choice of interventions - WASH Sector Coordination; - Mercy Corps Staff Interview prioritized activities that Minutes of weekly Za’atari – 4 September 2018 would serve vulnerable camp WASH sector - On site assessments and groups. coordination group. Sunday interviews in Za’atari, Azraq, - The existence of a feedback 24 November 2013 KAP, and Rukban. loop and monitoring system - 2017 Azraq KAP Survey - Interview with Gama that evolved with the Report Engineering programme. - 2014 WASH Sector Gender - Host community Interviews Analysis Document in Irbid and Ruwashaed

120

- Evidence that UNICEF used - REACH 2014 ITS - UNOPS Staff Interview evidence for learning. Assessment - UNICEF Jordan Country - World Vision PCS 19-13 Programme Document - ACF 2017 Azraq KAP - UNICEF WASH Actions in Survey Report Humanitarian Situations: - Rukban emergency plan Synthesis of Evaluations, 2010 - 2016 In the intial phase of the Programme, UNICEF determined refugee’s WASH needs through talks with partners such as The WAJ, Mercy Corps, Acted, and Relief International, and based on the assessments and information that they’d gathered. The tactics that they employed took into consideration the Convention of the Rights of the Child (CRC) and the Convention on the Elimination of All Forms of Discrimination against Women (CEDAW). Through these means, UNICEF created estimates of the best way to provide WASH services for the newly arriving refugees. UNICEF also consulted humanitarianresponse.info for guidance on issues such as accountability and feedback mechanisms.

The evaluation team did not find evidence that UNICEF consulted internal resources for guidance on WASH response in an emergency situation.

Another consequence of the need for quick decision making was that, in the initial emergency response phase, the camp’s effect on Jordanians living near the camp and in other host communities with high concentrations of refugees was not taken into consideration adequately.

In later phases, UNICEF used assessments and consultations to understand the needs of beneficiaries inside the camps. Azraq residents benefited from lessons learned in the establishment of WASH services in Za’atari and contributed their preferences to WASH set up.

After the emergency response phase, UNICEF worked with its partners to adapt WASH facilities and services to the observed needs and behavior of camp inhabitants.

121

Time pressure and the desperation of the Settlements’ inhabitants made consulting them about their preferences impractical.

UNICEF provided water and sanitation services to respond to the settlements’ exigencies. They moved quickly and effectively, with clear goals, such as ensuring that each person had at least 15 liters of water per day. The WASH programme also recognized that its intervention would have an impact on the larger Ruwaished host community, so they undertook to rehabilitate sanitation facilities there.

The WASH programme has maintained weekly updates since the initiation of work. The updates monitor operations and needs. In 2017, surveys and assessments became part of the monitoring mechanism. These tools ensure that WASH programme activities remain relevant and effective. UNICEF’s improvement of the wells and infrastructure in host communities is significantly important. However, the degree to which it addresses the needs generated by the refugee crisis and the needs of the most vulnerable is uncertain.

UNICEF reports that they have served 114 ITSs, but the total number of ITSs is unknown, and the level of that service is unclear.

The evaluation team could not find evidence that The WASH programme is structured as a learning organization. It does not appear to have taken guidance from UNICEF’s experience in WASH programming at the outset of the programme. While it has certainly made marginal improvements to its activities in terms of efficiency and responsiveness to people’s needs, it did not build monitoring networks or feedback loops that fed into a system that would allow for learning or informing strategic management

EQ2: To what extent did the interventions consider the needs and the priorities of the MWI and were they coherent with global references and the regional and national response to the Syria Crisis?

122

Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection JC 2.1: Degree to which - Evidence that MWI priorities - National Water Strategy - Senior Staff Interview – UNICEF took government were considered in 2016 – 2025 WAJ 30 July 2018 policy and strategy into intervention design and - Jordan Response Plans - Senior Staff Interview – consideration when implementation. MoWI 30 July 2018 determining programme - Government officials feel - Specific focus groups priorities. that UNICEF’s interventions JC2.2: Degree to which the were in line with policy and MWI and WAJ were regulations. satisfied that UNICEF responded to GoJ needs

UNICEF’s work in WASH is in line with the priorities of the National Water Strategy, for which The key reference document is the National Water Strategy 2016 – 2025. UNICEF is focused on the sustainable operation of boreholes to supply water and create resilience. UNICEF also recognizes the government’s concern that Jordan’s northern governorates have not benefited from the government’s efforts to increase water supply (Ministry of Water and Irrigation, 2016). In response, UNICEF has focused much of its host community interventions on improving WASH in the northern governorates.

Both the MoWI and WAJ reported in interviews that the GoJ-UNICEF partnership is successful. The WAJ praised UNICEF’s success at sector coordination despite budget cuts140. The MoWI appreciated UNICEF’s role in coordinating the sector141.

140 WAJ Senior Staff Interview – 30 July 2018 141 MoWI Senior Staff Interview – 30 July 2018

123

Interviews at MoWI and WAJ demonstrate the Government of Jordan’s struggles with its strategy and UNICEF’s response. Senior staff at the MoWI praised UNICEF’s responsiveness inside the camps, but sees that project’s host communities are more dependent on available funding. MoWI said that when the WASH programme first took on host community projects, there was a miscommunication regarding the meaning of the protocol UNICEF established with the MoWI. The protocol included a mandate and budget for UNICEF to improve infrastructure in governorates. The ministry didn’t realize that the budget was prospective, and moved those items off of its budget for the year. When fundraising fell short, those governorates were underserved.

EQ3: How responsive was the UNICEF WASH programme over time to changes in the external environment?

Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection JC 3.1: UNICEF - Evidence of adjusting - Jordan Annual Reports - Senior Staff Interview – 6 demonstrated flexibility in programme design and August 2018

124

decision making and the operations to changing - Terms of Reference for Service ability to reorient resources conditions Contracting - Rukban. as the external environment - - Focus group discussions in changes. Azraq, Za’atari, and KAP In the initial phase of the programme, the GoJ and INGOs in Jordan prepared for a temporary camp. This preparation included procuring tents, portable sanitary facilities, and trucked in water supplies. While these interventions were effective, their expense made them unsustainable. When it became apparent that the need to house refugees was more than temporary, UNICEF devised solutions that required investment, but would be cheaper and more sustainable to operate in the medium to long-term. These interventions included WASH Blocks, piped in water from nearby boreholes, and established hygiene campaigns. UNICEF also made these adjustments in the settlements, establishing water infrastructure that was both resilient enough to withstand difficult conditions and use, but also reduced operating funding needed for maintenance142. As Senior UNICEF Staff noted, “Transitioning from humanitarian response to systems and infrastructure…is incredibly challenging, particularly in environments like the berm (Rukban).143” One type of transition in which the WASH programme struggles was in implementing host community projects. As an interview respondent at the MoWI suggested, UNICEF did not originally plan on implementing host community projects, but turned to them in the face of external pressure from municipalities that felt that refugees were putting stress on local water resources 144. The WASH programme boldly took up the challenge and set high goals for itself such as increasing the percentage of the population in Jordan that has access to adequate sanitation facilities (2014), or increasing the number of inhabitants with safe water and storage facilities by 1.4 million (2015)145. While UNICEF missed these targets in each year, it continued to set high targets in this component. As mentioned in the Funding Environment section of this report, UNICEF blamed shortfalls in fundraising for missing host community targets in each year of the programme. However, the evaluation team did not find evidence that the WASH programme considered its funding outlook when planning these targets, or, given the regular shortfall in funding, how it might better spend these funds. We discuss this topic further in the Effectiveness and Efficiency sections.

142 UNICEF Staff Interviews – 10 October 2017 143 UNICEF Senior Staff Interview – 6 August 2018 144 MoWI Senior Staff Interview – 30 July 2018 145 See results framework, Annex H.

125

EQ4: To what extent did UNICEF’s response achieve its intended outcomes (as stated in UNICEF’s Country Programme Outcome and Outputs as given in Annex A)? To what extent did UNICEF achieve equity results, especially for children and women specific interventions?

Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection JC 4.1: The degree to which - Indicators based on results - RAM reports 2013- 2016 - Interviews with partner UNICEF set specific, frameworks. - UNICEF Jordan Annual organizations measurable and realistic Reports - Interviews with UNICEF targets. - Za’atari Hygiene Working staff Group KAP reports - Reviews of PCAs - The Sphere handbook JC4.2: The degree to which the evaluation team could find evidence to support UNICEF’s achievement measurements.

JC4.3 The degree to which targets were met. Component 1 Under this component, the WASH programme achieved high levels of effectiveness for the targeted population across all years.

126

Component 2 As detailed with citations below, the WASH programme consistently struggled to achieve targets under this component. Two issues created difficulties: 1) funding, 2) lack of strategic planning As described in the Funding Context section of this report, UNICEF Jordan struggled with funding issues. Maintaining services in the camps necessarily received first priority for funds at the expense of achieving targets in host communities. That said, after several years of missing targets, UNICEF doesn’t appear to adjusted its strategy to make it more effective or realistic. Also, indicators aren’t specific or timebound under this component, so it’s difficult to know what the programme is trying to achieve. For example, UNICEF’s Outcome 4, Indicator 2 target is that 1.5 million host community residents use improved drinking water as per Jordanian standards. In its 2015 RAM report, the WASH programme stated that “In Host Communities, 41% (634,858) of the target population of 1,550,500 benefitted from improved access to water facilities and systems as a result of rehabilitation of water infrastructure, as well as support to vulnerable households.” In its 2016 RAM report, for the same indicator the WASH programme reported, “In Host Communities in four governorates (Mafraq, Madaba, Balqa and Irbid), 237,360 additional people were reached in 2016. Since 2013, an estimated 1,482,402 people have improved access to water services …which equates to 96% of the Country programme target.” It is unclear if the target is that UNICEF serve 1.5 million per year under this outcome as the 2015 report implies, or 1.5 million cumulative people as the 2016 report implies. It is also important to consider the improvement. For xample, In Irbid UNICEF made improvements to the water distribution system. This should not imply that everyone in Irbid had better water services as a result of the intervention. Water services are still very limited, and there are still serious problems with the systems distribution network, water safety, and availability.

Component 3 Government officials at the MoWI and WAJ reported satisfaction with the WASH programme contribution to planning documents, strategies, and policies146. The MoWI said that, as head of the WASH Taskforce, UNICEF always has a good plan, and has a good partnership with UNICEF on technical committees. The Ministry also praised the work done by the consultant that UNICEF seconded to Ministry, and wished he could have stayed longer.

146 Source: Interviews at MoWI and WAJ

127

The ministry staff interviewed by the evaluation team mentioned that there were some unintended consequences to the work that the WASH programme has done in host communities. A specific example was the downstream effects of increasing water supply through a network. Increased water supply means quicker degradation on infrastructure, leading to the Ministry having to replace equipment sooner than expected. One interview respondent mentioned the opportunity cost of providing water to refugees, in that the water could have gone to Jordanian workers or industry. He estimated that for each refugee the government spends $650 on water and $7,700 on indirect costs included lost opportunities.

Component 4 Stakeholders and UNICEF partners felt that UNICEF was effective as WASH sector lead. One contractor that serves Rukban said “UNICEF staff are super cooperative and available anytime of the day or night”.147 The evaluation team also spoke with two of the WASH programme’s donors, The Bureau of Population, Refugees, and Migration (BPRM) at the United States State Department, and the German owned development bank Kreditanstalt für Wiederaufbau (KfW). BPRM commented that UNICEF was effective in leading the sector, and excellent to work with as a donor. BPRM felt that they had benefited from UNICEF’s guidance in setting their own strategy for addressing the crisis. KfW appreciated the guidance UNICEF could provide in constructing infrastructure in the camps and assistance in host communities. As sector lead, UNICEF has led WASH activity coordination in the Camps, including responses to the occasional emergency, such as the 2015 outbreak of Hepatitis A in Azraq. As part of camp coordination, UNICEF leads a bi-weekly Za’atari coordination meeting that has been effective in setting standards and avoiding duplication of effort. UNICEF also improved the MoWI’s use of data to create water management strategies, and led the sector’s WASH in schools initiatives. One area in which UNICEF could improve its coordination is in strategic planning and communication with partner organizations. Several partner organizations, particularly INGOs, mentioned that they were surprised by UNICEF’s decisions in the camps, often

147 Source: Interview with Gama Engineering

128

requiring that they make major adjustments on short notice, or conduct work for which they hadn’t planned or budgeted148. Examples included sudden decisions to hand operations over to other organizations or contractors with short notice, without guidance on the handover process. Other partners mentioned contracting processes that took a long time only to have a requirement that initiatives start days after a contract is signed. Decisions such as these, that do not appear as part of a strategically designed plan, may force risk and costs on to partner organizations. It may also create risks for UNICEF that its monitoring and accounting systems cannot bear. These risks could also be mitigated by an annual strategic planning process that would set expectations and guidelines based on clear analysis and reduce last minute decision making149.

EQ5: What were the strengths and weaknesses of the interventions in achieving results in the camps/settlements, ITSs and host communities on the following: 1. Water (Litre/day, supply, quality, consistency, reliability, accessibility, and equity)? 2. Sanitation (adequacy, appropriateness, accessibility)? 3. Hygiene promotion (appropriateness of the messages, timeliness, behaviour change)? 4. WASH in schools? 5. National level support?

Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection

148 Interviews with Mercy Corps, World Vision, ACF, and Oxfam Staff 149 The UNICEF Jordan Country Programme Document 2013-2017 mentions a two year rolling workplan which is measured through annual component reviews and a mid- term review in 2015. The evaluation team did not see evidence that the WaSH Programme had complied with this mandate.

129

JC 5.1: The degree to which - International WASH - Jordan response plan - Interviews with Partner Staff UNICEF met its own and standards - Global WASH Cluster - Interviews with UNICEF international standards. - WASH in Schools targets vs handbook staff accomplishment - Sphere handbook. - Site inspections of UNICEF - UNICEF Reports intervenions JC 5.2: Beneficiary opinions - Camp Assessments on stregths and weaknesses - Hygiene Promotion Working of interventions. Group reports.

Water For both Za’atari and Azraq, the interventions were effective. Residents of both camps people at least 35 liters of water per day, well in excess of the 15 liters per day that is the Sphere standard

Sanitation The system in Za’atari, described below, is an innovative approach that works well and is cheaper to install than a traditional sewer system. The same is true for the boreholes and water systems at both Zaatari and Azraq, where high quality, well designed systems are being run by local operators from the camps, which contributes to sustainability.

In Azraq, The WASH programme has implemented gradual improvements of basic service. In 2014, shelters in Azraq were not connected to a greywater network, meaning greywater and mud tends to accumulate in the ditches surrounding the shelters. UNICEF with partners constructed a plot-level grey water network, and provided income generating opportunities within the camp contributing to camp cleaning and sustainable waste management practices. Communal toilets remain unpopular in Azraq.

The absence of assessment data with respect to the early stages of the response makes it difficult to determine what the key public health issues were in relation to hygiene, and the behaviours that needed to be targeted. As noted earlier in the report, WASH

130

programme partner ACF noted a high occurrence of waterborne diseases in Za’atari in 2012/2013. A general observation from the evaluation team is that the no official documents referenced these key infectious diseases. The exception was the Hepatitis A outbreak that afflicted Azraq. The first reported measurement on the issue of Hygiene was the Baseline KAP survey in Za’atari in November 2012, four months after the camp opened. A notable feature for Za’atari and Azraq was the strong commitment to repeating KAP surveys in both camps on an annual basis enabling one to measure change over time. The evaluation team did not have access to hygiene data with respect to Host Communities, those in the ITSs, or the settlements (Hadalat and Rukban). However, the evaluation team understands that given the circumstances it was not feasible to gather hygiene data in those areas.

Over time the KAP survey results for Za’atari showed an improvement.

In the area of hygiene items, interview respondents indicated that they did receive hygiene items, but not enough of certain items.

The intended results at the settlements were to provide water to the Rukban and Hadalat Refugee Camps as effectively as possible while achieving value for money. The evaluation team inspected the equipment and operations of the water system at Rukban. The borehole and related equipment are of high quality, the design is appropriate, the construction is of high quality, and the operations and maintenance activities are performed properly and on schedule. Staff working for Gama, the UNICEF hired to supply and operate the system at Rukbank reported in an interview that UNICEF always purchases high quality equipment.

At Rukban, the treatment system is a sophisticated and costly, but warranted and necessary due to the concentration of certain constituents of the raw water.

Hygiene promotion in the settlements seems to be limited to the distribution of hygiene kits on a periodic basis.

131

EQ6: To what extent did the programme use available resources in an economic manner to provide WASH services and facilities in camps/settlements and host communities, to ensure child rights, equity and gender equality?

EQ7: To what extent were the costs to deliver water and wastewater services, as well as solid waste management, rationalized and optimized in the camps and settlements to ensure Value for Money? How do these compare to similar situations at the local or regional level? Could more cost effective operations/interventions have been undertaken at an earlier stage?

Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection JC 6.1: The degree to which - Demonstrated declining - Review of PCA agreements. - Compilation of budget data. the programme could operations costs. - Review of cost reports - Complication of cost data. demonstrate it had used - Demonstration of value for - - Interviews with financial resources in an economic money. management staff. manner. - Existence of an adequate - Interviews with UNICEF system. staff JC 6.2: Documentary evidence that costs were rationalized and optimized.

There is no indication that the cost of providing water services in the early stages of the intervention was rationalized.

UNICEF took steps to bring down the unit cost of providing water to refugees in the camps. As figure 10 below shoes, the evaluation team estimates that between 2013 and 2016, the cost of delivering water to refugees in the camps dropped from 3.34 Jod/m3 to 0.24 Jod/m3.

132

There is some indication that trucking water into Za’atari camp went on longer than it should have, draining the programme’s resources.

Initially wastewater management was inefficient. Issues such as contractors claiming to remove up to 40 percent more sludge than they actually removed burdened the programme. Drivers claimed trucks were capable of holding more volume than they actually did and sometimes carrying waste in to the camp from outside, which was subsequently charged to the UNICEF operation

The wastewater system in Za’atari evolved from unimproved pit latrines to a system of shared septic tanks, a wastewater collection network, and eventually a wastewater treatment and reuse system.

In Azraq, the wastewater treatment plant is not functioning and has been abandoned.

An innovation that UNICEF developed was the “Waste Taxi” also known for a time as Uber for Waste. It was a predictive model designed to address with the various different volumes of wastewater tanks that were introduced when people began constructing their own private washing and toilet facilities in Za’atari.

EQ8: To what extent did UNICEF try to minimize duplication/gap in WASH interventions in camps? And how early did UNICEF move to more cost-efficient interventions?

Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection

133

JC 8.1: The existence of a - Evidence of full coverage in - REACH Assessments - Focus groups among management plan to beneficiary locations - UNICEF camp management beneficiary populations minimize duplication and - Satisfaction with coverage documents. - Interviews with UNICEF and gaps. from beneficiary populations. - Working group meeting Partner staff. - 3 minutes - JC 8.2: The degree to which the plan was effectively implemented

UNICEF minimized duplication or service gaps in camps by dividing the camps into sectors and assigning different partner organizations responsibility for WASH services in each sector. UNICEF’s role as sector lead allowed it to unify service standards among the different partners.

In Azraq, a similar coordination structure was followed as Za’atari. ACTED, World Vision, and THW provided WASH services with Relief International providing hygiene promotion and WASH in schools. However, coordination among the three entities appears to have been difficult. There is far less documentation on coordinated efforts in Azraq. in Zaatari, the WASH programme has reduced costs and realized efficiency by reducing the numbers of partners that they are managing.

134

EQ9: To what extent were resources (financial and human resources) allocated for the range of intervention types, with particular attention paid to the respective allocations for humanitarian response and resilience, to achieve the intended results? Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection JC 9.1: The degree to which - Financial information - Partner organizations PCAs - Interviews with UNICEF UNICEF could demonstrate showing distribution of and budgets. management staff. allocation of resources resources - UNICEF Staff estimated that approximately 5 percent of UNICEF’s budget goes to host community interventions. The evaluation team’s estimate, which as mentioned is based on incomplete information150, is that from January 2013 - -July 2017, about 63 percent of project funds went to camp operations, about 15 percent is spent on host communities, and another 15 percent is spent in settlements, Rukban and Hadalat (figure 13).

EQ10: To what extent did UNICEF advocate with the Ministries to mobilise their available resources to address the identified needs? And how efficient was UNICEF’s support to ministries (i.e. MWI and MOE) to reaching most vulnerable people? Did the WASH programme maximise the potential collaboration with other UNICEF programme sections and partners?

150 The financial numbers used in this report were derived from UNICEF’s estimates of expenditure on contracts and PCA agreements provided by UNICEF, which were often in a format that was difficult to read. PCA numbers were budgeted amounts, not actual expenditure. These figures also do not include any indirect costs that may not have been allocated proportionally by beneficiary group.

135

Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection JC 10.1: Existence of - Satisfaction levels of - Programmatic reports. - Interviews with partners documentation government officials - GoJ strategy documents - Interviews with UNICEF demonstrating advocacy - Existence of reports detailing staff efforts. support initiatives. - Interviews with government - Partners report positive and officials. maximum collaboration JC 10.2: Degree to which GoJ officials agreed that UNICEF’s support assisted them in meeting most vulnerable people.

JC 10.3 Degree to which maximum potential collaboration was demonstrated Over the time frame that this evaluation covers, UNICEF’s primary engagement with the MWI was through a consultant UNICEF hired to assist the government in developing strategies and response plans. UNICEF also interacted with the government to support host communities. The evaluation team saw no documentation of WASH programme advocacy with the MoE during the period described by the scope of the evaluation. UNICEF Jordan has education programmes that may have had primary engagement with that government ministry. Still, a UNICEF partner reported that there is more demand for education services than the MoE can handle. UNICEF could work more to support the capacity of the MoE and should consider building those activities into its future strategy in this area.

136

EQ11: Was the infrastructure in the camps constructed in such a manner as to ensure long term functionality, and will continue to service the most vulnerable people even when UNICEF is no longer directly supporting? This should be investigated in terms of Water and Wastewater infrastructure.

Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection JC 11.1: Demonstration that - Evidence of quality material - Review of Partner PCAs - Onsite inspections infrastructure in camps was use. - Review of Contractor - Review of construction plans constructed with long term - Evidence that long term documentation and reports. functionality as a priority. sustainability was taken into - Interviews with engineering account. staff - Interviews with UNICEF JC 11.2 Degree to which contractors UNICEF balanced quality and price in construction decisions.

Following the emergency response phase of the programme, The infrastructure in the camps was constructed to ensure long-term, sustained functionality. The GoJ prohibited the construction of permanent structures at the beginning of the crisis, forcing UNICEF to make due with mobile sanitation and trucked in water. When it became apparent that the refugee crisis would not abate quickly, the government recognized the need for longer-term solutions and UNICEF provided lasting, sustainable solutions. UNICEF chose highly mechanized treatment systems for both Za’atari (trickling filter and MBR) and Azraq (the MBBR American system, which eventually failed). These systems are expensive, but the decision to use these systems was sound and valid. The national laws of Jordan specify stringent effluent quality from municipal wastewater sources. Even though camps are surrounded by vast unused parcels of government land, wastewater systems must comply with environmental standards. Also, the 4-month rainy season that makes it more difficult for passive wastewater treatment systems to produce quality effluent year round.

For Za’atari, the common septic tank and small diameter sewer is an appropriate and innovative wastewater collection model.

137

For Azraq, the installation of the MBBR system, translocated from Afghanistan and provided by US donors, turned out to be a poor choice, although the rationale for selecting it was valid. Unfortunately, the system did not meet the needs of Azraq’s inhabitants or the program in general. Wastewater strength was higher than anticipated. Following an expensive attempt to make it work, the system was eventually abandoned. Wastewater is now pumped and transported at great expense. There is a greywater management scheme that has yet to come on line fully, and until there is some centralized wastewater treatment option, septage will require hauling some 90 KM away. It is surprising that the Azraq camp would have more difficulty in establishing a wastewater system as it is neatly laid out, compared to Za’atari, which has developed an elegant and lower cost solution, which has evolved into something that could become sustainable over the long term as economic development in the camp increases. The water systems in the camps are designed, installed, operated and maintained to provide long term functionality, even if UNICEF were to leave, which is unlikely. The quality of the equipment is high quality, which adds to the longevity of the systems.

EQ12: To what extent did UNICEF take sustainability into account at the early stage for water, wastewater and solid waste management services (e.g. transition to local government institutions and local capacity)?

Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection JC 12.1: Evidence of - Strategic plans for - Workplanning - Interviews with UNICEF UNICEF’s Jordan sustainaibility. documentation. staff.

138

promotion of Sustainability - - Interviews with GoJ officials. Initiatives.

The evaluation team has no evidence that UNICEF has a transition plan to government or any other institution in place. The GoJ at all levels has made it clear that they will not take ownership or spend public funds on the crisis. The government understands that it is providing an international public service in using internal resources such as water to sustain the refugees and expects the international community to provide for the rest of their needs.

UNICEF’s approach to sustainability has three aspects. • Build high quality infrastructure: Building systems and infrastructure that have long term potential to serve Jordanians and the national government. • Leverage other, non-governmental resources to contribute to ongoing operations of the camps: This approach includes building the capacity of refugees to manage their own WASH issues and partnering with national entities to engage in management. • Become as efficient as possible: UNICEF continues to find ways to bring down costs so that resources are used as efficiently as possible, and available as long as refugees need WASH services.

EQ13: To what extent did UNICEF’s interventions increase the resilience of the Government and respective target populations? To what extent will UNICEF-supported projects (schools, ITSs, water and wastewater infrastructure, mobilisation) continue to be operational after the end of UNICEF’s financial support

Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection

139

JC 13.1: Evidence of planned - - Jordan Response Plan - Interviews with beneficiaries resilience in UNICEF - Partner descriptions of - Interviews with partner staff Interventions projects in PCAs - Focus group discussions This evaluation question is difficult to answer because it appears that target populations will require UNICEF’s financial support well into the future. UNICEF increased the resilience of supported populations and the GoJ in the following ways. • Designed projects in accordance with the JRP, linking refugee assistance in Za’atari and Azraq with resilience oriented projects that rehabilitate or extend systems in urban areas with large refugee populations. • Instituted consistent standards and policies around water access in camps to promote equitable access and reduce conflict. • Ensured that infrastructure was built using high quality materials. • Built capacity of stakeholders in camps to manage water, sanitation and hygiene within their communities. • Created mechanisms for hygiene and water conservation promotion that operate under the leadership of local stakeholders, such as WASH committees and the Lead Mother model in Azraq. • Improved water and sanitation infrastructure and services in some host communities with high refugee populations to reduce tensions. • Worked with the government to create Wash in School Standards to set a bar for water, hygiene and sanitation in Jordan’s often overburdened schools. • Transferred a process for creating response and water strategies to the GoJ that they can replicate in the future. • Promoted the use of data for strategy development at the MoWI.

EQ14: Does the MWI have the necessary mechanisms in place and capacity to ensure the water and sanitation needs of the most vulnerable people will be met as a result of UNICEF’s support and advocacy, if another large scale humanitarian crisis occurs in the future?

140

Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection JC 14.1: Evidence of - mechanisms in place at the Ministry level.

The evaluation team did not find evidence that MWI has the necessary mechanisms in place for a future, similar crisis. One reason for this is that the government does not want to own these crises or been seen as a refuge for at-risk populations.

JC 14.2:

EQ15: To what extent did UNICEF document its WASH interventions and make available to the sector learning on water, wastewater, solid waste management and mobilisation?

Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection JC 15.1: Evidence of the - Documentary evidence of - - Interviews with UNICEF documentation of lessons learned staff. interventions and their use - Evidence of propogation of - Interviews with partner for learning. materials. organizations. - Reports from annual reviews - Interviews with sector as per the Jordan Programme coordination group Document. participants.

141

Documentation of interventions for the purpose of sector learning was limited. Lessons learned were shared at coordination meetings. The evaluation team did not find evidence that the WASH programme documented its experience for the purpose of sector or institutional learning. This evaluation is the first attempt to document UNICEF’s experience implementing the WASH programme in Jordan.

EQ16: To what extent were environmental protection and sustainability taken into account in the design and implementation of programmatic interventions?

Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection JC 16.1: Evidence that - Completed environmental - Programme Reports - Interviews with partner staff. environmental protection assessments. - PCA Environmental - Interviews with donors. assessments were carried - Ongoing environmental protection descriptions - Onsite reviews out. protection efforts.

JC 16.2: Evidence that environmental protection

142

was built into programme designs.

Early in the programme, environmental protection was not taken into account sufficiently. Lack of focus on environmental issues led to Za’atari’s construction over an aquifer that accounts for 34 percent of Jordan’s water supply without proper facilities for managing wastewater and preventing it from leaching into the ground. As described in earlier sections of this report, the problem was exacerbated when open pits filled with graywater developed as camp residents began to create private sanitation facilities in their housing units. UNICEF corrected this issue in Za’atari by installing wastewater collection tanks, reaching 2,000 m3 of collected wastewater per day, utilizing backwash from borehole water filters for construction and cleaning purposes, and eventually constructing a water treatment plant. UNICEF applied the lesson it learned in choosing a site location and water system for Azraq. Prior to the construction of Azraq, UNICEF worked with the MoWI to conduct an environmental impact assessment. The assessment identified a high risk of underground water contamination. UNICEF incorporated protections in its design of Azraq’s WASH services. It commissioned construction and operation of containerized wastewater units. It also attempted to install a treatment plant and septage receival and dewatering units. As seen earlier in this report, the treatment plant ultimately did not work out, but the evaluation team recognizes that the decision to include it was appropriate at the time. The septage receival and dewatering units permitted the system to receive wastewater from trucks and bring down the cost of desludging operations. In 2016, a pond was also constructed at Azraq camp to store wastewater that could be used for agriculture. UNICEF also assisted the government in improving environmental protections as regards WASH services. For example, UNICEF built the capacity of the MoWI to conduct groundwater monitoring more effectively. Other initiatives that UNICEF implemented that took environmental protection and sustainability into account include: • Working with the Royal Scientific Society to study the most environmentally sensitive way to provide heading for camps. Providing heating with gas was chosen. • Promoting water conservation in camps and host communities through implementation partners. • Celebrating world events, such as World Environment Day. • Promoting recycling where feasible.

143

EQ17: Did UNICEF’s WASH response identify and reach the most vulnerable people? To what extent did specific interventions strike an equitable balance between interventions in camps/settlements and in host communities, to reach the most vulnerable people?

Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection JC 17.1: Sector satisfaction - Degree to which the most - - Interviews with WASH with UNICEF’s performance vulnerable’s needs were sector actors. as sector lead. covered. - Sector satisfaction with UNICEF’s achievements as JC17.2: Evidence that sector lead. UNICEF took the lead in sector coverage and ensured the most vulnerable’s needs were addressed.

As described elsewhere in this report, the evaluation team found that UNICEF ensured the needs of those in camps and at the settlements were addressed in a coherent manner. UNICEF organized sector coordination efforts and coordination efforts at the camp and settlement levels to design activities, assign responsibilities to avoid duplication, and set standards to ensure equity of results as far as that was possible. Following that, UNICEF worked in host communities and ITSs to the extent resources allowed. The evaluation team does not have evidence that UNICEF used its position as sector lead to make sure those in needs were identified and that their needs were addressed.

144

EQ18: Did UNICEF as Sector Lead ensure that the needs of those in most need were addressed in a coherent manner, particularly in comparison to alternative existing systems at the local or regional level (i.e. UNRWA camps)? If not, how?

Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection JC 18.1: Evidence that - UNICEF enhanced addressed needs in a coherent manner

EQ19: To what extent did UNICEF role as WASH sector lead enhance response processes and results to avoid duplication and gap in interventions?

Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection

145

JC 19.1: Evidence of - Identification of process - Vulnerability maps - Interviews with WASH processes that UNICEF improvements through - National WASH inSchools programme stakeholders. enhanced. documentation and strategy. interviews.

Regarding enhancing response processes, a sample of WASH sector meetings from May 2016 through April 2018 show 13 organizations regularly attending. Action points and focal points are thoroughly covered, and cover issues from The National Water Strategy to Schools Assessments and issues in the camps.

UNICEF also established a number of reporting mechanisms through third party monitors including water quality reporting, ad security incident reporting. UNICEF established the use of vulnerability Maps for Water and Sewage, though these have only been in use since 2017. The maps revealed Azraq as the most vulnerable place in both water and sewage. UNICEF partners that the evaluation team interviewed revealed partner satisfaction with UNICEF as sector lead. One partner said that, “UNICEF are very good coordinators. They always have staff in Azraq.” A donor said that they were very happy with UNICEF’s coordination and partnership. A staff member of one partner said, “UNICEF does a very good job of controlling the work. 95 percent approval rating for UNICEF.”

EQ20: Were partnerships effective and leveraged to the maximum extent to assist the most vulnerable population in camps/settlements, ITSs and in host communities? Judgement Criteria Indicators Sources of Documentary Methods and tools for data Information collection JC 20.1: Degree to which - Amount of partner in kind - Financial documents. - Partner interviews partners provide in kind contribution. - PCAs - Host community interviews contributions. - Degree to which reports - show effective collaboration.

146

JC 20.2 Degree to which vulnerable populations were served. l The evaluation team believes that partnerships were effective and leveraged to the maximum extent in the camps and settlements. As described in this report, UNICEF utilized its partners to ensure full coverage in the camps and compliance with standards. Based on the limited financial information that the evaluation team was provided, the team estimates that UNICEF also leverages over $27, 250,000 in in-kind contributions from the partners it worked with151.

UNICEF also effectively worked with partners in host communities. Partners were relied on to identify and manage key interventions. WASH in schools programs were also managed effectively through partners. One aspect of working with partners that UNICEF could have improved was communicating strategic decisions and medium-term vision for the project. The UNICEF partners that the evaluation team interviewed reported that they often heard about drastic changes to plans or operations with little notice or time to prepare. One partner noted “UNICEF take quick decisions – the change from Mercy Corps to ACF in Azraq could have been done in a better way. It was not a transparent process.”

151 Estimate based on a review of PCA’s during the life of the WaSH Programme

147

148

Annex H: UNICEF WASH programme Results Framework Outcome Indicators 2013/2014 2015-2017

National Institutions provide improved Quality WASH facilities are sustained and utilised, and health and neonatal care services and Objective Objective hygienic behaviours practiced, by the most vulnerable while quality early childhood care with a focus ensuring the protection of the environment on disadvantaged groups

Outcome Indicators Baseline Target Outcome Indicators Baseline 2015 - 2017 Target

Indicator 1: Existence of Indicator 1: Existence of a strategic planning capacity in comprehensive national Monitoring system Monitoring MoWI and sector and other monitoring system for school no clear strategy strategy in place needs improvement system upgraded relevant institutions in readiness to inform policy Emergency preparedness decision on child disparities and response

Indicator 2: Proportion of Indicator 2:Existence of a Target: 100% of 2015 population at camps and child and neonatal target 140,000 camps, No information Information host communities including 100,000 at camps, 225,000 at host information system in 1,550,000 host system existent system in place schools using improved communities and 188,000 students support of child and community, 200,000 drinking water as per Jordan maternal deaths audits students standards Indicator 3: Existence of Indicator 3: Proportion of updated and functional MoH 100% of 2015 target: population at camps and guidelines to focus on neo- 140,000 camps, 545,000 Not updated Updated host communities including 100,000 at camps, 575,000 at host communities and 180,000 students natal mortality and high risk host community, schools using improved pregnancies 200,000 students; sanitation facilities

Indicator 4: Proportion of Target: 100% of 2015 Indicator 4: % of health population practicing targets 140,000 camps, facilities providing neonatal 0 30% hygiene promotion at Baseline: 100,000 camps, 35,000 host150,000 communities, host and 188,000 students care services with the new targeted camps, host community, 200,000 MoH guidelines communities and schools students

149

Output Indicators

150

2013/2014 2015 2016/2017 Output 4.2: Institutions, utilities and Output 1.6: New Output: Support the Ministry of Water and Output 4.1: Government and WASH sector partners plan organisations provide equitable access to Irrigation and Sector partners to address the impact of and optimally utilize WASH resources and services for sustainable water and sanitation services for the refugee influx as well as chronic sector needs equitable access most vulnerable Output Indicators Baseline Target Output Indicators Baseline Target Output Indicators Baseline Target Comprehensive Fragmented WASH master Indicator 1: Availability WASH Indicator 1: Existence of WASH donor Indicator 1: Number of plan in place of effective planning, working Master 2016: comprehensive WASH plans, National people living in vulnerable aligned to monitoring and group, Plan/Strategy on 18,000 Master plan aligned to plan in place Households with access to National plan coordination scattered Humanitarian for 2017: National priority, NRP and (2022 strategy) improved household (2022) and mechanism for concept and development in 44,000 integrate Syrian Crisis not include WASH facilities and incorporate engagement of preliminary WASH, GIS/MIS Impact Syrian refugee services Syrian crisis government idea impact Impact Indicator 2 : % Reduction of water wastage/leakage 150,000 50 percent of to ensure targeted host Indicator 2: Number of refugees in the water 150,000 refugees Indicator 2: Number of 2016: population accessing Rreduce water population benefiting HC and supplied by the in HC and people with access to 600,000 sufficient quantity of waste/leakage from safe water and 225,000 Public water 1,400,000 host improved municipal water 2017: 3.2 water of appropriate by 20% storage facilities people in systems is Non- communities services million quality for drinking, improvement host revenue water. cooking and personal communities hygiene (2 cubic) Sanitation Indicator 3: % Increase of Coverage in targeted host population Jordan overall is 5% increased Indicator 3: Number of 2016: accessing sufficient using around 62 % Indicator 3: Number of 188,000 specifically in 200,000 students people with access to 400,000 appropriate sanitation while in students provided by students in the Northern in HC improved municipal 2017: 2.1 facilities Northern WASH services HC Governorate sewerage services million Governorates it is at the level of 37 %

Indicator 4: No of targeted KAP survey Host community Indicator 4: Number of Indicator 4: Number of 2016: population accessing available for (Nationally start population at host 150,000 Syrians in people with access to 60,000 hygiene promotion and Northern in 2 communities benefiting 575,000 at HC HC, 395,000 in HC improved WASH facilities 2017: hygiene education in Governorate Governorates) from sewage systems in institutions 114,000 schools and camps and 324 schools improvement

Indicator 5: Availability of Cost analysis Study None Evaluation study conducted (Y/N)

151

2013/2014 2015 2016/2017 Output 4.3: Urgent needs of vulnerable people Output 4.3: The most vulnerable women and IR 1.3: Relevant Core Commitments on WASH are met especially children are met through quality and children are aware of and practice key hygiene in all humanitarian settings lifesaving WASH services behaviours and practice water conservation IR Indicators Baseline Target Output Indicators Baseline Target Output Indicators Baseline Target Indicator 1: Number of WASH Indicator 1: Number refugees people in camps Indicator 1:Availability Working Formation of a 2016: provided with safe access to reached through of effective working group working group 100,000 140,000 50,000 sufficient water for drinking social/community group coordination functional for the sector refugees refugees 2017: and domestic use through mobilisation on key mechanism and ToRs response plan 50,000 water trucking WASH messages and developed water conservation

Indicator 2: Number of 345,000 Indicator 2:Number of people in institutions refugees (in emergency affected Indicator 2: Number of in camps reached 2016: camps and off population (male and refugees provided with safe 100,000 140,000 through social/ 20,000 250,000 camp) and female) provided with waste water disposal through refugees refugees community 2017: 180,000 access to drinking and dislodging of septic tanks mobilisation on key 20,000 affected domestic water WASH messages and Jordanians. water conservation

Indicator 3: Number of 345,000 Indicator 3: Number of people in the host refugees (in Indicator 3: Number of Emergency affected community reached 2016: camps and off refugees provided with safe population (male and 140,000 through 18,000 camp) and waste water disposal through 0 refugees female) provided with refugees social/community 2017: 180,000 provision of Waste Water access to soap and mobilisation on key 44,000 affected Network in Zaatari Camp other hygiene items WASH messages and Jordanians. water conservation

152

IR Indicators Baseline Target Output Indicators Baseline Target Output Indicators Baseline Target Indicator 4: Number of Indicator 4: Number of 345,000 people in institutions Emergency affected refugees (in Indicator 4: Number of in the host community 2016: population (male and camps and off refugees provided with safe 140,000 reached through 60,000 female) with access to camp) and 0 refugees solid waste management refugees social/community 2017: appropriately designed 180,000 facilities mobilisation on key 114,000 toilets and sanitation affected WASH messages and services Jordanians. water conservation Indicator 5:Number of Indicator 5: Number of 345,000 Indicator 5: Number of population (male and people in informal refugees (in refugees provided with safe female) covered settlements reached 2016: camps and off access to sanitation facilities by through face to face 100,000 140,000 through 15,000 19,300 camp) and operating and maintaining the (including HH and refugees refugees social/community 2017: 180,000 existing WASH Blocks through Group sessions) mobilisation on key 45,000 affected WASH Committees and/or hygiene promotion WASH messages and Jordanians. community involvement messages water conservation Indicator 6:Number of children (girls and boys) provided with access to safe water, Indicator 6: Number of refugee 19,108 45,000 sanitation and hygiene 72,985 230,633 Boys and Girls in camps students students facilities in their provided with WASH in schools learning environment and in child friendly spaces Indicator 7:Number of children (girls and boys) not practicing open NA 100% defecation in the camp settings and in host communities

153

2016/2017

Output 4.4: The urgent WASH needs of the vulnerable women and children are met through safe, predictable and sustainable water and sanitation services and facilities

Output Indicators Baseline Target

Indicator 1: Number of people in camps supplied with 2016: 140,000

water (tankering or network) 2017: 140,000

Indicator 2: Number of people in camps receiving 2016: 140,000

wastewater services (tankering or network) 2017: 140,000

Indicator 3: Number of people in camps receiving solid 2016: 140,000

waste collection services 2017: 140,000

Indicator 4: Number of people in informal settlements 2016: 15,000

with improved access to WASH facilities and services 2017: 45,000

Indicator 5: Number of people in institutions 2016: 20,000

benefitting from WASH facilities and services 2017: 20,000

154

Annex I: UNICEF Provided Financial Estimates Note: The following documents were provided after the evaluation period. Overview of examples of increased efficiency in the UNICEF Jordan WASH programme (June 2016, updated July 2016, updated October 2016)

Ongoing WASH services in Camps UNICEF has been providing WASH services to camps since July 2012. Once critical WASH services were established, UNICEF has consistently worked to reduce the cost associated with the provision of these life-saving interventions through the construction of infrastructure to reduce external tankering, establishment of third party monitoring mechanisms to ensure that only actual services delivered were paid for (verification of truck volumes, reduced recycling of wastewater etc) and the progressive rebidding of services to ensure that the unit costs are progressively reduced.

- Unit costs have been progressively reduced as outlined in the table (since 2012 or from the start of the service): Price Difference (JD/m3) % Reduction

Water Tankering - Za - internal 0.17 14% Water Tankering - Za - external 0.41 16% Water Tankering - Az - internal 0.12 8% Water Tankering - Az - external 0.2 6% Water Tankering - KAP - external 0.78 31% Water Tankering - CC - external 0.78 31%

Desludging - Za - internal 0.02 1% Desludging - Za - external 0.55 17% Desludging - Az - internal

Desludging - Az - external 0.61 10% Desludging - KAP - internal 0.73 29% Desludging - CC - internal 0.73 29%

Solid waste unit costs - Za 2.9 67% Solid waste unit costs - Az 8.46 85%

155

Solid waste unit costs - KAP 2.7 63% Solid waste unit costs - CC 2.7 63%

- Transitioned from external water sources to internal tankering (three wells in Za’atari and one well in Azraq) As the four boreholes are now operational, the water is supplied for most of the year from the internal boreholes with no need for external tankering. In Za’atari, Borehole 1 became operational in 2012, Borehole 2 became operational in 2013 and Borehole 3 in May 2015. In Azraq, the internal borehole started providing water to the camp in October 2015. Through these key infrastructural projects, the cost of supplying water has been reduced considerably For Za’atari, the difference between the highest external water tankering unit cost and the current price is 1.48 JD/m3. Likewise, for Azraq, the price difference is 2.05 JD/m3. On average, this equates to a monthly cost difference of 150,000 JD for Za’atari and 73,800 for Azraq (excluding the operational costs of the boreholes) - Improved Operation and Maintenance of the three boreholes has resulted in reduced downtime of the boreholes and their more consistent operation due to regular monitoring and maintenance (there was no daily external water trucking from November 2015 to May 2016). - Over the period July to November 2014, the average daily demand (including institutions) was 3,704 m3, of which 780 m3 (21%) was from Borehole 1, 1,160 m3 (31%) from Borehole 2 and the remaining 1,757 m3 (48%) was from external boreholes. Prior to 2015, there were no seasonal allocations and issues were regularly faced with Borehole 1 and turbidity. As a result, it was often necessary to further supplement the water supply with external water. Over 2015, the water needs at a block level were regularly recalculated (based on updated population numbers as well as a seasonal alteration to take account of increased needs in summer and lower needs in winter), and as a result, the volume of water required was recalculated and the volume reduced, over an annual period - UNICEF’s positive response to the water strike in May 2015 (alternative modes of delivery and the accelerated operation of BH3, as well as the construction of a mini network within a few days) sent a very strong signal to the Contractors and their ability to negotiate A mini-network was installed and this enabled 200 to 300m3/day to be delivered directly by pumping and as a result, there has been a reduction (of approx. 270 JOD/day, 8,100 JOD/month, 129,600 to date – the network cost less than 5,000 JOD to construct) - Construction of a connecting line to the main pumping distribution line in Za’atari as a Contingency supply (with the permission of the Minister of Water and irrigation) which can be activated in the event of an emergency - Mobilisation on water conservation to parents and children (to reduce children playing with the water) and reducing the losses – refugees have been engaged to disseminate messages at tapstands in Azraq where children often played with the taps, which has resulted in a considerable reduction in the water consumed, as well as the amount of standing water - Replacement of items which were prone to vandalism e.g. the ventilation pipes (triple bolted) and taps (taps removed and a basic design installed) which has significantly reduced the cost of replacement

156

There was a reduction of 58% in the costs for the replacement of taps, and 24% for the replacement of the ventilation pipes in Azraq - Transitioned from external desludging in Za’atari to the internal treatment plant and nearing completion of the Azraq wastewater treatment plant (unit transportation costs will reduce to less than one third of the external) The cost to manage 1 m3 of wastewater in Za’atari Camp has decreased from an estimated with the WWTP completion in February 2015 (the figures are currently being updated) - Extensive third party monitoring to accurately determine the volume of wastewater reported (physical inspection of the truck when it arrives and before it is offloaded) and to reduce the scope for ‘recycling’ of the waste See the figure below:

- Development of an Operational Plan to ensure the most efficient use of the desludging trucks (to ensure a smooth supply of wastewater to the treatment plant and to reduce the scope for external desludging) See the figure below:

157

- Expansion of the storage capacity at the Za’atari Wastewater Treatment to reduce the scope for external tankering The works to expand the storage facility (by 400 m3 storage + 200 m3 treatment capacity) cost approximately 250,000 JOD – this led to an additional 19,500 m3 of wastewater/month which could be treated at the plant (and not transported externally) which has led to a saving of c20,800 JOD/month on the tankering costs - Independent physical verification of the volume of the wastewater trucks – brought to a weighing station as often modifications have been made to the trucks (the registered volume was different to the actual measured volume) The actual measured figure was 27% but a compromise figure of 23% was agreed with the Contractor – this equates to a 23% saving in the monthly desludging costs for Za’atari - Construction of a weighbridge for the volume of the compacted solid waste in Azraq - Regular rebidding of contracts for improved unit rates as well as UNICEF’s Long Term Arrangements (as given in the first table) - Progressive reduction in Cash For Work (gradually reduced and in consultation with all agencies) while supporting alternative job opportunities (recycling) The first scale down of CFW took place in April 2016, which reduced the cost by about 20% with a second (20%) scheduled to take place from 9 October 2016 – these two reductions will reduce the monthly cost by approximately JOD 10,000 - Progression from partner implementation to UNICEF’s direct engagement of Contractors (networks, desludging, borehole operation and maintenance, construction etc) - The decommissioning of public WASH blocks has significantly reduced the cost of repairs, operation and maintenance. The installation of private toilets with no O&M costs for UNICEF and much appreciated by the refugees (significant cost reduction (over 50%) in the installation of private toilets due to the recycling/or reuse of materials from decommissioned public WASH blocks)

158

As of early October, 346 WASH Blocks have been decommissioned (out of 355) in the camp – the materials from the blocks are being recycled for Household toilets for vulnerable families, where possible - The phasing out of O&M of WASH facilities in schools and the subsequent engagement of another agency (by Education section) have led to reduced operational costs for WASH - Reduced cost of water quality monitoring due to internalized water supply - Functional water supply contingency plan have led to price stability; prices could have been most likely increased if we were more vulnerable It is hard to quantify this but the fact that alternative modes of transport and delivery were up and running within one day of the strike sent a very strong message, as did the construction of the tanks for the network – we consider that this ensured that the prices for the internal and external water did not increase in the subsequent bidding period - Reduced power consumption due to the installation of VFDs at the boreholes (installed on all four boreholes and already making a difference) This has just recently been introduced but is anticipated to have two impacts; a) on the energy consumed (c 7.5% for the reduction in the output) and b) more continuous water supply with less interruptions and shutting down/restarting - Construction of water reuse pond where contractors can access water for construction works at no cost. Otherwise, UNICEF would have been charged for construction water. The water was also used for cleaning of the water tanks, again contributing to reduced water needs During the peak of wastewater network construction, an estimated 56m3 (six trips of 8m3 truck) of water was used for construction, as well as for other purposes – on average, approximately 60m3 per day of water is collected and used, which has saved approximately 2,700 JOD per month - Reuse of water from the Wastewater treatment plant for agricultural purposes in Za’atari and under construction for Azraq Approximately 1000 to 1200 m3 of treated wastewater is produced by the Za’atari wastewater treatment plant, each day – if this had to be tankered to the site, the cost would be 25,000 to 49,500 JD/month, depending upon the source - Efficiency review of the energy requirements of all of the infrastructure (an initial review of the energy consumption of the Za’atari Wastewater Treatment Plant has been carried out) - Partner contribution to cooperation agreements (up to c10% but up to 70% in some agreements)

Za’atari Network Construction: UNICEF will directly engage the Contractors for second phase of Water and Wastewater for the Za’atari Networks - This will significantly reduce the number of staff maintained by partners and associated indirect costs and overheads, as well as reduce duplication in roles and responsibilities - UNICEF will combine the phase II water and wastewater network design and construction tender documentation, and will directly engage construction supervision consultants and construction contractor(s). This strategy will improve cost effectiveness through streamlining of the project management structure and reduction

159

in construction contractor costs through joint delivery of the networks. Joint delivery of the networks will also offer significant benefits to beneficiaries through reduced construction impact on communities, reduced construction safety and coordination risk, and improved construction quality outcomes It is estimated that by revising the project management structure and combining the water and wastewater work into distinct packages, the cost will be further reduced by an estimated 15-22% due to construction and management efficiencies. - The works have been designed to facilitate the operation of the water network in districts which will facilitate a gradual reduction in the costs of tankering It is planned that three districts will be connected per quarter from June 2017 (25% reduction per quarter) from when the pumping stations start to be commissioned – this will reduce the Za’atari costs by over 87,000 JD per quarter (for each three districts) - The costs to supply water across the camp via the network will eliminate the need for water tankering and as a result dramatically reduce the cost of water supply – directly (no costs for tankering) as well as indirectly (reduced monitoring, support staff, transportation, water quality monitoring etc) The elimination of tankering from Za’atari and Azraq will result in savings of over US$ 4M a year (over US$ 7M compared to external tankering) - Although there will still be a need for desludging of the wastewater network, the frequency and scale will be significantly reduced It is anticipated that there will be a reduction in the frequency of desludging from household tanks from once per week to once every 6 months (desludging from the communal tanks of the WASH blocks was daily) - Procurement of desludging trucks for the network desludging (once or twice a year) which can be done using our own trucks which we can procure The cost of desludging will be reduced by over US$ 1.5M/year - Bulk procurement of tanks for Phase 1 of the wastewater network (across all three partners) - Pipeline for Azraq – elimination of trucking - UNICEF’s Engineers oversight of the designs and revision of the BoQs Reduced reliance on partners and increased oversight of the technical requirements and strategic planning of progressive construction, commissioning and operationalization of the networks, will ensure that ongoing operational costs are progressively reduced as the camp transitions to a longer term sustainable operations and maintenance framework - Reduce the cost of the chemical used to treat the wastewater at Za’atari Wastewater Treatment Plant (ZWWTP) Review of the ZWWTP hydraulic and process efficiency in coordination with WAJ will ensure that plant energy and chemical usage requirements are optimised while meeting camp wastewater treatment requirements. UNICEF Engineers are working collaboratively with other agencies to investigate the potential for ZWWTP sludge to be used in a trial biogas generation facility that will process generated sludge (removing the need for external trucking and disposal of the sludge) as well as generate renewable energy for export to the local power grid and reduction in camp operational power costs.

Support to Host communities:

160

- UNICEF support the Water and Sanitation Vulnerability mapping to ensure that areas in most need were prioritised (based upon agreed criteria) - UNICEF supports the replacement of leaking water lines and inefficient equipment to reduce the operational costs in water systems - UNICEF supports the capacity building of operators of the utility companies to undertake maintenance and strengthened oversight of the equipment - UNICEF supports the repair, unblocking and replacement of sewer lines to reduce sewerage overflows

161

Annex J: Additional Financial Information The table below summarizes additional financial information provided by UNICEF in December 2018, after the evaluation period concluded. The table below summarizes UNICEF WASH programme reported expenditure by category for the years 2012 through 2017. The table include categories that accounted for at least 1 percent of total expenditure. These categories account for 96 percent of total expenditure. The top expenditures were for the categories “Sanitation and Solid Waste Management, and “Provide Access to Safe Drinking Water,” accounting for 35 percent of the total, though there are many categories with similar titles on the list. Items USD Percent SANITATION AND SOLID WASTE $47,113,983 18% MANAGMENT PROVIDE ACCESS TO SAFE DRINKING WATER $44,568,288 17% PREPAREDNESS & CONTINGENCY PLANNI $22,224,107 9% PROVIDE REFUGEES IN CAMPS WITH SAF $15,547,325 6% CONSTRUCT/EXTEND WATER NETWORK IN $13,650,381 5% PROVIDE DRINKING WATER-CAMPS $10,147,397 4% CONSTRUCT WASTEWATER NETWORK IN ZA $11,457,662 4% SEWAGE NETWORK-CAMPS $7,409,768 3% HYGIENE EDU AND MATERIALS $6,011,048 2% SOCIAL MOB IN CAMPS (GENERAL) $5,590,211 2% WASTEWATER DISPOSAL-CAMPS $4,210,105 2% SOLID WASTE MANAGEMENT-CAMPS $4,418,779 2% WASTEWATER COLLECTION (DESLUDGING) $4,806,943 2% COLLECT AND DISPOSE SOLID WASTE $4,997,696 2% TECH ASSIST FOR SUSTAINABLE W $4,732,505 2% PROVISION OF SANITATION AND HYGIENE $2,074,047 1% MERCY CORPS PCA/0/12 $1,625,658 1% ACCESS TO SAFE WATER $1,637,291 1% WASH SUPPLIES $3,408,290 1% WASH IN SCHOOLS $3,561,058 1% WATER SYSTEM REHABILITATION $1,715,156 1% SEWAGE NETWORK REHAB-HOST COMM $1,615,982 1% HYGIENE PROMOTION-CAMPS $3,152,868 1% WATER SUPPLY NETWORK-CAMPS $3,607,696 1% WASTEWATER TREATMENT-CAMPS $1,322,810 1% OPERATE/MAINTAIN WASH FACILITY $2,732,411 1% O&M WASTEWATER TREATMENT PLANTS IN $2,788,510 1% PROVISION OF WASH SERVICES IN ITS $3,037,358 1% WASH FACILITIES IN INSTITUTIONS I $2,339,079 1% WATER INFRASTRUCTURAL SUPPORT $3,060,999 1% WASH FACILITIES IN SCHOOLS, CFS, C $1,627,130 1% Total $246,192542 96%

162

163