Evaluation of the UNICEF Albania Country Programme 2017-2021
Final Report December 2020 Disclaimer: This is a publication by the independent evaluation team of IOD PARC. The analysis and recommendations of this report do not necessarily reflect the views of UNICEF. Readers are encouraged to use material from this report for their own publications. As copyright holder, UNICEF in Albania requests due acknowledgement and a copy of the publication.
For further information, please contact: UNICEF Albania Country Office Skenderbej Street, UN House Bld, 3rd floor Tirana, Albania Telephone: +355 4 45 48 400 Email: [email protected]
Suggested Citation of the Report: O’Callaghan, B. Duci, V. Kacapor-Dzihic, Z. (2020) “Evaluation of the UNICEF Albania Country Programme 2017-2021”. UNICEF / IODPARC / IMC Worldwide. Evaluation of the UNICEF Albania Country Programme 2017-2021
Final Report December 2020 Acknowledgements
Sincere thanks to Ashley Wax at UNICEF Europe and Central Asia Regional Office, Dr Roberto De Bernardi, UNICEF Albania Country Representative, Elda Hallkaj and the staff members of the UNICEF Albania Country Office, as well as the members of the Evaluation Reference Group for providing the evaluation team with feedback, information and support throughout the evaluation period.
The evaluation was undertaken by IOD PARC’s consortium consisting of IMC Worldwide, Child Frontiers, Development Analytics and M-Vector. The evaluation team members are Brian O’Callaghan (Team Leader, IMC Worldwide), Veronika Duci and Zehra Kacapor-Dzihic. Additional research was carried out by Joy McCarron. The evaluation was supported by Project Managers - Karolina Mclellan and Elisa Radaelli. The evaluation direction and quality assurance were provided by Jo Kaybryn.
Contents
Acronyms and Abbreviations vi Executive Summary viii Overview of the Country Programme viii Purpose of the Evaluation and its Audiences viii Evaluation Methodology ix Key Findings ix Relevance ix Effectiveness x Efficiency xii Sustainability xii Synergies and coherence xiii Conclusions xiv Lessons Learned xvi Recommendations xvi
1. Introduction 1 1. 1 Purpose, objectives and scope of the evaluation 1 1. 2 Country context 3 1.2.1 Geographic and demographic context 3 1.2.2 Political context and government priorities 4 1.2.3 Social development and gender profile 6 1.2.4 Children, youth and families 7 1.2.5 Governance and civil society 9 1. 3 Country Programme components and goals 9
2. Methodology 12 2.1 Evaluation framework 12 2.1.1 Departures from the Terms of Reference 12 2.2 Data collection, data analysis and sampling strategy 13 2.3 Limitations and constraints faced by the evaluation 14 2.4 Management of the evaluation 15 2.5 Ethical considerations 15
3. Findings of the Evaluation 17 3.1 Relevance 17 3.1.1 Evaluation Question 1 UNICEF’s strategic positioning 17 3.1.2 Evaluation Question 2 Implementation strategies 23 3.2 Effectiveness 26 3.2.1 Evaluation Question 3 Organizational structure 26 3.2.2 Evaluation Question 4 Achievement of results 30 3.2.3 Evaluation Question 5 Gender mainstreaming 52 3.3 Efficiency 55 3.3.1 Evaluation Question 6 Resourcing 55 3.4 Sustainability 61 3.4.1 Evaluation Question 7 Long-term alignment of programming 61 3.5 Synergies/Coherence 64 3.5.1 Evaluation Question 8 Internal coherence 64 3.5.2 Evaluation Question 9 Coordination 66
4. Conclusions 70 5. Lessons Learned 74 6.Recommendations 76
Annexes 80 1. Terms of Reference 2. Reconstructed Theory of Change/Evaluation Matrix 3. List of Interviewees 4. Documentary Evidence 5. Data Collection Instruments 6. Evolution of UNICEF Albania CPD Priority Results Areas 7. UNICEF Albania Programme Locations 8. Gender Integration and Targeting in AWPs, COARs, RAMs and SMQs 2017–2019 9. Results Reporting Analysis 10. UNICEF Albania Stakeholders 11. Evaluation Reference Group Members 12. Human Rights Analysis 13. Evaluation Team
List of Tables
Table 1: UNICEF Albania country programme outcomes and outputs 2017–2021 11 Table 2: Progress in realization of child rights reported by UNICEF stakeholders 31 Table 3: Summary of UNICEF Albania results assessment module reporting against indicators 32 Table 4: Selected UNICEF Albania results assessment module indicators 33 Table 5: UNICEF Albania expenditure by programme area (2017–2019) 56 Table 6: UNICEF Albania donor funding in 2017–2019 58 Table 7: UNICEF Albania financial breakdown by partners 60
List of Figures
Figure 1: Map of Albania 3 Figure 2: Young persons’ perspectives on rights in Albania (U-Report Survey 2020) 35 Figure 3: UNICEF Albania planned vs. utilized annual budgets 2016–2019 56 Figure 4: UNICEF Albania regular resources vs other resources vs management/admin 56 Figure 5: UNICEF Albania budgets 2017–2019 by programme outcome 57 Figure 6: UNICEF Albania financial allocations by partner type 60 Acronyms and Abbreviations
C4D Communication for Development CEDAW Convention on the Elimination of all Forms of Discrimination Against Women CFC Child Friendly City CFS Child Friendly Spaces CO Country Office (UNICEF) COAR Country Office Annual Report COVID-19 Coronavirus Disease 2019 CP Country Programme CPD Country Programme Document CPE Country Programme Evaluation CRC Committee on the Rights of the Child CRM Child Rights Monitoring CSO Civil Society Organization DAC Development Assistance Committee DaO Delivering as One DCM Decisions of the Council of Ministers DFID Department for International Development DHS Demographic and Health Survey DRR Disaster Risk Reduction ECARO Europe and Central Asia Regional Office (UNICEF) ECD Early Childhood Development ER Emergency Response ERG Evaluation Reference Group ESRC Economic and Social Research Council EU European Union EUROSTAT Statistical Office of the European Union EVAW End Violence Against Women FP Focal Point GAP Gender Action Plan GDP Gross Domestic Product GPR Gender Programmatic Review HACT Harmonized Approach to Cash Transfer HCT Humanitarian Cash Transfers HDI Human Development Index IFI International Financial Institution ILO International Labour Organization INSTAT Instituti Shqiptar i Statistikës (Albanian Institute of Statistics) IOM International Organization for Migration IR Inception Report JWP Joint Working Plan KAP Knowledge, Attitudes and Practices LNB Leave No One Behind M&E Monitoring and Evaluation MAPS Mainstreaming, Acceleration, and Policy Support MCH Maternal and Child Health MESY Ministry of Education, Sports and Youth MoHSP Ministry of Health and Social Protection of the Population MNCH Maternal, Newborn and Child Health Care NSDI National Strategy for Development and (European) Integration NGO Non-governmental Organization OECD Organisation for Economic Co-operation and Development OOSC Out-of-School Children OR Other Resources PF4C Public Finance for Children PoCSD Programme of Cooperation for Sustainable Development PSS Psychosocial Support RAM Results Assessment Module RO Regional Office RR Regular Resources SDGs Sustainable Development Goals SitAn Situation Analysis SMQ Strategic Monitoring Questions SMR Strategic Moment of Reflection SRH Sexual and Reproductive Health TB Tuberculosis ToC Theory of Change ToR Terms of Reference UMIC Upper Middle-Income Country UN United Nations UNDAF United Nations Development Assistance Framework UNEG United Nations Evaluation Group UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund WB World Bank WHO World Health Organization 10 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
EXECUTIVE SUMMARY FINAL REPORT 11 12 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
his report presents the findings of the contributions to realization of child rights in Albania, evaluation covering the entire UNICEF portfolio including inequities and gender; and during the first four years (2017- August 2020) 3. Identify good practices and lessons and draw T of the five-year programme as articulated in its forward-looking recommendations that can inform 2017–2021 Country Programme Document (CPD), and the development of the next country programme. successive implementation strategies derived from this. The intended primary users of this evaluation are the UNICEF Albania Country Office and the Government Overview of the Country of Albania, as well as other United Nations agencies, Programme UNICEF partners, and other UNICEF offices in the region, including the UNICEF Europe and Central Asia With the initial budget of US$ 16,345,000, and in Regional Office (ECARO). partnership with government ministries, agencies and departments at national, regional and district level, the overall goal of the Albania Country Programme by 2021 Evaluation Methodology is “the progressive realization of rights of all children in Albania, accompanied by the reduction of equity The evaluation Terms of Reference proposed nine key gaps, to be based on effective child-focused policies, evaluation questions (EQs), focusing on the OECD systematically enforced accountabilities with adequate Development Assistance Committee (DAC) criteria of public resources, and strengthened respect for and relevance, effectiveness, efficiency, sustainability and protection of children’s rights”. synergies and coherence. In the inception phase, some revisions were made to the evaluation sub-questions to With an equity focus on vulnerable and marginalized streamline data collection and analysis, with review and children (for example children with disabilities, from approval from the Evaluation Reference Group (ERG). ethnic minorities and/or out-of-school), the intended long-term impact of the country programme was that The evaluation matrix was developed around the all boys and girls in Albania have equitable access to evaluation questions and criteria to map out the data quality social services and that appropriate assistance sources, methods and analysis approach. A number is provided to their families to reduce the negative of assumptions for each question were also formed to outcomes of deprivation and vulnerability. reflect the elements of the question. An overarching programme theory of change (ToC – also referred to in The country programme (CP), as originally articulated the report as the “reconstructed” ToC) was developed within the CPD, is organized around three main that combined and summarized individual outcome- outcomes: level theories of change.
1. Strengthened governance for equity and social The evaluation used a theory-based, non-experimental inclusion; approach. The primary evaluation methods included 2. Protection and access to justice for children; and the wide-ranging document review and analysis of 3. Programme effectiveness. administrative data, key informant interviews with UNICEF and stakeholders, as well as a quantitative survey through U-Report, an online social media-based Purpose of the Evaluation and data collection system implemented by UNICEF. its Audiences Stakeholders for interview were selected from five The evaluation serves both accountability and learning municipalities of Albania where UNICEF is active. purposes, with primary emphasis on learning. As Selection was based on several programmatically defined by UNICEF in the terms of reference, the relevant criteria, such as: Country Programme Evaluation had three objectives: a. UNICEF activity within the current CPD time 1. Assess UNICEF’s strategic positioning and frame; programmatic choices made, given government b. Level of UNICEF engagement across the priorities and the changing socioeconomic and programme outcome areas; political context in Albania; c. Presence of marginalized/vulnerable groups; and 2. Provide an independent performance assessment d. Key health/demographic indicators (families of the Albania Country Programme 2017–2021 receiving economic aid or welfare, child mortality, seen in relation to expected results and UNICEF’s children with disabilities). FINAL REPORT 13
Consultations with the Albania Country Office led to EQ2. identification of approximately 120 internal and external Were the implementation strategies utilized stakeholders for interview during the data collection relevant for the context, and the way it has period. To incorporate the views of Albanian youth, evolved? a structured online survey was distributed through U-Report. Data collection took place between mid- Given the current context for children’s rights, June and mid-August 2020, with analysis and reporting to what extent should these implementation taking place between August and October. strategies be continued and/or refocused?
Finding 2.1 Key Findings Initially implementation strategies were derived from the CPD, but divergence over time was not RELEVANCE documented/justified in subsequent planning – there was no overarching theory of change (ToC) for the CPD, EQ1. and the Programme Strategy Note which included To what extent has UNICEF’s strategic positioning programme outcome ToCs was not fully aligned with remained fit-for-purpose to advance child rights for all children, considering the context? the CPD, leading to lack of clarity around programme plans. (Context should consider children’s needs, national priorities, country and regional (Balkan) Finding 2.2 situation, and partner landscape.) Annual review processes led to a more ‘organic’ approach to deciding on implementing strategies Finding 1.1 based on identified priorities. This approach enabled UNICEF to ensure programmatic relevance but lacked The evaluation found good evidence of the relevance systematic documentation or justification of its rationale of UNICEF Albania’s strategic positioning at the start for decisions. of the programme cycle. A substantial range of data (surveys, studies, evaluation reports, etc.) were used Finding 2.3 by the UNICEF Country Office to inform the design of the CPD and as a basis for appropriate downstream Priority programme results areas evolved year-on-year programming priorities. Justification for programme over the course of the country programme in response priorities could have been further strengthened by wider to the prevailing context in terms of perceived needs consultations with partners and children, bottleneck and available resources. and human and child rights/gender analyses, and a formal evaluation of the previous country programme. Finding 2.4
Finding 1.2 Programming choices and implementation strategies were well-aligned with both UNICEF’s mandate as UNICEF updated its programming strategies over articulated in the CPD and associated plans, and the the course of the CPD via annual reflection exercises priorities of (albeit not all) stakeholders. and the preparation of annual plans that documented consideration of the context and needs. However, more formalized UNICEF approaches, for example a EFFECTIVENESS Strategic Moment of Reflection or Mid-Term Review, EQ3. were not undertaken. The country programme would To what extent is the organizational structure vis- have benefited from such exercises by facilitating a à-vis programme design fit-for-purpose to deliver clearer rationale and justification for decision-making results for children? over the course of the programme period and providing an opportunity for more systematic inclusion of a wide Could any changes be made to increase effectiveness? range of stakeholders in consultations. 14 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
Finding 3.1 Finding 4.2
Evidence indicates that the organizational structure as Quantitative evidence of the outcomes of UNICEF’s of 2017 was systematically planned to appropriately initiatives on children is lacking, notably around key align with the CP and the results areas. The office developmental indicators, which lack recent data. structure, including the introduction of specific positions Evaluation evidence from youth (via a U-Report immediately prior to development of the 2017–2021 survey commissioned for this evaluation) suggests CPD, was designed to fit and serve the new CPD both that realization of rights has stalled or is not meeting organizationally and programmatically. expectations among young people.
Finding 3.2 Finding 4.3
Evidence suggests the organizational structure is fit UNICEF responses to crises have been prompt, timely and for purpose overall given the context, with some areas targeted, informed by assessments and in collaboration requiring consideration such as the ratio between with diverse stakeholders, although not directed by a pre- permanent and temporary consultants which may have established crisis emergency preparedness plan. disadvantages in terms of the workload for core staff. This in turn may limit the capacity to make changes Finding 4.4 or strengthen the programme to deliver, for example, through strategic intersectoral working or embedding There is considerable uncertainty around the future gender mainstreaming. trajectory of COVID-19 and its impact on both the rights of children and the operating environment for UNICEF. Finding 3.3 Nonetheless, some future directions of programming can be predicted with some reliability, for example There were no significant changes to the organizational increased emphasis on online activities (education, structure during the programme period. While the work, communication), socioeconomic challenges, organizational structure remained broadly relevant to social care and protection challenges. the country context, there were shifts within UNICEF globally. There is no evidence that these changes Finding 4.5 were reflected (or even considered) in country priority decision-making and therefore, these institutional While many resources are being applied to data (internal priorities appear to have had no impact on adapting the reporting, Demographic and Health Survey/DHS, programme design and the corresponding organizational child rights monitoring/CRM), internal data systems structure. incompletely track progress, both with respect to high- level outcomes/impact indicators (many of which lag by several years) and to ongoing reporting of data via results EQ4. assessment modules (RAMs) and strategic monitoring To what extent were programme results questions (SMQs). Similarly, external data sources are achieved? What were the key factors (internal and incomplete or out-of-date and cannot address the shortfall. external) influencing the achievement of results? This impedes the Country Office’s capacity to report results and justify programmatic choices adequately. EQ4a. How has the CP contributed to the observed progress in the realization of child rights? Finding 4.6 Particularly the most vulnerable boys, girls and youth at risk of being left behind? UNICEF has made significant and important contributions to the improvement of governance and policies for education, child protection and social protection. Finding 4.1 Finding 4.7 There are strong, albeit anecdotal, perceptions among informed stakeholders that UNICEF has contributed to The strong contributions on the policy/governance an improvement of the situation of children across the side are not yet matched by government performance dimensions of social and child protection, governance and in the operationalization of these achievements. The policies and in services. Many of these improvements need for greater government resources, leadership and are reported to be of specific benefit to vulnerable and accountability for delivering on policy, particularly at the marginalized children. local level, is potentially an advocacy target for UNICEF. FINAL REPORT 15
Finding 4.8 SUSTAINABILITY EQ7. In line with its planned strategies, UNICEF has also To what extent has gender been effectively supported capacity and standards among service mainstreamed in the CP design and providers and other partners in the country via the technical implementation and, if not, how should it be done? expertise, tools and policies, but this is not commensurate with the scale of ongoing needs for improved quality and quantity of service provision in Albania. Finding 7.1
UNICEF implementation strategies within the sectors EQ5. of engagement (education, social protection, health To what extent has gender been effectively and child rights monitoring) have been well-linked to mainstreamed in the CP design and implementation and, if not, how should it be longer-term national development strategies some of done? which were designed with UNICEF support. UNICEF Albania’s work is well integrated into the wider One UN programme. Finding 5.1 Finding 7.2 The Country Programme Document makes reference to the principle of gender equality, yet there is little UNICEF engaged in a number of sectors in response to evidence that gender equality was robustly integrated. government priorities. Its support brought sustainable results in strengthening the policy and normative Finding 5.2 framework, in particular for education and social protection. The foundations for future enhancement of the fulfilment While the Country Office has taken on board some of child rights were laid through strengthening the of UNICEF’s gender guidance, notably appointing a capacities of national stakeholders, and the establishment gender focal point, and conducted some work with an of institutional mechanisms and models. Overall, Albania’s explicit gender dimension (e.g. child marriage research) drive towards European Union (EU) accession has led to operationalization of the gender equality approach is a faster pace of reforms in certain sectors where UNICEF limited. The CO is more ‘gender-blind’ than ‘gender- can support the reforms and thus the achievement and focused’. sustainability of results.
Finding 7.3 EFFICIENCY Investment in strengthening legislative and policy EQ6. framework, along with investment in modelling To what extent are the resources (financial and innovative approaches, have contributed to better human resources) allocated by the Country Office appropriate to support the implementation of performance by local and central actors, particularly strategies and achievement of CP results and, through support to developing policy and legislation. if not, what could be done to ensure resources However, the lack of resources and reprioritization of match programmatic ambitions and needs? budgets, particularly at local level, has created risks for comprehensively translating policy and legislation into implementation, and even the potential for a reversal in Finding 6.1 gains made in relation to capability strengthening.
The Country Office has applied an efficient and effective resource mobilization strategy, with most of its year-to- SYNERGIES AND COHERENCE year needs met from a variety of sources. EQ8. To what extent do the individual CP components Finding 6.2 and implementation strategies demonstrate internal coherence and synergies i.e. Several key factors support UNICEF’s overall efficiency interlinkages and reinforce each other to fully in delivering results, such as maintaining a small leverage the contribution that UNICEF makes number of core consistent partnerships which can towards expected results? be considered a strength by reflecting investments in longer-term relationships for realizing rights. 16 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
Finding 8.1 Conclusions
Intersectoral coherence and synergies within UNICEF UNICEF’s strategic positioning. UNICEF Albania has programming were limited, reflecting the prevailing placed itself strategically as a trusted partner of the dynamic of government mechanisms. UNICEF Government but also of international, national and local faces limitations to its effectiveness by mirroring partners. It is recognized as a convenor and on the this rather than encouraging intersectoral linkages whole this is highly appreciated by all stakeholders. across government. Better horizontal integration is While UNICEF’s strategic positioning has, overall, been necessary to ensure a holistic approach to tackling the well aligned with needs, especially in response to the multidimensional needs of children, particularly those Albanian Government, it would be better placed to justify of most excluded groups (e.g. interlinkages between its programmatic decisions and priorities if it introduces social welfare, health and education). UNICEF has more systematic processes. This would provide the demonstrated the effectiveness of intersectoral working necessary check that all emerging data and evidence through discrete initiatives but there are opportunities inform decisions and that an appropriately wide range of to build on the limited mechanisms that are in place stakeholders are consulted. for aligning multi-sector policymaking to leverage more comprehensive and coherent approaches to realizing Implementation strategies. The choice of child rights. implementation strategies to deliver on the programme components was logically derived from the CPD. In examining the ways in which the Country Office EQ9. worked within its sectors, staff applied a wide range of To what extent is UNICEF coordinating with approaches. There was no evidence that the Country development partners and other UN agencies Office had wrongly identified strategies, and indeed to avoid overlaps, leverage contributions and stakeholders appreciated the high level of consistency catalyse joint work? and welcomed the way in which UNICEF approached issues and worked towards achieving its aims. However, the Country Office’s own rationales for strategies were Finding 9.1 less well documented. A missing component was an overarching theory of change for the programme UNICEF has a strong mediation (convener) role which which would support decision-making on programmatic helps to establish and maintain partnerships with choices and guide the most appropriate implementation national (government and public sector institutions, civil strategies. Introducing more systematic processes of society) and international actors (donors, development reflection would also allow the CO to appropriately adapt partners). Partnerships with national and international its programme priorities and implementation strategies stakeholders are generally positive and productive according to the changing context and new evidence. although UNICEF could have done more to nurture This would facilitate the implementation of policy, for relations with the EU. example in the context of decentralization.
Finding 9.2 Organizational structure. The organizational structure of the CO followed the CPD strategic outcome areas and Despite a well-embedded “Delivering as One” was designed to be fit for purpose to deliver results. The philosophy and good participation in the United Nations structure works well ‘on paper’ but the small number of Development Assistance Framework (UNDAF), there is permanent staff undertake significant responsibilities some evidence of competition within UN agencies. Joint including delivering results and managing temporary projects steered through the Sustainable Development consultants (these reached a peak during the programme Goals (SDG) Fund offer the possibility to overcome period of outnumbering the core staff by four to one). The partnership challenges. imperatives for strengthening UNICEF’s effectiveness in areas such as increasing intersectoral working, Finding 9.3 mainstreaming gender, and incorporating new priorities issued by UNICEF HQ, are inevitably hindered by the Private sector collaboration is nascent, emerging, but existing pressures and workload on the core staff. needs to be more structured and systematic to reflect the ambitions articulated within the CPD. Achievement of results in child rights, especially for the most vulnerable. There is good qualitative evidence to suggest that the work of UNICEF over the past three years has led to changes in the realization of children’s FINAL REPORT 17
rights. In terms of the most vulnerable, UNICEF has Efficiency. In terms of achievements, to date the Country focused its efforts on particularly at-risk children such Office has relatively successfully secured and utilized as those in institutions and children of Roma families. resources according to budgetary plans, even in the UNICEF has also responded rapidly to crises such as face of a challenging fundraising environment. The ratio the earthquake in 2019, and more recently COVID-19. of regular resources to other resources has not matched Feedback from stakeholders strongly supports the that planned in the CPD, but this has been a response to perception that the situation for children is improving in the dynamic nature of funding within the country rather Albania. However, robust quantitative data to evidence than any financial management issue. Funding sources this is severely lacking and in general disaggregated data are reported as secure in the short term (through to the (sex, age, vulnerabilities) is missing. In order to identify end of this country programme cycle), but the Country appropriate priorities, UNICEF requires data to guide its Office’s conservative approach to resource planning response to the most pressing needs, including those of is important to ensure that medium-term uncertainty the most vulnerable. Key indicators to measure where is mitigated. UNICEF’s human resources seem well progress is taking place or stalling are also needed. allocated and efficiently utilized. However, the ratio of core This is particularly important as the country implements to temporary consultants remains a problem. Partners its decentralization process and UNICEF works with expressed strong satisfaction with the openness of staff municipalities as well as national Government to ensure in responding to their programmatic needs with technical the downstream implementation of policy gains. expertise. Although responsiveness to the needs of key partners can be an important and positive driver, it can Contribution to improving the performance of also rapidly absorb the capacity of a small office and thus government institutions and service providers. compromise UNICEF’s core mandate to focus on the UNICEF has contributed to a range of robust and well- practical realization of children’s rights. designed policies and laws that move the child rights agenda forward in Albania. This represents a culmination Sustainability. When measured in terms of the CO’s of efforts by UNICEF over the past decade in line with alignment with national development strategies and Albania’s preparations for EU accession over a similar Agenda 2030, UNICEF has demonstrated approaches time frame. As such, UNICEF has successfully placed which promote sustainability in the sectors of itself in a key position at a key time in the history of child protection, social protection and education. A the country to leverage its position to advance the major factor in this, is UNICEF’s investment in the agenda of children’s rights. Similarly, UNICEF has enabling environment through its direct contribution been a fundamental partner in building the capacity to development of such policies and legislation. The of many sectoral actors in a range of areas that will subsequent step, after developing laws and policies, is potentially contribute to the downstream realization to support their implementation at subnational levels. of children’s rights. However, there are substantial UNICEF has already positioned itself to engage with challenges in ensuring and, importantly, demonstrating, municipalities and has had positive results in convening implementation of these policy and legislative changes multisectoral actors, for example through the child- via appropriate advocacy for resource allocation, capacity- friendly city initiative. But its experience and feedback building at subnational levels, and effective and timely from multiple stakeholders show that the decentralization monitoring of the realization of children’s rights. process poses further challenges as there is increased competition for scarce budget allocations. The extent Gender mainstreaming. This is clearly a gap within to which UNICEF is planning for this new context was the country programme. There were some important not strongly evident throughout the evaluation and the initiatives such as the qualitative knowledge, attitudes CO should not underestimate the challenges of seeing and practices (KAP) survey on child marriage, but overall through the implementation of policies. there was no systematic approach to gender equality principles. Staff themselves cited a lack of capacity and Further sustainability strategies include UNICEF’s linkages expertise. It is essential that gender equality and gender with other actors. In many areas this is highly visible, mainstreaming components become embedded into the and yet there are continuing demands from partners for thinking, planning and implementation of the country the CO to engage more cooperatively with national civil programme across all the staff. Gender mainstreaming society organizations (CSOs). As part of strengthening is everyone’s responsibility and not the sole domain of the national capability these stakeholders represent an the Gender Focal Point. UNICEF as an organization has important group in fostering further collaboration. Finally, expertise, and multiple analysis and planning tools that there has been some small but important engagement the CO can and should draw on. with the private sector and there are opportunities to capitalize on this. 18 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
Internal programme coherence. There were two good practices: Where an initiative has been successful, examples of intersectoral work being integrated into the further scale-up to expand UNICEF’s reach is a worthy Albania Country Office structure (Child Rights Monitoring investment, albeit with due consideration to UNICEF’s Specialist and Communications Officer) but otherwise own capacities and those of the different stakeholders, the CO generally works in silos. The sector-specific focus particularly of central and local government. aligns well with the way the Government works but it is not necessarily desirable, and UNICEF may be missing opportunities to increase its effectiveness through complementary, synergistic and joint ways of working. Recommendations Two key areas, early childhood development (ECD) and gender, offer opportunities to create synergies and Strategic recommendations 1 to 5 are of high to complementarities across programming. The CO can medium priority and they are intended to inform the achieve this internally within the programme, although overall country programme direction for the next the external limitations in terms of working intersectorally cycle. Recommendations 6 & 7 are of lower priority, with the Government are recognized. Multi-sector but still remain important to be implemented. coordination of policy development and implementation are limited in terms of the Government’s mechanisms, but UNICEF has the potential to add significant value if it RECOMMENDATION 1 can increase its own and the Government’s intersectoral approaches to promote more holistic responses to child Introduce a systematic approach to country rights and development. programme priority setting that is in line with an equity-based theory of change of the new Country Coordination with development partners. Coordination Programme Document and informed by evidence. with development partners is an area of strength for the Ensure regular periods of reflection. Country Office with multiple lines of inquiry evidencing and confirming UNICEF’s effective partnership and Programme priority setting should be rooted in the convening role. There are areas that UNICEF needs to governing theory of change and informed throughout make decisions on regarding where its role is to lead, and the programme cycle by robust evidence that supports what it contributes to. The expectation of development adaptive management. partners that UNICEF would take a stronger role in certain areas (e.g. on responses to Roma children and Programme planning should incorporate outcomes children with disabilities) is evidence of both UNICEF’s of needs assessments, existing or emerging UNICEF leadership role and also the lack of national coherence global or regional strategies or guidance, identification and response by development partners. Certain overlaps of synergies within the Country Office, clear and logical and competing mandates are noted, such as in trafficking outcomes, outputs and activities to equitably realize (UNHCR/International Organization for Migration-IOM), rights for children in Albania and ‘leave no-one behind’. and gender (UNFPA/UN Women), and UNICEF again Regular moments of reflection should be conducted needs to determine where and how its skills can best to ensure that programme direction and intervention be used. There are also opportunities in other areas (for strategies remain relevant or are adjusted according to example with other UN agencies, and funding from the new evidence, guidance and learning. EU) which should not be neglected.
RECOMMENDATION 2
Lessons Learned Increase focus on intersectoral work to maximize UNICEF has sought to strike an appropriate balance the coherence of Country Office programming with between its organizational mandate and priorities for the overall theory of change of the new Country realization of children’s rights and those of the Albanian Programme Document. Government, for which the rights of children are only a component. This is a dynamic and evolving process Building on a robust, logical and coherent theory of that requires regular revisiting. change and linked programme strategies, an increased focus on intersectoral working between the different People are the foundation of UNICEF Albania’s country programme components of UNICEF Albania should programme outcomes. Overall programme effectiveness form part of the new CPD process. This could include an is predicated on ensuring the right people in the right explicit focus on internal collaboration and leadership; place with the right motivation to perform. Promising delegating responsibility for intersectoral work FINAL REPORT 19
appropriately across all positions with well-defined RECOMMENDATION 6 responsibilities and balanced staffing/workloads; and address the challenge of promoting synergies and Focus on support to the ongoing evolution of the intersectoral work with government partners. normative framework and decentralization process in Albania, including advocacy for appropriate resource allocation at subnational levels and guided RECOMMENDATION 3 by data and evidence from monitoring of service provision, particularly to the most vulnerable and Plan for future and ongoing emergency responses. marginalized.
The experience of successive crises should be A solid base of accurate and up-to-date data and evidence utilized for more systematic planning for future crises. is crucial to ensuring operationalization of policies related Consideration should be given to the development of to realization of children’s rights through all levels of risk analysis as part of the CPD planning process as government (e.g. at municipality and local level). UNICEF well as minimum preparedness activities; designation should address the challenges in policy implementation via of an emergencies focal point; evaluation of the 2019 advocacy for resource allocation (technical and financial) earthquake response as a learning tool; and assessing for children in Albania at central and subnational levels. This various potential outcomes of the COVID-19 pandemic. could capitalize on the guidance already available from UNICEF in the area of Public Finance for Children (PF4C). UNICEF should also redouble efforts to build capacity of RECOMMENDATION 4 key child rights stakeholders at subnational levels and ensure that monitoring, evaluation and research activities Determine a clearer partnership strategy across are relevant to effective implementation of policies and different stakeholder groups and set partnership services for children at municipality level in line with the objectives for each for which progress can be decentralization process. tracked.
Programme strategies should incorporate commitments RECOMMENDATION 7 to, and practical approaches (including objectives, engagement modality, selection criteria, outcomes, Introduce a country programme-wide effort to performance metrics) around partnership with different embed gender equality principles and gender groups, and tailor approaches to each. This should mainstreaming. include groups such as private sector and academia, civil society, development partners and Government. The Country Office should embed its approach to gender responsive programming. This can be done by analysing the existing data and evidence, which could inform the RECOMMENDATION 5 basis of the new CPD. The CO should also consider a Gender Programmatic Review; programmatic alignment Invest in a clear and systematic approach to data with Gender Action Plan 2018–2021; implementation of and evidence generation. a capability strengthening plan for the CO; and use of external and regional expertise to gain technical skills on To support results-based management and strategic gender-responsive programming. decision-making there is a need for a clearer and more systematic approach to data and evidence generation for the equitable realization of child rights. The focus should be placed on timely monitoring of results and ensuring that programmatic priorities and implementation strategies are adopted on the basis of this and reviewed regularly. Additional considerations should be made to support external administrative data collection systems. Findings would then be linked to UNICEF’s advocacy work and monitoring data would then be utilized for ad- hoc evaluations and formative research. 20 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021 1. INTRODUCTION FINAL REPORT 21
his report presents the findings of the evaluation of the UNICEF Albania Country Programme (CP) 2017–2021. The evaluation was commissioned by the Country Office (CO) of UNICEF Albania and was managed by UNICEF Europe and Central Asia Regional Office (ECARO) in close collaboration with the T Country Office. It was carried out between May and October 2020 and covers the entire UNICEF portfolio during the first four years of the five-year programme. This is a strategic evaluation that complements the sectoral evaluations and studies that have been carried out by UNICEF and external experts during this period.
The CP had an initial budget of US$ 16,345,000. Its overall goal is to achieve, by 2021, “the progressive realization of rights of all children in Albania, accompanied by the reduction of equity gaps, to be based on effective child- focused policies, systematically enforced accountabilities with adequate public resources, and strengthened respect for and protection of children’s rights”. UNICEF’s main partners were government ministries, agencies and departments at national, regional and district level.
The evaluation assesses the CP’s collaboration with government structures, UN agencies, international bodies and donors, civil society organizations (CSOs) and implementing partners, and a small number of private sector representatives.
Section 1 of this report describes the purpose, objectives and scope of the evaluation, the context within which UNICEF has been operating, and outlines the main components and goals of the CP itself. Section 2 explains the methodology of the evaluation including the evaluation framework, data collection and sampling strategies, and limitations and challenges faced by the evaluation. This section also highlights considerations of child rights, equity and gender in the analysis, as well as ethical considerations. Findings resulting from the evaluation are presented in Section 3, organized according to the main evaluation criteria (relevance, effectiveness, efficiency, sustainability, synergies/coherence) and the evaluation questions refined in the inception phase. Based on the findings, conclusions are presented in section 4, lessons learnt in section 5 and recommendations in section 6.
1.1 Purpose, objectives and scope of the evaluation
The object of this evaluation is the UNICEF Albania Country Programme (CP), specifically the CP of cooperation 2017–2021 developed in partnership with the Government of Albania. The evaluation rationale is described as follows: “Aligned with corporate policy requirements, this formative evaluation of the CP in Albania serves both accountability and learning purposes. The CPE sets out to document and account for UNICEF’s performance and contribution towards national development goals. The CPE will look back and assess the relevance, effectiveness, sustainability, efficiency, and synergies/coherence associated with UNICEF’s portfolio, and programmatic and strategic choices made in the design and implementation of the CP, in order to identify good practices, and to draw lessons and forward-looking recommendations that can inform the 2022–26 CP planning process.”
1.1.1 Objectives
As defined by UNICEF, the Country Programme Evaluation (CPE) had three objectives which were to: 1. Assess UNICEF’s strategic positioning and programmatic choices made, given government priorities and the changing socioeconomic and political context in Albania; 2. Provide an independent performance assessment of the country programme 2017–21 seen in relation to expected results and UNICEF’s contributions to realization of child rights in Albania, including inequities and gender; and 3. Identify good practices and lessons and draw forward-looking recommendations that can inform the development of the next country programme.
1.1.2 Scope
The scope of the evaluation covered the following dimensions: • Temporal scope: The evaluation covers the first four years of the current country programme (2017–October 2020), 2021 is excluded from the scope; • Geographical scope: All areas in Albania where UNICEF has worked over the period from 2018 to 2020; • Portfolio scope: The entire CP portfolio, including cross-cutting issues, intersectoral support involving communication, data generation/child rights monitoring and evaluation, and gender; and 22 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
• Organizational scope: The organizational scope of the evaluation goes beyond the Country Office and includes provision of technical support and oversight from UNICEF’s Regional Office and, when relevant, UNICEF’s Headquarters in New York. • Excluded from the scope: The evaluation presents an assessment of the situation of children in Albania at large and does not provide a detailed assessment of any one CP component.
1.1.3 Audiences
The intended primary users of this evaluation are the UNICEF Albania Country Office and the Government of Albania, as well as other UN agencies, UNICEF partners, and other UNICEF offices in the region that may benefit from learning about the Albania experience and approaches. The evaluation report aims to assist the intended users in implementation of the current CP during its last year, as well as in the 2022–26 country programme planning process, which is now in its preliminary stages, and specifically in the development of the new Country Programme Document.
1.1.4 Evaluation Management
The evaluation was supervised by an Evaluation Specialist located within the UNICEF Regional Office and the ERG consisting of key UNICEF and national stakeholders which was convened to provide inputs on all main evaluation deliverables that are expected to strengthen the quality and credibility of the evaluation. The ERG also sought to facilitate access to key informants during the evaluation process, participated in interviews with evaluators as relevant, and supported the dissemination of results. A full list of the ERG Members is provided in Annex 11.
1.2 Country context Figure 1: Map of Albania
1.2.1 Geographic and demographic context Montenegro Kosovo* Located in south-eastern Europe, Albania borders with Montenegro, Kosovo, North Macedonia and Greece, with the Adriatic Sea forming a SHKODËR 420 kilometre western border (Fig.1). KUKËS
LEZHË
1 As of January 2020, Albania’s population amounted to 2,845,955 DIBËR inhabitants. Over the past two decades, the population has been DURRËS experiencing a general trend of decrease due to a low birth rate2 and Macedonia net excess of emigration outwards over immigration inwards. The TIRANË youth population of Albania is higher (median age 37.2 years) than the ELBASAN European (28-member EU) average of 43.1.3 Overall, 22 per cent of the population are 14 years old or under, with 31 per cent 19 years old or FIER under. BERAT KORÇË
The majority (82.6 per cent)4 of the population are ethnic Albanians, with a significant diaspora5 population in other parts of Europe (Austria, VLORË Germany, Greece, Italy, Switzerland), and beyond. Other ethnic GJIROKASTËR groups within Albania are Balkan Egyptians (Jevgits/Jevgs), Greeks, Macedonians, Roma, and Serbian-Montenegrins. Greece
*(UN SC resolution 1244)
1. Population of Albania Press Release, Institute of Statistics (INSTAT), January 2020. 2. At 1.64 births per woman (World Bank 2017), slightly higher than the European average of 1.59 (Eurostat 2017) but lower than the generally accepted replacement rate of 2.1 births per woman. 3. Eurostat, 2018 figures. 4. Census of population and housing, INSTAT, 2011. 5. Estimates vary between 3.5 million and over 9 million ethnic Albanians outside the country. Albanians have a long history of emigra- tion as a result of political and/or social upheaval and conflict, most recently as a result of successive insecurity in the early 1990s and the collapse of communist rule in the country. There has been some related inward/outward migration resulting from the Kosovo conflict in the late 1990s. FINAL REPORT 23
Following recommendation from the Council of Europe, the Government of Albania endeavoured to document the ethnicities of its inhabitants for the first time in 2011. The data recorded for the above subgroups were, however, small and a substantial minority of 15.6 per cent did not record their ethnicity. The results of the census were deemed unreliable by the Council of Europe.6
Other data7 estimates the populations of these ethnic groups as follows:
• Roma: 80,000–150,0008 • Greeks: 58,758 (per the 1989 census) • Balkan Egyptians: 200,000–250,000 (disputed claim by the Union of the Egyptians of Albania) • Macedonians: 120,000–350,000 (disputed claim by the Association of Macedonians in Albania) • Serbian-Montenegrins: 2,000–30,000 (claim by Assembly for the Diaspora, Serbia).
Among these, the Roma and Balkan Egyptian communities are the most politically, economically and socially neglected groups in the country.9 In addition to widespread societal discrimination, they are generally at a disadvantage due to high illiteracy, particularly among children; poor health conditions; lack of education; and marked economic disadvantages.
1.2.2 Political context and government priorities
Following 46 years of communism, Albania began its transition to a liberal democracy in 1991 and is now a unitary parliamentary constitutional republic. The President, who acts as head of state, is nominated by the Parliament which is elected every four years.10 The current Prime Minister, Edi Rama, came into office in 2013. The political environment in Albania has been marked by increasing polarisation in recent years. The first half of 2019 saw multiple opposition rallies and several members of the opposition resigned from their Parliamentary mandates.11 The opposition decided to boycott the local elections of 30 June 2019 and accused the ruling majority of widespread corruption and use of criminal networks to manipulate elections.12 Faced with threats of a boycott from opposition parties, the President, Ilir Meta, issued a decree to postpone the vote to October. This decision prompted an impeachment procedure by Parliament. The parliamentary inquiry committee concluded in July 2020 that while the President had overstepped his constitutional competences, the violations did not justify his impeachment.13 The local elections took place in June 2019, amidst the opposition boycott and allowed the ruling majority to extend its authority over nearly all of Albania’s 61 municipalities.14 The preliminary findings of the election observation mission concluded that “voters did not have a meaningful choice between political options”.15 This context of uncertainty led the EU, to postpone the opening of accession negotiations with Albania in 2019.
In March 2020, the members of the European Council endorsed the conclusions on enlargement and the stabilization and association process, therefore opening accession negotiations with Albania.16 The European Commission 2020 report found that “on fundamental rights, Albania complies with international human rights instruments and has developed its legal framework in line with European standards”.17 Rapid social, political, and economic changes have
6. Third Opinion of the Council of Europe on Albania, November 2011. 7. Data from Minority Rights Group International: https://minorityrights.org/ 8. In 2011, UNICEF reported data indicating approximately 15,000 Roma (including 5,000 children) scattered among some 108 communi- ties. The 2011 census recorded 8,301 Roma settlements across the country. 9. Minority Rights Group (2015, September 15). Albania. https://minorityrights.org/country/albania/ 10. Constitution of the Republic of Albania, 1998. https://www.ahjucaf.org/sites/default/files/Constitution%20albanie.pdf 11. Freedom House (2020) Albania profile. Nations in Transit 2020. https://freedomhouse.org/country/albania/nations-transit/2020 12. Ibid. 13. European Commission (2020) Communication on EU Enlargement Policy: Albania 2020 Report. October 2020. 14. Freedom House (2020) Albania profile. Nations in Transit 2020. https://freedomhouse.org/country/albania/nations-transit/2020 15. Election observation mission undertaken by the Office for Democratic Institutions and Human Rights (ODIHR) and the Organization for Security and Cooperation in Europe (OSCE): OSCE/ODIHR Election Observation Mission, Statement of preliminary findings and conclusions (July 1st 2019). https://www.osce.org/odihr/elections/albania/424433?download=true 16. European Commission. (2020, March 26). Albania – European Neighbourhood Policy And Enlargement Negotiations. https://ec.euro- pa.eu/neighbourhood-enlargement/countries/detailed-country-information/albania_en 17. European Commission (2020) Communication on EU Enlargement Policy: Albania 2020 Report. October 2020. 24 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
been accelerated by the prospect of EU integration since 2014,18 a main driver of domestic reform in the country and is the country’s highest priority.19
The current Government has developed and implemented the National Strategy for Development and Integration 2014–2020 (NSDI II), that has set out the road map to prepare the country to become a full member of the European Union with strong and sustained economic growth to improve the well-being of its citizens.20 One of the pillars of the NSDI II is investment in human capital and social cohesion. The main aims of this pillar are to expand access and improve the quality of education at all levels, ensure equitable access to health services with better service delivery quality, and improve financial efficiency of the health system.21 The Government is currently developing the third National Strategy for Development and Integration. Despite making up the largest share of public expenditure as a percentage of GDP,22 public investments in the social sectors (education, health and social protection) remain lower compared to other countries in the region and at a lower level compared to international standards for an upper middle-income country.23 As a result, these sectors lack adequate human, technical and financial resources needed for improved services for children.24 In 2018, 2.9 per cent of GDP was devoted to public investments in health care (5.1 per cent in Serbia, 3.5 per cent in Turkey,25 7 per cent in Italy26), 3.1 per cent to education (3.7 per cent in Serbia, 4.4 per cent in Turkey27).28 In 2016, Albania’s social protection spending was significantly lower than the EU average at 9.4 per cent of its GDP compared to 28 per cent for the European average. The country’s public financing of social sectors is made even more fragile by the recession resulting from the 2020 COVID-19 pandemic.
Following the first cases of COVID-19 in Albania, in early March 2020, the Government announced a series of restrictions on travel, public services and gatherings and a nationwide lockdown was instituted. The COVID-19 crisis is expected to have a significant impact on the economy as well as on vulnerable groups and families. Progress achieved in the reduction of poverty levels in Albania is likely to be put at risk by the consequences of the November 2019 highly destructive earthquake and the looming recession engendered by the COVID-19 pandemic. According to recent World Bank estimates, extreme poverty levels could see an increase of between 4 to 8 percentage points, reaching about 40 per cent in the first scenario and 44 per cent in the second scenario, which would amount to the poverty rates experienced in 2012 and 2005.29 Albania’s annual GDP growth is expected to contract by 5 per cent in 2020 and 8.4 per cent in 2021,30 with major sectors of the economy severely affected such as tourism and hospitality, manufacturing, and non-essential trade being impacted by containment measures.31 The impact of the recession and the implementation of economic support measures are projected to increase the country’s fiscal deficit to 5.4 per cent of GDP in 2020, pushing government debt up to 75.8 per cent of GDP.32 The unprecedented nature of the COVID-19 crisis heightens the uncertainty around economic previsions.
18. UNICEF Albania CO Annual Report 2018. 19. Ibid. 20. National Strategy for Development and Integration 2015–2020 (NSDI II) 21. Ibid. 22. 27 Mainstreaming, Acceleration and Policy Support for Achieving the Sustainable Development Goals in Albania (2018). 23. UNICEF Albania CO Annual Report 2018. 24. Mainstreaming, Acceleration and Policy Support for Achieving the Sustainable Development Goals in Albania (2018). 25. European Commission. (2019). Key figures on enlargement countries. Luxembourg: Publications Office of the European Union. https://doi.org/10.2785/080458 26. Mainstreaming, Acceleration and Policy Support for Achieving the Sustainable Development Goals in Albania (2018). 27. Ibid. 28. INSTAT 2019 Albania in Figures – 2018. 29. World Bank. (2020, May). Western Balkans Regular Economic Report. The Economic and Social Impact of COVID-19. Poverty and Household Welfare. (17). World Bank Publications. http://documents1.worldbank.org/curated/en/236311590680555002/pdf/The-Economic-and-Social-Impact-of-COVID-19-Poverty-and- Household-Welfare.pdf 30. World Bank. (2020, June). Global Economic Prospects: Europe and Central Asia. World Bank Publications. https://www.worldbank. org/en/region/eca/brief/global-economic-prospects-europe-and-central-asia 31. UNICEF Albania. (2020, July 15). UN Albania COVID-19 Socio-economic recovery and response plan. https://www.unicef.org/albania/ documents/un-albania-covid-19-socio-economic-recovery-response-plan 32. World Bank (2020). The Economic and Social Impact of COVID-19: Western Balkans Outlook (No.17). World Bank Publications. http://docu- ments1.worldbank.org/curated/en/606131588087679463/pdf/The-Economic-and-Social-Impact-of-COVID-19-Western-Balkans-Outlook.pdf FINAL REPORT 25
1.2.3 Social development and gender profile
Despite a significant improvement of living standards since 1991, Albania remains one of the poorest countries in Europe, with high unemployment rates especially for youth and women. Nationally, the estimated unemployment rate in 2015 was 17 per cent, with International Labour Organization (ILO) estimates for 2019 at 12.3 per cent of the labour force (11.6 per cent female; 12.8 per cent male).33 According to the most recent data, in 2015 youth unemployment34 stood at 32.8 per cent (32 per cent female; 29.8 per cent male),35 which is significantly higher than the EU average (10.4 per cent) and higher than neighbouring countries (Kosovo: 31.4 per cent; Macedonia: 24.7 per cent; Serbia: 20 per cent; Montenegro: 19 per cent; Greece: 17.2 per cent).36
In 2008, Albania was classified as an upper-middle-income country by the World Bank37 and in 2019 it ranked 69th out of 189 countries on the UNDP Human Development Index, placing it in the high human development category and above the average for countries in Europe and Central Asia.38 Despite significant socioeconomic progress, the country still faces substantial challenges. It remains one of the poorest countries39 in Europe with minority and vulnerable groups (people living in rural areas, the elderly, vulnerable women and people with disabilities) being among the most politically, economically and socially neglected groups in the country.40
Almost a quarter (23.4 per cent) of the Albanian population were assessed to be at risk of poverty in 2018, only ahead of Serbia and Romania in the European area (the EU average is 16.9 per cent). The risk of poverty is exacerbated for children, for households with dependent children, and for females, particularly younger females (17 or under).41 29.6 per cent of children were reported to be at risk of poverty in 2017 and 2018, with girls being affected more (30.6 per cent).42
The political transition of the 1990s led to changes in the social protection and health systems in Albania. According to the last Demography and Health Survey (DHS) 2017–2018, however, access to health care is strongly determined by the wealth quintile, with 62 per cent of women in the lowest wealth quintile reporting difficulties in accessing services.43 People with disabilities as well as people from Roma and Balkan Egyptian communities also face discrimination in accessing health services.44
The 2018 female Human Development Index (HDI) value for Albania is 0.779 in contrast with 0.802 for males, resulting in a Gender Development Index (GDI)45 value of 0.971. Albania has a Gender Inequality Index (GII)46 value of 0.234, ranking 51st out of 162 countries in the 2018 index. In the last decade, the Government of Albania has made important progress in establishing relevant institutional and policy frameworks for achieving gender equality and women’s empowerment, in line with international, regional, and national gender equality standards and obligations according to UN Women and UNDP.47 The national legislation on equality and non-discrimination stems from the fundamental rights and freedoms recognized by the Constitution of the Republic of Albania approved in 1998.
33. Ibid. 34. Share of youth not in education, employment or training as a percentage of the youth population. 35. International Labour Organization, ILOSTAT database, 2020. 36. Ibid. 37. World Bank data, 2020. 38. Human Development Report 2019, UNDP. 39. Jorgoni, Elira (2019). ESPN Thematic Report on In-work poverty – Albania, European Social Policy Network (ESPN), Brussels: Europe- an Commission. 40. Poverty in Albania, European Social Policy Network, 2019. 41. Income and Living Conditions in Albania, 2017–2018, INSTAT, December 2019. 42. INSTAT 2019 EU-SILC Key Results 2017, 2018. 43. UNICEF Albania COAR 2018, 44. Ibid. 45. The GDI measures gender gaps in human development achievements by accounting for disparities between women and men in three basic dimensions of human development – health, knowledge and living standards – using the same component indicators as in the HDI. The GDI is on a scale of 0 (lowest) – 1 (highest). 46. The GII is an inequality index. It shows the loss in potential human development due to disparity between female and male achieve- ments in three dimensions: reproductive health, empowerment and the labour market. Overall, the GII reflects how women are disadvan- taged in these dimensions. The GII ranges between 0 and 1. Higher GII values indicate higher inequalities between women and men and thus higher loss to human development. 47. UN Women & UNDP (2016). Gender Brief Albania 2016. Prepared by Monika Kocaqi, Ani Plaku and Dolly Wittberger. UN Women, Albania. 26 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
Albania has fully committed to the SDGs and Agenda 2030 in monitoring their achievements. The Albanian Government has convened an inter-ministerial committee on the SDGs, chaired by the Deputy Prime Minister of Albania and comprising key government institutions as well as other stakeholders from the business community, civil society, academia and international organizations, including UN agencies in Albania. Albania’s policy goals and objectives in the national strategies and policy documents link to 134 (of 170) SDG targets and the Government has commissioned a baseline for SDG achievement and published a voluntary national review of SDG progress in 2018.
1.2.4 Children, youth and families
Albania has significantly increased access to all levels of education (nine years of education are mandatory). In 2018 net enrolment rates in primary education (94.5 per cent) were close to the European average of 96 per cent, and similar to Albania’s neighbours of Serbia (94.5 per cent) and Montenegro (96.5 per cent). However, rates in lower secondary stood at 89.07 per cent, lower than the European average (91 per cent), Serbia (94.6 per cent) and Montenegro (89.9 per cent).48 Overall, the enrolment rates in education are higher for girls than boys, with the gap widening as children progress through the education system.
The out-of-school rate for Albania is in line with the European average, at approximately 2.8 per cent. However, UNICEF has identified minority populations (such as Roma and Balkan Egyptians) returnees, young carers, children contributing to the family income, early married girls and children with disabilities as having a high risk of not attending school.49
Nutrition and feeding practices remain a public health concern for children, especially of age 6–59 months.50 The DHS 2017–2018 found that only one in three (38 per cent) children born in the two years prior to the survey were exclusively breastfed in the first six months of life as per World Health Organization (WHO) and UNICEF recommendations.51 The survey noted that 25 per cent of children aged between 6 and 49 months were anaemic, with some more affected than others, especially in households of the low wealth quintile. The DHS results indicate improvements in the prevalence of stunting, which has decreased from 19 per cent in 2008 to 11 per cent in 2018. The survey noted that stunting in children is closely linked with mother’s education (22 per cent of children, whose mothers have no or solely primary education, are stunted, compared to 8 per cent of children, whose mothers’ education is higher).52 Immunization coverage for children is nearly universal at 97 per cent, although concerns remain on emerging vaccine hesitancy in the population.53
With regard to child protection, progress has been made in reducing numbers of children exposed to violence in various settings (at home, in school or in communities). The DHS found that the number of children experiencing non-violent methods of discipline increased from 22 per cent in 2009 to 37 per cent in 2018.54 Furthermore, 48 per cent of children have been subject to at least one form of psychological or physical aggression in 2018 compared to 75 per cent in 2009. Despite this progress, violence and harassment against children on the internet has been a growing source of concern with one in five children having seen violent content online and one in ten children reporting at least one unwanted sexual experience on the internet.55
Ending children’s institutionalization is a priority of the Government of Albania along with the reform of the childcare system: as of 2019 703 children were registered in residential care, of which 525 placed in public and non-public residential institutions.56 Moreover, 684 children with disabilities were attending segregated and often residential schools as reported in the State’s Party Periodic Report to the Committee on the Rights of the Child.57
48. All data UNESCO as retrieved May 2020. http://data.uis.unesco.org/ 49. Identification of out-of-school children reaching the compulsory school age in Albania, UNICEF, 2017. 50. UNICEF Albania CO Annual Report 2018. 51. Albania Demographic and Health Survey 2017–2018. 52. UNICEF Albania CO Annual Report 2018. 53. Ibid. 54. Demographic and Health Survey 2017–2018. 55. Dunja A, Gjergji O, Gvineria D, Hallkaj E, and Verzivolli I. (2019). One Click Away: Children’s Experience of Internet Use in Albania. UNICEF in Albania and IPSOS Strategic Marketing, https://www.unicef.org/albania/media/2486/file/one%20click.pdf 56. Evaluation ToR, page.4. See Annex 1. 57. Government of Albania 2019 5th and 6th Periodic Report to the Committee on the Rights of the Child. https://tbinternet.ohchr.org/_ layouts/15/treatybodyexternal/SessionDetails1.aspx?SessionID=801&Lang=en FINAL REPORT 27
Adolescents in Albania face a series of issues related to education, access to employment, and violence. According to the 2019 UN Country Team Common Country Analysis, 1 in 4 adolescents of ages 11,13 and 15 reported having experienced physical abuse once or twice in their lifetime with 6 per cent reporting multiple instances of physical abuse.58 Adolescents in Albania, girls in particular, are also vulnerable to instances of intimate partner violence and sexual harassment with 1 in 5 girls reporting having experienced intimate partner violence and 80 per cent reporting controlling behaviours from their partners.59 Furthermore, 22 per cent of adolescents of ages 11, 13 and 15 reported at least one instance of bullying at school.60
Adolescents and youth in Albania also remain concerned about the level and quality of education when considering access to employment and matching labour market demand. The survey “Youth of Albania 2018/19” noted that 38 per cent of young people feel ‘somewhat’ satisfied with the quality of education.61 According to the same survey, a majority of young people in Albania (43 per cent) have a strong desire to emigrate, this desire reaching 31 per cent of adolescents between the ages of 14 and 17. In 2019, the share of youth not in education, training or employment stood at 25.8 per cent (% of youth population) compared to 29.8 per cent in 2015.62 The limited economic prospects for young people seem to be the primary reason for wishing to emigrate to another country, matching trends seen in other categories of the general population.
1.2.5 Governance and civil society
In the past 20 years, decentralization reforms have been implemented in Albania following the ratification of the European Charter for Local Self-Government and the adoption of a National Decentralization Strategy in 1999.63 Today, the country is divided into 12 primary administrative divisions (counties, formerly prefectures), which are further divided into 61 municipalities with greater institutional capacity.64 Decentralization is one of the policy areas included in the NSDI II 2015–2020 in a strategy that aims to consolidate good governance, democracy and the rule of law with strong, effective and democratic institutions and promote a fully functional and open judiciary.65
With regards to governance, Albania has made significant progress over the last five years, including key milestones such as the territorial reorganization and new local elections in 2015, a decriminalization law and a series of constitutional reforms focused on justice.66 However, the country is still facing significant challenges in the areas of good governance and rule of law. According to a 2018 mainstreaming, acceleration, and policy support (MAPS) report, governance reforms have introduced significant uncertainties into Albania’s overall government structures.67 The report notes that questions around division of labour between and among central and local government bodies on the issue of child protection should be resolved through the implementation of appropriate sustainable development policy frameworks.68 Civil society has played a crucial role in improving the quality of governance in Albania and is a crucial element for the EU integration process. However, according to the 2018 MAPS report, civil society is still relatively under-developed, especially outside Tirana.69 The report advocates for a review of the policy and regulatory frameworks for civil society organizations (CSOs) in order to strengthen their ability to support governance reform and strengthen public sector accountability.
58. UNCT (2020) Common Country Analysis in Albania 2020 update. 59. Ibid. 60. Ibid. 61. Kamberi, G., Çela, A. (2019). Youth Study Albania 2018/2019. Tirana, Albania: Friedrich Ebert Foundation, e.V., Referat Mittel und Osteuropa. 62. International Labour Organization, ILOSTAT database. Data retrieved 21 June 2020. 63. Government of Albania, The World Bank and United Nations Albania (2018). Mainstreaming, Acceleration, and Policy Support for Achieving the Sustainable Development Goals in Albania. https://www.un.org.al/sites/default/files/MAPS_Report_web.pdf 64. Ibid. 65. National Strategy for Development and Integration 2015–2020 (NSDI II). 66. Programme of Cooperation for Sustainable Development 2017–2021. 67. Government of Albania, The World Bank and United Nations Albania (2018). Mainstreaming, Acceleration, and Policy Support for Achieving the Sustainable Development Goals in Albania. https://www.un.org.al/sites/default/files/MAPS_Report_web.pdf 68. Ibid. 69. Ibid. 28 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
1.3 Country Programme components and goals
As defined in the CPD document, the overarching goal of the Albania country programme is “to achieve, by 2021, the progressive realization of rights of all children in Albania, accompanied by the reduction of equity gaps, to be based on effective child-focused policies, systematically enforced accountabilities with adequate public resources, and strengthened respect for and protection of children’s rights”.70
The long-term overall intended impact of UNICEF’s country programme is (a) that all boys and girls in Albania, especially those in situations of marginalization (those affected by poverty, with disabilities, from Roma and Balkan Egyptian minorities, living in underserved regions and/or on the move) equitably access quality social services that fully meet international norms and standards; and (b) that children and their families receive appropriate assistance to overcome multidimensional deprivations. Effective governance at national and especially subnational/local levels is a prerequisite for services to be designed, planned, delivered and monitored.
The Albania CP was developed during the previous UNICEF strategic plan period (2014–17), but the implementation period stretches into the new strategic plan covering 2018–2021.
For the period of the current CPD, Albania has been faced with two crises: the November 2019 earthquake and the COVID-19 pandemic. In both instances, UNICEF supported the Government of Albania in addressing the crisis response. Following the earthquake, UNICEF supported the Government of Albania and provided timely response to the emerging crisis. During the COVID-19 pandemic, UNICEF assisted the Government in adjusting its education programme, and overall support to child protection during the time of crisis.
The key actors and partners are Albanian governmental bodies, primarily the Prime Minister’s office and the Ministry of Foreign Affairs, and include different ministries (Ministry of Health and Social Protection/MHSP, Ministry of Education, Sports and Youth/MESY) as well as local government units and municipalities. UNICEF also works with UN agencies such as UNFPA, UN Women, UNHRC and IOM as noted during the documentation review. UNICEF partners with human rights and independent institutions such as the Office of the Ombudsperson and the Commissioner for Protection against Discrimination and civil society organizations (CSOs). UNICEF has been engaging with representatives from the private sectors, think tanks and academia, bilateral donors and stakeholders such as the Swiss Development Cooperation and Austria Development Agency. A detailed stakeholder analysis was undertaken in the Inception Report and is included in Annex 10.
Geographically, UNICEF undertakes or supports activities in 20 of Albania’s 61 municipalities, although most activities are focused on Tirana, the capital. In nine municipalities, UNICEF supports activities in a single sectoral area (primarily social protection activities), with the remaining municipalities being the focus of multiple interventions. A breakdown of operational municipalities by sectoral area is presented in Annex 7. The CP contributes to three of the six UN Development Assistance Framework (UNDAF71) 2016–2021 outcomes, namely:
• Governance and Rule of Law (Outcome I) • Social Cohesion (Outcome II) • Environment and Climate Change (Outcome IV).
The Sustainable Development Goals (SDGs): The CP is intended to contribute towards 10 out of the 17 SDGs, that is SDGs 1, 3, 4, 5, 6, 10, 11, 13, 16 and 17.72
Albania is also a “Delivering-as-One” (DaO) country. It was one of eight countries to volunteer for the pilot DaO from 2007–2011. Evaluation of the DaO pilot in Albania in 2010 provided lessons and recommendations that shaped the 2012–2016 Programme of Cooperation that aimed to expand and deepen the DaO approach.
70. UNICEF Albania, Country Programme Document 2017–2021, para 13, page 5. 71. Formally entitled the “Government of Albania and United Nations Programme of Cooperation for Sustainable Development 2017–2021”. 72. 1 – No poverty; 3 – Good health and well-being; 4 – Quality education; 5 – Gender equality: 6 – Clean water and sanitation; 10 – Reduced inequalities; 11 – Sustainable cities and communities; 13 – Climate action; 16 – Peace, justice and strong institutions; and 17 – Partnerships for the Goals. FINAL REPORT 29
Table 1: UNICEF Albania country programme outcomes and outputs 2017–2021
Outcome 1: Outcome 2: Outcome 3:
By 2021, children in situations By 2021, children’s rights to The Government of Albania of vulnerability enjoy equitable justice and protection from – UNICEF programme of access to quality health, education violence, abuse, exploitation cooperation is effectively and social protection services and neglect are effectively designed, coordinated, supported by effective governance supported by a comprehensive managed and supported to mechanisms. system of multisectoral meet quality standards in prevention and response achieving results for children. Output 1.1: By 2021, health care mechanisms. policymakers and service providers are Output 3.1: Guidance, tools and equipped with knowledge, guidance, Output 2.1: By 2021, social and child resources to effectively and efficiently tools and mechanisms to effectively scale protection practitioners have increased design and manage the programme of up and implement the new maternal, capacity to assist families in situations cooperation are available to UNICEF newborn and child health care normative of vulnerability/at risk of separation, and its partners. framework, focused on the most to provide care for their children, marginalized. preventing institutionalization. Output 3.2: Guidance, tools and resources to effectively generate, Output 1.2: By 2021, education sector Output 2.2: By 2021, child protection analyse, and utilize statistical and policymakers and practitioners at practitioners are empowered with qualitative information for a child rights central and local levels are equipped legal and normative frameworks, monitoring system are available in the with knowledge, guidance, tools and operational standards and tools country. mechanisms to effectively implement to effectively prevent and address and scale up the reformed, Early Learning situations of child abuse, violence, Output 3.3: Guidance, tools and and Development Standards-based and neglect and exploitation. resources for effective communication equity-sensitive, early learning education on child rights issues with stakeholders framework. Output 2.3: By 2021, justice sector are available to UNICEF and its policymakers, practitioners and partners. Output 1.3: By 2021, social protection independent human rights institutions policymakers and practitioners are have the capacity to fully align Output 3.4: Strategies to address cross- equipped with knowledge, guidance, the Justice for Children normative cutting issues related to child rights are tools and mechanisms to effectively framework to international standards developed and applied throughout the implement the reformed social protection and to effectively protect the rights of programme of cooperation. policy (combining cash assistance with children in conflict and in contact with decentralized care services). the law.
EU priorities: the Albania country programme is aligned with national priorities of EU integration, including democracy and rule of law; independent and accountable judicial institutions; consolidated good governance; social cohesion based on a modern educational system; universal and quality health care coverage; expanded employment opportunities; a stronger social protection system; gender equality; and social inclusion.73
The CP is organized around three main outcomes (strengthened governance for equity and social inclusion; protection and access to justice for children; and programme effectiveness), which are summarized in Table 1 below. A summary of the results of the country programme is included in EQ4a and a full analysis is in Annex 9.
Two additional outcomes (Outcomes 4 and 5), related to internal systems (governance, financial resources, human resources) and “special purpose” (country office facilities), were noted in the Annual Management Plans for 2017 and 2018, with both being combined into one outcome (4) in 2019. As these outcomes are not specifically related to the evaluation questions, they did not form an explicit part of this evaluation.
73. Government of Albania, National Strategy for Development and Integration, 2015–2020. 30 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021 2. METHODOLOGY FINAL REPORT 31
2.1 Evaluation framework
The evaluation used a theory-based, non-experimental approach. The primary evaluation methods were qualitative including extensive analysis of documentation, administrative data and key information interviews with UNICEF and stakeholders. A quantitative survey captured the views of U-Report users.
The process of carrying out the evaluation took place between May and October 2020 during the COVID-19 pandemic. The approach for the data collection was adapted to reflect the changing situation.
The evaluation took a strategic, overall view of the programme and UNICEF’s Strategic Plan 2018–2021 (as reported on through the Country Office Annual Report/COAR and results assessment modules/RAM) and the accompanying Strategy Note developed in 2016, in attempting to understand and assess the contribution of the different programmatic interventions to the anticipated chain of results (short-term outputs, medium-term outcomes) and the contributions towards longer-term impacts, as well as their potential sustainability.
An overarching programme theory of change (or “reconstructed” ToC) was developed to summarize the individual outcome-level theories of change. At the same time, the policy and institutional context of implementation were also considered for relevance to the needs of Albanian children and the effectiveness, efficiency, sustainability, and coherence of the approaches taken was assessed. This reconstructed ToC can be found in Annex 2.
The theories that the evaluation sought to test were:
• That the reconstructed programme ToC delivers UNICEF’s contribution to advancing children’s rights; • That implementing strategies were integrated and applied effectively to achieve results; • That the data reported in the COAR and RAM map onto and enable effective monitoring of UNICEF’s programmes in Albania (via the reconstructed ToC); and • That together these enable UNICEF to both track the contributions made by the programmes and the progress towards children’s rights in Albania.
An evaluation matrix was constructed around the evaluation questions, setting out the data sources, methods and analysis approach (Annex 2a). Revisions were made to the evaluation sub-questions, to reflect the current situation and adjustments were made to the approach and methodology. The evaluation criteria are based on UNICEF’s adaptations of the OECD DAC and United Nations Evaluation Group (UNEG) evaluation standards.
Each of the evaluation questions was interrogated to dissect its meaning(s) and one or more assumptions developed to reflect the elements of the question. For example, EQ1 asks about the extent to which UNICEF’s strategic positioning remained fit for purpose to advance child rights for all children, considering the context. The question has been broken down into two assumptions – the first focusing on the relevance of programme priorities to actual needs, and the second examining change over time and incorporating stakeholder and partner perspectives.74
2.1.1 Departures from the Terms of Reference
The most significant departure from the ToR was that, due to COVID-19, it was not possible to carry out an in- country inception visit nor to carry out in-country fieldwork, both of which were done virtually.
Revisions were made to the evaluation sub-questions, notably around EQ4, which was originally split into three separate sub-questions in the Evaluation Matrix, to reflect the current situation by including questions aimed at capturing UNICEF’s responsiveness and resilience to emergency situations. On commencement of the data analysis and report writing phase of the evaluation, the three EQ4 sub-questions (4a, 4b, 4c) were combined into two (4a, 4b) for the purposes of avoiding duplication of findings and brevity of the overall report. As it had been finalised prior to the data collection, the Evaluation Matrix was not revised in retrospect to reflect this.
74. Nkwake, A. Working with Assumptions in International Development Program Evaluation, 2013. 32 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
2.2 Data collection, data analysis and sampling strategy
As per the assignment ToR, the evaluation team selected five municipalities to conduct subnational data collection activities (remotely). The selection of these municipalities was on the basis of the following criteria:
• UNICEF activity within the current CPD period; • Level of UNICEF-supported activity across the CPD programme areas; • Presence of marginalized groups, minorities, etc.; • Highest number of children living with families receiving economic aid; • Highest number of deaths of children aged under 1; and • Highest number of children with disability allowance.75
A data spreadsheet was set up to weight these criteria and those municipalities that best fit these criteria were selected for data collection. On the basis of this analysis, the municipalities of Berat, Durres, Korça, Shkodra and Tirana were selected by the evaluation team. This selection was validated with the UNICEF CO and Evaluation Reference Group.
2.2.1 Interview sampling
The full list of potential key informants compiled during the stakeholder analysis process formed the sampling frame for interviews. This task commenced during the inception phase and was updated throughout the data collection phase of the evaluation. It was based on data provided directly by UNICEF Albania and was refined on the basis of the desk review of documentation by the evaluation team.
The evaluation team shortlisted external and internal stakeholders (approximately 120 individuals) with the input of the evaluation manager and CO, and respondents were selected on the basis of the level of engagement (i.e. number of years of experience and breadth of activities with UNICEF) and capacity to access remotely (via phone or VOIP).
By completion of the evaluation, 120 individuals had been interviewed via 88 separate remote interviews. These included 21 interviews with UNICEF staff (15 females, 6 males) and 6 interviews with UN agencies (2 females, 4 males). The evaluation team interviewed 30 respondents from the national Government (22 females, 8 males) as well as 16 respondents at the subnational level (10 females, 6 males). Nine respondents from donor and international agencies were also interviewed (8 females, 1 male) as well as from CSOs (14 females, 2 males). Nine interviews were undertaken with respondents from the judiciary, NHRI, the private sector and academia (7 females, 2 males). The majority of interviews were conducted via video link (Zoom or Skype) and a detailed list of interviewees can be found in Annex 3.
2.2.2 Online stakeholder survey
A structured online survey using the U-Report system was distributed to youth who participate in U-Report.76 UNICEF Albania has been a member of U-Report since 2018 and regularly disseminates polls on topics of interest to its mandate. The system was made available to the evaluation as an additional method to collect data directly from its members. There are some limitations to the U-report tool notably its limited capacity to collect detailed information, and an inherent bias in participants towards those with the requisite technological capacity and interest. Hence, the evaluation used it to obtain a general sense of perception of the realization of young people’s rights across UNICEF’s key mandate areas. In total, the U-Report survey for the evaluation generated 200 responses. 11 per cent of respondents are aged between 15 and 19; 12 per cent are aged between 20 and 24; 14 per cent are aged between 25 and 30; 16 per cent are aged between 31 and 34; and 46 per cent of respondents are aged over 35. 58 per cent of respondents (108) are male and 42 per cent (72) are female.
75. All data for setting sampling criteria were taken from the State Agency for Child Rights and Protection data dashboard, available at http://smartprocesses.org/children/statistikatestim/index.php 76. U-Report is a free and anonymous digital tool, created by UNICEF that directly interacts with young people aged 15 and above, (with a focus on the age group 15 to 24), who sign up to become U-Reporters through simple polls sent via Facebook Messenger or Viber. See https:// westernbalkans.ureport.in/opinion/1841/ for questions and responses. FINAL REPORT 33
2.2.3 Data analysis
All findings from the data collection process were aggregated into an evidence database and coded across the nine evaluation questions (and three sub-questions) nested under five criteria and 23 accompanying assumptions to produce an evidence table that provides a repository of all of the evidence available to the evaluation team for analysis and drawing findings and conclusions. Illustrative indicators were associated with each assumption to guide the data collection and analysis process (dependent on availability of data). Analysis of the cleaned, coded and anonymised data with reference to its congruence or divergence from the evaluation questions and assumptions provided the evaluation team with the basis for the evaluation findings presented in the report. Coded data was reviewed by successive team members to identify evidence that supported or contradicted specific assumptions derived from the evaluation questions. This evidence was collated, and findings derived and discussed on the basis of it.
2.3 Limitations and constraints faced by the evaluation
The evaluation was undertaken between May and October 2020 during the COVID-19 global pandemic, which substantially affected the context, background, and approach to the evaluation. Due to the COVID-19 outbreak, the evaluation team did not travel to Albania and therefore could not carry out in-person inception and data collection visits. In order to mitigate this, the evaluation team adapted the approach and questions, by carrying out semi- structured online interviews and consultations and augmenting data collection through a survey and the use of U-report to reach young people.
As anticipated in the inception report, access to stakeholders could be limited due to several reasons, including access to technology, lack of time due to holidays or other commitments, or more pressing priorities (for example, prioritization by government stakeholders of the COVID-19 response over other activities). To overcome this, the team addressed the data collection process with flexibility, adjusting their schedules to respond to the different needs of stakeholders. The data collection process started in July to reach as many stakeholders as possible before the holidays, though it was stretched out to mid-August so that the team could reach more people. The evaluation team also consulted with the UNICEF Albania Country Office to organize and schedule the interviews. The UNICEF Albania Country Office supported the team in reaching out and following up with key external stakeholders.
The timely presentation of findings and submission of the draft report have been a priority for the evaluation, and the time available between the conclusion of the data collection and the submission of the first draft of the evaluation report has been limited. Time constraints have been placed on the team, especially in the final stages of the evaluation, to meet UNICEF internal deadlines and to inform the process of the new country programme in a timely manner.
2.4 Management of the evaluation
Overall management of the evaluation was by the Multi-Country Evaluation Specialist from the UNICEF Europe and Central Asia Regional Office, with liaison management from the CRM Specialist in the UNICEF Albania Country Office. An external ERG was constituted to provide overall technical oversight for the evaluation. It did not have any formal evaluation management responsibilities but acted in an advisory capacity and provided inputs on all main evaluation deliverables to strengthen the quality and credibility of the evaluation. The composition and responsibilities of the ERG are listed in the ToR for this evaluation which can be found in Annex 1.77
77. Evaluation ToR. page 16. Available in Annex 1 page 21. 34 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
2.5 Ethical considerations
The evaluation team observed ethical standards set in IOD PARC’s ethical code. IOD PARC’s ethical framework is based on international guidelines for all contexts, in particular UNEG Ethical Guidelines for Evaluation 2008;78 UNEG Code of Conduct for Evaluation in the UN System 2007;79 Department for International Development (DFID) Ethics Principles for Research and Evaluation 2011;80 and Economic and Social Research Council (ESRC) Framework for Research Ethics Principles 2012.81
The evaluation team adhered to UNICEF’s Procedure for Ethical Standards in Research, Evaluation, Data Collection and Analysis (2015) and the UNICEF-Adapted UNEG Evaluation Reports Standards. This means that the evaluation team upheld the appropriate obligations of evaluators, including maintaining the independence, impartiality, credibility and accountability of the individual team members and the evaluation process as a whole. The evaluation team was not subject to any conflicts of interest and confirmed that they were able to carry out the evaluation without any undue interference.
There was no requirement for formal ethical review board approval for this evaluation because no in-person consultations or data collection with children, young people or vulnerable adults took place. The evaluation approach as outlined in the Evaluation Terms of Reference did present a rationale for community-level participation in a CPE, but due to the travel restrictions in place as a result of the COVID-19 pandemic, community level engagement did not take place.
With respect to ethical approaches to managing client and evaluation participant data (applying to the content of interviews, focus group discussions and surveys), the evaluation specifically ensured the following:
• Confidentiality: participants’ anonymity was protected, and all participants were assured of the confidentiality of any information they shared; • Preventing disclosure of identity: we took appropriate measures to prevent individual data from being published or otherwise released in a form that would allow any subject’s identity to be disclosed or inferred; and • Informed consent: we obtained informed consent from all participants, and ensured that participants had the contact details of the evaluation team to withdraw or change their consent at any time. • Data security: data was kept secure on servers; once analysed, all data was anonymised. All personal data is destroyed within six months of the completion of a project. Only team members will have access to password-protected folders containing the data of the participants for the duration of the project. Raw data (for example interview transcripts) are stored in a protected folder in IOD PARC’s document sharing site on the secure server. This folder is accessible only by the evaluation team. The evaluation team and all users of the shared folders have signed confidentiality and data protection agreements.
78. United Nations Evaluation Group, Ethical Guidelines for Evaluation. 2008. http://www.unevaluation.org/documentdownload?doc_ id=102&file_id=548 79. Code of Conduct for Evaluation in the UN system. UNEG http://www.unevaluation.org/document/download/547 80. DFID Ethics Principles for Research and Evaluation. 2011. https://www.gov.uk/government/publications/dfid-ethics-principles-for-re- search-and-evaluation 81. ESRC Framework for Research Ethics (FRE) 2010, updated September 2012. Economic and Social Research Council. https://esrc.ukri. org/files/funding/guidance-for-applicants/esrc-framework-for-research-ethics-2010/ FINAL REPORT 35 3. FINDINGS OF THE EVALUATION 36 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
3.1 Relevance
3.1.1 Evaluation Question 1 UNICEF’s strategic positioning
EQ1. To what extent has UNICEF’s strategic positioning remained fit-for-purpose to advance child rights for all children, considering the context?
(Context should consider children’s needs, national priorities, country and regional (Balkan) situation, and partner landscape).
Finding 1.1
The relevance of the CPD 2017–2021 was based on evidence from a substantial range of data (surveys, studies, evaluation reports etc.) with appropriate priorities set on the basis of these. The justification for programme priorities could have been further strengthened if they were informed by wider consultations with partners and children, human and child rights/gender and bottleneck analyses, and a formal evaluation of the previous country programme.
National developments and priorities have considerably informed the CP development for 2017–2021. A number of key events took place at a strategic level prior to and during the preparation of the UNICEF Albania CPD related to economic progress (being designated as an upper middle-income country – UMIC), being granted EU candidacy status in 201482 with formal approval for opening negotiations received in March 2020,83 and adopting the SDGs in 2015. The Government issued its National Strategy for Development and (European) Integration (NSDI II) 2015–2020,84 and territorial reform followed by a decentralization process with new roles and responsibilities for local governance, commenced in 2014–2015, though at the time of this evaluation these roles are still unclear.85 This landscape framed UNICEF’s partnership development and programmatic priorities with respect to central and local government partners as well as with development partners (for the latter see EQ7 and EQ 9).
Within the context of national developments and in an uncertain external environment, the CPD noted that there were no accompanying advances in children’s rights. There is reliable evidence that UNICEF’s prioritization of programmatic areas in designing the CPD was appropriately informed by the analysis available at the time. For instance, Child Rights Situation Analysis Reports conducted in the period before CPD development show the general need to improve the normative framework, enhance coordination among different stakeholders at national and subnational level, promote research and the use of reliable data to improve accountability of institutions, and to strengthen evidence-based interventions and policymaking, both at central and local levels. Additionally, the specific needs of children are highlighted in the same documents, for example in protection referral in the context of child protection, justice for children and deinstitutionalization. The situation is the same for the education and health sectors. Engagement with local government and intersectoral collaboration was another noticeable suggestion.86 UNICEF incorporated these findings appropriately into its CPD priorities.
The CO also conducted an analysis of donor priorities as part of the UNICEF COAR 2015 and alignment can be seen with UNICEF’s programme areas and its equity-focused approach in relation to reforming Albania’s system of social care services, promoting education and social inclusion of Roma, combating violence against children and ensuring children’s safety online.87 The Regional Knowledge and Leadership Agenda also indicated a focus on the
82. UN Programme of Cooperation for Sustainable Development, p. vii. 83. Council of the European Union, Conclusions 25 March 2020. https://data.consilium.europa.eu/doc/document/ST-7002-2020-INIT/en/pdf accessed on 24 August 2020 84. UN Programme of Cooperation for Sustainable Development, p. vii. 85. Albania EU Progress Report, 2015, p. 7. 86. See e.g. Cunninghame, C. and Hallkaj, E. (2015). Child Rights Situation Analysis Report 2012–2015, p. 49, Section on Recommen- dations; Byrne, K. (2014). Analysis of Policies and Reforms Affecting the Situation of Children in Albania. UNICEF, p. 118 Section on Recommendations and throughout the document. 87. UNICEF Albania COAR 2015, p. 12. FINAL REPORT 37
result areas of social protection, deinstitutionalization of childcare, advancement of juvenile justice, promotion of early learning and development, and support to preventive mother and child health care.88
To respond to these unmet needs, UNICEF articulated its overall goal as “to achieve, by 2021, the progressive realization of rights of all children in Albania, accompanied by the reduction of equity gaps, to be based on effective child-focused policies, systematically enforced accountabilities with adequate public resources, and strengthened respect for and protection of children’s rights”. This was to be achieved via the three country programme outcomes.89 UNICEF key informants and document review show that UNICEF Albania built its new strategy not just on a perceived gap in the realization of children’s rights, but also during the previous cycle of collaboration with the Government, CSOs and other development partners. This includes other UN agencies, under the mantle of the UNDAF and “Delivering as One” planning, implementation, monitoring and evaluation mechanisms.90
This attests to the relevance and appropriateness of UNICEF’s priorities, although the Country Office did not conduct a formal evaluation of the previous programme cycle (2012–2016), which could have provided further analysis and reflections on lessons learned to inform the CPD.91 UNICEF stakeholders noted an absence of a sectoral or overall bottleneck analysis to further inform the CPD design. This may have contributed to understanding common sectoral or intersectoral bottlenecks. However, document review shows that this missing analysis is partially mitigated by the Situation Analysis of Albania (2015) and the Analysis of Policies and Reforms Affecting the Situation of Children in Albania (2014), which includes a detailed section on “Social Determinants’ Analysis”.92
Given Albania’s UMIC status, the UNICEF country programme focused more on systems’ strengthening than direct service provision, in line with general guidance for UNICEF’s programmatic approach in middle income countries.93 Thus, UNICEF has adopted an appropriate overall approach to its programming. The approach was endorsed as appropriate by evaluation respondents at both local and central levels of government.
Perspectives from wider stakeholders, however, queried the comprehensiveness of this approach with respect to ongoing unmet needs of children, a lack of government resources and insufficient prioritization of responses to these needs. Development partners and donors raised their concerns about whether UNICEF was getting the balance right between contributing to the work of the Government or practically “replacing” it (see more detailed analysis on this in EQ4 and EQ7). Even though the UN and UNICEF are preferred partners for international organizations and the EU, some external stakeholders noted that the relation of UNICEF to the EU is more of a donor-contractor type.
A Programme Strategy Note was commissioned by UNICEF in parallel to CPD development in 2015/2016. An internal audit of the Country Office for the 2016–2017 period indicates that the utility of the Strategy Note was limited, however, because of time constraints around submission of the new CPD.94 Given that programme strategy notes can help sharpen programmatic thinking and inform strategic progress and potential programming shifts, this was a missed opportunity.95 The approved country programme document diverges from the Programme Strategy Note, particularly in the results framework (discussed further under EQ2).96
UNICEF Annual Plans and Reviews attest to human rights principles being integrated into the country programme and the associated management plan, including:Alignment to national development targets and Albania’s human rights/child rights obligations;
88. UNICEF Albania COAR 2016, p. 2. 89. CPD 2017–2021, p.1 and p. 5. 90. Ibid, p. 10. 91. Country Programme Evaluations are not required for every programme cycle, particularly for smaller COs. 92. “Children Situation Analysis 2012–2015” by Save the Children, Albania; “Analysis and Reforms Affecting the Situation of Children in Albania”, 2014 by UNICEF Albania; and “Child Notice”, 2015 by UNICEF Albania 93. Issued by Division of Policy and Strategy in May 2010, in Internal Audit of Country Office in Albania, 2018, p.7. 94. UNICEF Office of Internal Audit and Investigations, 2018, Internal Audit of the Republic of Albania Country Office. 95. UNICEF Albania COAR 2017, p.12. 96. UNICEF. Guidance on Developing Strategy Notes, 2015, p.1. https://washenablingenvironment.files.wordpress.com/2016/05/unicef-guidance_programme-strategy-note-8-december-2015.pdf ac- cessed on 24 August 2020 38 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
• Emphasis on accountability of duty-bearers and democratic oversight in relation to their actual performance; • Supporting the ability of rights-holders to claim their rights; and • Highlighting inequities through disaggregated data.97
Gender equality is also mentioned in the CPD, but not based on systematic gender-related evidence and guidance for gender mainstreaming in CPDs (see EQ5).
Stakeholder consultations
There is some, but limited, evidence from key stakeholders regarding UNICEF efforts to consult with government, CSOs and development partners in its policymaking processes during the design of the country programme. A limited number of stakeholders reported that UNICEF Albania had conducted consultations with them during the design process, but the majority of them could not provide specific details of issues discussed or feedback incorporated into the final document. Still, this can be attributed also to the time passed since that period (recall bias).98 This evidence contrasts with the CPD itself, which notes that broad consultations took place during preparation. Even though central government stakeholders noted that UNICEF aligns its interventions with their priorities based on regular consultations conducted with them, evidence from the document review suggests that this is not a systematic and embedded approach towards the whole range of stakeholders – for example at the local level, CSOs, implementing partners – nor a holistic approach to the overall programme design.99
On the other hand, the majority of relevant respondents confirmed that even if they were not significantly involved in UNICEF strategic planning, UNICEF staff engages with them on implementation of activities on a regular basis with the aim of aligning interventions.100 One stakeholder noted, “it goes without saying that the UN Programme of Cooperation should definitely be aligned with national priorities, but UNICEF’s added contribution here is that children’s rights are increasingly included as an intersectoral issue in many strategic documents”.101 Overall, the evaluation evidence from key stakeholders confirms the relevance of UNICEF strategic priorities.
Rights-holder consultations
There was no documentation or respondent confirmation of any formal process of consultation with children or vulnerable communities (see EQ2 below for further analysis). Although UNICEF internal stakeholders stated that they tended to consult children on their field missions, this is a rather informal and sporadic approach, which may not necessarily result in incorporating their needs and thoughts in planning processes. A recent UNICEF report confirms that many UNICEF commissioned reports ranked “partly” on child participation.102
National and subnational relationships
The document review and interviews across different categories of stakeholders confirm that since the start of the new programme cycle 2017–2021 UNICEF’s effective strategic positioning is evident through its status as the partner of choice of the Government in supporting reforms across social protection, education, justice and child protection sectors. Most external stakeholders stated that UNICEF has aligned its work to the existing strategic/ normative framework, much of which was acknowledged as having been developed with UNICEF Albania support and initiatives. Multiple stakeholders view UNICEF’s support as the provision of highly qualified expertise of its own staff and national and international consultants to provide technical assistance in developing important strategic documents and carrying out reforms. Government respondents consistently recognized and appreciated the work of UNICEF staff and reported confidence to revert to UNICEF for advice and support.
97. See for example UNICEF Albania COAR, 2017, p.13 and Key Performance Indicators Annex, UNICEF Country Programme Manage- ment Plan 2017=2021 p.3. 98. 103 Government key informants. 99. The Annual Work Plan of 2017 included sectoral discussions with stakeholders during its development, but these did not take place in 2018 or 2019. This was also noted in the Internal Audit of the Republic of Albania Country Office, 2018 100. Government and local-level key stakeholders. 101. UN key informants. 102. Cunninghame, C. (2020). Child Rights Issues in Child–Participatory Focused Documents. UNICEF Albania. FINAL REPORT 39
An example of this can be seen during the process of decentralization, during which UNICEF collaborated with government for this programme cycle. Decentralization reform, and the related impacts on the legal and regulatory framework, aimed to empower local government, although without clarity on the definitions, duties and responsibilities of the central and local-level actors. While this reform has given increased powers to municipalities to plan and administer pre-school, social and child protection services, it has not been accompanied by the required financial transfers and mechanisms for implementation.103 Local government stakeholders confirmed that after the decentralization process of 2015 they were still not clear about their functions. They lacked the knowledge and skills in developing local Social Care Plans, and UNICEF assisted them in this process through technical expertise. The majority of stakeholders confirmed that UNICEF contributed to the development of a normative and policy framework in social protection, for example, the Social Protection Strategy 2014–2020 (and its update in 2019), and Law no 121/2016 on Social Care Services.104
Stakeholders from the education sector largely confirm that UNICEF is one of the most serious strategic partners that invested in the education system, a sector mostly targeted by small project-based NGOs. In contrast, UNICEF has a strategic approach to the education system. It has contributed to the previous Strategy on Pre-University Education 2014–2020 and its evaluation; development of the new Strategy on Education 2021–2025 – a review and assessment of the education system in Albania, which was highly appraised by key informants;105 a case study on out-of-school children; among others.106 Key informants stated that UNICEF’s focus on the most vulnerable children aligns with the strategic pillars of education strategies on inclusive education, dropout prevention, prevention of violence in schools, ongoing teacher qualifications, curricula reform, etc.107
Albania started a process of comprehensive justice reform in 2014 that addressed the long-standing deficiencies of the system in respect to independence, accountability, efficiency, and professionalization of the justice system. Focusing on justice for children, UNICEF is a key partner, as it has been investing in this sector prior to this programme cycle. At the beginning of the CP 2017–2021 the Criminal Code of Justice for Children was developed and the Justice for Children National Strategy and Action Plan (2018–2021) was formulated.108 Stakeholders from government institutions and CSOs acknowledge UNICEF’s contribution to this process.
Document reviews showed that the child protection sector was fragmented before CPD development.109 Key stakeholders confirm that UNICEF has supported the State Agency for Children’s Rights and Protection to develop the legal and policy framework; updated the law on Children’s Rights and Protection110 and other relevant sub–legal documents; and participated in the National Agenda on Children’s Rights 2017–2020. Both documents are based on the Convention on the Rights of the Child (CRC). Specifically, the Law on Child Rights and Protection defines duties, institutions, structures and mechanisms to ensure respect for children’s rights by individuals, family and state. The Law also strengthened the system of child protection from violence, abuse, exploitation and neglect at central and local level.111
103. UNICEF Albania Country Office Annual Report 2019; Massey, C. Poni, M. Sammon, E. (2020). “Evaluation of UNICEF’s Contribution to the Normative Policy Framework of the Social Care Services Reform 2013–2019 in Albania”. UNICEF Albania. 104. Local government key stakeholders. 105. UNICEF and OECD, OECD Reviews of Evaluation and Assessment in Education, 2020. 106. UNICEF Out of School Children Country Solutions, Albania. 2019. 107. Government key informants. 108. See: https://www.unicef.org/albania/media/1496/file/National%20Justice%20for%20Children%20Strategy%20Eng.pdf; Result Assessment Module 2018, Output 2.3. J4C. 109. Council of Europe, Future of an Integrated Child Protection System in Albania, 2015. https://rm.coe.int/1680681ebb 110. Law no 18/2017 on “Child Rights and Protection”. 111. CRC 5th and 6th Periodic Report for Albania. 40 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
While overall there is strong evidence in relation to UNICEF’s strategic positioning, there is also, to a lesser degree, some conflicting information. On the positive side, the existing data and evidence regarding the unmet needs of children provides a good basis for UNICEF’s selection of programmatic areas. Further, UNICEF is considered a strategic and trusted partner by many national and subnational institutions. This puts them in a strong position to act as an ‘honest broker’ to guide reforms and system change.
However, the fragmentation of the child protection sector and the clear legislative/policy needs indicate a requirement to strike a balance between assisting/contributing to the needed changes and “replacing” government work, thus maintaining the ‘honest broker’ role of advocating for, facilitating and assisting in change, even where the data may suggest there is a conflict with government priorities. Systematic and formal consultations with a broad range of stakeholders could contribute to the improvement of UNICEF’s strategic positioning towards the whole landscape of stakeholders.
Finding 1.2 Strategy updates took place through annual reflections and the preparation of annual plans which did consider the context and needs. But more formalized approaches of using a Strategic Moment of Reflection or Mid-Term Review – which are typically conducted in other UNICEF Country Offices – were not implemented. The CPD may have benefited from such a formalized reflection/review exercise. This would have evidenced more clearly the rationale and justification for decision-making during the course of the programme period and helped to ensure more systematic inclusion of a wide range of stakeholders in consultations.
A review of UNICEF documentation shows that no significant (i.e. outcome level) programme adjustments were made during the period of the current CP. UNICEF Albania did not use the mechanism of a Strategic Moment of Reflection (SMR) during the period of implementation of the country programme. To address a reported increasingly difficult funding situation in 2017/2018 that was noted in the internal audit report from 2018,112 the Country Office prepared a document that set out potential programmatic adjustments in case of funding gaps.113 This document proposed a mid-term review of the country programme in 2019 as an appropriate moment to potentially revise financial planning assumptions.114 It also recommended the formulation of a fundraising strategy and an advocacy strategy that would take into consideration risks and factors mentioned in the document, both of which were developed.115 Besides these two documents, no further action took place and no mid-term review was initiated.
UNICEF internal respondents confirmed that the annual process of reflection and updating of programme strategies led to only minor adjustments, mainly to address emerging crisis-related needs. Evidence from stakeholders of other UN organizations corroborated these processes, which they noted serve as monitoring tools and opportunities for reflection by the Country Office.
Evidence from UNICEF and government informants indicates that ongoing consultations with stakeholders were held annually on a sectoral basis to reach mutual agreement for the upcoming year. Also, in the framework of DaO and UNDAF annual coordination meetings and thematic groups aimed to ensure a fine-tuning of different programmatic priorities and interventions. UNICEF key informants reported that they had the ability to be flexible within the programme in response to needs (facilitated by country office management), primarily around programme activities level; the CPD provisions at outcome-level are therefore respected. Many examples were provided to the evaluation team of specific initiatives across the programme sectors that were developed over the course of the country programme in response to emerging needs.116 UNICEF partners corroborated this, highlighting the ‘approachability’ of UNICEF Albania with respect to needs and opportunities.
112. UNICEF Office of Internal Audit and Investigations, 2018, Internal Audit of the Republic of Albania Country Office. 113. UNICEF Albania. Adjustment of programme strategies and targets, in case of substantial funding gaps, 2018. 114. Ibid, p. 5. 115. Ibid. 116. An example of this flexibility came from the education sector, where key informants indicated that UNICEF CO responded quickly to their needs for school opening after lockdown; justice sector participants also indicated UNICEF CO flexibility to respond to their needs for the infrastructure of “child-friendly interview rooms”, even though initially it was not part of UNICEF objective. FINAL REPORT 41
An important example of how UNICEF programming is responding to needs and changing contexts is the response to COVID-19. The pandemic has been a “huge game changer for education”,117 with specific issues emerging of internet access for vulnerable children and remote learning opportunities. UNICEF has sought to adjust implementation strategies to reflect the new realities of educational opportunities under the pandemic (see also Finding 4.4. on Effectiveness).
Use of new evidence to align with needs
While there is limited evidence of consultations with rights-holders at the design stage of the CPD as noted above,118 there is good primary and secondary evidence of the Country Office conducting or accessing various ongoing needs assessments. This has taken place over the course of the country programme and has incorporated the perspectives and needs of a range of stakeholders, notably with respect to the realization of child rights (see Annex 4 – Documentary Evidence). As an example, the recent evaluation of UNICEF’s contribution to the reform of social care services noted that “the intervention was adjusted to evolving national priorities: it embraced the local self-governance reform, and the corresponding new legislative environment, and was adjusted to the growing demand from government for expertise”.119
This process did miss some important global UNICEF developments that could have served as potential drivers of strategic re-positioning, however.120 Notably: UNICEF Strategic Plan 2018–2021; a paper on Theory of Change in UNICEF Strategic Plans 2018–2021; and the UNICEF Gender Action Plan 2018–2021, were all key developments that did not appear to be incorporated into planning and strategic decision-making over the CPD period.
3.1. 2 Evaluation Question 2 Implementation strategies
EQ2. Were the implementation strategies utilized relevant for the context, and the way it has evolved? Given the current context for children’s rights, to what extent should these implementation strategies be continued and/or refocused?
Finding 2.1
Initially implementation strategies were derived from the CPD, but divergence over time was not documented/ justified in subsequent planning – there is no overarching theory of change (ToC) for the CPD, and the Programme Strategy Note which included programme outcome ToCs is incomplete/unclear.
To achieve UNICEF’s programme outcomes within the component areas, a range of implementation strategies were proposed; these were described more generally within the CPD but the accompanying Programme Strategy Note provided more detail.121 This document notes the following “flagship strategies … critical for the ambition of any breakthrough achievements.”122 :
117. UNICEF key informant. 118. The CPD Programme Strategy Note cites use of “findings of the nationwide consultations with the communities, children and youth on the country’s Post-2015 Agenda”. Also, UNICEF key informants noted ‘informal’ consultations with children on specific programme elements at the time of the CPD design. 119. E Massey, C. Poni, M. Sammon, E. (2020). “Evaluation of UNICEF’s Contribution to the Normative Policy Framework of the Social Care Services Reform 2013–2019 in Albania”. UNICEF Albania. 120. For example: Albanian Demographic and Health Survey 2017/2018; Situation Analysis on Children Living with Disabilities “We all matter” (2018); Mainstreaming, Acceleration and Policy Support for Achieving the Sustainable Development Goals in Albania (2018); KAP qualitative study on “Child Marriage” (2018); UNICEF Strategic Plan 2018–2021; Paper on Theory of Change on UNICEF Strategic Plans 2018–2021; UNICEF Gender Action Plan 2018–2021. 121. UNICEF Albania, Programme Strategy Note 2017–2021. 122. Ibid, section 3, p. 23. 42 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
• Evidence generation, research and analysis to generate better quality, more disaggregated data; • Advocacy and high-level policy dialogue to create space for children’s issues in the national agenda; • Capacity building, primarily of government partners at higher-level managerial functions, (rather than trainings of frontline service providers) via “transfer of methodologies, tools and guidance, rather than directly through trainings”; and • Communication for Development targeting duty-bearers (regarding their accountabilities in relation to children) and children (with knowledge of their rights) and families (with knowledge of their rights and obligations as caregivers).
Strategies are again articulated under each programme outcome/component in the document, via narrative and individual theories of change for each of the three main programme outcomes. No overall country programme theory of change was presented as part of the CPD or the Programme Strategy Note. This would have been useful for capturing the overall vision and direction of the country programme. Importantly, theories of change are also used to guide decision-making within a programme period. The theory of change paper123 that accompanies UNICEF’s global 2018–2021 strategy may provide useful guidance for the future CPD.
The above-mentioned flagship strategies are not explicitly followed in each of the three outcome ToCs, but restated/ summarized as: advocacy and policy dialogue; convening partnerships; using Communications for Development (C4D) to influence social norms and behaviour. These high-level strategies are deconstructed into a detailed list of “implementation strategies” which are worded more akin to intermediate outcomes/outputs, for example: redesign of systems; stakeholders being empowered with skills and knowledge; data collection systems supported; surveys/ studies supported; among others.124 Thus, actual programme implementing strategies, while derived from and coherently linked to the specific outcomes within the CPD, are general in nature as articulated within the Programme Strategy Note, and intermediate outcomes/outputs have been detailed imprecisely as implementation strategies within the theories of change.
As noted in the EQ1 discussion, the Programme Strategy Note was not fully utilized and the result was that it was not updated to reflect some of the changes made to the final draft of the CPD, hence some divergence between the two documents exists. Indeed, there is inconsistency between the three programme outcomes as articulated in the CPD (strengthened governance for equity and social inclusion; protection and justice for children; programme effectiveness) and an additional outcome in the associated Programme Strategy Note (child rights monitoring and public oversight, included under outcome three – programme effectiveness) in annual reporting. Such inconsistencies should be avoided or at least clarified in foundational strategic and programmatic guidance.
Finding 2.2
Annual review processes have led to a more ‘organic’ approach to deciding on implementing strat- egies based on identified priorities. This approach has supported UNICEF to maintain its relevance although without systematically documenting or justifying its rationale for decisions.
Finding 2.3
Priority programme results areas have evolved year-on-year over the course of the country programme in response to the prevailing context in terms of perceived needs and available resources.
There is evidence of UNICEF adopting a flexible approach to both programme priorities and associated implementation strategies across the life of the CPD. Subsequent to the establishment of the CPD in 2017, the Country Office utilized annual reporting and planning processes to update the intended implementation strategies for the coming year. The strategies are articulated across both annual management plans and annual workplans (AWPs) that set out clear change strategies/activities linked to programme outcomes, the stakeholders responsible, and budgets. It is evident that the programme priority results areas over the course of the country programme have evolved each year in
123. https://www.unicef.org/about/execboard/files/2017-EB11-Theory_of_Change-2017.07.19-EN.pdf 124. UNICEF Albania, Programme Strategy Note 2017–2021, section 12 (annexes), figures 3 to 5 – theories of change for outcomes 1 to 3. FINAL REPORT 43
response to the prevailing context (needs/resources). This is a positive example of how UNICEF adapts its approaches (see Annex 6 for a comparison of priority areas year-on-year).
The reconstructed theory of change prepared by the evaluation team (see Annex 2) and validated by the Country Office and evaluation reference group summarized the range of implementation strategies articulated by the Country Office over the course of the country programme:
• Generation, validation, utilization and dissemination of knowledge; • Mobilization of technical expertise; • Identification and promotion of intersectoral cohesion opportunities; • Mainstreaming disability, gender and/or ethnicity-based exclusion; • Advocacy with government; • Raising awareness of duty-bearers/rights holders of child rights and equity-sensitive normative frameworks and services; • Learning and domesticating innovative approaches and techniques; • Mainstreaming emergency preparedness and response; • Capacity strengthening; and • Leading, convening and coordinating efforts.125
The evidence from key informants and secondary data is that most strategies are originally articulated within the CPD, were well-chosen and clearly met the context for child rights in Albania at the time of the CPD development insofar as:
• They were grounded in the published contextual analyses available at the time; • They were in line with the needs and priorities as determined by other key child rights stakeholders in Albania at the time; and • They were developed as part of a continuum of reflection and planning on the part of the Country Office.
These three determining factors were applied to Country Office programming over the course of the CP and thus choice of strategies continued to be relevant to the changing context and to UNICEF’s mandate. For example, the strategy of learning/domestication of innovative approaches and techniques (as well as dissemination of knowledge) via modelling was cited by several stakeholders as important and impactful, particularly with respect to providing a path to sustainable financing of services at municipality levels.
“They [UNICEF] brought many models, which … we contextualized and decided on the one we would use.”
Key informant (education), Government of Albania
This modelling approach was used as a low-cost method to simulate equitable, sustainable and ‘marketable’ (by local authorities to the Government) approaches to management of a range of sectors, specifically:
• Pre-school education; • Social services; • Home care: • Transformation of public residential care institutions into “family strengthening hubs”.
Modelling of the latter was cited in the 2017 CPD as showing “promising results” and successive workplans between 2017 and 2019 included it as a strategy for scaling up.
125. C4D, a central pillar of the original draft of the CPD Programme Strategy Note, was not referenced in subsequent annual manage- ment plans from 2018–2020. 44 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
Finding 2.4
Programming choices and implementation strategies were well-aligned with both UNICEF’s mandate articulated in the CPD and associated plans and the priorities of key (albeit not all) stakeholders.
Respondents at municipality and central levels variously described UNICEF’s programming choices and implementation strategies as “well-aligned” (with their own and UNICEF’s mandates), “satisfactory” in terms of how approaches are structured, and confirmed that UNICEF’s approaches were, and remain, in line with the priorities established in the CPD. This external evidence triangulates well with that of UNICEF respondents, many of whom noted an ongoing process of updating and amending the above strategies to reflect priorities and needs over the course of the CP on the basis of what they perceived to be children’s needs. An example is how UNICEF’s technical expertise, capacity- strengthening, advocacy and innovation strategies have been progressively developed in the area of risks for children in online access over the course of the country programme period. While not necessarily adopted after full stakeholder consultation (as discussed under EQ1), this area of programme focus for UNICEF has been progressively refined via research across different dimensions (directly with children, among legislators, police/judiciary etc.) to shape UNICEF’s response modalities around this issue.126
In considering the Country Office’s effectiveness, EQ4 identifies a number of achievements and factors that have been key to UNICEF’s contribution to these gains. They partially map across the implementation strategies as articulated above:
• Fostering close relationships with and convening key stakeholders/decision makers at all levels; • Promoting mutual learning, working relationships and participation by all; • Support to civil society (particularly those representing minority/vulnerable groups); and • Research initiatives that highlight gaps and opportunities.
These factors are explored further under EQ4.
3.2 Effectiveness
3.2.1 Evaluation Question 3 Organizational structure
EQ3. To what extent is the organizational structure vis-à-vis programme design fit-for-purpose to deliver results for children? Could any changes be made to increase effectiveness?
Finding 3.1
Evidence indicates that the organizational structure as of 2017 was systematically planned to appropriately align with the CP and the results areas. The office restructuring, including the introduction of specific positions, was designed to fit and serve the new CPD 2017–2021 both organizationally and programmatically.
Finding 3.2 Evidence suggests the organizational structure is fit for purpose overall given the context, with some areas requiring consideration such as the ratio between permanent and temporary consultants which may have disadvantages in terms of the workload for core staff. This in turn may limit the capacity to make changes or strengthen the programme to deliver, for example, through strategic intersectoral working or embedding gender mainstreaming.
126. CSO key informants. FINAL REPORT 45
Based on the CPD, the UNICEF Programme Plan shows that the country programme proposed an organogram comprised of 20 staff members, with two new positions created.127 Several smaller changes were applied to programmatic staff positions, mainly in the title of the officer/specialist (e.g. ‘Social Protection Specialist’ to ‘Social Policy Specialist’) and reporting lines.128
Prior to this programme cycle, annual reviews starting from 2014 show that UNICEF Albania systematically and strategically planned the organizational structure for the CP 2017–2021 even before its design phase. UNICEF internal stakeholders stated that several positions were abolished or changed, to introduce more relevant expertise, and this was confirmed by document review. The development of the Country Programme Management Plan for the new cycle (2017–2021) was preceded by a review of existing office positions to assess capacities to sustain the mix of skills required to achieve the results set out by the new country programme. This assessment started in 2014 after the mid-term review the Government of Albania-UN Programme of Cooperation 2012–2016.129
Feedback from external key stakeholders was limited for this finding. Internal stakeholders confirmed that the organizational structure was adapted to the needs of the new programme. The roles of programmatic staff were intended to support the overall approach of systems strengthening and liaising with the Government to do so, the latter being influenced then (and now) by other dynamics (as mentioned in EQ 1) such as the EU accession process. A secondary role was to support the need for technical expertise for upstream policy work. Each programmatic result/output should have a staff member who was the equivalent of a highly qualified external consultant but also with a background in planning and management, and with good representational skills for liaison with their government counterparts.130 All these changes took place before the new programme cycle, so by the start of the 2017–21 CP cycle, UNICEF Albania was prepared organizationally and programmatically.
Decision-making around staffing structure
As mentioned, the organizational structure served the CP cycle 2017–2021. The CPD notes that the staffing structure brought together for the CP was founded on lessons learned and experiences from preceding phases. These included: the importance of increasing intersectoral collaboration; the need to strengthen accountability mechanisms and key performance indicators for national and subnational institutions; performance standards; formalized job descriptions; and management and monitoring tools.131 The associated Programme Management Plan 2017–2021132 added the need to upgrade the Child Protection profile and distinguish this area from Social Protection, based on consultations with external stakeholders. However, there is no evidence of this need in previous Annual Reviews, or in the CPD per se, so it is not clear what this decision was based on.
Internal UNICEF stakeholders indicated that the structure primarily addressed the need for staff to respond to UNICEF’s approach to systems strengthening. Therefore, within the internal structure programmatic staff is usually responsible for a specific sector, e.g. education, justice, health and social protection. In more detail:
• Outcome 1: Strengthened governance for equity and social inclusion. There are three programme specialists responsible for the respective outputs on health, education and social protection, besides the overall responsibility of the Deputy Representative. • Outcome 2. Protection and justice for children. Currently, there are two programme specialists, one international and one national (justice for children) and two programme officers. All three (national specialist and officers) report to the international programme specialist, who is responsible for the overall child protection portfolio.
127. These two positions were: an international child protection specialist (at P-3 level) to boost UNICEF’s programme analysis and articulation of sensitive child protection issues, based on organizational positions; and a programme officer (Harmonised Approach to Cash Transfers/HACT) to provide professional technical, operational and administrative assistance throughout the programme cycle. This would take place through the support to HACT planning, implementation of HACT activities, the review and use of HACT reports, networking and managing the inter-agency partnership. 128. UNICEF Albania Country Programme Management Plan 2017–2021, p. 5. 129. UNICEF Albania COAR 2016, p. 16. Section on Human Resources; also UNICEF Albania COAR 2014, p. 14, same section. 130. UNICEF Albania COAR 2014, p. 14. 131. UNICEF Albania CPD, pp. 4–5. 132. Albania Country Office Country Programme Management Plan 2017–2021, April 2016. 46 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
• Outcome 3: Programme effectiveness. This outcome has a programme officer for communications and a programme specialist on child rights monitoring who is responsible for the four outputs of this outcome, as well as the Deputy Representative.
Despite one of the rationales for the structure being to increase intersectoral working, the document review of Annual Working Plans and Annual Reviews and contributions of internal UNICEF stakeholders indicate that in practice there is limited intersectoral cooperation between the different target sectors of UNICEF Albania, even within the same programmatic area (see EQ8). For example, prior to 2018 there is limited evidence of more than one programmatic staff member being responsible for intersectoral results.133 Analysis of internal documentation indicates a shift in annual planning and review reports after 2018. A positive outcome of this is clearer evidence on internal collaboration, where more than one staff member was responsible for the same priority result. However, it is noted that certain staff members with a more intersectoral profile, e.g. the Child Rights Monitoring and Communications Officers, were allocated responsibility for more intersectoral priority results, adding to workloads rather than necessarily intersectoral collaboration.134 Therefore, even though intersectoral collaboration within the office structure, is emphasized in the CPD and Programme Strategy Note, the Country Office has not achieved this effectively and there is still a need to invest in this direction.
Finding 3.3 There were no significant changes to the organizational structure during the programme period. While the organizational structure remained broadly relevant to the country context, there were shifts within UNICEF globally. There is no evidence that these changes were reflected (or even considered) in country priority decision-making; and therefore, these institutional priorities appear to have had no impact on adapting the programme design and the corresponding organizational structure.
In terms of “fitness for purpose” to achieve the overall goals of the CP based on internal staff technical expertise versus that of external consultants, evidence from internal documentation indicates that UNICEF frequently outsources expertise to address internal shortfalls in capacity. Even though they are mainly lower-cost consultants (low-value per the CO nomenclature), in 2018 as many as four times more consultants than programme staff were retained (35:23 national/12 international). In 2019 the number was lower (18:8 national/10 international), but still high in comparison to permanent staff.135 In one instance a national consultant was part of the team responsible for delivering on a priority area.136 Taking into consideration that consultants may have limited authority and sense of ownership for specific results, a high reliance on consultants for delivering programme elements may place core office staff under management pressure in addition to achieving results for their portfolios. At an institutional level, the temporary nature of consultants can have a negative impact on institutional memory, which impacts also intersectoral collaboration, as in the case of gender mainstreaming. In general, permanent staff has an enhanced capacity gained over the years in intersectoral issues compared to external consultants. Issues such as these need to be considered by the CO, in particular to assess whether there are challenges to the overall effectiveness of UNICEF’s interventions. Evidence from external stakeholders triangulates with this, with informants recommending mitigation strategies such as developing long-term agreements with regional think tanks.137
Being part of the UN’s “Delivering as One” (DaO) initiative is a further responsibility which places pressure on the workloads of core staff. Albania has been one of the pilot countries for the initiative since 2007 and all evidence indicates that UNICEF has strived to uphold the UN coherence principles. However, the 2017 internal audit notes that this effort contributes to staff overburdening.138
Firstly, given that the strategic direction for the Albania CO did not substantively change since 2017, the evaluation did not expect to see any changes reflected in the organizational structure. That said, document review and evidence
133. UNICEF Annual Review Meeting, 2018. 134. UNICEF Albania Annual Work Plan, 2019. 135. UNICEF Albania Operations Team annual meeting presentations, 2018 and 2019. 136. UNICEF Annual Plan 2018, p. 8. 137. UN stakeholders. 138. UNICEF Office of Internal Audit and Investigations, 2018, Internal Audit of the Republic of Albania Country Office. FINAL REPORT 47
from internal key informants indicates that throughout the country programme period there were ongoing changes to the Country Office core team and of incumbents in positions subsequent to the organizational restructuring of 2014–2016.139 Even though the size of the core team of programmatic staff remained overall the same, and there was no structural change, the changes at managerial level impacted implementation pace and added to the burden of the core programmatic and general staff, due to inevitable differences in leadership styles and the time required for role familiarization. Positive changes in the general staffing structure were also noted, e.g. a division of the human resources and finance roles and responsibilities was welcomed by informants as it clarified roles and responsibilities.
As explained in EQ1 the Country Office did not initiate a formalized mid-term review or SMR and its activities were based mainly on annual reviews to update the CP. The same is true for the organizational structure: the evaluation did not identify any plans or reports of strategic changes in the staffing composition and structure either in document review or in responses from stakeholders. Yet during 2018 there were several important strategic changes in the external UNICEF environment. New strategies were approved by UNICEF at the global level, for example, the UNICEF Strategic Plan 2018–2021, the Gender Action Plan 2018–2021, and a new paper on Theory of Change. The latter paper on Theory of Change suggests an overarching ToC for the CP and brings updated change strategies. These included gender-responsive programming (for more information see EQ5), harnessing the power of business and markets for children, and fostering innovation for children. These were not originally included in the Albania CPD and there is no evidence that they were incorporated into CP strategic decision-making during the course of the programme cycle. Therefore, there was no rationale at country office level to reconsider the staffing structure which it may have done if these new areas had been incorporated.
3.2.2 Evaluation Question 4 Achievement of results
EQ4. To what extent were programme results achieved? What were the key factors (internal and external) influencing the achievement of results?
Evaluation question 4 (revised for clarity from the original question structure in the evaluation Terms of Reference) is expanded by two sub-questions against which five assumptions were tested. Each of the two sub-questions is treated distinctly. EQ4a looks at the realization of child rights including for the most vulnerable, and EQ4b considers the strategies that the CO implemented to improve the performance of duty-bearers. In total there are nine key findings across the two sub-questions within EQ4.
EQ4a. How has the CP contributed to the observed progress in the realization of child rights? Particularly the most vulnerable boys, girls and youth at risk of being left behind?
EQ4b. To what extent have the strategies contributed to improving the performance of government institutions/ service providers (at national and subnational/local level)?
Finding 4.1
There are strong, albeit anecdotal, perceptions among informed stakeholders that UNICEF has contributed to an improvement of the situation of children across the dimensions of social and child protection, governance and policies, and in services. Many of these improvements are reported by stakeholders to be of benefit to very vulnerable and marginalized children.
139. For example, in 2017 a new international Child Protection Specialist was appointed, in 2018 the office had a new Country Representa- tive, and the Deputy Representative position was held by four different people from 2017 to 2020. 48 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
Table 2: Progress in realization of child rights reported by UNICEF stakeholders
Protection Governance Services
Improved quality of parenting skills New/improved laws and policies More inclusive education for minority groups* and norms for protection of children* Changes in attitudes towards physical Improved justice services and Improved capacity of teachers* discipline legal outcomes for children Reductions in bullying in schools Reductions in child statelessness Reduced dropouts and increased registrations in schools/preschools* Improved awareness of child rights* Improved government monitoring Minorities and children with disabilities have of child rights improved access to education* Improved awareness of the dangers Improved treatment of children in Victims of crises have better access to essential of online child abuse* contact with the law* services Increases in children making formal Improved health policies (e.g. Increased educational performance/grade complaints to justice services breastfeeding and nutrition) promotion Increases in children using informal Improved skills of child protection/social workers complaints services Reducing numbers of children in Increased numbers of children using social institutions* assistance Improved quality of online education Improved capacity of health personnel to apply standard protocols for maternal and child health and nutrition (preventive services) * Triangulated with secondary quantitative or qualitative data – see following section
The primary evidence gathered by the evaluation team from UNICEF Albania stakeholders comprising informed feedback from representatives of the Government, judiciary, civil society, national institutions and other international agencies presents a broad (albeit anecdotal) consensus of an improved situation for children within Albania as a result of UNICEF’s interventions and support.
A substantial number of evaluation informants reported benefits to children across different dimensions and different services within Albania. These informants are drawn from all stakeholder groups with which UNICEF works, including those at municipality level. The following table presents a summary of responses from interviewees on the areas where they have perceived benefits of UNICEF support and activities over the past three years.
Finding 4.2 Quantitative evidence of the outcomes of UNICEF’s initiatives on children is lacking – notably around key developmental indicators, which lack recent data. Data from youth (via U-Report) suggest that realization of rights has stalled or is not meeting expectations among young people.
It is important to note that the evidence base on which informants reported the above is largely anecdotal; only a few respondents were in a position to cite quantitative data to support their assertions. Examples of where data was available were educational performance (via PISA international education rankings), numbers of children held in detention/institutions and numbers of children accessing support services (such as telephone or online help services).
“We understood our role and what verdict we should give to the prosecution office after inspecting the mediation efforts and documents.”
Juvenile justice key informant
Other respondents provided specific examples of how UNICEF support since 2017 has improved the quality of their work in relation to children, with the understanding that improved policies, governance, services and supports are FINAL REPORT 49
leading to an improved realization of rights among children. However, the wide range of positive responses to the evaluation questions regarding UNICEF’s outcomes on children’s rights, when viewed in aggregate, is compelling evidence of progress in the realization of positive results.
Key factors identified by the evaluation that drive UNICEF’s contribution to these gains are:
• Fostering close relationships with key stakeholders and decision makers to drive policy improvements and leverage UNICEF’s convening power between different stakeholders; the central role it plays in realizing children’s rights for both government and non-government stakeholders; and its capacity to engage in dialogue with these actors and bring them together (see finding related to emergency response and finding 8.1, below); • Promoting a mutual learning approach that increases understanding and focuses on performance, working relationships and participation by all, rather than a didactic or ‘top-down’ approach (discussed under capacity- building below); • Directly funding support to poorly resourced actors, particularly civil society and minority group organizations that may otherwise struggle to address their mandate (a concern repeated by many CSO key informants); • Targeting research initiatives that highlight gaps and opportunities for future work that can be grounded in robust and recent analysis, e.g. mapping of social protection, risks to children online, child marriage analysis, etc.; and • Building relationships with levels of governance i.e. good engagement with municipalities in line with the decentralization agenda, which requires new competencies and skills among subnational leaders, but with as- yet limited guidance from central government, leading to gaps in services and responses to needs (discussed further under Sustainability, EQ7).
Key development indicators
Analysis of secondary (i.e. reported) data in relation to intended results of UNICEF’s programming provides an opportunity for more quantitative description of the extent to which results may have been achieved and a point of triangulation with the above qualitative data.
The first point of analysis is with respect to UNICEF’s targets as articulated in the CPD and associated results framework and other programme documents (including the CPD Strategic Note, notwithstanding programmatic misalignments between the CPD and Strategic Note discussed above).
An analysis of the UNICEF RAM reports from the 2017–2019 period (the time frame for which data was available) indicates that there has been very limited internal reporting on results against either baselines or targets. The table below presents a summary of reporting against indicators across the available data period (for the full analysis see Annex 9).
Table 3: Summary of UNICEF Albania results assessment module reporting against indicators
Number of Indicators Baseline Number of years reported Outcome 1: Strengthened Governance 1 year for each of 3 indicators for Equity and Social Inclusion 7 7/7 0 years for 4 indicators Outcome 2: Protection and Justice for 2 years for 2 indicators Children 5 5/5 1 year for 3 indicators Outcome 3: Programme Effectiveness 3 years for 6 indicators 2 years for 2 indicators 12 8/12 1 year for 3 indicators 0 years for 1 indicator Outcome 4: Programme Management 3 years for 3 indicators (incorporated Outcome 5 in 2019) 4 4/4 2 years for 1 indicator
While baselines values (and end-of-programme cycle targets) were established for most indicators (using nationally- available data at the time) when the CPD was established, the reporting on these indicators was limited across the 50 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
Table 4: Selected UNICEF Albania results assessment module indicators
Indicator Baseline Target Most recent data
Share of GDP allocated for education (Outcome 1) 3.5% 5% 3.5% (2018)
Existence of a national management information system that Weak Strong Medium (5 points (2017) includes disaggregated data on nutrition (Outcome 1) (0 points) (7 points) Rate of children (age 0-17) in residential (public and non-public) 84 42 83 (2017) care per 100,000 child population (Outcome 2) Rate of children with disabilities (age 0-17) in public institutions 7 3 8 (2017) per 100,000 child population (Outcome 2) Rate of children in detention (per 100,000 population aged 14-17) 179 105 108 (2018) (Outcome 2) Of child-related valid complaints, % of complaints for which reme- 65% 70% 57% (2018) dial action taken by the People’s Advocate, annually (Outcome 2) Existence of systems/mechanisms/capacities to routinely collect, weak/ established underway analyse and publish child-rights indicators (Outcome 3) fragmented
subsequent years, presenting a challenge to the Country Office in ascribing results from its activities. Further, while baselines and targets were established during the design phase of the CPD results framework, they and subsequent reporting were not disaggregated by sex, age or vulnerability status. This undermines the ability of the Country Office to report on whether changes to child rights are realized equitably.
Those programme-related indicators for which reporting took place show limited results in terms of positive progress. The table below presents some examples of the indicators and their most recent data compared to the baseline and the indicator targets (the full dataset comparison is in Annex 9). Only two indicators (the rate of children in detention and a national management information service for nutritional data) have seen a substantive positive change, with other indicators seeing no change, or even regressing. The evaluation is careful to note that lack of recent data (i.e. 2019 or later) prevents any significant determination of a trend. There is also a challenge of attribution of changes (either positive or negative) of many of these macro-level indicators to UNICEF actions alone.
The evaluation undertook a similar analysis on secondary data from a range of publicly available administrative data sources, including several that UNICEF supported or committed to contribute to over the course of the country programme.140 The comparison of these data sources with UNICEF’s headline indicators cited in the CPD present the same challenges as UNICEF’s internal reporting, in that data are unavailable for the UNICEF indicators, or incomplete for the time period of the country programme (up to 2018 at the latest, and in many cases predating 2017) and thus cannot contribute to answering the evaluation questions. The analysis table is presented in Annex 9. Nonetheless, they present some useful data points on positive changes to the realization of children’s rights, as follows:
• Good level of children’s awareness of their rights (82 per cent of children aged 10-12);141 • Increased feelings of efficacy among youth to make Albania more democratic and more inclusive of human rights;142 • Increased confidence among youth in their ability to positively influence their future;143 • 25,000 pre-school children in four municipalities benefiting from improved teaching methodologies;144 and • Improvements in Albanian children’s learning outcomes, as measured by PISA results for mathematics, with a 2018 average of 437 points compared to 413 points in 2015.145
140. The databases reviewed were the Demographic and Health Survey (DHS, 2018), the Global Health Observatory, Multiple Indicator Cluster Survey (MICS, 2005), Social Institutions and Gender Index (SIGI), INSTAT, TransMonEE, the Survey on Income and Living Condi- tions (SILC), and the Program for International Student Assessment (PISA). 141. Save the Children (2019), “Children’s World - A National Survey of Children’s Well-Being”. 142. World Vision Albania and Kosovo, 2019, Baseline and Action Research Report: Youth Standing Up for Human Rights in Albania. 143. Ibid. 144. UNICEF Albania Annual Report, 2018. 145. UNICEF Albania Annual Report, 2019. FINAL REPORT 51
The data available from the U-Report survey conducted as part of the evaluation in July 2020 indicates that, among young people, there is a perception that realization of rights and delivery of services has stalled or is regressing (see Figure 2 below).146
As the chart illustrates, in relation to perceptions of whether authorities respect the rights of young people, more respondents felt that the situation was worsening over the past three years than any of the other categories, although only 38 per cent of respondents noted this.
Aggregated across all of the services for children and young people, most respondents felt there was no change – for better or worse). This paints a somewhat negative picture of the perceptions of the young people surveyed, over a time when progress on many policy and structural issues has been reported. Notwithstanding that much of the work of UNICEF focuses on younger age groups than those who participate in the U-Report surveys, the results of this survey triangulate with other evidence which indicates that many of the policy changes that have taken place over the past several years have yet to gain significant traction at rights-holder (i.e. community) level.147
Previous U-Report surveys conducted among youth in Albania provide some perspectives on opportunities for UNICEF for the future. Examples include:
In 2018, 92 per cent of youth agreed that a specific law on youth was required in Albania, even though youth employment, social protection and health are regulated by special laws;
• In 2019, 91 per cent of respondents felt that the quality of pre-primary school programmes needed improvement; • A follow-on poll in 2019 saw 59 per cent of respondents stating that children with economic difficulties need more support to achieve success in education. Only 9 per cent said the same for Roma/Egyptian children, 20 per cent for children with disabilities, and 13 per cent for girls in remote areas; and • A 2019 poll on online bullying indicated that 33 per cent of respondents (slightly more female than male) were victims of online bullying, mostly via social media platforms, underscoring the need for work in this area.
Figure 2: Young persons’ perspectives on rights in Albania (U-Report Survey 2020)
Authorities Respecting Rights Health & Nutirition Services Learning & Education Services Welfare & Social Care Services Legal, Police, Justice Services Protection Services
Better Worse The same Don’t know
146. Note that U-Report respondents are not likely to be representative of the overall youth population (noted in the limitations section), so this data should be viewed cautiously. 147. The small (200 respondents) and unrepresentative (due to self-selection by respondents as U-Reporters) sample means this data was interpreted with caution. 52 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
Contributions to equity
UNICEF in Albania, as globally, has placed equity considerations to the forefront of its plans and work. The CPD highlighted the persistence of equity gaps in Albania, specifically noting the risks to children from vulnerable or marginalised groups related to poverty, ethnicity, disability, migrant/refugee status, detention or residential care, including children out of school (for whatever reason).
The CPD notes that children from these groups “remain chronically deprived of developmental opportunities in health, education, civic participation and other areas. This deprivation hampers the realization of their rights and narrows their prospects to thrive.”148
To address these inequities for children, UNICEF stated that it would focus on strengthening policy frameworks and implementation mechanisms in the health, education, social protection and justice sectors. It would also seek to eliminate bottlenecks in data (i.e. statistical/administrative data and qualitative analyses) or resources that hinder equitable realization of child rights. Learning assessment systems, in particular, were noted as a means to help eliminate ethnicity/gender-based inequities.
The CPD also alludes to promotion of pre-existing equity-focused standards of care and services for children throughout Albania, thus at subnational/local level, leveraging the decentralization process. In practice, over the course of the country programme, UNICEF has integrated these elements into much of its programme plans and strategies, with a specific equity-related governance outcome reflecting the language of the CPD (as well as integrating equity considerations across other programme outcome areas).
As discussed under EQ4b, UNICEF has contributed strongly to positive equity-based policy change for children over the country programme period. Feeding the basis for policy change have been several data-related initiatives (in line with the stated aims of the CPD) that provide a robust basis for future work. Two strong examples are the 2017 joint UNICEF-UNFPA assessment of the child marriage phenomenon in Albania and a 2019 analysis of living conditions of children with disabilities and their families.149 UNICEF also supports institutional data collection activities in Albania via the National Institute of Public Health, and INSTAT – both key actors in the generation and dissemination of data related to vulnerability and marginalization.
With respect to programme performance, primary qualitative data from UNICEF stakeholders at all levels, summarized in Table 4 above, provides anecdotal evidence of positive equity-related changes in the areas of children in institutions, in detention, at risk of statelessness, inclusive education, school attendance/dropout, social assistance services (both quality and availability) and risks of online harm.
Reported progress in some of these areas can be triangulated with secondary data to provide insight into changes to the realization of children’s rights, with some specific elements related to equitable realization for the vulnerable/ marginalized groups targeted by UNICEF. Examples include:
• Increase in Roma children’s (aged 3–6 years) access to early education from 26 per cent in 2011 to 66 per cent in the school year of 2016–2017;150 • A C4D campaign targeting discriminatory social norms against Roma led to an improvement from 36 per cent to 46 per cent in the awareness of Roma parents of their responsibility for enrolling children in school and an increase of 15 per cent to 36 per cent among non-Roma parents approving of their children developing friendship with Roma children;151 • Advocacy for changes in civil registration law leading to registration of 1,000 children without legal documentation and at risk of becoming stateless and unable to exercise their rights;152
148. UNICEF Albania Country Programme Document 2017-2021, pg. 4 149. Rogers & Sammon, 2019, We All Matter! Situation Analysis of Children with Disabilities in Albania 150. UNICEF Albania Annual Report, 2017 151. UNICEF Albania Annual Report, 2018. 152. Ibid. FINAL REPORT 53
• A 32.5 per cent decrease in the number of children in detention and pre-trial detention between 2017 and 2019;153 and • Improvement in the situation of students at risk of school dropout in four target municipalities.154
However, in addition to these positive reports of realization of children’s rights, there is also data which indicates regression or ongoing challenges in terms of equity in relation to realizing rights to education, health, social welfare and other services. Examples include:
• DHS reports of a decrease in primary school attendance from 95 per cent to 93 per cent between 2009 and 2018;155 • Ongoing challenges among minority groups (Roma and Egyptians) in accessing education, birth registration (and related public/social services); • Poor quality of education in small and remote schools using multi-grade teaching;156 • Neonatal mortality (8 per 1,000 live births), infant mortality (9 per 1,000 live births) and under-5 mortality (10 per 1,000 live births) all showing an increasing trend in the last three years;157 • Anaemia rates among children under five increased from 17 per cent in DHS 2008/09 to 25 per cent in DHS 2017/18; • Disparities in child nutrition outcomes by household wealth, mothers’ education, and urban/rural residence; • Albanian social protection programmes falling short of addressing in a comprehensive manner the multiple vulnerabilities faced by children and families;158 • Social care is limited to cash transfer schemes, subsidies and tax exemptions, while services are fragmented, with limited coverage;159 • No specific programmes to ensure income security for children (although many are covered by programmes targeting poor and vulnerable households);160 • Only 43.2 per cent of children aged between 6 and 17 receiving disability allowance registered for school;161 • Disparities in educational attainment between urban and rural regions estimated to differ by two years of schooling;162 • Disparities between the number of referrals of potential child abuse materials from international law enforcement agencies (5–15,000/year) to Albanian police and the actual number of cases investigated in the country (3–4/year);163 and • Emerging vaccine hesitancy amongst parents, with MM1 coverage rates below 94 per cent in 2019, including regions with MMR1 vaccine coverage as low as 64 per cent.164
There is evidence to suggest that these examples reflect the challenge of ensuring that the strong policy and capacity achievements that UNICEF has contributed to in Albania translate into robust and equitable realization of children’s rights. This challenge and its associated needs have been cited by numerous key informants to this evaluation, supported by secondary data. In addition, a 2019 evaluation of UNICEF’s contribution to social care services reform in Albania noted that “the normative policy framework is not yet sufficiently implemented to have had a measurable impact on rights holders, beyond anecdotal evidence and individual stories.”165 This is particularly relevant when the intersectional nature of many of these challenges are taken into account – in other words,
153. Ibid. 154. Ibid. 155. UNICEF Albania Communication and Advocacy Strategy 2018–2021. 156. Ministry of Education, Sports and Youth assessment, cited in UNICEF Albania Annual Report, 2019. 157. United Nations Inter-Agency Group for Child Mortality Estimation (UN IGME), 2020. 158. UNICEF Annual Report 2019. 159. Ibid. 160. Ibid. 161. Rogers and Sammon, 2018, cited in UNICEF Albania - Education Sector Appraisal, 2019. 162. UNICEF Albania - Education Sector Appraisal, 2019. 163. 2020 Albania - UNICEF input to 2020 EU Enlargement Package. 164. WHO/UNICEF Joint Reporting Form 2019. 165. Massey, C. Poni, M. Sammon, E. (2020). “Evaluation of UNICEF’s Contribution to the Normative Policy Framework of the Social Care Services Reform 2013-2019 in Albania”. UNICEF Albania. 54 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
children from poor families or otherwise marginalized are more likely to come from rural areas with fewer and worse public services and welfare mechanisms and thus are less likely to enjoy full realization of their rights.166
For example, government key informants working in the social welfare and protection sectors noted that they were still familiarizing themselves with the Code of Justice for Children and the Law on Protection of the Rights of Children, and not yet in a position to operationalize many of the provisions. They did note achievements in terms of improved equity/access of vulnerable groups to services such as education, via school fee waivers for children with disabilities and from Roma and Balkan Egyptian minorities, and funding for child protection units, but implementation of many of the provisions of the laws is hampered by a lack of resources.
Other examples cited by informants were:
• Children’s justice: informants noted a lack of resources available to assign specific prosecutors to cases relating to child abuse. • Children with disabilities: The UNICEF-supported analysis of living conditions of children with disabilities and their families noted difficult conditions for this vulnerable group and “provides a good baseline for future work of the Government” but a respondent noted “no progress” as yet in addressing the needs clearly identified in the report.167 Further, Albania has not ratified the optional protocols to the Convention on the Rights of Persons with Disabilities, a potential target for future UNICEF advocacy.
“Albania is in a space where good laws are in place, but they are not implemented,”
NGO key informant
Finding 4.3 UNICEF responses to crises have been prompt, timely and targeted, informed by assessments and in collaboration with diverse stakeholders, although not directed by a pre-established emergency preparedness plan.
Finding 4.4 There is considerable uncertainty around the future trajectory of COVID-19 and its impact on both the rights of children and the operating environment for UNICEF. Nonetheless, some future directions of programming can be predicted with some reliability – e.g. increased emphasis on online activities (education, work, communication), socioeconomic challenges, social care and protection challenges.
There is considerable primary and secondary evidence that UNICEF was quick to respond to the crises faced by Albania in 2019 and 2020 and has had a positive impact on affected populations.
Earthquake 2019
UNICEF Albania participated in several responses to crises over the period of the country programme, most notably a magnitude 6.4 earthquake in November 2019 centred in Durres that affected 202,291 people in the country, 47,263 directly and 155,029 indirectly, with disruption of schooling for more than 12,000 children and displacement of 17,000 people. The earthquake caused 51 fatalities and injured at least 913 people.168
UNICEF responded with support to the Government via six Child-Friendly Spaces and four Temporary Learning Spaces that served up to 1,400 children. It also reported building the capacity of national and local actors involved in the child protection response, disaster risk reduction and shock-responsive social protection, improving their knowledge and skills in implementing and monitoring humanitarian standards through technical assistance and mentoring.169
166. Development partner, CSOs, government key informants. 167. CSO key informant. 168. Albania Post-Disaster Needs Assessment Report - February 2020. 169. UNICEF Albania Country Office Annual Report 2019. FINAL REPORT 55
For the 2019 earthquake, UNICEF led the education component of a joint Post Disaster Needs Assessment. This was used as the main reference for a donor conference on international assistance to the earthquake. UNICEF also commissioned a study to assess the post-earthquake situation of families and children residing in temporary shelters and their needs for recovery in December 2019, which informed ongoing response activities throughout 2020.170
Key informants cited as a positive practice UNICEF’s engagement with service providers to work on the provision of a direct cash transfer programme as a post disaster response intervention. A key strength noted was that UNICEF could use its convening power to bring together stakeholders with the capacity to undertake cash transfer approaches and ensure effective coordination.171
As part of its response to the 2019 earthquake, UNICEF, under the overall One UN mechanism, also participated in the coordination mechanisms for the response by co-chairing the Working Groups for Education, Child Protection and Humanitarian Cash Transfers (HCT). Testimony from key informants is that UNICEF worked quickly and efficiently with other actors to respond to needs, indeed, is “among the best” of the various agencies that were engaged in the response.172 As part of its response, UNICEF also piloted an HCT programme through municipal social protection delivery systems in Korça, Shkodra and Durres.
COVID-19
In 2020, Albania experienced nationwide disruption to movement, employment, the economy and social services subsequent to the restrictions adopted in response to the COVID-19 pandemic. While simultaneously adjusting its day-to-day operations to safeguard the health of its own staff, UNICEF moved quickly to adjust its programming in response to the challenges presented to families and children as a result of the restrictions imposed on movement, and the economic shocks caused by the COVID-19 pandemic response in Albania in March 2020.
While the evaluators note that the changing nature of the pandemic at the time of the evaluation presents challenges to an in-depth and considered appraisal, it is evident that the UNICEF Albania COVID-19 response was based on and/or informed by a range of needs assessments and analyses of the likely impact of the pandemic response in Albania (discussed under EQ2 above). Examples include:
• An online survey of 363 youth aged 15 to 29 on their needs during the COVID-19 pandemic related to health, education, participation and socialization services (June 2020); • A March/April 2020 survey by NGO Terre des Hommes of 779 household members from eight cities in Albania on the impact of the pandemic and quarantine on families in need; • World Bank projected poverty impacts of COVID-19 (June 2020); and • Ministry of Health and Social Protection (MoHSP) and Institute of Public Health COVID-19 Action Plan (March 2020).
Such assessments (and real-time data on the impact of the pandemic and the quarantine) were used to guide UNICEF’s responses, including joint advocacy (with the Albanian Child Protection Coalition) to the MoHSP which led to issuance of a Ministerial Instruction on the case management of children at risk or in need of protection during the COVID-19 emergency.173
UNICEF Albania published its own COVID-19 response plan in April/May and updated it in early July 2020, setting out the key needs and a response strategy according to its mandate areas, specifically health/hygiene, protection, education, coordination, communications as well as internal safeguarding/protection and business continuity considerations.
From the One-UN perspective, in mid-July 2020 the UN Albania COVID-19 Socio-Economic Recovery and Response Plan was published, setting out key areas for support to minimize the impact of COVID-19 and
170. UNICEF, January 2020, Assessing the Post-Earthquake Situation of Families with Children Residing in Temporary Shelters in Albania. 171. Donor, UN, local government key informants. 172. UN key informant. 173. Ministerial Instruction No. 253, https://www.unicef.org/albania/media/2756/file/Instruction%20253.pdf 56 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
maximize the potential for recovery. Health and protection are prioritized within this strategy, with significant contributions from UNICEF noted.
“UNICEF had a very quick [COVID-19] response, with provision of good practices and experiences from other regions.”
Key informant, Government of Albania
Respondents to the evaluation noted UNICEF support during the COVID-19 pandemic to online education (via TV and other media platforms) and in provision of hygiene kits and other medical items (e.g. thermometers) to pre-schools and schools. Evidence from key informants and secondary data indicates that the UNICEF response was both relevant to needs and proactive. As noted in the UN Albania COVID-19 Socio-Economic Recovery and Response Plan, the “UN (UNICEF specifically) took measures immediately to support MoES to adapt to new learning modalities”.
Respondents highlighted the capacity of UNICEF to respond flexibly and proactively with educational and health protection measures in comparison with other stakeholders (including donors) which can be “very bureaucratic”.174 Stakeholders also underscored the technical capacity of UNICEF education team members that underpinned its response, as well as its overall leadership support to the concrete needs of Albania’s education system.175
Evidence from interviews with a range of stakeholders at government, civil society and international agency levels indicated that UNICEF’s response was well aligned to country priorities, that it has capacity and willingness to assume leadership where required for emergency responses and ensure good coordination in its active areas and according to its mandate on child protection.
Pre-positioning and monitoring ongoing responses
Although all evidence indicates that the UNICEF response to these major crises (and other responses, e.g. the migrant crisis of 2018) was timely, effective and well-coordinated, it was not undertaken (to the extent that this evaluation can determine) on the basis of any pre-prepared contingency plan or risk assessment (a UNICEF key informant noted that such a plan was initiated, but not completed). Given the relative frequency of crises of various natures in and around Albania, implementation of such contingency planning may be warranted.
Further, as the Country Office does not have an emergency component within its operational plans, specific results-related metrics are not systematically established or collected. Therefore, good practices and learning from the responses may be lost in the absence of any in-depth documentation of or reflection on practices and experiences.
The evolution of the pandemic in terms of ongoing restrictions on movement and access to health, education, employment and other facilities – and the growing consensus that such restrictions may be needed for the medium-term, has led to increasing recognition of the value of initiatives such as UNICEF’s work on remote and online learning for schools. In education, while many relevant stakeholders have made the decision to continue work with teachers and children remotely, activities have been adjusted to reflect this modality, and include additional measures related to the COVID-19 pandemic, including how to cope with new circumstances.176 As of mid-August 2020, the Albanian Government had announced that schools would reopen for children, although they would operate in shifts to create conditions for social distancing and mitigate pre-existing issues of overcrowding.177
Key informants note that UNICEF has anticipated this via dissemination of school reopening guidance following COVID-19 and noted that such guidance (originating from the UNICEF Regional Office) is “useful for Albania”.178
174. Donor key informant. 175. Ibid. 176. NGO key informant. 177. MoESY via social media (Facebook), reported in Balkan Insight, August 20, 2020. 178. NGO key informant. FINAL REPORT 57
Respondents also noted UNICEF provision of hygiene kits and health items but have underscored the ongoing need among schools and children to adapt to either online education and/or effective hygiene practices. They expressed their views that UNICEF has a role to play in training teachers to reach children remotely and sharing lessons effectively, particularly in remote and rural areas that lack appropriate technology. Further, even if schools re-open in September, it is unclear how the pandemic will evolve throughout 2020 and 2021 and if schools will remain open.179
Outside the education sector, other UNICEF partners and stakeholders have been required to reconceptualize work modalities to reflect the pandemic-related restrictions, and UNICEF is supporting these, as well as managing its own internal changes.
What is clear is that there is unlikely to be a return to pre-pandemic modalities of public services in the short to medium term, and thus UNICEF will need to maintain flexibility to respond to the emerging and evolving needs. Some examples of such needs expressed by stakeholders are:
• Personal protective equipment for reopening of schools; • Work on online education/digital skills, particularly digitization of pre-university education via television (98 per cent of households have access to television vs 72 per cent access to internet180); • Use of social media and internet communications platforms (Zoom, WhatsApp) to deliver and improve education services;181 • Internet safety and safeguarding the rights of children with respect to bullying, sexual abuse etc. The economic impact of the crisis may exacerbate all these issues;182 and • Increased prevalence of gender-based violence/intimate partner violence resulting from social and economic stresses arising from the crisis.183
While many stakeholders were positive with respect to the role and activities of UNICEF during the COVID-19 crisis, there was some feedback that suggested that meaningful preparations were not in place (as discussed under the previous finding). The main driver of good performance was (and still is) the ability and willingness of staff to react very quickly and flexibly (e.g. ongoing field presence) and strong partners.184 However, the concern was expressed that this cannot be relied upon and appropriate planning processes are needed. Such plans should embed the experience and lessons from the 2019 earthquake as well as COVID-19 on logistics and procurement, particularly with respect to procurement processes mediated through the UNICEF Supply Division, which can be slow in responding to needs.185 This challenge is of course not unique to the Albania CO. The CO now has first-hand experience of both international and national/local procurement and codifying this experience into preparations/contingency planning will pay dividends in the eventuality of another crisis, or to respond to changes in the COVID-19 crisis.
Finding 4.5 While many resources are being applied to data (internal reporting, DHS, CRM), internal data systems incompletely track progress; both with respect to high-level outcomes/impact indicators (many of which lag by several years) and ongoing reporting of data via results assessment modules (RAMs) and strategic monitoring questions (SMQs). Similarly, external data sources are incomplete or out-of-date and cannot address the shortfall. This impedes the Country Office’s capacity to report results and justify programmatic choices adequately.
179. Government key informant. 180. International Telecommunication Union via World Bank, 2017 data. 181. Government key informant. 182. NGO key informant. 183. Government key informant. 184. UNICEF key informant. 185. UNICEF key informants. 58 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
Internal data for measuring results
The Programme Strategy Note which accompanies the CPD is clear in its emphasis on data and evidence as a means for ensuring the rights of children in Albania: “... stronger evidence base and more regular presentation of ‘facts and figures’ on children in Albania.”186
As discussed above, a review of UNICEF Albania reporting documentation and data from the country programme period indicates that the Country Office has diligently followed internal planning and monitoring/reporting processes, as exemplified by the publishing of:
• Annual reports for each year; • RAM reports for programme outcome and output areas to accompany the annual reports; • Annual work plans; • Annual management plans; • Strategic monitoring questions (SMQ) reports annually; • Annual management reviews; and • Key Performance Indicator reports annually.
These systems are in line with UNICEF global procedures and provide a relatively clear picture of the plans and, to a lesser extent, past performance, of the country programme.
However, as discussed above, the analysis of RAM reports from 2017–2019 (i.e. the CPD period, not prior to this) shows very limited quantitative reporting on results against baselines or targets, meaning that these data are unavailable to guide the choice of programme or implementation strategies.
While narrative and qualitative data in reports (notably annual reports) is richer than quantitative data, this is mainly focused on UNICEF activities (inputs or outputs), rather than on outcomes and impacts on the realization of the rights of children. Reports lack information on “what works”, via analyses of bottlenecks, external evaluation evidence, reflection and data on child rights. The Country Office has commissioned or utilized a range of external evidence sources of qualitative data (i.e. ad-hoc situational analyses, needs assessments etc. – see EQ1 and EQ2) over the course of the 2017–2019 period. These may well represent the main evidence base for programmatic choices, although it is not clear whether the studies drive the programme choices, or vice versa.
There is also a lack of evaluation evidence to support knowledge generation and determine programme effectiveness. The 2017–2021 CPD makes a commitment (in line with a UNICEF global commitment) to “research, studies and evaluations ... to ascertain the effectiveness of programme interventions and enhance knowledge generation practice ...”187. However, over the course of the CP to date only three evaluations were planned:
• Evaluation of UNICEF’s contribution to Social Care Reform; • Evaluation of the Country Programme; and • Evaluation of the Human Trafficking Response Programme.
Of these, only the country programme evaluation has been undertaken to date, with the remaining evaluations planned for the 2020–2021 period, although the COVID-19 pandemic may prevent this.
External data for measuring results
The paucity of quantitative data for programmatic choices and implementation is not only manifested in the internal reporting modalities of the Country Office, but across other sources of data available for Albania. For example, as reported by UNICEF in 2019, “independent mapping of child-targeted indicators for Strategic (sic) Development Goal (SDG) implementation revealed that 30 percent of the SDG indicators have not been measured, 24 percent of those that have been measured have not been reported and made available in any published report, and only 32 percent have been integrated into the national strategic documents.”188
186. UNICEF in Albania - Programme Strategy Note 2017-2021, section 3. 187. UNICEF Albania Country Programme Document 2017–2021, p.11. 188. UNICEF Albania Annual Report 2019. FINAL REPORT 59
Also discussed above, analysis of administrative data from a variety of sources highlights major gaps, including several that UNICEF has supported or committed to contribute to over the course of the country programme189 with respect to key CPD indicators.
UNICEF Albania has acknowledged that these gaps are a particular challenge in the effective rollout and implementation of its mandate, particularly the downstream implementation of policy achievements. As noted by the Country Office in 2019, there was “… no change in the data gaps of 2018; these continue to remain a critical barrier to assessment of the extent to which new laws and policies have been implemented, and, most importantly, whether progress has been achieved in realization of children’s rights.”190
“The issue is less with the indicators themselves and more the availability of data – administrative data systems are weak in Albania.”
Key informant, UNICEF
This challenge is likely to grow as the body of legislation and policies in Albania is brought further in line with EU accession protocols and the decentralization process in the country gathers pace.
Among external stakeholders, there was good awareness and appreciation of UNICEF’s role in supporting external data collection processes,191 many under UNICEF’s child rights monitoring programme area. Specific examples include:
• UNICEF’s contribution to the DHS in 2017/2018 – this is a strong success of the country programme within the period of this evaluation, and evidence from Country Office staff and government key informants attests to the contributions of UNICEF to this. Indeed, the use of DHS data as direct or proxy indicators is potentially a useful option for efficient and effective measurement of the outcomes of UNICEF’s activities (e.g. health/ nutrition and education outcomes for children), assuming that there will be a commitment to repeat the DHS at the appropriate intervals (normally every five years, so 2022/23); • Technical and funding support to Albanian Government institutions (e.g. the Institute of Public Health, INSTAT, Ministry of Foreign Affairs, municipalities) on a range of data collection, analysis and dissemination activities related to key commitments and rights-related indicators;192 • The adoption of U-Report in Albania in 2018 to provide a direct feedback link from young people to decision- makers; • Work with EUROSTAT, the statistical office of the EU, on health indicators related to child deprivation and redefining the statistics on crime prevention; and • Support to a network of statistical officers who collect child protection data for TransMonEE – a platform for monitoring the situation of children facing inequities in the realization of rights.
These efforts are worthwhile and indeed may well contribute to improving the data landscape in Albania, but they are output-oriented (rather than results-oriented), ad-hoc and not fully commensurate with the need for a systematic and clear strategy of knowledge and evidence generation to yield results. The emphasis placed on data and knowledge by UNICEF via the CPD and subsidiary strategies is unambiguous, as is the lack of progress to address key gaps.
There is a need for a clear and systematic approach to evidence generation via internal/external monitoring, situational/ needs analyses, evaluations, administrative data support, bottleneck analyses, case studies etc. This approach must place ongoing and timely monitoring of the realization of child rights at its centre to ensure that programmatic priorities and implementation strategies are adopted on the basis of complete, accurate and timely data.
189. The databases reviewed were the Demographic and Health Survey (DHS, 2018), the Global Health Observatory, Multiple Indicator Cluster Survey (MICS, 2005), Social Institutions and Gender Index (SIGI), INSTAT, TransMonEE, the Survey on Income and Living Conditions (SILC), the Programme for International Student Assessment (PISA). 190. UNICEF Albania Annual Report, 2019. 191. Evidence of actualisation of one of its “flagship” strategies from the CPD programme strategy note:Evidence Generation, Re- search and Analysis to Generate Better Quality, More Disaggregated Data. 192. UNICEF key informant. 60 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
EQ4b. To what extent have the strategies contributed to improving the performance of government institutions/ service providers (at national and subnational/local level)?
Finding 4.6 UNICEF has made significant and important contributions to the improvement of governance and policies for education, child protection and social protection.
There is extensive primary and secondary evidence to support the finding that UNICEF has both successfully advocated for and contributed to the development of a range of foundational policies governing the rights of children, and their equitable realization, in Albania during, and prior to, the period under evaluation. These have taken place via strong partnerships with a range of actors, including parliamentarians, government ministries, judiciary actors, national human rights institutions, civil society organizations (national and international), donors, international bodies and other UN agencies.
Key policies enacted (or underway) that UNICEF has contributed to and have improved the realization of the rights of children are:
• Code of Criminal Justice for Children (2017); • Law on Protection of the Rights of Children (2017); • Agenda for Children’s Rights 2017–2020; • October 2018 amendments to the Law on Civil Status; • National Agenda for Children’s Rights (2017); and • Justice for Children Strategy (2018–2021).
The Code of Criminal Justice for Children and the Law on Protection of the Rights of Children, in particular, have been cited by many respondents as being hugely significant to the rights of children in Albania. For the former, which helps children who are victims, witnesses or perpetrators of crimes, the concept of child-friendly justice is reported193 as being very new. The latter instrument is acting to fundamentally change perceptions amongst duty- bearers of children as unique rights-holders. UNICEF has also contributed to amendments or updates to a range of pre-existing instruments, such as:
• The Law on Pre-University Education (ensuring increasing school autonomy and school leadership); • Amendments to the Law on Domestic Violence (from a child rights and protection lens); • Analysis of gaps/shortcomings of the existing Universal Salt Iodization law and support to revision of this law, approved in February 2020; • Renewal/extension of National Social Protection Strategy 2019–2022; and • Development of all secondary legislation for the Law on Social Care Services approved in December 2016.
UNICEF also supported the Government of Albania in a range of strategies, reviews and implementation plans over the course of the evaluation period, notably the 2019 submission of the combined fifth and sixth Periodic Report on implementing the CRC (and two optional protocols) to the UN Committee on the Rights of the Child.194 This report details many policy measures that the Government of Albania has committed to in relation to child rights realization, monitoring and enforcement, including appointment of an Ombudsperson for child rights monitoring in 2018, adoption of child-focused indicators by INSTAT, social welfare measures for vulnerable children and specific measures to equitably realize the rights of Roma and Balkan Egyptian children.195
These achievements have been part of a continuum of efforts by UNICEF over the past decade to influence the governance and policy environment within Albania. They have complemented and supported the evolution of policy
193. UNICEF key informant 194. UNICEF Albania Annual Report 2019. 195. Ministry for Europe and Foreign Affairs, 2019, Fifth and Sixth Periodic Reports on the Convention on the Rights of the Child and its two optional protocols. FINAL REPORT 61
within Albania in line with its preparations for EU accession over a similar time frame. As such, UNICEF has successfully placed itself in a key position at a key time in the history of the country to advance the agenda of children’s rights.
UNICEF’s reported contributions to the above laws, policies, studies and strategies (via successive annual reports) are corroborated widely and comprehensively by direct evidence from many government and non-governmental partners in Albania.
“UNICEF as a partner has been the most serious, the most invested.”
Government of Albania key informant
Precisely attributing each of UNICEF’s efforts during the 2017–2020 time period to these achievements – many of which are the culmination of years of advocacy and support – is challenging. However, the evidence of key stakeholders indicates that the development of such policies would not have happened to the same standards, nor to the same time frame, if at all, without the efforts of UNICEF.
“The child protection system that has been built up since the launch of the Law on child protection in 2017 was a major turning point in Albania’s child protection normative framework and was primarily due to UNICEF’s role in supporting the legislation.”
NGO key informant
While such achievements are commendable, the need to strike a balance between technical support/advocacy for change and generating ownership and sustainability is also evident. The overarching priority of the Albanian Government to ensure its legislation and institutions are in line with the EU accession acquis196 may result in key partners relying on UNICEF to lead on initiatives that they should drive themselves.
Policy and governance
While there have been a number of keystone policy achievements over the period of evaluation, such accomplishments represent the first step in ensuring that the rights of children they enshrine are realized. UNICEF also reported providing a range of inputs to secondary legislation (via Decisions of the Council of Ministers/DCMs) that guides officials and professionals throughout Albania in implementing the new laws and applying the progressive approaches to childcare that they have introduced.197 UNICEF reported in 2018/2019 that 23 by-laws had been drafted in collaboration with local legal expertise, of which eight had been approved, while the rest awaited the approval of the Council of Ministers.198 This secondary legislation further details the actions that Child Protection and Justice for Children professionals, as well as those from other sectors, need to take to ensure the rights and protection of the child in implementation of the laws. In 2017/2018 UNICEF also supported a decree on labelling of breastmilk substitutes and a ministerial order regulating the contents and approval of infant and young child feeding educational materials.
Responses from key informants corroborate the ongoing process of operationalization of signature policies into by- laws and DCMs across areas such as justice and education.199 Examples include UNICEF’s work with the Friends of Children parliamentary group and the Albanian Legal Aid Society, noted by representatives of both groups interviewed for this evaluation.
196. The body of EU legislation, comprising 35 chapters, that is negotiated between an accession country and the European Council and incorporated into the candidate state’s accession treaty. 197. UNICEF Albania Country Office Annual Report 2018, UNICEF key informants. 198. UNICEF Albania Country Office Annual Report 2018. 199. CSO key informants. 62 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
Finding 4.7
The strong contributions on the policy/governance side are not yet matched by government performance in the operationalization of these achievements, with a requirement for greater government resources, leadership and accountability for delivering on policy, particularly at the local level, potentially an advocacy target for UNICEF.
However, a range of key informants across most relevant sectors noted ongoing challenges in implementation of primary legislative instruments and downstream legislation. Many such respondents to the evaluation were consistent in their message that Albania has good laws in place, but they lack implementation, particularly at municipality level, many of which do not have adequate staff, skills or other resources to effectively implement laws on children’s rights and protection. Some municipalities (notably Tirana) are better equipped than others, but smaller and more rural municipalities face particular challenges.200
“UNICEF has supported social care plans introduction and law adoption – assisting the Government also in implementing strategies and laws. If UNICEF steps down from the process the agenda and reforms will stop.”
Development partner key informant
A final risk is that the formal approval of Albania’s EU accession in 2020 may lead to an intensification of a focus by decision-makers on the most visible and more easily achieved elements of the EU accession acquis, i.e. legislation and policies. Without a credible and authoritative voice advocating for the less tangible and more challenging aspects of operationalizing policies that lead to improvements in the lives of the population, realization of the rights of children may by stymied.
The evaluation evidence suggests that there is a definite need for greater or more holistic focus on operationalization of policies (via direct support – funding and/or technical capacity-building – and advocacy for national resource flows to promote equitable public finance for children with greater attention to disadvantaged groups and areas) at the local level. This is harder work and more small-scale, but more pragmatically impactful on the rights of children.201 The manner in which UNICEF can support such operationalization based on its strategic position (or strength in implementing strategies) should be a key consideration of the focus and content of the forthcoming CPD.
Implementation strategies work effectively in combination
UNICEF Albania has been committed to assisting the Government and independent bodies in building systems for the protection of child rights, simultaneously engaging in advocacy for policy development and modelling services and mechanisms at local level. There is common agreement among all stakeholders that a mix of implementation strategies202 has appropriately targeted the most vulnerable populations. Evaluation evidence indicates that the Country Office has made considerable achievements in advocacy and policy development, particularly in terms of a strong normative role for children’s rights. Conversely, evidence generation was less pronounced and mainly focused on capacity-building of national institutions.
The document review and interview data highlights UNICEF’s strong advocacy and policy dialogue and advisory role contributing to systematization and sustainability of protection of children’s rights through the adoption of legislation and strategies: the new Law on Child Rights and Protection (in February 2017); the Criminal Justice for Children Code (in March 2017): the National Agenda for Child Rights 2017–2020; and the Justice for Children Strategy 2018–2021. Government stakeholders testified that these documents were developed with extensive technical support from UNICEF and in close consultation with a broad range of government and civil society partners. Other examples include changes in the updated law on pre-university education, ensuring increasing
200. Parliament of Albania key informant. 201. UN key informants. 202. A range of implementation strategies are utilized to achieve outputs and outcomes in line with UNICEF‘s core roles: 1) evidence generation, policy dialogue and advocacy: 2) promotion of intersectoral linkages; 3) leveraging and promotion of partnerships; 4) south- south cooperation: 5) identification and promotion of innovation and models of good practice; 6) capacity development; and in times of crisis 7) service delivery. FINAL REPORT 63
school autonomy and school leadership; amendments to the Law on Domestic Violence from the child rights and protection perspective; advocacy and policy advisory for adoption of the Universal Salt Iodization (USI) Law; and a number of by-laws operationalizing the legislation.
The evaluation found that UNICEF’s knowledge generation and child rights monitoring role somewhat shifted towards providing technical assistance to INSTAT and independent bodies to develop their evidence generation and child rights monitoring capacities. Document review and stakeholder interviews noted the contributions to strengthening these institutions, though persistent problems with data availability and validity were emphasized, particularly by civil society and development partners. While support to national efforts to build an evidence base is commendable and appropriate, in particular taking into account Albania’s EU accession status, primary and secondary data indicates persistent gaps in key data on children that may be affecting policymaking processes and monitoring of child rights.
Finding 4.8
In line with its planned strategies, UNICEF has also supported capacity and standards among service providers and other partners in the country via technical expertise, tools and policies, but this is not commensurate with the scale of ongoing needs for improved quality and quantity of service provision in Albania.
As discussed above, UNICEF has contributed strongly to advocacy, provision of technical expertise, knowledge- sharing and modelling (of services such as pre-school education and home care) in improving governance and policies for service provision. There is considerable evidence from key informants across different sectors to attest to UNICEF’s inputs in these areas, for example:
• Valuable support for the provision of materials, translations and facilitating trainings (CSO respondents); • Successful advocacy by UNICEF and the Child Rights Observatory for a DCM authorizing scholarships to increase equitable access to education for vulnerable children (government respondent); • Mapping of vulnerable and out-of-school children to help target them for education services and increase equity of opportunities among children; • Positive feedback from teachers to the national quality assurance agency on the quality of UNICEF’s capacity- building activities, particularly on improving teaching norms; • Improvements in the delivery of policing services to children such as the introduction of child-friendly facilities and capacity-building in interviewing skills; • Capacity development of health-care workers regarding universal progressive home visiting – a holistic approach to childcare focusing on the most vulnerable. This included preparation of standardized tools for service delivery and development of mechanisms for intersectoral collaboration; and • Joint monitoring visits for borders, assessment of unaccompanied children, trainings with local police and mapping issues relating to foreign/unaccompanied children arriving in Albania.
Respondents acknowledged that specific data relating to any improvements in accessibility or quality of services is difficult to obtain, but many asserted (anecdotally) that they felt that such positive changes were, at the very least, likely to be taking place, due to the visible high level of support and engagement that UNICEF provides.203
However, the evidence indicates that there are still gaps in operationalizing commitments from the Government of Albania and ensuring accessibility and quality of services. For example, one respondent cited the Social Pact project: a commitment from 2016/2017 by the Government of Albania promising 70 new services across the country, particularly for 27 municipalities with no previous social services. UNICEF notes that the challenge is “to translate such legislation into concrete action” and therefore is focusing on supporting national partners in “planning action and to take gradual steps towards making the system work”.204
203. Donor key informant. 204. UNICEF Albania: https://www.unicef.org/albania/social-protection 64 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
This is particularly vital with respect to the equitable realization of the rights of the most vulnerable children. A European Commission report on the status of Albania’s legislative framework in relation to EU accession in 2019 noted that “despite efforts to ensure inclusive education, the quality of education for disabled children, especially deaf children, is a cause for concern”. Indeed, in the same report, the Commission notes the gap in implementation of human rights instruments and recommends that Albania “continue to improve access to services for vulnerable populations, including: women; Roma and Egyptian minorities; people with disabilities; and populations in rural or remote areas.”205
Much of the data gathered by the evaluation indicates that work is ‘ongoing’ to improve services. Given that many of the government policy commitments and achievements made date back several years (to 2016/2017 in the case of the Social Pact), the absence of evidence of concrete progress suggests that there are missing or mismatched commitments of resources to implement policies.
“Some municipalities do not fulfil the Social Pact. Organizations were waiting for release of the Social Fund but thus far [there is] no provision of funds for services.”
NGO key informant, Tirana
Partners suggest that UNICEF should increase its focus on monitoring of service provision vis-à-vis commitments by the Government. Without this, they note that it is impossible to determine the outcomes of UNICEF’s work on children and the realization of their rights to quality services.206 Such monitoring of service provision could form part of the child rights monitoring efforts of UNICEF.
“We cannot say if these efforts are translated into concrete, tangible results for children.”
Donor key informant
“Capacity building of social service directorates and budget finance departments is critical to enhance collaboration and better prepare responses to social service needs.”
Development partner key informant
Part of the challenge is the longer-term approach required to see changes in services, particularly in educational outcomes, noted by several respondents. Such changes take longer than a one-year project horizon, therefore sustainability via commitments that are met by the Government are an essential adjunct to smaller, ad-hoc initiatives.
“Without UNICEF’s focus on school safety, the situation would not be improving. Working with teachers is important to change their mindsets not to beat or shout at children. This is slowly changing thanks to interventions supported by UNICEF, bringing a positive culture to schools. However, the project is small – only for one year. To see real results there needs to be five years engagement.”
NGO key informant
Provision of expertise, tools, training and resources to ensure effective data and programming
UNICEF has devoted considerable resources over the course of the country programme period to building the capacity of stakeholders on sectoral initiatives, development and dissemination of tools and guidance, and promotion of data gathering initiatives. Evaluation respondents reported that the specific trainings supported or provided by UNICEF Albania span the full range of country programme sectors, with respondents almost unanimously highlighting the value of such training to their work.
Stakeholders that were not recipients of trainings (such as donors or UN agencies) also attested to the usefulness of the expertise of UNICEF, its willingness to share it and the value of the guidance and training provided to Albanian
205. European Commission, 2019 Communication on EU Enlargement Policy, Albania 2019 Report. 206. NGO key informant. FINAL REPORT 65
actors. One respondent highlighted the useful modality of mutual learning used by UNICEF: supporting initiatives that provide opportunities for increased understanding and approaches to work, with a focus on performance, good working relationships and full participation.
Some specific highlights of tools/guidance/training shared by respondents are:
• Support to in-progress monitoring of the child rights and protection system; • Support to the Institute of Public Health to establish and scale up a national web-based data collection system, using administrative data, for monitoring of child growth and infant and young child feeding practices;207 • Technical assistance to the Albanian Institute of Statistics (INSTAT) on the 2020 census, with particular technical focus on children with disabilities and other vulnerabilities, overall data quality and an analysis paper on key issues regarding the last census, and additional work on a five-year data plan and a national strategy; • Technical assistance to the Ministry of Foreign Affairs in preparation for the next state’s Periodic Report to the Committee on the Rights of the Child; • Technical assistance to municipalities to improve quality of data and improve data-sharing protocols; • Training in mental health in emergency settings; • Training on cybercrime/bullying; • Training on restorative justice; • Developing of training curricula on child protection for different stakeholders, including on integration of health and social services in child consulting centres; • Technical support/training on child rights monitoring by national stakeholders.
UNICEF in Albania has also produced a number of commissioned reports and studies. One of the most important studies produced during the current CP as a joint effort of UNICEF, UN Women and UNFPA is the 2017–18 DHS that offers an insight on important indicators used to measure the country development. Further, UNICEF and OECD supported a review of the assessment and evaluation system and also the mid-term review of the National Strategy of Pre-University Education; the National Assessment: System Response to Child Protection Online in Albania carried out in 2018; and Situation Analysis on Children with Disabilities in Albania in 2018. The DHS was an important data collection effort that followed the last UNICEF Multiple Indicator Cluster Survey (MICS) in 2005, and was the first or similar comprehensive study of children since then to support stronger advocacy and a more visible overview of situations of particularly vulnerable population groups.
Notwithstanding the widespread satisfaction expressed by most respondents, examples of areas that respondents highlighted as requiring ongoing technical or capacity support are:
• Service providers in the area of children with disabilities to further promote equity among the most vulnerable and marginalized; • Government and non-government service providers still responding to the effects of the 2019 earthquake and the increased needs due to the ongoing COVID-19 crisis; • Stakeholders at subnational levels that require financial management and budgeting training to allocate or effectively implement decentralized budgets related to projected social or other service initiatives (many thanks to successful UNICEF advocacy);and • Overall systems and capacity of the civil society sector in Albania, which is not well-developed.
With respect to data, as discussed above, there are many gaps that have yet to be addressed (and are acknowledged by UNICEF). Firstly, while UNICEF has undertaken considerable work in provision of expertise, guidance and training on all sectoral areas within its country programme, there are clear needs f wider capacity development related to implementation/decentralization in Albania.
Secondly, while child rights monitoring (CRM) is a priority of UNICEF’s programming, respondents reported to the evaluation team that it sits “in the shadow” in terms of national priorities.208 Key informants reported that the Albanian parliament was moving forwards with rolling out CRM, but that it is on the basis of one or two parliamentary “champions” and the work of the Ombudsman rather than a systematic allocation of priority and
207. UNICEF key informant. 208. UNICEF key informants. 66 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
resources. Testimony from key informants on the need for more, and timelier, data, coupled with the lack of up-to- date information in UNICEF Albania’s own reported results and among other data sources is a clear indication that this is an area for prioritization.
One key informant expressed that prior to the current country programme, UNICEF placed a higher priority on a robust evidence base with respect to children’s rights and setting priorities on that basis, and that this emphasis should be restored, even if that means diminished alignment with government needs.209
3.2.3 Evaluation Question 5 Gender mainstreaming
EQ5 To what extent has gender been effectively mainstreamed in the CP design and implementation and, if not, how should it be done?
Finding 5.1 The country programme document makes reference to the principle of gender equality, yet there is little evidence that gender equality was robustly integrated.
UNICEF’s 2018–21 Gender Action Plan (GAP)210 defines a two-track approach to gender equality.211
• Integrated gender results – where gender is embedded across UNICEF programming sectors (health, nutrition, education, child protection, water, sanitation and hygiene (WASH), and social policy); and • Targeted gender results – areas that focus on the empowerment and well-being of adolescent girls.
The CPD specifically mentions contributions to SDG 5 on gender equality. The CPD narrative refers to terms like “gender-based inequities, gender-based violence, gender-sensitive case management and gender-disaggregated data”, but without further elaboration of them. Also, in its results framework UNICEF’s contribution to UNDAF Outcome 2 (Social cohesion), health, education, social protection, child protection and gender-based violence are specified.212
However, further analysis of the CPD, Programme Strategy Note and programme plan for 2018–2021 indicates few or no references to international standards and norms (notably CEDAW). In its 2017 annual report the Country Office noted that the empowerment of girls and women was systematically promoted throughout all its interventions. The next two annual reports for 2018 and 2019 do not include specific sections on gender equality.213 UNICEF’s own documents note that the UN Sustainable Development Framework for 2017–2021 was “informed by the Common Country Assessment 2012–2016 that included a special gender-focused analysis of the situation and challenges in Albania in which gender gaps and under-empowerment of women have been identified among the major drawbacks of Albania’s current socioeconomic development model”.214 Another document said to have informed the UNICEF CPD was a 2010 gender review of the UNICEF country programme and a gender scorecard by UN Women as part of the One UN programme in Albania.215
However, evidence from key stakeholders and analysis of documentation indicates that the Country Office did not undertake a gender analysis or gender programmatic review (GPR) in order to guide the formation of the current CPD 2017–2021. Gender analyses and GPRs, a cornerstone of gender mainstreaming, have not aided mainstreaming of gender in the CPD/results framework design and throughout the programme cycle. This includes strategies for gender equality; monitoring implementation; and holding individuals and institutions accountable for
209. NGO key informants. 210. Which was preceded by a Gender Action Plan for 2014–17. 211. UNICEF Gender Action Plan 2018–2021, p. 5. 212. UNICEF Albania CPD 2016–2021, p. 12. 213. UNICEF Albania COARs 2017, 2018 and 2019. 214. UNICEF Albania COAR 2017. 215. Country-Lead Evaluation, “Delivering as One” Albania, Final Report, 7 July 2010, “Albania Gender Scorecard 2014”. https://unsdg. un.org/sites/default/files/Albania-Gender-Scorecard-Report-2014.pdf FINAL REPORT 67
results.216 Key stakeholders noted some limited efforts by UNICEF to integrate gender mainstreaming components in its strategic document and results framework, for example the KAP survey on Child Marriage (Priority 1 of GAP 2018–2021 “Ending child marriage and protecting girls from multiple risks that limit life opportunities”). However, additional interventions based on the recommendations of this study were limited to an unspecified number of awareness trainings.
As a UN key stakeholder said, “they tend to scramble for gender data when reporting comes around – then it just becomes disaggregation of data, and not analysis of underlying gender issues – little is said on the barriers and bottlenecks that the country is trying to address”.
Finding 5.2 While the Country Office has taken on board some of UNICEF’s gender guidance, notably appointing a gender focal point, and conducted some work with an explicit gender dimension (e.g. child marriage research) operationalization of the gender equality approach is limited. The CO is more ‘gender-blind’ than ‘gender-focused’.
An analysis of Country Office AWPs, COARs, RAMS and SMQs for the years 2017, 2018 and 2019 (presented in detail in Annex 8) indicates that for 2017 and 2018 gender-related indicators were not documented in UNICEF Albania interventions. Neither document narratives nor annual output indicators use gender-aware language or a gender-sensitive approach. Based on the WHO Gender Assessment Scale,217 the Country Office results are at the “gender-blind” level, in other words, they do not take into account the different roles and diverse needs of men/boys and women/girls, maintain the status quo, and do not help transform the unequal structure of gender relations.218 The details of this analysis are presented in Annex 8.
A Gender Equality Marker was introduced by the CO in 2018 for reporting, but analysis of documentation indicates inconsistencies in its reporting, as it is for some areas where other UN key stakeholders support this marker. For instance, in 2017 the CO indicated that it included reporting on gender targeted areas through the following means:
• Promoting gender-responsive adolescent health; • Advancing girls’ secondary education; • Ending child marriage; • Identifying and responding to gender-based violence; • Disaggregating data on adolescent girls in national Out-of-School Children (OOSC) statistics; and • Implementing, in 2017, a joint UNICEF-UNFPA assessment of the child marriage phenomenon in Albania.
However, an analysis219 of reports on outcomes/outputs/indicators does not strongly validate this statement, as there is no reference to UNICEF’s input to gender-responsive adolescent health besides being included in the national SRH (sexual and reproductive health) strategy; it does not explain how gender-based violence is targeted and there is no reported disaggregation of data of OOSC. More positively, a qualitative KAP (knowledge, attitudes and practices) study on child marriage was conducted jointly between UNICEF-UNFPA in 2017, which was the only explicit gender-targeted indicator. The analysis shows that there is a growing effort to include more gender-aware language in planning and reporting and gender sensitive interventions after 2018. This was also acknowledged by some external stakeholders, which highlighted recent UNICEF efforts to be more vocal in including gender principles in their interventions. For example, local government stakeholders noted that their social care plans will be reviewed and revised through gender lenses and sex-disaggregated data are used more in reporting (although not yet in planning). Such efforts are laudable, but are ad hoc and not systematic nor strategic, with much remaining to be done to achieve UNICEF’s gender goals.
In terms of human resources, the CO has had a gender focal point position since 2017 – a member of the first cohort of UNICEF gender focal points to be accredited. Evidence from internal stakeholders suggests that gender
216. UNICEF Gender Toolkit: Integrating gender in programming for every child in South Asia. 217. Ibid, p. 78. 218. Ibid, p.6. 219. See Annex 8. 68 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
mainstreaming is considered more the “work of the gender focal point” rather than a common effort for all staff members. Annual planning documents indicate that, for 2020–2021, the Country Office has planned a mandatory “GenderPro” training for all staff, which is intended to be a significant step towards integrating gender in programmatic and organizational outcomes and in line with guidance from GAP 2018–2021 on programme excellence for gender equality.220 A further positive indication of improving gender mainstreaming is assignment of responsibility for gender mainstreaming to the Deputy Representative, in line with global-level guidance on gender-responsive programming.
A regional capacity assessment survey of Country Offices, conducted by the Regional Gender Advisor in UNICEF ECARO in 2019/2020 to create a baseline for future approaches to gender regionally, highlighted areas that need attention within the Albania Country Office.221 The three main reasons (from CO staff) for not addressing gender were insufficient knowledge, budgetary resources and a busy work schedule. Staff also noted that the top needs for support in addressing gender are opportunities to learn or to develop technical skills, and funds to be allocated for gender work.222
Analysis of programme financial data by the evaluation team did not identify any “gender tagged resources” allocation.223 The Gender Toolkit for Programming suggests an output level Gender Equality Markers linked to Specific Intervention Code (SIC)-level gender tags through percentage of expenditures allocated to gender tagged activities. The minimum level set is 15 per cent of gender tagged expenditures at activity level, to be classified as marginal.224
Institutional results for 2019 show that the Country Office is lagging with respect to gender parity, staffing, expenditure and responsibilities across sectors. The document review and current ratio of female to male staff confirm this finding.
3.3 Efficiency
3.3.1 Evaluation Question 6 Resourcing
EQ6. To what extent are the resources (financial and human resources) allocated by the Country Office appropriate to support the implementation of strategies and achievement of CP results and, if not, what could be done to ensure resources match programmatic ambitions and needs?
Finding 6.1
The Country Office has applied an efficient and effective resource mobilization strategy, with most of its year-to- year needs met from a variety of sources.
CP Funding Requirements
From a financial perspective, UNICEF has been effective and efficient in securing and utilizing resources according to budgetary plans set out in the CPD, in the face of a challenging fundraising environment. The overall budget proposed in 2017 for the country programme was US$16,345,000, of which US$4,345,000 (26.6 per cent) was to be regular resources (RR) and US$12,000,000 (73.4 per cent) other resources (OR).225 Although the country programme period is not due to conclude until 2021 (i.e. financial data from 2021 cannot be included within the evaluation analysis), available financial data (as of October 2020) indicates that the Country Office was funded to the
220. Gender Action Plan 2018–2021, p. 16. 221. UNICEF ECARO. Results of Online Assessment of the Gender Capacity and Needs of UNICEF Programme Staff In Europe and Central Asia. 222. Ibid, p. 72. 223. Financial data from VISION provided by the UNICEF Albania Country Office for 2014–2019. 224. Ibid, p. 85. 225. UNICEF Albania Country Programme Document, 2017–2021. FINAL REPORT 69
amount of US$18,163,104 – thus has reached 111% of target.226 This is corroborated by evidence from UNICEF key informants, who note the alignment with plans, and indeed a likely increase in the level of OR for 2020 as a result of UNICEF’s response to the 2019 earthquake, and donor funding that may be available to respond to the challenges of COVID-19 in Albania. Indeed, a readjustment of the planned RR/OR ratio may be required for 2020/2021.227
The chart below illustrates the planned versus utilization of budgets by the Albania Country Office during the programme period under evaluation. As can be seen, there has been very close alignment between the two.
Figure 3: UNICEF Albania planned vs. utilized annual budgets 2016–2019
$11,072 $11,073 $12,000 $10,000 $8,000 $4,170 $2,793 $4,170 $6,000 $1,725 $2,384 $2,792 $4,000 $1,727 $2,384 $2,000 $0 2016 2017 2018 2019 Total
Total Utilized Budget (in ‘000 USD) Total Planned
A similar analysis of the actual spending of regular resources versus other resources over the country programme period (see figure 4 below) shows that, typically, OR exceeds RR, although the ratio has varied over the course of the 2016–2019 period from between 25.3 per cent (RR as a percentage of total programme funding) in 2016 to almost parity (48.9 per cent RR to 51.1 per cent OR in 2017). Over the course of 2017 to 2019, RR has averaged at 41 per cent of programme funding and OR at 59 per cent. This is a higher proportion of RR than was originally planned in the CPD, so additional OR funding in the pipeline for 2020 may adjust the ratio more in alignment with plans. Management and administration costs average at approximately 12 per cent of total budgets, which is high compared to similar UNICEF offices; for example, the UNICEF Serbia Country Office, with a country programme budget of US$22.8 million over the 2016–2020 period, has a management/administration overhead rate of 5 per cent.228
Figure 4: UNICEF Albania regular resources vs other resources vs management/admin
$4,000 $3,579 $3,008 $3,000 $2,497
$2,000 $1,645 $1,223 $1,017 $915 $901 $826 $1,000 $681 $710 $146 $334 $246 $247 $0 2016 2017 2018 2019 Total
RR (’000 USD) OR (’000 USD) Management, Admin & Misc (’000 USD)
226. UNICEF Country Office unpublished financial data. 227. UNICEF key informant. 228. Institute for Development Impact, 2019, Evaluation of the UNICEF Serbia Country Programme 2016–2020 70 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
The table below presents UNICEF expenditure for the period 2017–2019 by programme area. Table 5: UNICEF Albania expenditure by programme area (2017–2019)
Outcome 1: Governance for Equity 2017 2018 2019 Output 1.1 — Equitable MCH Care $171,392 $328,183 $244,881 Output 1.2 — Education and Early Learning $348,203 $474,269 $766,270 Output 1.3 — Social Protection $90,452 $289,689 $313,515 Outcome 2: Protection and Justice Output 2.1 — Deinstitutionalization $38,531 $205,004 $336,632 Output 2.2 — Child Protection Systems $295,520 $441,981 $384,399 Output 2.3 — Justice for Children $105,298 $115,009 $177,356 Outcome 3: Programme Effectiveness Output 3.2 — External Relations $2,418 $0 $0 Output 3.3 — Child Rights Monitoring $105,173 $89,799 $115,061 Output 3.4 — Intersectoral Approaches $136,688 $166,573 $185,141 Output 3.5 — Evaluation $22,350 Outcome 4: Management Governance and Systems $78,281 $10,781 $20,995 Financial Resources and Stewardship $209,410 $207,037 $192,444 Human Resources Management $41,760 $27,954 $33,468
The chart above also presents the data. It shows a steadily increasing allocation of budgetary share to outcomes 1 and 2 of the CPD, and relatively flat allocations to the programme effectiveness outcome (which includes CRM) and programme management. This data triangulates well with the finding that CRM and data has not received the same level of attention as other outcome areas, and is thus underfunded given the stated ambition of the country programme with respect to CRM. Otherwise, the budgetary allocations are in line with the CPD.
Perspectives of Country Office and Regional Office staff is that individual programme component budgets are often insufficient, but some areas (e.g. C4D) can leverage the budgets of the other programme areas to meet requirements. In other areas (health, education/ECD, CRM) there are perceptions of inadequate funding to meet the needs in Albania, or fully meet UNICEF’s stated goals/plans.
Figure 5: UNICEF Albania budgets 2017–2019 by programme outcome
$1,400,000
$1,200,000
$1,000,000
$800,000
$600,000
$400,000
$200,000
$0 Outcome 1 Outcome 2 Outcome 3 Outcome 4
2017 2018 2019 FINAL REPORT 71
CP Funding Sources
In terms of external funding (OR) sources, UNICEF has demonstrated a reasonably wide spread of donors across different programme areas. Over the course of the programme period (2017–2019), UNICEF financial records indicate eight external direct donors (in the case of UNDP, the Multi-Partner Trust Fund system may collate funding from different donors and is intended for multi-agency operations; it acts as the fund administrator for the UN system).229
Table 6: UNICEF Albania donor funding in 2017–2019230
Direct Donor Total funding 2017–2019 UNDP–MPTF $1,036,858 (multi-partner UN funding) End Violence Fund $686,662 European Commission $275,973 Government of Norway $109,222 Government of Austria $76,506 UNDP $73,183 USAID $36,628
In addition to the above, additional funding from donors was generated via UNICEF at regional (ECARO) and global levels from other external donors (i.e. counting as OR). For example, in 2019 UNICEF Albania received donor support via ECARO from Deutsche Telekom and The David Beckham UNICEF Fund.231 These sources represent a broad range of donors, with limited reliance on any one source, and thus a spread of risk. Testimony from key informants indicates that implementation of funding across programme areas and partners generates mixed levels of satisfaction. Some senior stakeholders expressed a concern that Albania is not doing as well as leveraging opportunities for funding related to the EU accession as some of the other countries (Serbia, Montenegro) and that this is an area that would benefit from additional focus.
Despite this, the Country Office has demonstrated a flexible and proactive approach to resource mobilization and planning. For example, concerns expressed by the Country Office regarding unpredictability for generating OR for the 2017–2021 period led, in 2017, to development of an internal programme strategy note covering:
• Potential mitigation measures i.e. adjustment /prioritization of programming ambitions to the actual level of funding captured; • A reorganization of programme strategies, to focus on those most important and/or least cost;232
While the concerns articulated in 2017 did not come to pass, the Country Office has maintained its focus on the challenges around funding. The Country Office noted in 2019 that progress towards OR fundraising targets was uneven across programme sectors, with Child Protection, Education and Social Protection being more successful than Health and CRM/programme effectiveness, reflecting donor priorities for Albania.233 In 2019 it published a detailed funding strategy for the 2018–2021 period, noting that, as of November 2019, “the total allotment of Other Resources (OR) to the Country Programme had reached US$6.02 million, equivalent to almost 50% of the
229. UNDP, Multi-Partner Trust Fund Office Factsheet, mptf.undp.org. 230. OR data provided by the Country Office, updated May 2020. 231. UNICEF Albania Country Office, 2019, Resource Mobilisation Strategy 2018–2021. 232. UNICEF Albania Country Office, March 2018, Adjustment of programme strategies and targets, in case of substantial funding gaps. 233. UNICEF Albania Country Office, 2019, Resource Mobilisation Strategy 2018–2021. 72 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
Executive Board-approved US$12 million OR ceiling for the five-year programme cycle.”234 Indeed, uncertainty around the medium-term funding outlook means that a conservative approach to human resources is warranted.
Human resources
Human resources/expertise within the Country Office is – overall – strong. UNICEF also relies on a range of external consultants to carry out work (see EQ3). The consultants are regarded by key external and internal stakeholders as representing a good body of external expertise and an efficient way to leverage resources, although this modality poses an additional burden on the operations team. Nonetheless, use of external consulting expertise is an effective way to drive results and also brings new ideas into the team. UNICEF staff noted that their resources have permitted them to take on younger professionals with a fresh perspective.
Country Office staff are very well-connected to the Government and have very useful channels of communication and influence. The disadvantage of this relationship is that the Government can then use UNICEF for its own agenda, and this can create a conflict of interest – noted by some non-governmental stakeholders.
Some additional human and financial resources would benefit data/evidence generation efforts in relation to children’s rights – Albania hasn’t conducted a MICS since 2005, and efforts to operationalize CRM are progressing slowly and could move faster with additional resources.
Perspectives from external stakeholders are that UNICEF programme staff are technically skilled, experienced and dynamic. The longevity among some staff makes them very well-positioned to cover their programme areas and priorities.
Peer UN agencies expressed the general view that UNICEF has historically been a small player but has more investment in technical capacity than some other (UN) actors, a strength of the Country Office.
Figure 6: UNICEF Albania financial allocations by partner type
100% 78% 79% 80% 69% 60%
40% 22% 21% 20% 13% 18% 0% 0% 0% 2017 2018 2019
CSO Partners Government Partners UNFPA
Finding 6.2 A number of factors support UNICEF’s overall efficiency in delivering results such as maintaining a small number of core consistent partnerships which can be considered a strength by reflecting investments in longer-term relationships for realizing rights.
The allocations of funding across partners by type of partner (government, CSO or, for 2018, UNFPA) are consistent across the three years of data, with almost 80 per cent of resources going to civil society and 20 per cent to government. The 2018 allocation to UNFPA was part of UNICEF’s contribution to the DHS survey for that year. Private sector partnerships (discussed further under finding 9.3) were negligible.
234. Ibid FINAL REPORT 73
UNICEF’s ten largest partners (in financial terms) over the course of the county programme period were: UNICEF provided smaller amounts to 18 other partners across civil society and government.
Table 7: UNICEF Albania financial breakdown by partners
UNICEF Partner Type of Partner Total US$ 2017–2019 The Quality Assurance Agency of Pre-University Education Government $271,892 (part of MoESY) The Albanian National Children’s Helpline (ALO 116) Civil Society $227,902 The Children’s Rights Observatory Civil Society $221,018 Albanian Social Services Association Civil Society $152,544 UNFPA UN $140,000 Europartners Development Civil Society $95,408 Terre Des Hommes Civil Society (international) $88,647 ESA Consulting Civil Society $89,511 Institute of Public Opinion Studies Civil Society $88,779 Fondacioni Partneritet Per Zhvillim Civil Society $78,112
Over the course of the country programme, UNICEF demonstrated reasonable consistency of partnerships; five partners were maintained throughout the 2017–2019 period, ten for two of those three years, and the remainder for one year only. However, the VISION data made available to the evaluation team on sub-grants to partners represents between 25 per cent and 35 per cent of annual UNICEF Albania Country Office budgets. The modality of the expenditure of the remaining portions (whether directly by UNICEF or via another disbursement mechanism) is unclear.
From the perspective of UNICEF’s implementing partners, partnership modalities are largely valued, though the nature of contractual agreements is short term, and lags between successive contracts have led to serious continuity issues, notably amongst smaller partners who do not have substantial reserves.
A point of consensus from the above evidence (across multiple stakeholders) is that a key determinant of UNICEF’s efficiency has been the technical competence and dedication of the Country Office team, contributing to the efficient implementation of the selected interventions. While satisfaction with the mix of full-time staff and consultants, the partnership agreements and contractual modalities vis-à-vis UNICEF’s programme strategies was not universal, the preponderance of evidence (from UNICEF, government and other partners) indicates that the Country Office’s openness to respond to the needs that its key partners highlight is a key strength.
Implementation and management of funding
From a compliance perspective, the UNICEF Office of Internal Audit and Investigations conducted an internal audit of the Albania Country Office in 2017. The objective of the audit was to assess the office’s governance, risk management and internal control processes, with a focus on key risks to delivering UNICEF’s objectives. The audit covered the period from January 2016 to October 2017, as the Country Office was entering into the new country programme cycle.
The audit was largely positive and noted a number of areas which were functioning well including a “well-prepared … CPD that had received one of the highest scores from the quality review of CPDs commissioned by UNICEF’s Field Results Group in 2016”.235 The audit also reported financial management as “generally satisfactory”.
235. UNICEF Office of Internal Audit and Investigations, 2018, Internal Audit of the Republic of Albania Country Office. 74 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
3.4 Sustainability
3.4.1 Evaluation Question 7 Long-term alignment of programming
EQ7. To what extent has gender been effectively mainstreamed in the CP design and implementation and, if not, how should it be done?
Finding 7.1
UNICEF implementation strategies within the sectors of engagement (education, social protection, health and child rights monitoring) have been well-linked to longer-term national development strategies some of which were designed with UNICEF’s support. In addition, UNICEF’s work is integrated in the wider One UN programme.
Finding 7.2 UNICEF engaged in a number of sectors in response to government priorities. Its support brought sustainable results in strengthening the policy and normative framework, in particular for education and social protection. The foundations for future enhancement of the fulfilment of child rights were laid through strengthening the capacities of national stakeholders, and the establishment of institutional mechanisms and models. Overall, Albania’s drive towards EU accession brought a faster pace of reforms in certain sectors where UNICEF can support the reforms and thus achievement and sustainability of results.
There is clear evidence that the UNICEF Albania Programme 2017–2021 was designed to align with the Albanian Government priorities. The overall goals that UNICEF in Albania aims to achieve by 2021 align closely with Albania’s national priorities and its international and EU accession obligations towards a progressive realization of rights of all children in Albania. The main focus of UNICEF support has been to assist national partners to progressively align national legislation, policies and action plans with EU accession priorities and the 2030 Agenda. Evidence from analysis of UNICEF’s main planning documents (the CPD, Strategic Note, annual workplans, etc.) and reports, as well as interviews with stakeholders from all key groups indicates good alignment between UNICEF’s interventions and government strategies. These linkages were found across areas related to child rights, including access to education, health and social welfare, juvenile justice and the protective environment. Investment in these areas in particular has brought significant contribution to shaping the government agenda in relation to protection and promotion of child rights.236
Greater foundations for sustainability in some but not all sectors
A major component of sustainability was UNICEF’s achievements in working with the Government to develop legislation and policies which create and strengthen the enabling environment. There has been investment in these foundations across UNICEF’s target sectors. There is consistent agreement among UNICEF’s stakeholders, and evidence in the literature and documentation, that the Country Office’s contribution to policy and the normative framework was needed and has led to the establishment of strong and sustainable foundations. These should provide for a more coherent and adequate government response to the needs and rights of children (See EQ 4). Accompanying the investment in the policy and legislative environment, UNICEF also focused on strengthening capabilities, for example, a range of manuals, curricula, guides and assessment instruments which should provide a reliable basis for ongoing capacity-building and quality assurance activities.
It is evident that UNICEF’s system-level approach and focus on the policy and normative framework aims to create a sustainable framework in the absence of future support (financial or technical). Despite this, the systems that UNICEF has invested in are not yet robust or self-sustaining. There are political, institutional and financial constraints on national and local-level government affecting implementation and operationalization of legislation.
236. Interviews with key government stakeholders at central and local levels, CSOs and development partners. FINAL REPORT 75
This situation was reflected in the literature237 and was confirmed by stakeholders among government, CSOs and development partners.
The evidence from documents and the feedback from stakeholders indicates that the Country Office has made important contributions to strengthening the capability of national and local stakeholders (see EQ4, Finding 4.8). The consensus among stakeholders was that UNICEF’s efforts were of particular importance in the sectors of social services (e.g. home visiting), education and justice, and provided the best coverage given available resources.
However, the institutionalization and therefore sustainability of strengthening capabilities faces limitations. There was a strong sense that there was a positive impact at individual level, with cohorts of trained practitioners and decision makers increasing their personal understanding, knowledge and familiarity with sector specific subjects and approaches. Stakeholders from central and local government noted that trainings, workshops, manuals and guidelines prepared with the support of UNICEF help them to understand better the needs and possible approaches to respond to the needs of children. However, sustainability of acquired knowledge and skills at an institutional level is challenged by the above-mentioned political, financial and institutional constraints. For example, the 2019 European Commission Country Report for Albania noted weak institutional mechanisms for protection of vulnerable groups.238
Another important sustainability factor is the decentralization process, which the CP aligned itself with strongly during its implementation. This process is generating important changes to the organization of social, educational and health systems. New competencies and obligations have been introduced, but without clear guidance on how to fulfil these new mandates, as noted by stakeholders from municipal governments, UNICEF (and other UN agencies) and CSOs. Key informants noted that this weak capacity of municipalities creates gaps in responses to needs and access to services.239
Finding 7.3
Investment in strengthening legislative and policy framework along with investment in modelling innovative approaches have contributed to better performance by local and central actors, particularly through support to developing policy and legislation. However, the lack of resources and reprioritization of budgets, particularly at local level, create risks for comprehensively translating policy and legislation into implementation, and even the potential for a reversal in gains made in relation to capability strengthening.
While decentralization, as defined in its related normative framework, brings a number of positive aspects with regard to the organization of educational, health, social and child protection sectors, in the short term it jeopardizes the sustainability prospects of results already achieved at local level. This is because a number of modelled services, capacities and approaches that were supported by UNICEF are at risk due to reorganization of duties and related financial allocations, as noted by municipal level stakeholders and CSOs.
A major barrier to the sustainability of results achieved is the Government’s financial resource constraints. Budget limitations and the competing budget allocation priorities of the Government have repercussions on financing services for children and thus the sustainability of UNICEF’s efforts.240 Throughout the country programme period, UNICEF advocated for effective and equitable budget allocations for health, education and social protection, including child protection. For example, UNICEF provided technical assistance for the revision of the legal framework on childcare to include an analysis of costing of the new service models that would need to be developed to replace the existing institutions within the deinstitutionalization process.
237. For example, consecutive EU Progress Reports for Albania; Situational Analyses; available evaluation reports and other UN and international and local organizations’ studies that were reviewed. 238. European Commission (2019); Country Report: Albania 2019. 239. Government, CSO key informants. 240. Ibid. 76 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
Further, UNICEF reporting indicates that, as a result of UNICEF’s advocacy efforts, the Government allocated an additional US$1.5 million in 2019 from the national social protection budget to fourteen municipalities for services for the most vulnerable population, including children. This is a positive outcome, although the evaluation could not find data on utilization of these funds for their intended purpose.
Public investment for children has been relatively low in Albania, with 3.1 per cent of GDP directed towards education (one of the lowest in the region); 2.9 per cent of GDP for health; and social assistance represented only 1.2–1.3 per cent of GDP, mainly comprising cash transfers rather than targeted services to families and children in need.241 Evidence from key informants indicates a concern with a shift of policy focus from child rights and related reforms to economic revival post-COVID-19, which may reverse fragile gains that have been achieved in this country programme period.242
Municipal-level key informants noted that the resourcing of municipal-level services and legislative provisions is the most important challenge. For instance, with UNICEF support, targeted municipalities developed Local Social Care plans, which are often not costed resulting in difficulties in implementing the planned activities. In some municipalities, the activities outlined in the Local Social Care plans were resourced by decisions of the local councils, but this was an ad-hoc rather than systemic approach.
In addition, municipal stakeholders reflected on the challenges of continuing models that have been promoted and demonstrated with UNICEF financial and technical support. One such example is the strengthening of local mechanisms for justice for minors. While the promoted models showed important results during implementation, local actors emphasized that they lacked the necessary human and financial resources and capacity to operationalize and continue with such practices without UNICEF’s support. This points to challenges to ensure sustainability and thus a continued dependence on UNICEF’s support.
3.5 Synergies/Coherence
3.5.1 Evaluation Question 8 Internal coherence
EQ8. To what extent do the individual CP components and implementation strategies demonstrate internal coherence and synergies i.e. interlinkages and reinforce each other to fully leverage the contribution that UNICEF makes towards expected results?
Finding 8.1
Intersectoral coherence and synergies are limited, leading to a silo approach to sector engagement. The silo approach reflects the Government’s ways of working but UNICEF faces limitations to its effectiveness if it mirrors this rather than aiming to increase both its and the Government’s intersectoral linkages. Such a silo approach does not allow for a more holistic approach to tackling the needs of children, particularly those multidimensional needs of most excluded groups (e.g. interlinkages between social welfare, health and education). UNICEF has demonstrated the effectiveness of intersectoral working through discrete initiatives but there are opportunities to build on the limited mechanisms that are in place for aligning multi-sector policy making to leverage more comprehensive and coherent approaches to realizing child rights.
An analysis of country programme planning documents shows that UNICEF organized its intervention logic along sectoral lines with little intersectoral integration. This approach was applied in the articulation of individual theories of change for each of the three main programme outcomes, while an overarching ToC was not developed. The country programme results framework provides little clarity on intersectoral synergies or wider reflection on how the implementation of individual sector strategies would or could contribute to the overarching goals of the programme.
241. UNICEF Albania COAR 2019. 242. Government, CSO key informants FINAL REPORT 77
Within each thematic sector, a mix of implementation strategies helped to mutually promote results of interventions within respective sectors.243 This internal component coherence was considered by stakeholders from government (central and local) and civil society to have contributed to more coherent results. For instance, the social policy pillar of the country programme invested both upstream (supporting legislative and policy changes, and generation of evidence) and downstream (investing in capacities of service providers and local institutions, modelling services). Attention was paid to devising interlinkages ensuring that gains in one area of intervention (e.g. adoption of laws) inform and/or leverage investment in other areas (e.g. capacity development and modelling). This comprehensive and integrated approach, while laudable, led to stakeholders expressing concern that it might create greater challenges in trying to follow up all the activities implemented.
Vertical approach to engagement
The main focus of this evaluation question, however, is coherence between programme components. While there was both documentary evidence and stakeholder feedback that there were efforts to increase coherence and complementarity of CP components and strategies (e.g. seeking synergies between health and social policy components; child rights monitoring), there were gaps visible across all thematic areas. This more vertical approach to sector engagement also reflected the context that the Country Office was working in i.e. the vertical sector organization of the Government. It is important that the Country Office aligns with the Government’s ways of working on the one hand, but does not necessarily mirror it completely on the other. The limitations of vertical working with little intersectoral cooperation and coherence can limit effectiveness for both the Government and for UNICEF. The very sector-specific focus by the Government and its lack of intersectoral cooperation or joint initiatives among the government institutions are reflected in the way UNICEF organizes its own work. While there is clear evidence that the country programme has delivered strong results in working with the Government, UNICEF should be cognizant of the limitations of poor horizontal integration, particularly where the Country Office team is small and relies on intersectoral work to be efficient and effective.
Analysis of both primary and secondary data indicates stronger sector-specific results than intersectoral engagement or results. But there were some programmatic examples of intersectoral engagement such as UNICEF’s support to home visiting by social workers. UNICEF interventions in this area supported expansion of social service to include a variety of services to families spanning health, social and child protection.244 In particular, UNICEF invested in building capacities of home visiting nurses to employ a holistic approach in their work and consider elements of child development, disabilities, child protection, and gender, in addition to standard health elements of care.245 A further positive example of intersectoral work is the Child Friendly City (CFC) initiative which UNICEF led. This established mechanisms for intersectoral collaboration following consultations with health, social care, child protection and local government representatives. The CFC initiative served as an entry point at the local level for introducing a more intersectoral perspective in planning and delivering services for children in the Korça municipality, resulting in the first municipal CFC Action Plan 2020–2021.
Such initiatives with a strong intersectoral perspective underline and point to the comparative advantage UNICEF has to perform this effective mediation and convening role. The example of the CFC initiative shows the benefits of leveraging UNICEF’s convening role, with a greater range of direct services being provided. For example, this could include establishing a model to provide integrated social services. In practice, the CFC is localized in terms of reach which limits its scope, but it presents a useful model for the consolidation of UNICEF’s learning in intersectoral effectiveness and coherence.
243. Within sectors, UNICEF applied a coherent mix of its core roles, ensuring that an evidence base informs the policy advisory and advocacy, and leads the planning and implementation of capacity building efforts. For instance, in preschool education, UNICEF support- ed generation of evidence to be used in policy dialogue and advice (e.g. the Mid-Term Review of the National Strategy of Pre-University Education produced jointly by OECD and UNICEF), while at the same time investing in advocacy, capacity development and modelling of approaches and services. 244. UNICEF Albania Country Office Annual Report 2019; Massey, C. Poni, M. Sammon, E. (2020) “Evaluation of UNICEF’s Contribution to the Normative Policy Framework of the Social Care Services Reform 2013-2019 in Albania”. UNICEF Albania. 245. UNICEF, Government and CSO key informants. 78 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
The evaluation also identified some instances of integration of social and child protection services at local level. UNICEF used its convener role to support intersectoral collaboration between professionals coming from child and social protection institutions (See EQ 4 for details).246
In other countries UNICEF is traditionally strong in the intersectoral area of early childhood development (ECD), but this is not at the forefront of UNICEF’s work in Albania. Evidence from UNICEF informants at national and regional levels indicates that, despite a range of important individual initiatives over the country programme period, the operational framework for ECD is limited in Albania, due the Government’s lack of recognition of the need for intersectoral engagement on ECD.247 According to stakeholder interviews with UNICEF and CSOs, limited mechanisms are in place to align multi-sector policymaking to promote the holistic development of children younger than six years old. An institutional anchor to coordinate early childhood development across sectors does not yet exist, and there is a lack of definition of responsibilities, which are seen as the main obstacles for UNICEF’s work in this field.
An opportunity to further align its programmes and work is presented in terms of gender integration across Country Office sectors of focus (see EQ5). The absence of a gender programmatic review is a key finding that will feed into integration of gender components of the country programme. The integration of various dimensions of the UNICEF Gender Action Plan into the future country programme would increase its level of attention across multiple instruments for gender related issues.
3.5.2 Evaluation Question 9 Coordination
EQ9. To what extent is UNICEF coordinating with development partners and other UN agencies to avoid overlaps, leverage contributions and catalyse joint work?
Finding 9.1
UNICEF has a strong mediation (convener) role which helps to establish and maintain partnerships with national (government and public sector institutions, civil society) and international actors (donors, development partners). Partnerships with national and international stakeholders are generally positive, although UNICEF could have done more to nurture relations with the EU.
As discussed across other evaluation questions, UNICEF has applied its strength as a convening power and positioned itself strategically with both government institutions and civil society in Albania. Through its effective convening roles across sectors and partners, UNICEF’s positive role was praised in terms of bringing together diverse stakeholders to discuss and provide inputs in legislative drafting and modelling of approaches and services. Such opportunities for exchange also enabled different partners to meet, discuss and exchange on a number of issues of relevance to child rights, which benefited mutual learning from the transfer of results and practices.248
“UNICEF motivates us to spread the results of our practices to other organizations.”
CSO Key Informant
The strong coordination of efforts was noted by stakeholders in the humanitarian response to the 2019 earthquake, with UNICEF’s role praised in terms of response to urgent needs of children and leveraging strengths of partners towards better targeting of children.249
246. Government and CSO key informants. 247. UNICEF key informants. 248. As confirmed by stakeholders from national and local level government institutions, CSOs, UN Agencies and development part- ners. 249. UN, government key informants. FINAL REPORT 79
As a result of a flexible, partnership-based approach, the goals and objectives of UNICEF in Albania were, at a minimum, complementary, but often synergetic with the vision and strategies of Government, civil society actors and other international stakeholders acting together. This was made possible by systemically addressing CP priorities with partners from government institutions within the respective sectors (although as noted above, less so at intersectoral level) as gathered from interviews with the government stakeholders. Government informants, in particular, expressed widespread appreciation for the work and leadership of UNICEF in many sectors and praised UNICEF’s ability to address policy needs and services, a dividend of UNICEF Albania’s four- year partnership strategy. Further, UNICEF’s efforts to assist the Government in advocacy and communication campaigns strengthened its mediation role. This was documented in UNICEF’s annual reports and reflected in feedback from government stakeholders at both central and local levels.
UNICEF’s relations with civil society were of diverse nature and depth depending on type of CSO partner and type of engagement. UNICEF engaged with local CSOs in support to implementation of local level initiatives and models. There were also cases where UNICEF engaged with national level CSOs on advocacy relating to child rights. There are diverging voices when it comes to UNICEF’s approach to cooperation with CSOs. Positive examples identified via document review and corroborated by local CSOs of UNICEF’s local level partnerships were noted not only in terms of policies but also work on the ground, particularly social service provision, for example in migrant tent cities, dumps, welfare centres, etc.
Stakeholders from both civil society and other development partners raised concerns however, about potential overlaps of UNICEF’s work with other CSOs in Albania, for example in social service provision modelling at local level. In these instances, CSOs expressed the perception that they were the more appropriate implementers at local level than UNICEF, due to the nature of their work and approaches. A further criticism was that UNICEF’s sub-grants sometimes created overlaps and should be better coordinated.250
The evaluation also noted some diverging evidence in relation to coordination. Document review and stakeholder interviews provided a positive example of UNICEF’s convener role in response to the earthquake in 2019. The coalition, convened by UNICEF, brought together over 20 international and local actors on child protection issues within the scope of the Child Protection in Emergency Working Group. The way it was managed was perceived as very positive with benefits such as facilitating the exposure of national organizations to many tools and experiences that they would not otherwise have obtained.
Conversely, some national and international respondents noted that a child protection advocacy coalition of five international organizations established in 2018 led to perceptions of exclusion of national CSOs. This was clearly not UNICEF’s intention, but underscores the importance of UNICEF’s role. It implies a need for clarity and transparency about decisions continue to effectively find common ground with all partners or act as neutral broker.251
Evidence points to a rather fragmented interaction between UNICEF and donors and development partners, predominantly on sector-specific issues.252 For example, the EU is the biggest stakeholder in Albania both in terms of policy formulation as well as in terms of direct assistance and support to both government institutions and CSOs. A review of financial allocations and partnerships shows that UNICEF had limited use of EU funds compared to overall EU investments in Albania,253 and while the CPD is aligned with the EU accession agenda, there is limited engagement and partnership with the EU delegation in the country.254 There seems to be a clear opportunity for the Country Office to increase efforts of partnership with the EU and to position itself strategically for complementary and synergetic work with it.
A theme that was raised by some development partners was work with Roma and children with disabilities as an area with potential for UNICEF to reduce overlap by coordinating with other development partners and UN agencies. Such partners expressed a desire for UNICEF to take a more active role in coordinating efforts in this
250. UNICEF, CSO key informants. 251. Government, civil society and development partners key informants. 252. Development partners key informants. 253. From 2017–2019 EC funding amounted to 5 per cent of UNICEF’s donor portfolio. 254. UN key informant. 80 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
area. Some interlocutors (especially UN agencies and donors) noted that rights and access to services for the most vulnerable groups (e.g. Roma children and children with disabilities) were within the mandate of a number of international organizations active in Albania. It seems evident that UNICEF’s proven role as a convenor and coordinator on issues concerning children (and families) belonging to vulnerable groups is much needed.
Finding 9.2 Despite a well-embedded “Delivering as One” philosophy and good participation in the UNDAF, there is some evidence of competition within UN agencies. Joint projects steered through the SDG Fund offer the possibility to overcome partnership challenges.
Albania is one of the countries where the “Delivering as One” initiative was implemented (as a pilot) since the start of the UN reform process. Within this framework, UNICEF has been investing efforts to align and embed its CP within the Government of Albania and the UN Programme of Cooperation for Sustainable Development (PoCSD) 2017–2021 framework, as confirmed by comparative review of these two strategic documents and evidence from interviews with key stakeholders from other UN agencies.
UNICEF has been providing its technical (and also reporting) contributions to the PoCSD process and bringing a strong children rights dimension to the development partnership framework, which was viewed as a positive contribution by other UN agencies.255 UNICEF participates in the Output Groups with other UN agencies, which leads to better coordination and also joint activities, such as the DHS with UN Women and UNFPA and with the WHO on immunization, nutrition and emergencies.
However, the evidence gathered by this evaluation triangulates well with the findings of the PoCSD Evaluation conducted in 2020, which found that “UN agencies work well together to the extent that they need and have to. Programme complementarities have been incorporated already at design stage, alongside a careful consideration of a clear division of labour between the different agencies, but synergies are not actively sought out over the course of implementation. There is inherent competition and territorialism over the PoCSD (and donor) resources and scope of activities. Nevertheless, smaller agencies are keener to cooperate and explore synergies than bigger ones.”256 For instance, key informants noted an overlap of UNICEF’s work on protection of children from violence with other UN interventions on gender-based violence/violence against women activities.257 Stakeholders from UN agencies noted that the UNICEF Country Office could have played a complementary role in an ongoing initiative on violence against women by UN Women, UNFPA and UNDP. This could entail cooperation at municipal and community level where UNICEF has engaged with the same stakeholders in activities related to violence against children (leaders, teachers, social workers).
Despite competition, which is not limited among UN agencies to Albania’s context, there were also positive examples of joint efforts. For instance, UN agencies implemented a joint resource mobilization strategy (developed in 2019), resulting in joint programme resources being channelled through SDG Acceleration Fund initiatives such as Leave No One Behind aiming to assist the MoHSP to update its Social Protection Strategy and Action Plan (2020–2022), as well as an Elderly Action Plan (2020–2024); and the UN Joint Programme to End Violence Against Women in Albania. Stakeholders across all stakeholder groups highlighted these positive measures and the coherence of UN joint efforts within these projects. Another example is intersectoral coordination between UN agencies on crisis responses with a focus on the rehabilitation and reintegration of children, where all sectors from health, psychosocial support, employment, training, education, justice etc. are brought together to cooperate.
Finding 9.3 Private sector collaboration is nascent, emerging, but needs to be more structured and systematic.
255. UN key informants. 256. Evaluation of the Government of Albania and United Nations Programme of Cooperation 2017–2021. 257. UN key informants. FINAL REPORT 81
The evaluation identified a small number of examples of UNICEF’s partnership efforts with the private sector, primarily in the field of online safety of children. For instance, UNICEF organized the VIRAL Summit in Tirana in November 2018, bringing together more than 250 participants including private sector groups258 to exchange best international practices on the benefits and risks children may encounter when using the internet. It also aimed to explore partnerships between government and the private sector (the ICT industry in particular) in Albania on providing solutions to risks faced by children online.
Another initiative reviewed by the evaluation was the Friendly Wi-Fi Albania initiative that was launched to offer a technological response to prevention of online child sexual abuse and harmful online content while accessing free Wi-Fi in public spaces. This endeavour brought together local governments and private businesses to support five Wi-Fi hotspots in the capital Tirana, with safe access for children. UNICEF’s role was rather limited and indirect, ensuring that the national government and private sector partners gain access to online safety filters provided by an international company. UNICEF reporting indicates that one friendly Wi-Fi spot managed, over five months (June–October 2019), to block 43,000 Internet requests to access blacklisted or adult sites, and 4,000 sites containing child sexual abuse and pornography,259 and this is also confirmed by stakeholder interviews.
Beyond these individual initiatives, document review and stakeholder interviews did not reveal any systematic effort by UNICEF to develop a more comprehensive partnership and also resource mobilization strategy towards the private sector. The initiatives, while useful and important, were ad-hoc activities and not undertaken on the basis of a more comprehensive strategy of private sector engagement. There are no specific instances in UNICEF’s CPD on partnership strategy with the private sector, although the accompanying Programme Strategy Note outlines ambitions to partner with private sector operators in several places, albeit with no further elaboration. The 2019 Resource Mobilization Strategy mentions private sector a number of times, but these references are very general and do not articulate any clear plan to approach the private sector and/or to tap their resources. The lack of an operational strategy for private sector engagement is a systemic weakness, taking into account the limited donor pool in Albania but also the well-recognized (by UNICEF) potential of investment in such partnerships (with impetus coming from UNICEF HQ guidance during the programme period260). However, the above examples demonstrate the potential of including private sector support in such areas as online safety, social services and innovation in education.
258. E.g. Facebook, Google, Microsoft. 259. UNICEF Albania COAR 2019 260. As already mentioned in relation to making business and markets work for children in EQ3. 82 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021 4. CONCLUSIONS FINAL REPORT 83
he evaluation findings show a clearly dedicated and responsive Country Office that has built and maintained close working relationships with its key stakeholders, especially the Government but also international and national partners. This has the effect of building trust through continually T demonstrating that UNICEF is a reliable and active partner. The Country Office regularly (annually) revises its plans for the coming year, and some emerging data and evidence is incorporated into planning and adjusting priorities on an annual basis.
Several processes are lacking, and this reduces the ability of the CO to apply a systematic and transparent approach to priority selection. For example, there is a lack of systematic analysis of all emerging relevant data, and a lack of systematic consultation with the full range of stakeholders. Changes within UNICEF at corporate level do not seem to inform thinking by incorporating emerging global advancements in addressing the rights of children. Without programme reporting data for indicators and especially without disaggregated data, the CO cannot know whether it is achieving or even aiming for the right equity commitments.
In a crisis (for example in response to COVID-19) the CO was more systematic as regards the above factors; it rapidly adjusted its activities based on data-informed needs and systematically created an action plan with the appropriate stakeholders. Some criticisms in this area are that the CO did not complete its advance contingency planning, and neither did it establish or collect data to check that results were being achieved.
Overall, the findings indicate that there is a tendency to not plan ahead comprehensively, or document how learning from past experience was incorporated into plans, or to manage the data environment for evidence – neither for documenting progress nor for using data comprehensively to inform decision-making.
The feedback from stakeholders is overwhelmingly positive. UNICEF has positioned itself strategically for its upcoming programme cycle in two key ways: the Country Office has succeeded in building relationships and trust; and it has invested in the policy and legislation enabling environment providing UNICEF and Albania with the necessary framework to follow through with creating tangible changes for children. To do this effectively, there needs to be a shift towards strengthening its evidence base to track actual changes or divergences from targets. This is needed to be able to make decisions on where to focus. But it is also essential for the CO to document progress and its own results via results for children.
There are opportunities and the need for introducing systematic approaches to the following:
• Data disaggregation as part of the equity agenda; • Collecting data on programme indicators; • Incorporating data and evidence into programme priority decision-making during the CP; • Mainstreaming gender across the staff and programme components; • Pre-positioning and pre-planning for emergencies; • Consulting a full range of stakeholders; • Planning for core human resources to reduce reliance on temporary consultants; and • Incorporating UNICEF HQ initiatives and emerging thinking on child rights.
UNICEF’s strategic positioning
UNICEF Albania has placed itself strategically as a trusted partner of the Government but also of international, national and local partners. It is recognized as a convenor and this is mostly highly appreciated by all stakeholders. There are certain issues, for example related to equity and a greater focus on the second decade, that development partners would like UNICEF to take a greater lead on. The investments that UNICEF has made in the enabling environment were much needed, as is the follow through, particularly in the context of decentralization. While overall UNICEF’s strategic positioning has been well aligned with needs, especially in response to the Government, it would be better placed to justify its programmatic decisions and priorities if it introduces more systematic processes. This would provide the necessary check that all emerging data and evidence informs decisions and that an appropriately wide range of stakeholders are consulted. 84 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
Implementation strategies
The choice of implementation strategies to deliver on the programme components was logically derived from the CPD. In examining the ways in which the Country Office worked within its sectors, staff applied a wide range of approaches. There was no evidence that the Country Office was getting this identification of strategies ‘wrong’, and indeed there was a high level of consistency that stakeholders appreciated and the different ways that UNICEF approached issues and ways of working towards achieving its aims was welcomed. What was less well documented was the Country Office’s own rationales for strategies. A missing component was an overarching theory of change for the programme which would support decision-making on programmatic choices, and therefore the most appropriate implementation strategies. Introducing more systematic processes of reflection would also allow the CO to check whether the changing context and new evidence affects its evolving programme priorities and implementation strategies, for example for the emerging needs of seeing policy implemented in the context of decentralization.
Organizational structure
The organizational structure of the CO followed the CPD strategic outcome areas and was designed to be fit for purpose to deliver results. The structure works well ‘on paper’ but the small number of permanent staff take on significant responsibilities including the delivering of results and the managing of temporary consultants (with a peak during the programme period outnumbering the core staff by four to one). The imperatives for strengthening UNICEF’s effectiveness such as increasing intersectoral working, mainstreaming gender, and incorporating new priorities established by UNICEF HQ, are inevitably hindered by the existing pressures and workload of the core staff.
Achievement of results in child rights, especially for the most vulnerable
There is good qualitative evidence to suggest that the work of UNICEF over the past three years has led to changes in the realization of rights of children. In terms of the most vulnerable, UNICEF has focused its efforts on particularly at-risk children such as those in institutions and children of Roma families. UNICEF has also responded rapidly to crises such as the 2019 earthquake and more recently COVID-19. Feedback from stakeholders strongly supports the perception that the situation for children is improving in Albania. However, robust quantitative data to evidence this is severely lacking. The essential component of disaggregated data (sex, age, vulnerabilities) is generally missing. To be sure that UNICEF is responding to the most pressing needs, including those of the most vulnerable, it needs the data to identify appropriate priorities. UNICEF further needs key indicators to be measured to see where progress is taking place or stalling. This is particularly important as the country implements its decentralization process and UNICEF works with municipalities as well as national Government to ensure the downstream implementation of policy gains.
Contribution to improving the performance of government institutions and service providers
UNICEF has contributed to a range of robust and well-designed policies and laws that move the child rights agenda forward in Albania. This represents a culmination of efforts by UNICEF over the past decade in line with Albania’s preparations for EU accession over a similar time frame. As such, UNICEF has successfully placed itself in a key position at a key time in the history of the country to leverage its position to advance the agenda of children’s rights. Similarly, UNICEF has leveraged its position to build the capacity of many sectoral actors in a range of areas that will potentially contribute to the downstream realization of children’s rights. However, there are substantial challenges in ensuring and, importantly, demonstrating, implementation of these policy and legislative changes via appropriate advocacy for resource allocation, capacity-building at subnational levels, and effective and timely monitoring of the realization of children’s rights.
Gender mainstreaming
This is clearly a gap within the Country Programme. There were some important initiatives such as the qualitative KAP survey on child marriage, but overall there was no systematic approach to gender equality principles. Staff themselves cited a lack of capacity and expertise. It is essential that gender equality and gender mainstreaming components become embedded into the thinking, planning and implementation of the country programme across all the staff. Gender mainstreaming is everyone’s responsibility and not the sole domain of the Gender Focal Point. UNICEF as an organization has expertise, and multiple analysis and planning tools that the CO can and should draw on. FINAL REPORT 85
Efficiency
In terms of achievements, to date the Country Office has relatively successfully secured and utilized resources according to budgetary plans, even in the face of a challenging fundraising environment. The ratio of regular resources to other resources has not matched that planned in the CPD, but this has been a response to the dynamic nature of funding within the country rather than any financial management issue. Funding sources are reported as secure in the short-term (through to the end of this country programme cycle), but the Country Office’s conservative approach to resource planning is important to ensure that medium-term uncertainty is mitigated. UNICEF’s human resources seem well allocated and efficiently utilized with, however, the question of the ratio of core to temporary consultants remaining. Partners expressed strong satisfaction with the openness of staff in responding to their programmatic needs with technical expertise. However, although responsiveness to the needs of key partners can be an important and a positive driver, it can also rapidly absorb the capacity of a small office and thus compromise UNICEF’s core mandate to focus on the practical realization of children’s rights.
Sustainability
When measured in terms of the CO’s alignment with national development strategies and Agenda 2030, UNICEF has demonstrated approaches which promote sustainability in the sectors of child protection, social protection and education. A major factor in this, is UNICEF’s investment in the enabling environment through its direct contribution to development of such policies and legislation. The subsequent step after developing laws and policies is to support their implementation at sub-national levels. UNICEF has already positioned itself to engage with municipalities and has had positive results in convening multisectoral actors, for example through the child-friendly city initiative. But its experience and feedback from multiple stakeholders show that the decentralization process poses further challenges as there is increased competition for scarce budget allocations. The extent to which UNICEF is planning for this new context was not strongly evident throughout the evaluation and the CO should not underestimate the challenges of seeing through the implementation of policies.
Further sustainability strategies include UNICEF’s linkages with other actors, which are highly visible in many areas, and yet there are continuing demands from partners for the CO to engage more cooperatively with national CSOs. As part of strengthening the national capability, it seems important to foster further collaboration with this group of stakeholders. Finally, there has been some small but important engagement with the private sector and there are opportunities to capitalize on this.
Internal programme coherence
There were two good examples of intersectoral work being integrated into the Albania Country Office structure (Child Rights Monitoring Specialist and Communications Officer) but otherwise the Country Office generally works in silos. The sector-specific focus aligns well with the way the Government works but it is not necessarily desirable, and UNICEF is probably missing opportunities to increase its effectiveness through complementary, synergistic and joint ways of working. Two key areas, ECD and gender, offer opportunities to create synergies and complementarities across programming. It can achieve this internally within the programme, although the external limitations in terms of working intersectorally with the Government are recognized. Multi-sector coordination of policy development and implementation are limited in terms of the Government’s mechanisms, but UNICEF has the potential to add significant value if it can increase its own and the Government’s intersectoral approaches to promote more holistic responses to child rights and development.
Coordination with development partners
Coordination with development partners is an area of strength for the Country Office with multiple lines of inquiry evidencing and confirming UNICEF’s effective partnership and convening role. There are areas that UNICEF needs to make decisions on regarding where its role is to lead, and what it contributes to. The expectation of development partners that UNICEF would take a stronger role in certain areas (e.g. on responses to Roma children and children with disabilities) is evidence of both UNICEF’s leadership role and also the lack of national coherence and response by development partners. Certain overlaps and competing mandates are noted, such as in trafficking (UNHCR/ IOM), and gender (UNFPA/UN Women), and UNICEF again needs to determine where and how its skills can best be used. There are also opportunities in other areas (for example with other UN agencies, and funding from the EU) which should not be neglected. 86 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021 5. LESSONS LEARNED FINAL REPORT 87
UNICEF has sought to strike an appropriate balance between its organizational mandate and priorities for realization of children’s rights and those of the Albanian Government, for which the rights of children are only a component. This is a dynamic and evolving process that requires regular revisiting.
UNICEF Country Offices have an organizational mandate to advocate for the protection of children’s rights, to help meet their basic needs and to expand their opportunities to reach their full potential. At the same time, they must align themselves with the reality of host government priorities in order to be an effective force for positive change for children.
UNICEF Albania has aimed to achieve a balance between the Government’s priorities – most notably those related to ensuring its legislation and institutions are in line with the EU accession acquis – and being an independent advocate for all children’s rights according to its mandate.
As explained in EQ4 (findings 4.6 and 4.7) UNICEF has contributed considerably to the development of policies and legislation in line with these government priorities. However, the progress on a legislative and policy framework was not accompanied by the same progress in realization of the rights of all boys and girls and achievement of their full potential. This is particularly with respect to downstream implementation and operationalization of major policy/legislative changes to a decentralized level. The risk is that the Albanian Government will address wider EU political goals but not apply the same urgency to downstream or more peripheral policies that reflect the best interests of all children, such as the management of migrants (many of which are children) via the Western Balkan corridor.The key lesson is to dynamically assess its role as children’s advocate and ensure momentum towards actual and equitable realization of children’s rights.
People are the foundation of UNICEF Albania’s country programme outcomes. Overall programme effectiveness is predicated on ensuring the right people in the right place with the right motivation to perform.
While the full-time Country Office staff are highly experienced, UNICEF Albania still relies extensively on temporary consultants to achieve planned programme work. As discussed under finding 3.2, the temporary consultants have limited authority and sense of ownership for specific results and frequently place core office staff under management pressure in addition to achieving results for their portfolios. It also presents challenges to intersectoral collaboration (e.g. child rights monitoring, gender mainstreaming). There is good evidence of permanent staff being overburdened in their multiple roles to the detriment of programme effectiveness and overall staff welfare.
The lesson for UNICEF is to maintain an effective balance between workloads of permanent staff and those of temporary consultants. Permanent staff have significant added value in relationship management, intersectoral work and ensuring good alignment with UNICEF’s strategic goals, whereas temporary consultants can add more value in discrete, technical roles, such as policy drafting, crisis response or capacity-building. Balancing the cost-benefit (in management and quality- assurance as well as financial terms) of temporary consultants (or even long-term agreements with national/regional “think-tanks”) presents ongoing opportunities and challenges that the Country Office must negotiate.
Promising practices: Where an initiative has been successful, further scale-up to expand UNICEF’s reach is a worthy investment, albeit with due consideration to UNICEF’s own capacities and those of the different stakeholders, particularly of central and local government.
Promising practices in UNICEF are those that are designed, implemented and evaluated as programmatically and administratively (i.e. in cost terms) successful. All interventions have the potential to become promising practices and thus the Country Office should be careful to invest in their design (e.g. using theories of change), implementation, evaluation and strategic decision-making around scale-up or scale-out (in consultation with relevant partners) either directly or via handover to partners.
Examples highlighted during the evaluation research by local-level key stakeholders demonstrate some promising approaches that UNICEF has introduced in different areas, e.g. online education during the response to COVID-19, work with local government in social protection, justice for children and child protection (see findings 3.3–3.4). All these interventions were met very positively by stakeholders. The initiatives themselves, combined with UNICEF’s convening power, present opportunities for UNICEF to facilitate and support expansion of these practices to other situations or other locations, including more remote areas which have a disproportionate number of children in need. In considering these opportunities, as with all potential growth areas, it is critical to ensure that any strategies for expansion or new partner relationships are given the appropriate due care and consideration. This includes basing decisions on evidence of needs, costing analysis, UNICEF’s comparative advantages, analysis of alignment with the CO’s overall strategy and absorption potential of central and local government. 88 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021 6. RECOMMENDATIONS FINAL REPORT 89
Recommendations for UNICEF action were developed on the basis of the key findings of the report and in consultation with the Evaluation Manager and ERG. As not all findings suggested a need for substantive change, not all led to recommendations for action. The evaluation team has focused the recommendations on areas of potential change that are of greatest significance and utility for UNICEF Albania.
Recommendations also include a description of their area of relevance (strategic or operational), their priority (high or medium) and time horizon (short/medium/long). In the interests of maximizing the utility of this section, low-priority recommendations were not made.
RECOMMENDATION 1
Introduce a systematic approach to country programme priority setting that is in line with an equity-based theory of change of the new Country Programme Document and informed by evidence. Ensure regular periods of reflection.
(strategic, high priority, short-term time frame)
Programme priority setting should be rooted in the governing theory of change and informed throughout the programme cycle by robust evidence that supports adaptive management. Suggested key aspects of this approach to programme planning should incorporate:
• Outcomes of needs assessments, emerging data sets/new evidence, consultations with external stakeholders/rights’ holders and/or formative research (on key equity-related dimensions of marginalisation or vulnerability); • Key UNICEF global or regional strategies or guidance, such as the UNICEF Strategic Plan 2018–2021 or the UNICEF Gender Action Plan 2018–2021 and other emerging factors (including sectoral and equity-related guidance) during the lifespan of the new CPD; • Identification of synergies and mutually beneficial ways of working within the Country Office to equitably realize rights for children in Albania and ‘leave no-one behind’; • A distinct and logical classification of outcomes, outputs and associated implementation strategies to achieve these; • Conduct regular reflections to ensure that programme direction and intervention strategies remain relevant or are adjusted accordingly. A Strategic Moment of Reflection or mid-term review, or similar, is suggested for consideration in the new country programme to provide an opportunity to incorporate strategic changes based on new evidence, guidance and learning emerging after country programme approval; and • Any updates or amendments to this governing strategic framework should be appropriately documented and propagated across all of UNICEF’s strategic planning.
RECOMMENDATION 2
Increase focus on intersectoral work to maximize the coherence of Country Office programming with the overall theory of change of the new Country Programme Document.
(strategic, high priority, medium-term time frame)
Building on a robust, logical and coherent theory of change and linked programme strategies, an increased focus on intersectoral working between the different programme components of UNICEF Albania should form part of the new CPD process. This could include:
• An explicit focus on internal collaboration and ensuring that more than one programmatic staff member is responsible for intersectoral results (notably for equity and gender considerations); 90 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
• Ensuring that the responsibility for intersectoral work is not only delegated to positions with an inherently intersectoral profile but appropriately distributed across all positions with well-defined responsibilities in the staff members yearly deliverables in Achieve; • Strong leadership and management to ensure appropriate staffing/workloads to balance sectoral and intersectoral responsibilities and eliminate sectoral isolation; • Introduce programmatic tools for planning and reporting that articulate the horizontal links across programmatic sectors necessary for effective intersectoral work; and • Recognize and explicitly seek to address the challenge of promoting synergies and intersectoral work with and between government partners.
RECOMMENDATION 3
Plan for future and ongoing emergency responses
(strategic, medium priority, medium-term time frame)
Country Office staff have leveraged their experience of successive crises of varying sizes and natures to be able to respond quickly and appropriately. This experience should be leveraged for more systematic planning (potentially leveraging the skills and resources of ECARO) for future crises (including the evolution of the COVID-19 crisis). Such planning could include:
• Consideration of an emergencies risk analysis as part of the CPD planning process to assess internal skills and capacity, analyse children and women’s vulnerability, and assess capacities of Government, humanitarian actors, national partners and communities; • Develop minimum preparedness activities for different types of crisis (sudden/slow-onset, acute/chronic) to fill the capacity gaps and ensure UNICEF readiness to respond in line with UNICEF global emergency preparedness guidance; • Designation of an emergencies focal point and appropriate training of this position and all other staff; • Adoption of standard operating procedures/critical actions for crisis response; • Integration of interagency/One-UN approaches and similar risk analyses/planning in place under either the One-UN approach or individual planning processes among sister UN agencies and/or CSOs; • Consideration of an evaluation of the 2019 earthquake response may be a useful learning tool (although the value of this should be considered in the light of COVID-19 restrictions); and • Consider the different potential outcomes of the COVID-19 pandemic, which could feasibly extend into the coming years and fundamentally alter social, education, welfare and economic dynamics in Albania, including the emergence of new risks to children and challenges to equitable development.
RECOMMENDATION 4
Determine a clearer partnership strategy across different stakeholder groups and set partnership objectives for each for which progress can be tracked
(strategic, medium priority, medium-term and long time frame)
Programme strategies should incorporate commitments to, and practical approaches (including objectives, engagement modality, selection criteria, outcomes, performance metrics) around partnership with different groups, and tailor approaches to each. UNICEF should maintain a small number of core consistent partnerships that reflect investments in longer term relationships for realizing rights equitably. Specific groups to be covered are:
• Private sector/academia: modalities should go beyond simple service provision (by private sector providers) or fundraising to look at modalities for private sector action to equitably realize children’s rights or advocate/ influence for children; FINAL REPORT 91
• Civil society: A specific and transparent strategy will ensure much-needed support to the national-level civil society sector in general, build on the successes of partnerships to date and ensure fair and equitable relationships; • Development partners, notably the EU delegation: UNICEF should position itself strategically for complementary and synergetic work with EU priorities to take advantage of increased resources available as Albania prepares for EU accession. Interagency opportunities with other UN agencies should be actively sought bilaterally or as part of DaO/One-UN/UNDAF; and • Government: clear partnership strategies that ensure UNICEF complements, builds, supports and engenders ownership rather than replaces or does the work of government counterparts in key mandate areas.
RECOMMENDATION 5
Invest in a clear and systematic approach to data and evidence generation
(strategic, medium priority, medium-term time frame)
To support results-based management and strategic decision-making there is a need for a clearer and more systematic approach to data and evidence generation. This approach should:
• Place ongoing and timely analysis of the equitable realization of child rights at its centre to ensure that programmatic priorities and implementation strategies are adopted based on complete, accurate and timely data and evidence – this includes establishing effective results indicators as part of the monitoring and evaluation system; • Support external administrative data collection systems to effectively and regularly collect relevant data, minimizing gaps where possible, and clearly identifying ongoing gaps and needs; • Plan the data/evidence generation exercises to link their findings explicitly to UNICEF’s advocacy work and revisit advocacy (and therefore programming) targets when new data emerges; and • Annually plan and utilize an appropriate mix of ongoing (e.g. annual) monitoring activities (for performance measurement and mid-course adjustments), studies and evaluations exercises (for setting advocacy targets or programme strategies).
RECOMMENDATION 6
Focus on support to the ongoing evolution of the normative framework and decentralization process in Albania, including advocacy for appropriate resource allocation at subnational levels and guided by data and evidence from monitoring of service provision, particularly to the most vulnerable and marginalized.
(operational, high priority, medium-term time frame)
A solid basis in accurate and up-to-date data and evidence is crucial to ensuring operationalization of policies related to equitable realization of children’s rights and changing the normative framework through the levels of government (e.g. at municipality and local level). UNICEF should address the challenges in policy implementation and be closer to services and children via:
• Appropriate advocacy for resource allocation (technical and financial) for children in Albania at both central and, importantly, subnational levels in order to promote adequate, timely, efficient, equitable and accountable downstream spending. This could capitalize on the guidance already available from UNICEF in the PF4C area:261
261. See https://www.unicef.org/sites/default/files/2019-12/UNICEF_Public_Finance_for_Children.pdf 92 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
• Redouble efforts to build capacity of key child rights stakeholders at subnational levels, for example among officials and services in the social welfare and protection sectors who require capacity and resources to operationalize the Code of Justice for Children and the Law on Protection of the Rights of Children (see Finding 4.2); • Ensure that monitoring, evaluation and research activities are relevant to setting and promoting effective implementation of policies and services for children at municipality level in line with the decentralization process and to be closer to children as rights-holders; and • Act as a monitor of and advocate for equitable changes to the normative framework and roll-out of policies, notably around the less tangible and more challenging aspects of operationalizing policies that may be neglected in the drive to EU accession.
RECOMMENDATION 7
Introduce a Country Programme-wide effort to embed gender equality principles and gender mainstreaming
(operational, high priority, short-term time frame)
This recommendation is noted as ‘short-term time frame’ in order to indicate that it needs to be initiated with a high priority but will also require ongoing attention throughout the programme cycle and beyond. The Country Office needs to embed its approach to gender-responsive programming and this will take time. Suggestions for increasing its efforts in gender mainstreaming across its country programme include:
• Draw on existing data and where needed generate specific evidence that will inform and be the basis of the new CPD, with respect to children and women (e.g. Children and Women SitAn) to assess the manifestations of child rights’ shortfalls, and disparities and inequities in child outcomes for both girls and boys (including for adolescents); • Consider a Gender Programmatic Review or similar exercise prior to the development of the new CPD and/ or in moments of strategic reflection to guide gender mainstreaming in programming; • Institute deeper and proactive engagement with external and regional expertise to gain technical skills on gender-responsive programming; • Programmatically align with the Gender Action Plan 2018–2021 and use Regional or Global Gender- Responsive Programming Tools to achieve gender–transformative results. This should include use of output level Gender Equality Markers (GEM) linked to Specific Intervention Code-level gender tags through the percentage of expenditures allocated to gender tagged activities; and • Put in place a capability strengthening plan for the Country Office to build expertise and embed learning on gender-responsive programming over the course of the country programme. This should focus on building the capabilities of both core and temporary consultants. FINAL REPORT 93
Annexes
Terms of Reference
Reconstructed Theory of Change/Evaluation Matrix
List of Interviewees
Documentary Evidence
Data Collection Instruments
Evolution of UNICEF Albania CPD Priority Results Areas
UNICEF Albania Programme Locations
Gender Integration and Targeting in AWPs, COARs, RAMs and SMQs 2017–2019
Results Reporting Analysis
UNICEF Albania Stakeholders
Evaluation Reference Group Members
Human Rights Analysis
Evaluation Team 94 EVALUATION OF THE UNICEF ALBANIA PROGRAMME 2017-2021
UNICEF Albania Country Office Skenderbej Street, UN House Bld, 3rd floor Tirana, Albania
Telephone: +355 4 45 48 400 Email: [email protected]
© UNICEF Albania, Tirana December 2020