Evaluation of the UNICEF Education Programme in Uzbekistan Final Report
Contract no.: 43292733 January 2021
Prepared by: Zehra Kacapor-Dzihic, Team Leader Dr. Johann (Hans) Schustereder, International Education Specialist Shukrat Amanov, National Evaluator Herma Majoor, Quality Assurance
Contractor: EPRD Office for Economic Policy and Regional Development Ltd. Szkolna 36A Street 25-604 Kielce, Poland
DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the UNICEF or the Government of Uzbekistan.
1
TABLE OF CONTENTS List of tables ...... 4 List of acronyms ...... 5 Executive Summary ...... 7 1. Introduction ...... 11 2. Background and context ...... 11 2.1 Country context ...... 11 2.2 Contextual issues affecting education in Uzbekistan ...... 12 2.2.1 Policy issues ...... 12 2.2.2 Institutional issues ...... 12 2.2.3 Enrolment ...... 13 2.2.4 Socioeconomic issues...... 14 2.3 Sustainable Development Goals and education reform in Uzbekistan...... 14 2.4 Recent legal initiatives in education ...... 14 3. The UNICEF Education programme ...... 16 4. Purpose, Objectives and Scope of the Evaluation ...... 20 4.1 Purpose and Objectives of the Evaluation ...... 20 4.2 Intended Uses of Evaluation and key stakeholders- users of the evaluation ...... 20 4.3 Evaluation scope ...... 21 4.4 Stakeholders analysis ...... 21 5. Methodology ...... 22 5.1 Evaluation design and approach ...... 22 5.2 Evaluation criteria and questions ...... 23 5.3 Data collection methods ...... 24 5.4 Data Analysis Methods ...... 25 5.5 Ethical considerations ...... 26 5.5.1. Protocols for Safety, Confidentiality, and Data Protection and mitigation measures ...... 26 5.6 Limitations and mitigation measures ...... 28 6. Evaluation findings ...... 29 6.1 Relevance ...... 29 6.1.1. Alignment of the programme with global, regional and national priorities in education sector, including SDGs (EQ 1, EQ 2, EQ 3) ...... 29 6.1.2. Relevance of the programme implementation strategies (EQ 4) ...... 31 6.2 Effectiveness ...... 32 6.2.1. UNICEF’s contribution to strengthening of evidence-based legislative and policy frameworks to improve access to equitable and inclusive quality pre-school and general secondary education services (Original Output 2.1/revised Output 2.1 (also relates to original outputs: 2.5, 2.6) ...... 33 6.2.2. UNICEF’s contribution to enhancement of government capacity to provide increased access to inclusive and quality early learning services (Original Output 2.2 and 2.3/revised Output 2.2) ...... 35 6.2.3. UNICEF’s contribution to enhancement of tools and skills to better formulate and implement programmes for inclusive and quality general secondary education for all children (revised output 2.3/original outputs 2.2; 2.5; 2.6) ...... 37 6.2.4. UNICEF’s contribution to enhancing tools and skills to promote safe behaviour practices among children and communities before and during emergencies (revised Output 2.4/original output 2.7) ...... 37 6.2.5. Consideration of gender and vulnerability in programme planning and implementation ...... 39
2 6.2.6. Unintended negative and positive outcomes and their management ...... 41 6.2.7. Factors contributing to and hindering the delivery of outcomes ...... 41 6.3 Efficiency ...... 43 6.3.1. Review of UNICEF’s implementation strategies (EQ 9) ...... 43 6.3.2. Adequacy and cost-efficiency of financial, material and human resources to meet the programme objectives (EQ 10 and EQ11) ...... 45 6.4 Sustainability ...... 48 6.5 Impact ...... 50 6.6 Coherence ...... 50 6.6.1. External coherence (EQ 16) ...... 50 6.6.2. Internal coherence (EQ 17) ...... 51 7. Conclusions, lessons learned and recommendations ...... 53 7.1. Conclusions ...... 53 7.1.1. Relevance ...... 53 7.1.2. Effectiveness ...... 53 7.1.3. Efficiency ...... 54 7.1.4. Impact ...... 54 7.1.5. Sustainability ...... 55 7.1.6. Coherence ...... 55 7.2. Lessons learned ...... 55 7.3. Recommendations ...... 56 8. List of Annexes ...... 62
3
List of tables
Table 1. Ethical Issues and Mitigation Measures ...... 27 Table 2. Overview of the fulfilment of UNICEF roles (implementation strategies) ...... 43 Table 3. Financial overview 2016-2020 (as of December 2020) ...... 46 Table 4. Recent legislation relating to education in Uzbekistan ...... Error! Bookmark not defined.
4 List of acronyms
C4D Communication for Development CEDAW Convention on the Elimination of all Forms of Discrimination Against Women CO Country Office CP Country Programme CPD Country Programme Document CRC Convention on the Rights of the Child CSO Civil Society Organization CwD Children with Disability DPG Development Partner Group DRR Disaster Risk Reduction ECD Early Childhood Development ECE Early Childhood Education ECCE Early Childhood Care and Education ELDS Early Learning Development Standards EMG Evaluation Management Group EMIS Education Management Information System EQAF Education Quality Assurance Framework ERG Evaluation Reference Group ESA Education Sector Analysis ESC Evaluation Steering Committee ESP Education Sector Plan ET Evaluation Team GD Group Discussion GDP Gross Domestic Product GPE Global Partnership for Education HRBA Human Rights-Based Approach IEC Information, Education and Communication KAP Knowledge, Attitudes and Practices KII Key Informant Interview LEG Local Education Group M&E Monitoring & Evaluation MICS Multiple Indicator Cluster Surveys MOPE Ministry of Public Education MOPSE Ministry of Pre-school Education NCF National Curriculum Framework NGO Non-governmental Organization OECD Organisation for Economic Co-operation and Development's Development DAC Assistance Committee OR Other resources PISA Programme for International Student Assessment PPP Public-Private Partnership PSE Pre-school Education QA Quality Assurance RBM Result-based Management RR Regular Resources SBM School-based Management
5 SDGs Sustainable Development Goals SISQE State Inspectorate for Supervision of Quality in Education SMR Strategic Moment of Reflection SP Strategic Plan TALIS Teacher and Learning International Survey ToC Theory of Change UN United Nations UNDAF UN Development Assistance Framework UNEG United Nations Evaluation Group UNICEF United Nations Children's Fund
6 Executive Summary
1. This report presents the end of cycle evaluation of the Uzbekistan Education Programme 2016-2020 for UNICEF in Uzbekistan. This evaluation is commissioned by the UNICEF Country Office and covers the period from 2016 to September 2020, assessing the entire UNICEF’s Education portfolio, both the upstream policy work and capacity development, evidence generation and advocacy activities. The findings and recommendations of this evaluation will strategically inform the implementation of the 2021-2025 Uzbekistan Country Programme.
2. Uzbekistan is the largest country in Central Asia with a population of around 34.3 million, around one third being under 18 years of age. After the presidential elections in 2016, Uzbekistan adopted the National Development Strategy 2017-2020, which envisions economic liberalization, administrative reforms, judiciary reforms and also reforms of the social and education sectors.
3. Since the beginning of the current UNICEF country programme in Uzbekistan, and particularly following political change in 2016, education sector underwent significant regulatory and institutional changes, some of which were supported by UNICEF. Notably, legislative and policy framework underwent changes, with adoption of new Laws on Education (adopted in September 2020) and a new Law on Pre-school Education (adopted in December 2019). Additionally, two consecutive Education Sector Plans (2013-2017, 2019-2023) were in place during the implementation of UNICEF CP within this period. Besides, on January 25, 2018 President of Uzbekistan Shavkat Mirziyoyev signed a decree "On measures to radically improve the system of general secondary, secondary specialized and vocational education." The decree stipulates that from the 2018-2019 academic year, compulsory general secondary education is carried out on the basis of a continuous and free 11-year cycle. Following the prioritization of early childhood education by the government, a new Ministry of Pre-school Education (MOPSE) was established in 2017. Likewise, a new State Inspectorate for Supervision of Quality in Education under the Cabinet of Ministers (SISEQ) has been created in 2017 to oversee and coordinate government efforts to promote quality education. The Ministry of Secondary Specialized and Higher Education stays in charge of provision of vocational and higher education.
4. While at the classroom level textbook-centered approach continues to prevail, there is increasing awareness at all levels of the education system of the need to shift to a competency- based approach. Since 2018 the Government has, with the establishment of the Fund for the Development of None-State Pre-school Educational Institutions, actively promoted the construction and refurbishing of private pre-school institutions. Additionally, a public-private partnership mechanism for pre-school education was introduced. In 2017, the Government announced its plan to significantly expand coverage of pre-school education to 100%. As of the academic year 2021/2022, pre-school education will be mandatory for all children aged six. In 2019, all levels of education developed a Concept for Development by 2030. The increased awareness of the importance of education for the country’s development was reiterated in the President’s address on January 25, 2020, where, among others, the development of students’ strategic thinking skills throughout the education system was highlighted.
7 5. The main focus of the UNICEF Education programme lays on overall pre-school education (3-6 years) and general secondary education, with particular focus on technical assistance to Government of Uzbekistan to strengthen legislative and policy framework for access to pre- school and general secondary education; increased Government capacity to ensure access to inclusive and quality pre-school education; increased Government capacity to formulate and implement inclusive and quality general secondary education; and increased Government capacities for the promotion of safe behaviour of children in emergency situations. The UNICEF Country Programme 2016-2020 was designed based on the Country Programme 2010-2015 evaluation, and further refined as a result of the Strategic Moment of Reflection in 2018, when the Education programme was adjusted reflect the development of the country’s priorities in education. The adjustment resulted, in particular, in a reduction of the original seven outputs to four, nevertheless maintaining the original scope of intervention.
6. The current evaluation covered the above UNICEF Education Programme, which was implemented through three Work Plans (2016-2017, 2018, 2019-2020). The objectives of the evaluation were to: assess the relevance, effectiveness, efficiency, and sustainability of the Programme, its long-term changes and the created synergies; provide recommendations for policy-level decisions; identify successes, challenges and lessons learned; and provide recommendations for the better incorporation of gender and equity issues in future programmes. The evaluation used a variety of data collection methods, namely desk study of project documentation, UNICEF’s and other external reports, research studies and analyses; interviews with UNICEF team from the country and the regional office; interviews and group discussions with representatives of national institutions, education professionals and experts, development partners, CSOs. Due to the situation resulting from the COVID-19 pandemic, interviews and discussions were held remotely.
7. During interviews and discussions with stakeholders, the Evaluation Team respected all norms and standards to ensure impartiality, confidentiality and ethical behaviour. Key considerations in this regard included security considerations for participants, the participation of females and opportunities to express themselves as well as data protection considerations. Besides, the evaluation underwent the Ethical Committee Review, receiving ethical clearance during the inception phase.
8. The evaluation delivered a comprehensive and reliable picture of the Programme in terms of the criteria relevance, effectiveness, efficiency, sustainability, its long-term changes and the created synergies.
9. In terms of relevance, the Programme is deeply rooted in national priorities and the unmet educational demands of children in the country. During the implementation period, UNICEF showed a high degree of flexibility and adaptability to the changing policy environment and arising new needs in the education sector. UNICEF managed to maintain its relevance and strategic position under the circumstances of political transformation.
10. Parallel to the implementation of the Programme, the Government frequently approached UNICEF for support in a variety of other education-related issues. Such requests testify to the good cooperation between the Government and UNICEF and the trust UNICEF enjoys. UNICEF was the driving support force in government’s efforts to develop legal and policy documents in the field of quality pre-school and general secondary education. Through data collected in studies and reviews UNICEF informed both state policy and UNICEF decision- making processes, ensuring relevance of offered technical assistance measures.
8
11. When it comes to efficiency, UNICEF exercised due diligence in financial and portfolio management. The education section team is lean and recorded increasing thematic and sector expertise, with a gradual shift from outsourcing expertise to supporting the reform with their strengths. UNICEF’s responsiveness helped maintain strategic partnerships, but at the same time, enhanced the risk to overstretch the portfolio. UNICEF reports cover key activities, but do not always include all of the additional support that was provided to the Government.
12. In terms of effectiveness, UNICEF made a direct contribution to the improvement of the legal and policy framework for pre-school and general secondary education. Through studies and reviews, UNICEF helped the Government to better understand needs and priorities in this regard. While UNICEF has shown substantial and reliable advisory and convening power, its contribution to capacity building in stakeholder institutions was limited to institutions at the central level. Gender and equity issues, which were addressed throughout programme implementation, will require additional attention throughout the implementation of future programmes.
13. Changes in the legal and institutional setup brought about with UNICEF support are likely to result in long-term change in the country’s educational environment, particularly with regard to access and the quality of education. However, education targets deeply rooted problems and efforts to reform education system and processes take inter-generational shift to show real outcomes and impact, which prevents efforts from showing immediate results, a final assessment on long-term changes is not possible at this stage.
14. The sustainability of the UNICEF programme will depend to a major extent on the Government’s efforts to fully implement the policy and legal environment created with UNICEF support, particularly at the subnational level. Factors like a high staff turnover across ministries and state agencies, frequent leadership changes and at times a too fast pace of reform potentially threaten sustainability.
15. Interventions and support measures under the Programme were coherent both among themselves and with interventions of other development partners. While there was no overlap with other interventions, there is potential for improvement between the UNICEF Country Office’s education section and other sections within the UNICEF Country office, particularly the social policy and child protection section. This presupposes a certain focus of social policy interventions that have effect on education as well.
16. The implementation strategies used under the Programme (evidence generation, policy dialogue and advocacy, partnership building, and capacity building) were appropriate and ensured participatory and timely implementation of the three Work Plans. The programme did not have any unintended negative effects. The key unintended positive effect is improved communication between state partners that resulted through UNICEF’s convener role.
17. The evaluation identified important lessons learned, as follows:
1) Stable and reliable support to the government in situations of rapid reforms is critical to ensure main principles enshrined in CRC and those underpinning equality and equity in education are integrated in reform priorities. In response to fast-paced reforms that aim at overhauling education system in Uzbekistan, UNICEF provided unreserved support to building evidence base
9 to inform policies and regulations, supporting institution and regulatory framework building, Reliability and commitment over a longer period of time enhanced government’s trust in UNICEF as a partner, providing also the space to UNICEF to raise critical issues from the perspective of a trusted friend. National partners underlined UNICEF’S calm and considerate approach to requests (including those labelled “urgent”). 2) Demand driven programmes need to come with a caveat of what is possible to provide within time and resource limitation. Possibility to be on the forefront of support to overhauling reforms in a country is attractive, as it offers opportunity to influence the direction, scope and breath of reform priorities and principles. However, demand driven nature provides a trap of doing too many things at the same time, affecting overstretching of resources (human, financial), resulting in fragmented and ‘shallow’ support. Responding to the requests needs to be balanced to ensure that partnerships are maintained but are founded on realistic expectations about the extent to which assistance is feasible given resource limitations. In some cases, not addressing a request may be better that trying to cover too many issues at the same time. 3) Reform support initiatives including modelling of approaches or mechanisms need to integrate scale-up and institutionalisation mechanisms from the onset to ensure sustainability. The institutionalization of assistance is key to ensuring sustainability and bringing about long-term change. Institutionalization tends to face administrative and legal barriers, which need to be recognised and addressed in the risk-mitigation strategy which should underpin the implementation of the reform support initiative.
18. The evaluation delivered two types of recommendations to UNICEF for the planning and implementation of future Country Programmes: recommendations related to strategic planning and thematic recommendations. 19. Recommendations related to strategic planning include the following:
1) Consolidate the education programme to ensure full support to deepening of reform gains 2) Ensure cross-sector and cross portfolio synergies are in place to provide for holistic approach to child rights. 3) UNICEF should invest in further collaborative approaches with other development partners
20. Thematic recommendations include the following: 1) Continue supporting curriculum reform, ensuring also that gender (as well as disability) dimension of education is integrated 2) Continue building local expertise through investment in Training of Trainers (ToTs) and their multiplication across the country. 3) Promote, advocate and offer technical assistance to inclusive education and the deinstitutionalization process 4) Continue investing in modelling of alternative educational services, particularly for PSE 5) Invest efforts in DRR and climate change resilience (also COVID)
10 1. Introduction
21. This Evaluation Report presents the end of cycle evaluation of the UNICEF Education Programme in Uzbekistan for the period of 2016-2020. This evaluation is commissioned by the UNICEF Uzbekistan Country Office and covers the period from the start of the programme to September 2020, assessing the totality of the UNICEF’s education portfolio, including support to pre-school and general secondary education reforms and their long-term impact. The findings and recommendations of this evaluation will strategically inform the implementation of the 2021-25 Uzbekistan Programme. 2. Background and context
2.1 Country context 22. Uzbekistan is the country with the largest population in the Central Asian region, which gained its independence in 1991. As of September 2020, the country’s population was estimated to be 34.3 million people,1and the 36 per cent of total population were children under 18 years old2.In 2010, Uzbekistan was re-classified by the World Bank as a lower-middle income country, with high rates of reported economic growth, but coupled with existing regional and income disparities. Following presidential elections in 2016, the country embarked on a new path of ambitious economic and social reforms as outlined in newly adopted National Development Strategy 2017-2020. These include, among others, ongoing economic liberalization, administrative and judiciary reforms, and reforms of social sector programmes and strategies, including in educations sector. 23. In the June 2020 issue of the World Bank World Economic Outlook report, data on world GDP growth for 2020-2021 were announced. The impact of the coronavirus pandemic (COVID-19) is projected to lead to a 1.7% decline in the Central Asian economy. It is notable that among the 24 states in the region of Eastern Europe and Central Asia, Uzbekistan will become the only country where economic growth of 1.5% is currently expected in 2020 according to World Bank projections. World Bank expects 6.6% increase in Uzbekistan’s economy in 2021.3 According to the latest data presented in the UNICEF’s Situational analysis of Children 2019/2020, the national poverty was reported at 11.4% with three-quarters of the people in poverty live in rural areas, while the relative poverty was estimated to be at a level of 21% of the overall population and 24% among children in 20184. The repercussions of the COVID-19 pandemic on poverty are yet not clear, but stakeholder interviews conducted within the scope of this evaluation show that significant negative outcomes are to be expected. 24. Like many other countries, Uzbekistan is facing significant challenges in the field of education. Key issues include the enrolment rate in preschool education, the quality of education at all levels of the education system, the widespread use of traditional teaching methods, the insufficient development of critical thinking, communication and problem-solving skills, inclusive education, assessment of student achievement, Government capacity to plan and implement reform programmes, and low salaries and the professional development of teachers and school managers.5 In its latest Programme, UNICEF addressed those issues by promoting preschool education, enhancing the quality of basic education, strengthening the
1 https://stat.uz/ru/ 2 https://stat.uz/ru/ofitsialnaya-statistika/demography 3 http://pubdocs.worldbank.org/en/344691588788182868/Global-Economic-Prospects-June-2020-Regional-Overview- ECA.pdf 4 UNICEF (2020); Situational Analysis of Children in Uzbekistan 2019/2020, p. 38 5 Cf., among, others https://www.unicef.org/uzbekistan/en/education
11 legislative and policy frameworks, promoting access to inclusive education, and providing tools and skills for the formulation and implementation of education reform programmes.
2.2 Contextual issues affecting education in Uzbekistan
2.2.1 Policy issues
25. During the implementation of the current UNICEF Country programme 2016-2020, a range of far reaching reforms have been undertaken, specifically since the political transformation that unfolded since 2017. The education sector was one of the sectors where significant reforms were taking place, with reformulation of education system, primarily pre-school education but also other education level. 26. UNICEF coordinated the preparation the Education Sector Plan (ESP) 2013-2017 that marked the first effort to prepare a medium /long term education plan with support from the Global Partnership for Education (GPE). The Plan provided an overarching policy framework, designed through an inter-sectoral effort, that offers the long-term vision and strategic objectives for developing the education sector and its sub-sectors over a five-year period. In 2018, UNICEF supported the Government of Uzbekistan to develop the second ESP 2019- 2023 by bringing together an entire spectrum of education stakeholders in the country. The ESP 2019-2023 continues to underline the importance of equal access to students of both genders, while it places main focus on quality education and related interventions. 27. Secondary education (basic education) has been reformed from two-tier 9+3 (9 years of general secondary education + 3 years of secondary specialised professional education through either academic lyceums or professional colleges) to 11 years. Starting with the 2017- 2018 school year, compulsory public education was expanded from 9 to 10 years of study and was again expanded to 11 years of study in 2018-2019. 28. In October 2020, the presidential decree “On measures to further improve the education system and upbringing of children with special educational needs” and the Concept for the Development of Inclusive (Integrated) Education in Uzbekistan until 2030 were adopted.6 These were, among others, the result of continuous UNICEF advocacy and support to the Ministry of Public Education. In accordance with the document, as an experiment in the 2021/2022 academic year: The system of inclusive education is being introduced in one of the general education schools in Tashkent; primary basic correctional classes for children with special educational needs will be opened in one of the secondary schools of the Autonomous Republic of Karakalpakstan, the country’s regions and Tashkent; and specialized groups are being created in selected professional schools of the Kashkadarya, Fergana and Khorezm regions for 9th grade graduates of specialized educational institutions and children with special educational needs.
2.2.2 Institutional issues
29. Until the establishment of the Ministry of pre-school education (MOPSE) and the State Inspectorate for Supervision of Quality in Education under the Cabinet of Ministers (SISEQ) in 2017, the Ministry of Public Education (MOPE) was responsible for the overall strategy and
6 https://uznews.uz/ru/article/25320
12 implementation of the state policy for both general secondary7 and pre-school education. The establishment of the MOPSE highlights the prioritization of early childhood education by the government. The task of the SISEQ is to oversee and coordinate government efforts to promote quality education. The Ministry of Secondary Specialized and Higher Education continues to stay in charge of the provision of vocational and higher education. 30. Students can choose to attend academic lyceums instead of public education schools for grades 10 and 11, although the study program for academic lyceums has been condensed to two years. Hence, compulsory education could involve 11 years in a general secondary public education school or 9 years in a public education school plus 2 years in an academic lyceum. Starting in the 2019-2020 academic year, the graduates of grade 11 can choose to continue their studies in vocational colleges. The reformed structure of the system means that students now have three pathways to complete their pre-university education: (i) 11 years of public education; (ii) 9 years of public education + 2 years in an academic lyceum or professional schools; or (iii) 11 years of public education + 0.5-2 years in a vocational college.
2.2.3 Enrolment
31. Uzbekistan maintains high enrolment rates in education. In 2019, there were a total of 6.1 million students enrolled in 9,942 schools.8 According to national data, the gross enrolment ratio of school-age children in primary and secondary school was around 97.4% in 2018/2019 and has varied between 97-99% during the period of 2000-2017.9 There were no significant differences in school enrolment by gender. In terms of transition rates, 98.3% of grades 1 - 4 students in Uzbekistan complete grade 4 and transition into grade 5. Nearly 95% of boys and 94% of girls complete grade 9. In 2019, the overall gross graduation ratio from grades 5 - 9 in Uzbekistan was 91%.10With almost universal enrolment, the overarching focus is mainly on equity and quality of education. The new Law on Education of 2020 redirects the focus to educational quality issues and includes additional articles, focusing on, inter alia, a more rights- based approach; more attention directed at realizing inclusive education; lifelong education; a greater role for the private sector. Likewise, the new Law on Preschool Education is based on right-based approach and provides focus toward universalization of preschool education. 32. The government recognizes the importance of investing in early childhood education, and it has announced an ambitious plan to expand Early Childhood Education (ECE) services with the aim of achieving 100% enrolment for children. With this goal, it established the MOPSE in September 2017 to play the lead role in the expansion of ECE in Uzbekistan. Currently the biggest increase in enrolments is in ECE – before 2017, enrolment was less than 20%, while before the COVID 19 lockdown it increased to more than 50%. This significant increase is likely to result, among others, from a Public-Private Partnership (PPP) mechanism and alternative methods of preschool education that UNICEF designed with MOPSE. 33. Under national legislation, all children with disabilities in Uzbekistan have the right to education. According to Education Sector Plan 2019-2023 in 2016/17, 20,794 children with disabilities attending Specialised Preschool educational institutions (children with disabilities included only in 188 of 5,735 [3.2%] Preschool institutions), 28,890 children with disabilities
7 In the education system of Uzbekistan, the concept of “general secondary education” refers to all stages of compulsory education, i. e. it includes what in other education systems is commonly referred to as “primary education”, “lower secondary education” and “upper secondary education”. “General secondary education” is to be understood as a level of educational attainment. 8 Ministry of Public Education. http://uzedu.uz/ru/statistika 9 UNESCO: http://uis.unesco.org/en/country/uz 10 Education Sector Plan 2019-23.
13 attending General secondary schools (children with disabilities currently included in 5,900 schools = 60.7% of the total of 9,719 secondary schools in the country), 18,600 children attend special schools and approximately 10,000 children with disabilities attend home schooling.11
2.2.4 Socioeconomic issues
34. In February 2018, the Fund for Development of Non-State Preschool Educational Institutions has been established in Uzbekistan, which is providing soft loans for the acquisition, construction, reconstruction and repair of buildings and structures for private kindergartens. Considering the importance of expanding the network of non-state kindergartens, a Public- Private Partnership (PPP) mechanism has been widely introduced in the sector. On April 5, 2018, a Presidential Decree, which approved the Regulations on PPP in the field of pre-school education, was signed. It provided various models of PPP, as well as a number of benefits, including tax and customs, (e.g. zero taxes for 10 years) and the possibility of obtaining preferential (zero percent) loans for the organization of pre-school educational institutions. 2.3 Sustainable Development Goals and education reform in Uzbekistan 35. The Government of Uzbekistan has committed to the implementation of the 2030 Agenda for Sustainable Development. At the Government level, an interdepartmental Coordination Council, headed by the Deputy Prime Minister of the Republic of Uzbekistan, was established. Also, a corresponding Roadmap for the implementation of national SDGs was adopted. With regard to SDG 4 (“Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all"), in close cooperation with the ministries and departments responsible for the implementation of the SDGs, as well as with UN agencies, a national list of about 200 SDG indicators was established. The list includes indicators recognized as relevant for Uzbekistan and reflecting the country's national priorities. The implementation of the SDGs in Uzbekistan coincides with large-scale reforms within the framework of the national Action Strategy for 2017-2021. This strategy and its five priority areas have been recognized as consistent with the global SDGs. At the national level, the priorities for SDG 4 are improving the quality and coverage of education at all levels, the development of the lifelong education system, ensuring the flexibility of the training system based on the needs of the economy, as well as increasing investments, including through public-private partnerships.
2.4 Recent legal initiatives in education 36. Over the past years, the Government has adopted a number of legislative documents aimed at structural and system reforms in education sector. The main areas include: 37. A new edition of the Law "On Education" which was published in September 202012replacing the outdated laws "On Education" and "On the National Program for Personnel Training" in force since 1997. The main changes brought about by the new law are: • the classification of forms of education has been expanded, including inclusive education for children (persons) with physical, mental, sensory or mental disabilities. • the powers of the Cabinet of Ministers, the State Inspectorate for Supervision of the Quality of Education, the Agency for the Development of Presidential, Creative and
11 Education Sector Plan (ESP) of Uzbekistan 2019-2023: https://www.globalpartnership.org/content/education-sector-plan- 2019-2023-uzbekistan 12 https://www.norma.uz/novoe_v_zakonodatelstve/novyy_zakon_ob_obrazovanii_korotko_o_glavnom
14 Specialized Schools, as well as relevant ministries and local authorities have been detailed. • more attention is paid to the legal status of all participants in the educational process: • issues related to state educational standards and requirements, the introduction of curricula and programs into the educational process have been settled. • the instruments of state regulation and control are regulated. 38. In December 2019, the law "On preschool education" was signed.13 The document defines preschool education as a type of continuous education aimed at teaching and educating children, their intellectual, spiritual, moral, ethical, aesthetic and physical development, as well as preparing children for general secondary education. The law gives definitions to such concepts as: children with special educational needs - children with individual needs, regardless of whether there is a disability or not, who need special attention from families, teachers, specialists, society and the state; inclusive education - process that ensures equal access of children to education and upbringing, considering their special educational needs and individual opportunities. 39. In December 2016, through a Presidential Decree (PP# 2707 on “Measures Aimed at Further Improvement of the Pre-School Education System in 2017-2021”), the Government approved a Programme for Further Improvement of the Preschool Education System from 2017 through 2021, with the goal of reforming the quality of preschool education. This national program aims at (i) creating conditions for a comprehensive intellectual, emotional, aesthetical and physical development of children, based on best international good practices, (ii) improving the quality of preschool education, and preschool children readiness, based on widely adopted international practice, (iii) establishing half day groups in preschools for children aged 5- 6, (iv) improving the curricula and syllabi for pre-service and in-service training of preschool teachers through modern educational technologies and methods, and (v) improving the material and technical infrastructure conditions of preschools institutions, including the construction of new preschools in rural settlements, provision of equipment, furniture, teaching and learning materials and multimedia tools compliant with modern requirements. 40. In 2017, the Government of Uzbekistan announced an ambitious plan to expand access to preschool education with the aim of achieving 100 percent enrolment for children 5 to 6- year-old, by 2021. Starting with the 2021-2022 school year, it will be compulsory for all 6- year-old children to be enrolled in pre-school education. This measure will be implemented in a phased approach, starting in few regions of the country in 2019-2020 school year. Within this context, the Government established the MOPSE in September 2017 to play the lead role in the expansion of Early Childhood Development) ECD in Uzbekistan and govern this core sub-sector of the whole education system.14 41. Within wider measures to invest in inclusive education, the government drafted the Decree “On Measures to Further Improve the Education System for Disabled Children”15 and the Concept for the Development of Inclusive (Integrated) Education in Uzbekistan until 2030, inter alia, with UNICEF’s support. This Decree envisages gradual implementation of inclusive education, approach resulting from UNICEF’s advocacy and support. 42. The Presidential Decree on the Concept for the development of the public education system of the Republic of Uzbekistan until 2030 was adopted in April 2019 (UP-5712). In accordance with this document, in order to implement the Concept, Uzbekistan is introducing new state educational standards and curricula for general secondary education, with particular attention
13 https://www.gazeta.uz/ru/2019/12/18/education/ 14 https://www.globalpartnership.org/content/education-sector-plan-2019-2023-uzbekistan 15 https://regulation.gov.uz/uz/document/17141
15 to the STEAM disciplines and the development of a competency-based approach in general secondary education. 43. In response to the 2017-2021 Action Strategy of Uzbekistan, the Concept for the development of the public education system of the Republic of Uzbekistan was developed. Similar concepts for pre-school and higher education followed. Table 4 in Annex 6 provides an overview of relevant legal documents of the past years. 3. The UNICEF Education programme
44. Within Country Programme (CP) 2016-2020, UNICEF in partnership with the Government of Uzbekistan is working on strengthening the education system. The CP builds upon the previous Education Programme Theory of Change (ToC) 2014 (See Annex 7) and was updated in 2018 to reflect the changing government priorities and UNICEF’s response. The original education programme consisted of seven outputs. After the Strategic Moment of Reflection (SMR)16, the Output/ results areas were mainstreamed and reduced to four to better reflect the UNICEF-regional priorities and also align them more closely with the Strategic Plan (SP) outcomes as shows in table below. Some of the activities included in the initial list of outputs spilled over to 2019, while some activities were reformulated for implementation by new partners such as the SISEQ and MOPSE. The revised Education programme that the Evaluation will organize the assessment of encompasses the following Outcome and Outputs:
Outcome 2: By 2020, girls and boys benefit equitably from improved pre-school education and enhanced quality of basic education including in emergencies
Original outputs Outputs revised after SMR Output 2.1: By 2018, MOPE has revised Output 2.1: By 2020, Government has Early Childhood Education policy strengthened legislative and policy frameworks for improved access to frameworks to improve access to equitable equitable quality pre-school services and inclusive quality pre-school and general secondary education services Output 2.2.: By 2019, MOPE has Output 2.2.: By 2020, Government has increased capacity to deliver and measure improved capacity to provide increased learning in line with a competency-based access to inclusive and quality early learning curriculum services Output 2.3.: By 2020, local education Output 2.3.: By 2020, Government of is authorities and communities are equipped equipped with tools and skills to better with tools and skills to encourage demand formulate and implement programmes for for quality basic education and pre-school inclusive and quality general secondary services education for all children Output 2.4.: By 2020, line ministries are Output 2.4.: By 2020, Government is equipped with tools and skills to promote equipped with tools and skills to promote safe behaviour practices among children safe behaviour practices among children and communities before and during and communities before and during emergencies emergencies
16 Strategic Moment of Reflection aims to serve as space to critically review the current Country programme document. It is a significant milestone to address programme progress related to country programme results (outputs and outcomes), implementation strategies, to assess the extent to which progress has been made towards reaching CP results.
16 Output 2.5.: By 2020 key relevant stakeholders have drafted a revised teaching workforce policy that promotes quality of education in line with the national quality education framework Output 2.6.: By 2020, key relevant stakeholders have devised a unified system of teaching workforce development that supports the national quality education framework Output 2.7: By 2020, MOPE has increased capacities to promote healthy, safe and protective school environments in line with the child-friendly schools (CFS) principles 45. As a result of the wider discussions within the UNICEF on strategic approaches and implementation strategies, a set of eight core roles (implementation strategies) was set forth for UNICEF offices in the wider CEE/CIS region as follows: 1) advocacy; 2) policy dialogue and advice; 3) knowledge generation and child rights monitoring; 4) convening partnerships and leveraging resources for children; 5) capacity development of professionals and organisations; 6) modelling and testing innovations; 7) horizontal cooperation beyond borders; and 8) ensuring proper internal controls and risk management17. UNICEF in Uzbekistan noted the following implementation strategies in its CPAP 2016-2020 to achieve education programme results, i.e.: capacity development; evidence generation, policy dialogue and advocacy; and partnership building. The total available funding for the UNICEF’s Education Programme was 4.474.600 USD over the period of 2016-2020. The Education programme had national coverage, with main focus on central level institutions, and capacity building interventions including both national and subnational level educational institutions. The Programme, hence, works primarily with central level duty bearers – Ministries and other institutions in charge of education reform. These institutions and policy makers are UNICEF’s primary target group. Besides, UNICEF invests in capacity building of wider pool of educational institutions. Final beneficiaries of UNICEF’s interventions are children and young people (particularly girls, Children with Disabilities, those in rural areas and from low-income families).
46. As outlined in the ToR and also as understood from document review, the Education Programme included a variety of programme interventions within the period of 2016-2019, as listed in paragraphs below.
47. Revision of Legal and Policy Frameworks. UNICEF performed the role of the coordinating agency for the Local Education Group (LEG) and Development Partner Group (DPG) for the implementation of the first Education Sector Plan 2013-2017, which UNICEF had helped to develop in 2012. At the expiry of the first ESP, the LEG entrusted UNICEF with the task of developing the second ESP for the next five years. In 2018, UNICEF supported the development of Education Sector Plan 2019-2023.
48. In 2018, UNICEF also provided technical expertise to the Government to prepare the new Law on Pre-school Education as well as to revise the 1997 Law on Education. The new Law
17 UNICEF Regional Office, ‘Discussion Note on Core Roles and Achieving Results for Children in the CEE/CIS Region’, 2015.
17 on Education redirects the focus onto educational quality issues and includes additional articles, focusing on, inter alia, a more rights-based approach; more attention directed at realizing inclusive education; lifelong education; a greater role for the private sector. Likewise, the new Law on Pre-school Education is based on right-based approach and provides focus toward universalization of pre-school education. Law on Pre-school Education has been approved by the Senate on December 14, 2019.18
49. During the period of 2016-2019, UNICEF carried out a number of policy reviews to facilitate systemic reforms in education sector. For example, the ECE Policy Review led to the Government coming up with several policies and programmes for enhancing equitable access to quality pre-school education. The review of Inclusive Education Policies facilitated the government to draft a roadmap for developing inclusive education policies which is under progress, as well as it contributed to the wider UN-SitAn on people and children with disabilities. The Teacher Workforce policy review has contributed to the Government coming up with a more detailed analysis of teacher workforce related issues and as well as with requests to develop teacher professional standards and qualification framework, which is currently ongoing with UNICEF support. UNICEF also prepared two policy notes, one on mobilizing additional resources for expanding pre-school education as well as another on ways to engage private sector through Public-Private Partnership mode in expanding pre-school access.
50. Equitable Access to Quality Pre-school Education. During the period of 2016-2019, UNICEF supported the newly established Ministry of Pre-school Education by strengthening its institutional and technical capacity. This process was preceded by an ECD policy review, which resulted in establishing the MOPSE’s roles and functions. The Early Learning Development Standards (ELDS) have been revised and a new curriculum has been developed using a competency-based approach. Capacity building of key staff in Ministry of Pre-school Education as well as master trainers for imparting the ELDS and curriculum to pre-school teachers through in-service training has been developed with support from UNICEF. To ensure access to equitable and affordable quality pre-school education, UNICEF has supported the development of alternative and innovative forms of pre-school education. As a result, a Conceptual Framework for Alternative Forms of ECE and a workplan for modelling of these forms in most deprived regions was developed and initial activities for the piloting in two regions have started recently (Namangan and Khorezm Regions). Quality assurance framework and mechanism were also developed for pre-school education. Furthermore, UNICEF supported MOPSE in developing a Pre-school EMIS, which is expected to facilitate systematic, robust data collection for monitoring and evidence-based planning. The system was piloted on the Open EMIS platform and is planned to be further extended to the general secondary education. UNICEF jointly with MOPSE also designed a study to help understand the Knowledge, Attitude and Practices (KAP) and to reveal the underlying bottlenecks and barriers for families and communities in accessing ECE services. The study was finalised in 2020. UNICEF also supported the MOPSE in the development of six guidebooks for in-service teacher training.
51. Promoting Inclusive and Quality General Secondary Education. During the period of 2016-2019, UNICEF jointly with MOPE initiated the process of revising the curriculum in general secondary education to go beyond a content-driven approach to adopt a more competency-based learning approach. UNICEF also supported the development of the
18 №ЗРУ-595 16.12.2019
18 national quality education framework for both pre-school and general secondary education and subsequently worked on the development of a quality assurance framework and mechanism to identify and address the gaps in provision of quality education. To support reforming existing national assessment systems and capacity building for participation in international assessments such as PISA, UNICEF jointly with MOPE carried out a study on the profile of children with low learning levels (4th graders). This study is the first effort in the country to do a nation-wide assessment of children’s learning levels at the end of primary grades, using tools and methods adopted from international assessment programmes. Furthermore, UNICEF jointly with MOPE designed a study to understand the KAP of the community regarding school-based management (SBM) and to identify and address the bottlenecks and barriers in the family and community’s participation in school decision making. UNICEF supported the development of competency-based teacher professional standards and qualification frameworks for pre-school and general secondary education teachers.
52. Promoting Safe Behaviour and Disaster Risk Preparedness. During the period of 2016- 2019, UNICEF jointly with MOPE and the Ministry of Emergency Situations has worked on disaster risk preparedness interventions within the education programme. This includes integrating disaster risk preparedness and safe behaviour into the regular school curriculum in a more holistic manner, support development of emergency preparedness plans and strengthening monitoring system for emergency preparedness at school level.
53. Review of programme documents of Unicef points that UNICEF partners with central government institutions to invest institutional and professional capacity development of national educational institutions, teachers and other education professionals across the board of pre-school and primary education system (target groups). UNICEF’s focus is primarily on vulnerable groups of children, in particular children (girls and boys) in remote areas who do not have access to pre-school and primary education, while UNICEF’s upstream policy works affects children across all societal strata.
19 4. Purpose, Objectives and Scope of the Evaluation
4.1 Purpose and Objectives of the Evaluation
54. As stipulated in the TOR, the findings of the evaluation will be used as a basis for discussion, planning and programming between UNICEF and key national and international stakeholders. As such, the evaluation has a formative - forward-looking - perspective to support strategic learning and decision-making; and a summative - backward-looking - perspective, to support enhanced accountability for development effectiveness and learning from experience. The main objectives of this evaluation were to: • assess the relevance, effectiveness, efficiency, sustainability, long-term changes, and synergies; • provide recommendations to guide policy level decision-making by relevant stakeholders on education sector reform and its sub-sectors; • identify and document successes, challenges and lessons learnt; • provide recommendation to better incorporate gender equality and other equity issues in the programme.
The main objectives of the evaluation were confirmed during the inception phase and remained unchanged.
4.2 Intended Uses of Evaluation and key stakeholders- users of the evaluation
55. As stipulated in the TOR, this summative evaluation will come at the end of the Country Programme of Cooperation between the Government of Uzbekistan and UNICEF 2016-2020. In this context, the evaluation offers the opportunity to critically assess UNICEF education programme’s contribution to education sector reforms and provision of more equitable and quality education services in the country. The findings of the evaluation will be used as a basis for discussions, planning and programming between UNICEF and key national and international stakeholders. 56. As noted in the Evaluation ToR, the evaluation will engage with primary stakeholders in charge of implementation of the state policy in education sector and involved in UNICEF’s education programme interventions, with detailed overview presented in stakeholder analysis section below. 57. Users of the evaluation include: • UNICEF, which will take stock of lesson learnt and best practices by the end of CP 2016-2020 to inform its future programing; • SISEQ, Ministry of Public Education, Ministry of Preschool Education and other key national and international stakeholders, who will use the evaluation to further inform education sector reform agenda and policy making for improving inclusiveness and quality of education services.
20 4.3 Evaluation scope
58. As stipulated in the ToR and confirmed in the inception phase, the evaluation focused on the UNICEF education programme and covered most of the programme implementation period, from January 2016 to December 20120 (three UNICEF workplans in education with MOPE, MOPSE and SISEQ: 2016-2017, 2018 and partly 2019-2020). The geographical scope was nationwide. As per the corporate guidance for programme evaluations, the evaluation will assess the totality of the UNICEF Uzbekistan Education portfolio including cross-cutting issues and inter-sectoral support aimed at addressing system level bottlenecks (enabling environment, supply and quality of services, demand for services and social norms). The evaluation aims to strategically inform the next CP operationalization when it comes to the education sector engagement of UNICEF. The organizational scope of the evaluation goes beyond the UNICEF Education programme in-country and includes provision of technical support and oversight from UNICEF’s regional office in Geneva and, when relevant, UNICEF’s headquarters in New York. The evaluation covers the entire education portfolio including cross-cutting issues related to equity – in particular as these contribute to inclusion of either most vulnerable groups of children (including children with disabilities, boys and girls, and address disparities (geographic including urban/rural).
4.4 Stakeholders analysis 59. The Evaluation Team grouped the stakeholders into several categories. The Annex 4 includes a list of all consulted stakeholders, the different categories, and the level of priority for consultation. Besides, Annex 11 provides a full overview of stakeholders of the UNICEF’s education programme. This composition of stakeholders represented an appropriate mix of key informants and representation and allowed for the triangulation and validation of information. The selection of the groups for consultation was based on criteria, such as the roles, engagement and extent of information and familiarity of persons and groups with UNICEF’s work and education sector reform in Uzbekistan.
60. The list of primary stakeholders in charge of implementation of the state policy in education sector and involved in education programme interventions include: State Inspectorate for Supervision of Quality in Education under the Cabinet of Ministers, Ministry of Public Education, Ministry of Pre-school Education, Ministry of Secondary Specialized and Higher Education.
61. Other key relevant national and international stakeholders include members of the Local Education Group (LEG) and Development Partners Group (DGP) such as Cabinet of Ministers, Ministry of Finance, Ministry of Economy, Ministry of Employment and Labour Relations, Ministry of Justice, Ministry of Health, State Committee on Statistics, Youth Union, Chamber of Commerce and Industry, “Makhalla” Foundation, Centre for Economic Research, UN RC, UNESCO, GIZ, EU Delegation, ADB, World Bank, JICA, British Council and USAID.
62. As noted in section 3, final beneficiaries of the UNICEF’s education programme are vulnerable groups of children, in particular children (girls and boys) in remote areas who do not have access to pre-school and general education such as children from poor households, children with disabilities, children in rural areas and from certain deprived regions19. UNICEF’s
19 See for instance UNICEF (2020); Situational Analysis of Children in Uzbekistan 2019/2020
21 upstream policy focus on design and implementation of policies and regulation comprehended that such efforts affect children across all societal strata.
5. Methodology
5.1 Evaluation design and approach 63. The evaluation was based on a non-experimental design, using a variety of qualitative data collection tools, including stakeholder mapping and analysis; context mapping; in-depth and structured document review of the UNICEF CP and education programme design, implementation approaches and results; policy documents and legislative frameworks; contribution analysis to determine factors that promoted or impeded the progress against intended results; systems analysis of management, monitoring, quality control and assurance strategies; mapping of risks analyses and mitigation measures; analysis of sustainability strategies and systemic barriers to sustainability; in-depth interviews with different partners, independent experts, and concerned UNICEF staff, as elaborated in the Stakeholder analysis in Section 1.4 above.
64. To assess UNICEF’s contribution to overall reform of education system towards more equitable access to education, the Evaluation Team conducted a Contribution Analysis through mapping and assessing pathways from interventions to results as elaborated in the programme ToC to help identify how contributions have evolved over time and to what degree the observed changes have been the result of UNICEF-supported interventions or externalities.20
65. Equity, gender equality and a human rights-based approach: The evaluation methodology is founded on a human rights-based approach (HRBA) – i.e. an assessment of achieved results towards realisation of the rights of children and women, as laid down in the Convention on the Rights of the Child (CRC) and in the Convention on Elimination of all Forms of Discrimination Against Women (CEDAW).21 The HRBA for this evaluation was guided by five core principles: normativity, participation, non-discrimination, accountability and transparency, as well as by the Common Understanding on HRBA to Development Cooperation and Programming approved by the United Nations Development Group in 2003. Evaluation of gender equality and mainstreaming was integrated into the evaluation approach and includes equity, gender-sensitive, child-sensitive and HRBA components, in line with Uzbekistan’s international commitments, most notably the CRC and the CEDAW, as well as the UNICEF Gender Policies and plans. The evaluation assessed the programme also through a gender lens, in particular with regards to the programme design, achievement of results and the contribution to the main outcomes as set out in the programme TOC. Overall, the United Nations Evaluation Group (UNEG) Code of Conduct for Evaluation in the UN System (March 2008)22 was strictly respected (See the Section on Ethical Considerations).
66. Participation: The evaluation was conducted in a participatory manner and participation of key stakeholders was ensured in all phases of the evaluation, including the planning, inception,
20 Contribution Analysis (Mayne, J, 2011) is designed to help managers, researchers, and policymakers arrive at conclusions about the contribution their program has made (or is currently making) to particular outcomes. 21 https://www.unicef.org/policyanalysis/rights/ 22 United Nations Evaluation Group (2016). Norms and Standards for Evaluation. New York: UNEG.
22 fact-finding, reporting as well as the management response phases. These elements are intertwined throughout the evaluation design and summarized in the evaluation workplan. Prior to the evaluation, UNICEF had established an evaluation reference group (ERG) comprised of representatives of SISEQ, Ministry of Public Education, Ministry of Pre-school Education, UN RCO, World Bank and UNESCO and UNICEF Programme Staff that acted as sounding board for the evaluation and be consulted at critical times during the evaluation process.
67. Appreciative inquiry: The evaluation adopted an appreciative inquiry perspective, meaning that evaluators focused on strengths and on weaknesses, on existing good practices and learning processes to envision UNICEF values and transformational opportunities for the education system in Uzbekistan. During the data collection phase, consultation with key stakeholders including government counterparts, educational professionals, other UN Agencies’ and donor representatives was be carried out to ensure design was most relevant to utilization interests. 5.2 Evaluation criteria and questions
68. The evaluation assessed the programme against the following criteria23: relevance, effectiveness, efficiency, sustainability, impact24 and coherence25. The evaluation against each of these criteria was guided by a set of questions: 1. Relevance 1.1. To what extent the programme was aligned with global, regional and national priorities in education sector, including Sustainable Development Goals (SDGs)? 1.2. To what extent was the programme aligned with the UNICEF global and regional education priorities? If the programme differed from the UNICEF global and regional priorities, what were the reasons for it? 1.3. Did the programme stay relevant during political, social and economic transformation of the country? How was the programme revised to stay relevant? 1.4. Were the programme implementation strategies relevant to the context? If not, why?
2. Effectiveness 2.1. To what extent were the planned programme results achieved? What were the factors that promoted or hindered the programme implementation? 2.2. To what extent did the programme effectively mainstream gender? 2.3. To what extent the needs of vulnerable children were considered in the programme planning and implementation? 2.4. Were there any unintended negative or positive outcomes and, if so, were they appropriately managed? 2.5. Which implementation strategies worked or not worked? Why?
3. Efficiency 3.1. Were the available financial, material and human resources adequate to meet the programme objectives? What were the gaps in the originally proposed funding and later actual funding?
23 These criteria have been established by the OECD Development Assistance Committee (DAC). 24 In the ToR, the relevant evaluation criterion was defined as ‘long term change’ and per request of UNICEF, this was changed to assessment of “impact’. 25 In the ToR, the relevant evaluation criterion was defined as ‘synergies’. Per request of UNICEF, this term was changed to assessment of “coherence”.
23 3.2. To what extent is the programme cost-efficient? Are there any potential efficiency gains to make?
4. Sustainability 4.1. Did the programme lead and/or contribute to improvement in the allocation and use of resources for the programme priorities? Were these improvements sustainable? 4.2. Did the programme interventions lead and/or contribute to improvements in legislative and policy frameworks in educator sector? 4.3. What are the opportunities for and risks to the sustainability of the programme in the short and long term?
5. Impact26 5.1. To what extent can the programme be considered to have made a contribution to reforms of the education system in Uzbekistan?
6. Coherence27 6.1. To what extent did UNICEF coordinate with development partners and other UN agencies to avoid overlaps, leverage contributions and catalyse joint work? 6.2. To what extent the education programme contributed to cross-sectoral synergies with other sectoral programmes within UNICEF?
5.3 Data collection methods
69. The evaluation used an evidence-based approach, combining a variety of qualitative data collection tools, which helped to ensure that the limitations of one type of data were balanced by the strengths of another. The main quantitative data sources were secondary data and existing documentation, including UNICEF’s annual reports and data sources (cf. the Annex 3 for a list of all consulted documents). The Evaluation Team generated qualitative data through key informant interviews and group discussions. Triangulation was be used to strengthen the reliability and robustness of all findings.
70. Secondary data: the evaluation utilized methods such as: • Stakeholder mapping and context analysis: to frame UNICEF educational programme within the operational context and assess relevance and opportunities for long-term change. • In-depth and structured documentary review: through the evaluation process, further documentation was provided, which made the document review a continuous process.
71. Primary data: The primary data collection process served the purpose of testing preliminary assumptions, hypotheses and findings resulting from the document review and filling in the data gaps encountered during the document review. Following desk review, the Evaluation Team combined different data collection approaches to ensure that a variety of views are captured. Due to the COVID-19 pandemic all scheduled meeting had to be held remotely, using zoom as a main communication tool. UNICEF provided the necessary technical support
26 As noted above, the ToR defines this evaluation criterion as ‘long term change’ and per request of UNICEF, this was changed to assessment of “impact’. 28 As noted above, the ToR defines this evaluation criterion as ‘synergies’. Per request of UNICEF, this term was changed to assessment of “coherence”.
24 to arrange meetings and help reach out to wider group of informants. During the primary data collection phase, Evaluation Team performed:
• Key informant interviews with use of the Interview and Group Discussions Guide (See Annex 5) with stakeholders in Tashkent and the regions including UNICEF Education programme team, government stakeholders, NGOs, UN Agencies, and other development partners. At the national (central) level, top-level interviews were conducted with senior representatives of government institutions and other public bodies and development partners. For these interviews, the evaluation team focused particularly on questions related to policy content and its relevance to achieving ownership and sustainability, particularly with regards to reform processes, implementation issues, challenges and plans etc. Table below presents the stakeholder groups interviewed for the purpose of the evaluation.
Stakeholder group Number of representatives interviewed UNICEF CO staff members 13 Regional Office representatives 11 National stakeholders (government, educational institutions) 25 Development partners 8
• Three (3) Group discussions were organized whenever the stakeholder was represented by several individuals or where a certain dynamic in the discussion was desirable. In particular, teachers and education administrators were consulted on their view of the results of the education programme. They were considered as informants who can provide evaluators with a ‘reality check’ on whether and how policy change supported by the UNICEF has affected practice. In total, three group discussions were conducted during the primary data collection phase.
5.4 Data Analysis Methods
72. The evaluation team applied specific methods for the analysis of quantitative and qualitative data collected through the document review and stakeholder interviews: a. Thematic narrative analysis through structured desk review to enable identification of key themes covered by the Education programme that are of relevance to the indicators outlined in the evaluation matrix (See Annex 1). b. Descriptive quantitative analysis with cross tabulation for evaluation indicators, but also criteria of interest (such as gender or geographic location where relevant. c. Qualitative iterative data analysis allowed to connect, and structure key thought units related to each evaluation question deriving from stakeholder interviews into clusters and identifying the key themes within each cluster. These formed emergent themes from each category for further analysis. Triangulation of data was conducted determine if inputs were coming from multiple sources and stakeholder levels and multiple stakeholder categories. Observations or comments that only come from a single source or a single category of stakeholder were given less weight during the building of the analysis. d. The Programme Theory of change formed the basis for the Contribution Analysis which assisted in assessing the degree to which programme actions have contributed
25 to the perceived outcomes. Initial document review showed that the programme developed a theory of change for the intervention with key assumptions. The activities in the programme were implemented to a large extent in line with the theory of change. Assessment of performance and achieved results was conducted based on these, accompanied with analysis of external factors influencing results. This type of analysis assisted in interlinking the findings of the above outlined different types of analysis and helped lead to definition of conclusions regarding effectiveness, relevance, and sustainability.
73. The evaluation utilized two types of triangulation that served to highlight any inconsistencies between document analysis and the feedback from key informants, i.e. how external parties perceive the results of the measures undertaken within UNICEF’s Education programme. These were: i) Methods triangulation, both qualitative and quantitative data was used to elucidate complementary aspects of the same subject; and ii) Data sources triangulation, which involves examining the consistency of different data sources within the same methods.
5.5 Ethical considerations 74. This evaluation underwent the Ethical Committee Review, receiving ethical clearance during the inception phase. Throughout the process, the Evaluation Team (ET)were obliged to follow UNEG Norms and Standards and UNICEF’s Procedure for Ethical Standards in Research, Evaluation, Data Collection and Analysis, and in particular with respect to dignity and diversity, right to self-determination, fair representation, compliance with codes for vulnerable groups, independence of judgement, impartiality, honesty and integrity, accountability, respect, the protection of the rights and welfare of human subjects and communities, confidentiality, avoidance of risks or harm to, participants or other types of burdens on those participating in the evaluation. Besides, principles of accuracy, completeness and reliability of report, transparency were strictly followed as obliged by UNEG and UNICEF’s Norms and standards. Further, the Evaluation team was sensitive to religious beliefs and practices, gender roles, disability, ethnicity, manners, culture and local customs. 75. The ET conducted Stakeholder mapping and analysis to identify and prioritize institutions/organisations to be engaged in the evaluation process. The stakeholder analysis integrated prioritization of those institutions/organisations to be interviewed (critical/desirable/less priority). This list was further discussed with UNICEF to reach agreement on those institutions to be interviewed. Based on the final list of critical and desirable interlocutors, UNICEF has been in charge of identifying representatives of these institutions/organisations and arranging meetings. UNICEF followed their internal procedures in arranging meetings.
5.5.1. Protocols for Safety, Confidentiality, and Data Protection and mitigation measures
76. In line with informed consent principles and confidentiality, interviewees were informed at the start of the interview regarding the purpose of the evaluation, assurances of voluntary participation, and confidentiality of all responses. The Evaluation Team applied the principle of “do no harm” – both for the evaluation team and evaluation participants together with the standard ethical requirements of any evaluation data collection process.
26 77. Confidentiality: In agreement with UNICEF, Key Informant Interviews (KIIs) and Group Discussions (GDs) were arranged by the UNICEF Team. This assisted in the protection of confidentiality of participants as most of their contact information was not shared with the evaluation team. The Evaluation Team did not use audio or video recordings of KIIs and GDs. All notes taken by Evaluation Team have been stored in Evaluation Team computers with password protection. 78. No compensation for participation in the evaluation process was provided. The Evaluation Team ensured that informed consent protocols were verbal, to avoid signing of any type of printed forms that might put participants at risk. For all interviews and group discussions, personal data, including contact details were stripped from the data before it was shared to ensure further confidentiality. All interview notes from the evaluation team are kept electronically on password encrypted computers. Any potential personal identifiers have been removed from the data prior to analysis. Data analysis was carried out only by the evaluation team members to ensure confidentiality. Data compiled in reporting was aggregated so that individual responses cannot be traced to specific locations or individuals. Both quantitative and qualitative information will be maintained on evaluation team computers only until the finalization of the report, at which time it will be deleted to further protect individuals from possible identification. 79. Children Involvement: The evaluation did not obtain responses from children in sampled communities, due to the fact that UNICEF worked primarily at the system level. 80. Gender Norms: The Evaluation Team ensured that gender roles were respected and provided space for women to share their views in safe and enabling environment. The following table profiles in more detail additional mitigation measures for the ethical issues.
TABLE 1. ETHICAL ISSUES AND MITIGATION MEASURES Ethical Issue Mitigation Measures • Verbal informed consent protocol discussed with participants with voluntary participation emphasis. No signatures or documentation to be collected to minimize risk to participants. • Names of KIs and participants of GDs were collected by the Evaluation Team and stored confidentially Security • The decision of all participants who refuse to take part in the Considerations evaluation process were respected. Interviewers respected the for decision of the respondents to refrain from an interview or to stop Participants the interview at any point if they do not feel comfortable. • The Evaluation Team considered language barriers and aspire to conduct interviews without translation in the language of the participants. Evaluation Team members, proficient in Russian, helped with translation in cases of need to avoid external translation service. Interviewing • Interviews and group discussions with children and adolescents underage were not organized. children Female • Evaluation Team ensured that both women and men had the Participation opportunity to voice their opinions and feedback Considerations Data Protection • All personal identifiers are stripped from quantitative data Considerations
27 • Qualitative interview notes are kept only on evaluation team member computers under password protection • Data analysis only carried out by evaluation team members • Reported data aggregated to avoid place or personal identifiers • After finalization of the evaluation report, all quantitative and qualitative data will be erased to protect respondents
5.6 Limitations and mitigation measures 81. The evaluation was significantly affected by the COVID-19 pandemic, which resulted in a lack of opportunity for the evaluation team to travel to Uzbekistan and conduct a standard field mission, which was originally planned for March 2020. In lieu of field mission, the evaluation team conducted the entire primary data collection remotely through virtual meetings, using communication channels such as zoom, phone, etc. After extensive consultations with the UNICEF team, it was agreed that the remote primary data collection phase would happen in September 2020. A longer timeframe for primary data collection was set forth to allow wider participation taking into account remote data collection approach. Eventually, the evaluation team was able to keep to the intended plan and methodology, which ensured confidence in the robustness and quality of data collected to inform the assessment. One remaining limitation that the evaluation encountered related to the remote approach is a lack of opportunity to observe and gain insights from observation and site visits, as it would have done in other evaluations. 82. Another limitation was noted in terms of the use of the Programme ToC as a guide for contribution analysis. The original ToC was changed during the SMR exercise in 2018 – marking robust adjustments of outputs (from initial seven (7) to four (4) outputs for the programme. This evaluation took into account both the original set out outputs and revised TOC into account, to ensure the full scope of programme activities was assessed. Besides, very limited data or internal reflection on UNICEF’s contributions to desired outcome in education sector were available, which limited the ability to draw evidence-based findings to track progress on outcomes and outputs. To mitigate these limitations, the evaluation team drew information from other secondary sources and interviews to establish evidence to be used within contribution analysis.
28 6. Evaluation findings
83. This section presents a set of corroborated findings and responses to the evaluation questions agreed with UNICEF Education Programme in Uzbekistan. The presentation of findings is structured according to the evaluation matrix and evaluation questions. Some evaluation questions are combined into a single section. When this occurs, a footnote clarifies which questions are being combined.
6.1 Relevance
6.1.1. Alignment of the programme with global, regional and national priorities in education sector, including SDGs (EQ 1, EQ 2, EQ 3)28
84. There is evidence of strong rooting of UNICEF’s Education programme in national priorities and unmet needs of children in Uzbekistan. A review of UNICEF’s Country programme sections on education shows that the selection of UNICEF’sareas for assistance was based on national priorities, set forth by the government before the political transformation. Further analysis of priorities of the new government shows that respective focus themes of UNICEF, i.e. early childhood and pre-school education, and basic inclusive education have been maintained priorities as set out in successive ESPs. Besides, UNICEF’s education programme was informed by the findings, conclusions and recommendations of the previous Country Programme 2010- 2015 evaluation. Importantly, the “It is remarkable that UNICEF always feels our needs. So, evaluation of the previous CP while developing each [annual] plan, we conduct careful recommended UNICEF tofocus at study first so that we can end up with the solution for system level for its education problems.” component. Stakeholder interviews and Key informant representing central level government group discussions with national government representatives and “Among all our development partners, UNICEF is the first development partners highlighted that one to address when we need assistance in education- the alignment of UNICEF has been related issues.” strong and boosted as a result of UNICEF’s programming during the Key informant representing central level government
reference period of this evaluation, which they perceived to be demand-driven. There has been an agreement among key informants across the spectrum of stakeholder groups that UNICEF’s strategic positioning as a partner of the government made it possible for UNICEF to align with and in many areas (e.g. PSE) to drive agenda setting and national priorities (see also section 6.1.2 below on implementation strategies). 85. Sweeping reforms resulting from political transformation that Uzbekistan faced from 2017 have included important advancements of the education sector reform agenda. The reforms
28 Full EQs are as follows: EQ 1: To what extent was the programme aligned with global, regional and national priorities in the education sector, including SDGs?; EQ 2: To what extent was the programme aligned with the UNICEF global and regional education priorities? If the programme differed from the UNICEF global and regional priorities, what were the reasons for it? EQ 3: Did the programme stay relevant during political, social and economic transformation of the country?How was the programme revised to stay relevant?
29 have brought new momentum for reform in particular when it comes to pre-school education with establishment of the MoPSE and reinstating the roles of the SISEQ. UNICEF was recognized as partner of choice in support to institutional capacity development of the MoPSE and SISEQ and policy adviser across education sector themes, as confirmed by interviewed national stakeholders and development partners, and corroborated by group discussions. In particular praised by national stakeholders has been UNICEF’s swiftness and adaptability to changing policy environment and new arising needs within the sector. One of the reasons for this was the ability of UNICEF to adapt its sector strategies and plans to the new policy environment thanks to its broad mandate and strategic focus laid out in the Country Programme Document (CPD) 2016-2020. However, such approach also brought certain efficiency challenges as discussed in section 6.2 below. 86. Enhancing access to and quality of education is one of the avenues of realization of the rights of every child, especially the most disadvantaged. These priorities were presented as key regional and global priorities as set out in UNICEF’s Strategic Plan 2014-2017 as well as Strategic Plan 2018-2021. Review of programming documents shows that selected intervention areas of the education programme in Uzbekistan29contribute to achievement of the Strategic plan goals, in particular addressing multiple and complex barriers to access to education for girls and boys from early childhood to adolescence and strengthening education systems and investment in quality of teaching and learning30. Comparative review of Uzbekistan’s Education programme of UNICEF and the successive UNICEF Strategic Plans did not find evidence of any noticeable departure from Strategic directions set out in this global document.31 87. In particular relevant from the perspective of the goals set out in the Strategic plans are interventions responding to the unmet needs of children, particularly when it comes to access to (quality) education. Uzbekistan has invested strongly in its education sector, yet significant barriers to universal access to pre-school education continue to persist in the country, with particularly visible gender and vulnerability divide. The increasing awareness of the relevance of gender and vulnerability issues across stakeholders is an encouraging sign, including the awareness of the need to involve more women in DRR as expressed by stakeholders. The Education programme has been able to use its strategic positioning to advocate for and offer policy advisory on education rights and standards, with particularly prominent role in PSE. 88. While advocacy and push for equal access to education by boys and girls was visible, significantly weaker focus of UNICEF was noted when it comes to vulnerable groups (e.g. children with disability (CwD), minorities, children from poor households, children in rural areas and from certain deprived regions), despite the needs of children from these groups (see detailed assessment of consideration of gender and equity in planning and implementation in Section 6.2.5).In particular, UNICEF studies (e.g. KAP studies, UNICEF’s Situational analysis on children with disability or the most recent UNICEF’s Situational Analysis of children in Uzbekistan) provide evidence and raise issues of vulnerable groups. Specifically, the UNICEF’s Situational Analysis of children in Uzbekistan raises issues of the vulnerabilities of girls in terms of access to education, noting that “fewer girls than boys enroll to early childhood education, so they are less prepared than boys for general secondary education. There are more girls than boys among out-of-school children, in particular out-of-school children (who are often girls) often have disabilities”32. For this
29 i.e. early childhood and pre-school education, basic inclusive education 30 See detailed presentation of Theory of Change underpinning UNICEF’s Strategic Plan 2017-2021 at https://www.unicef.org/about/execboard/files/Strategic_Plan_Theory_of_Change_Draft_Paper_June_22_v4_combined.pdf 31 In response to sub-question of the EQ 2 32 UNICEF (2020); Situational Analysis of Children in Uzbekistan 2019/2020, p. 95
30 reason, specific focus is called for when it comes gender and equity issues in education. UNICEF’s upstream policy work was perceived to bring contributions to the children across all population groups and gender through strengthening of policies and systems. Conceptually, this is a right approach. However, taking into account the immediate needs of girls in particular but also the most vulnerable groups in the context of the education system which still faces incremental change on the ground (in schools and wider educational environment) resulting from system level changes, stronger focus on meeting the needs of such groups would have brought more immediate returns in terms of access to education. 89. The education programme upstream interventions align with the aspirations laid out in the SDG 4 on education, which aims to “ensure inclusive and equitable quality education and promote lifelong learning opportunities for all.33”. Document review, stakeholder interviews and feedback received through group discussions confirmed that the programme was developed in conjunction with the 2016-2020 United Nations Development Assistance Framework (UNDAF) and contributes directly to the achievement of UNDAF outcome5 relating to education, i.e. “Continuous quality education as well as lifelong learning for all improved by 2020”. 90. Review of the most recent interventions of UNICEF Education programme in response to the COVID-19 pandemic shows high flexibility, openness and responsiveness to the arising and urgent needs of the education sector to respond to the changed reality of education approaches. The most praised initiatives of UNICEF was support to the government to introduce DL using television lessons. Within this support, more than 11000 TV lessons have been prepared and broadcasted since March 2020. In addition, guidelines on designing blended learning programmes and customized “catch-up” programmes have been developed and tested for implementation in the 2020-2021 academic year. Besides, UNICEF worked closely with the government and wider educational professionals’ pool to develop an education sector response plan, risk communication and psycho-social support; along with developing guidelines and strategies for safe pre-school/school re-opening; and analyzing data from rapid assessments and school/pre-school WASH to inform the strategies for safe school reopening. Agility of UNICEF to respond to the needs of the education sector within the COVID-19 pandemic related restrictions is also an indicator of value added of UNICEF as a partner but also of its demand driven approach.
6.1.2. Relevance of the programme implementation strategies (EQ 4)34
91. Document review, feedback received through group discussions and stakeholder interview note the adaptability of the programme to changing situation in the education sector in Uzbekistan, particularly following the political transformation. In line with the changing context in the field, UNICEF updated its 2014 Education programme ToC in 2017, building a new theoretical framework underpinning the programme on seven outputs (See ToC in Annex 7). Review of the TOC shows that the ToC that was updated to reflect and integrate RBM principles. The programme was further refined as a result of the Strategic Moment of Reflection (SMR), by mainstreaming Output/ results areas bringing the framework to four outputs to better reflect the UNICEF-regional priorities and also align them more closely with the Strategic Plan (SP) outcomes. Document review, feedback received through group discussions and stakeholder interviews noted that such revisions also assisted reflecting the institutional context whereby new institutions were established, i.e. SISEQ and MOPSE.
33 https://sdg4education2030.org/the-goal 34 Full EQ is: Were the programme implementation strategies relevant to the context? If not, why?
31 92. With UNICEF being the Government’s key partner in education reform, it is obvious that UNICEF is frequently approached with requests for assistance in areas which are not directly or only remotely covered by the programme and work plans. It seems that UNICEF has handled the numerous additional requests for assistance in a balanced manner, particular considering the limitations it faces in terms of human and financial resources. Assessing the character of similar requests and clearly identifying their purpose remains critical for the effective planning of resources in this regard. 93. UNICEF’s Regional Office for Europe and Central Asia for established a set of eight (8) core roles35 that Country Offices/programmes should undertake, translating them into programme implementation strategies. The most prominent implementation strategies that the Education programme undertook relate to its predominantly upstream engagement (system strengthening through policy advice, knowledge generation, advocacy and UNICEF convener role), along with some elements of downstream engagement (e.g. capacity development and modelling innovations). Assessment of the programme showed that the implementation strategies were relevant for the overall reform process and its pace in Uzbekistan. In particular appreciated by the national stakeholders were the UNICEF’s policy advisory and convener role which were seen as particularly crucial in development of new cycle of the Education Sector Plan2018-2021 and legislative documents prepared within this cycle of the country programme. National stakeholders and development partners also noted that studies and analytical papers presented by UNICEF served as relevant evidence base assisting planning and selection of policy alternatives. UNICEF’s strategic positioning was also highlighted to result from its strong convener role. Stakeholders emphasized UNICEF’s ability to serve as a bridge between the government ministries and other development partners, bringing together various stakeholders to discuss and consider reform priorities and support streams. Elaborated assessment of effectiveness of UNICEF’s implementation strategies is presented in Section 6.2 below. 6.2 Effectiveness
94. The overall CPD and in particular the education programme have been continually adapted over the course of the CPD cycle to reflect the changing environment, particularly following the political transformation since 2017. In particular, UNICEF conducted a Strategic Moment of Reflection in mid-2018, whereby education programme was reviewed in detail leading to revision of its outputs from seven (7) outputs to four (4). The changes in the output scope and targeting happened in line with the Government’s Strategy of Actions on the five priority areas for the development of the Republic of Uzbekistan in 2017–2021, promoting policy advocacy and institutional strengthening at national level as primary strategies for UNICEF’s engagement with the national stakeholders. The next sections profile the key achievements and contributions of the UNICEF’s educational programme in response to the EQ 5: To what extent were the planned programme objectives and results achieved? What were the factors that promoted or hindered the programme
35 UNICEF Regional Office, ‘Discussion Note on Core Roles and Achieving Results for Children in the CEE/CIS Region’, 2015. Established eight core roles as follows: 1) advocacy; 2) policy dialogue and advice; 3) knowledge generation and child rights monitoring; 4) convening partnerships and leveraging resources for children; 5) capacity development of professionals and organisations; 6) modelling and testing innovations; 7) horizontal cooperation beyond borders; and 8) ensuring proper internal controls and risk management
32 implementation?In order to ensure that full analysis of achievement of results is presented, this section presents analysis per four output areas36of the programme for easier reference.
6.2.1. UNICEF’s contribution to strengthening of evidence-based legislative and policy frameworks to improve access to equitable and inclusive quality pre-school and general secondary education services (Original Output 2.1/revised Output 2.1 (also relates to original outputs: 2.5, 2.6)37
95. National reviews and studies of the education sector preceding the CPD development in 2015 noted significant gaps related to accessibility and quality of education, and in particular early childhood and pre-school education.38 UNICEF’s response to the needs within the sector were rooted in the recognition of these main needs and directed towards applying UNICEF core roles in support of the government’s reform agenda. Evaluation found evidence of ambitious and comprehensive programmatic support to strengthen the education system, to provide increased access and quality education to all children. Support was rather comprehensive and integrated a number of core UNICEF roles39 to ensure that the results were stable and addressing the main needs of children. Interventions within the education sector have been implemented within a conducive operating space, which was characterized by established partnerships between UNICEF and the government with strong positioning of UNICEF as the Government’s partner of choice in this sector. 96. Document review and “Reliable data is key to evidence-based decision making. In the stakeholder interviews with national country’s educational context, access to relevant data for stakeholders at all levels of the education system remains a government partners showed that critical issue. UNICEF has made an important contribution to UNICEF’s support in particular to improving processes related to the collection, analysis and legislative and policy development was interpretation of education-related data.” directed towards ensuring that the Key informant (government representative) evidence base was in place to inform policy/legislative planning processes of the government. Taking into account the stable interaction and advisory to the government, in particular following the political transformation since beginning of 2017, UNICEF planned and produced a number of studies to inform policy/legislative processes within pre-school and general secondary education system. Within these efforts, two critical studies/reviews were prepared as input for decision making processes, as emphasized by stakeholder interviews. National stakeholders and development partners praised in particular the National Learning Achievement study, which brought important inputs to understand the state of education and areas where the government
36 For reference, the four outputs are: Output 2.1: By 2020, Government has strengthened legislative and policy frameworks to improve access to equitable and inclusive quality pre-school and general secondary education services Output 2.2.: By 2020, Government has improved capacity to provide increased access to inclusive and quality early learning services Output 2.3.: By 2020, Government of is equipped with tools and skills to better formulate and implement programmes for inclusive and quality general secondary education for all children Output 2.4.: By 2020, Government is equipped with tools and skills to promote safe behaviour practices among children and communities before and during emergencies 37 In response to EQ5 38 See Context analysis section of this report or National review of the Republic of Uzbekistan on achievements of the Education for All goals – 2015 on https://unesdoc.unesco.org/ark:/48223/pf0000230329 39 UNICEF’s core roles, as defined in the Discussion Note on Core Roles and Achieving Results for Children in the CEE/CIS Region include: Knowledge generation and child rights monitoring; Convening partnerships and leveraging resources for children; Capacity development of professionals and organizations; Modelling and testing innovations (See: UNICEF Regional Office, ‘Discussion Note on Core Roles and Achieving Results for Children in the CEE/CIS Region’, 2015.
33 should invest to ensure that education provides basis for future development of its population. This study stirred discussions on how to reform education in order to increase learning potential of students leading to reviews of Laws on Education and pre-school education. According to UNICEF’s records and interviews with UNICEF Education team, the National Learning Achievement Study was classified within ten finalists in the “Best of UNICEF Research & Evaluation” of 2019 by the UNICEF-Innocenti office. 97. Another important investment by UNICEF was the Education Sector Analysis (ESA) (in 2018) to inform preparation of the new ESP cycle. Initially, in coordination with the government, the ESA was supposed to be conducted by the World Bank, while UNICEF was supposed to use the findings of the ESA to assist the government in preparation of the ESP. However, due to delays in production of the ESA, UNICEF took over this responsibility and finalized the ESA as input for ESP. This measure was taken to ensure that the government could respond timely to the requirements of the GPE. The UNICEF team also led the drafting of the new ESP document, which was adopted at the beginning of 2019. UNICEF’s role in assisting the government in preparation of the ESP was crucial as noted by the UNICEF team and corroborated by national stakeholders, as it helped inform the planning processes with availability of current data on education system. The successful completion of ESP 2019-2020 created an opportunity for the MOPSE to mobilize 10 million USD from the GPE multiplier grants along with the World Bank project worth 64 million USD. This is an example of UNICEF’s direct leveraging of resources through appropriate intervention.Stakeholders noted that these efforts were important as they filled in the gap in capacity of relevant ministries to prepare such comprehensive document. On the other hand, some concerns with stretching UNICEF resources to fulfil this task were noted by UNICEF and government staff. The evaluation found that the investment in the Learning Achievement study and the Sector Analysis helped catalyse two important results: 1) creation of legislative and policy base to lead the education reform; and 2) positioning of UNICEF as the trusted and able partner and adviser of the government to implement such reform. 98. Within the CPD cycle, UNICEF also prepared the Review of National Curriculum and the National Textbook Policy Review as well as a range of other studies (e.g. Joint UN SitAn on People/ Children with Disabilities; KAP study on parents’ demand for ECE (finalized); a KAP study on school quality and School-based Management (draft); a KAP study on Disaster Risk Reduction and Safe Behaviour (draft)), as well as Early Childhood Education Policy Review, Inclusive Education Policy Review and Teacher Policy review. All these studies and reports serve as evidence base on a range of issues to be tackled within the education sector informing drafting important reformist sector plans. 99. The evidence generation efforts were quite successfully combined with UNICEF’s policy advisory and advocacy roles as evidenced through review of documentation, feedback received through group discussions and stakeholder interviews. UNICEF’s interactions with the government to inform about international best practices in education laws (e.g. analytical summary of laws from around 30 countries); convening consultations /discussions;40 and provision of regular inputs and comments to SISEQ/ MOPSE on their various drafts on the Laws led to overall strengthening of legislative and policy framework for pre-school and general secondary education field. As noted by government stakeholders, direct contributions of UNICEF led to adoption of the following documents: the Education Sector Plan; the Law
40 Throughout the CPD implementation, UNICEF convened a number of workshops, conferences and round table discussions on education to enable discussions among policy makers and practitioners informing legislative development. Some of such examples include, but are not limited to, Conference on PPP held in March 2018; Workshop on Alternative Models of ECCE; Conference on quality ECE held in November 2018; NQEF and NQEF M&E tools development Workshops; Workshop on teacher workforce policy review February 2018, etc.
34 on Pre-school Education (in December 2019); the revised Law on Education (in August 2020); finalization of National Quality Assurance Frameworks for Pre-schools and General secondary schools; Teacher Competency Standards Framework for Pre-schools & General Secondary Schools; the Education Quality Assurance Framework (EQAF); the Draft National Curriculum Framework (NCF) and the Early Learning Development Standards; Conceptual Framework for Alternative Forms of Pre-school Education. 100. Document review and stakeholder interviews with national government and development partners emphasised that these documents created a strong foundation for wider reforms of education sector. The UNICEF insistence on a rights-based approach in the development of such documents has raised the understanding of the national stakeholders of implications of education on children (boys and girls) and fulfilment of their potential through access to better quality education across the country. These legislative changes have helped the government to gear the reform to areas where reforms are most needed (tackling curriculum, quality of textbooks; equity and access to education, and most importantly the role of PSE in childhood development and fulfilling the full potential of children).
6.2.2. UNICEF’s contribution to enhancement of government capacity to provide increased access to inclusive and quality early learning services (Original Output 2.2 and 2.3/revised Output 2.2)41
101. Document review, “We did not have set standards for early child education development feedback received through or state education programs on this issue. UNICEF helped us develop group discussions and them from scratch, and now we have them on the new curriculum. We stakeholder interviews show have for the first time introduced standards, we have the role between that UNICEF’s bulk of all three subjects of education introduced; and the information system support was directed at for managing the education.” Key informant upstream, policy support, with little engagement in subnational level or in more extensive capacity development efforts. Two main reasons underpinned such strategy: 1) the political transformation brought new energy and renewed commitment of the government to the education sector reform demanding support to enhancing legislative/policy basis for any deeper structural reform; and 2) UNICEF’s education sector budget cuts following SMR process, which demanded more focus and consolidation. 102. UNICEF’s main partners across the implementation of the current cycle of the education programme were the MoPE, the newly established MoPSE42 and SISQE. The latter two were tasked to implement a five-year pre-school education programme that was developed in 2016 with indirect input from UNICEF Uzbekistan. UNICEF provided stable and useful technical assistance to MoPSE and SISQE to build their institutional practices and capacities, a support area that was highly appreciated by these institutions. Stakeholder interviews with both the UNICEF team and the national stakeholders also noted that UNICEF filled in the gap in institutional capacity to draft policy and legislative documents, which was an important contributor to ensuring that the documents are viable and tackle the most important reform priorities, well as helping the government to integrate equity and child rights and issues. However, there was a concern that, while such contribution was of critical importance in the short term, in the long run it may continue government deficiency to respond to national priorities and dependence on external support.
41 In response to EQ5 42 The ministry and SISQE were established in late 2017.
35 103. Bringing international “UNICEF’s support to the development of teacher standards experience and expertise in support to for pre-school education was crucial in the improvement of the reform was one of the support quality of education at this level of the education system. In the interventions that UNICEF used to country’s education system, the concept of the standard is a key concept and any support in this regard is likely to be widely strengthen national capacity. Through accepted show long-term results.” organizing conferences and workshops Key informant on a variety of themes (See footnote 23 above), UNICEF helped share international standards and practices, convening discussions of how such international best practices could be applied in Uzbekistan. This work, as noted in Section 6.2.1 above, helped in policy work but also strengthened awareness and knowledge of national stakeholders, government and wider pool of educational professionals, on innovation in education. 104. In particular noted were efforts to Quality assurance in education is high on the enhance capacity for PSE formulation, where agenda of the national partners. UNICEF international practices were shared. The contributed to improving the quality of education through five areas of support: resulting outputs in the form of pre-school • The development of quality standards for pre- curriculum contributions and a Teacher Training school and general secondary education; package including six guidebooks for training on • A quality framework for pre-school and general various aspects of Early Learning and secondary education; Development Standards (ELDS) and the new • Support in the development of the new Law on pre-school curriculum were highlighted as direct Education; capacity strengthening efforts. The training • Training of more than 400 experts on QA; • Training of teachers on the new standards for package was used for Training of Master Trainers pre-school and general secondary education. on ELDS and Pre-school Curriculum, creating a pool of trainers to scale up trainings across the country. Stakeholder interviews and group discussions with government and educational professionals noted that this pool of Master trainers, besides potential to scale up the training across the country, also serve as a critical mass of PSE professionals who can assist in planning and delivery of PSE reform interventions across the country. This core team of National Master Trainers in delivering pre-school curriculum and ELDS thus far trained as many as 38,400 pre-school teachers in cascade model.43 From that perspective, UNICEF’s investment was seen as very positive, and has strong catalyst prospect when it comes to enhancement of PSE access and quality. 105. Focus groups underlined the fact that training organized under the UNICEF programme sharply contrasted with those organized solely by national partners. The duration of training events was appropriate, materials well developed, trainers well prepared and participants given the opportunity to participate actively and express their views freely. Hence, implementing training in line with a small set of quality criteria makes a significant difference. 106. Another capacity building effort aimed at promoting Public-Private Partnerships for PSE and modelling of PPP in PSE as alternative means of delivery of PSE services across the country. UNICEF helped the MoPSE to learn about international experiences (e.g. Conference on PPPs in PSE) and design PPP models. Such investments were highlighted as important capacity building effort which lead to scaling up of this model to increase access to PSE. Document review, feedback received through group discussions and stakeholder interviews in particular highlight the relevance of engagement in PPPs for PSE, as means to enhance access to PSE for children, particularly those whose access to PSE is not possible
43 COAR 2019, p. 4
36 due to lack of institutional capacity to provide such service in remote regions and locations outside of the capital and larger regional centers. Government stakeholders, educational professionals and development partners noted that such investment helped change the relatively low access to early age children development programmes, particularly girls and children in remote areas. While the access to PSE is gradually changing, interviewees across stakeholder groups noted an ongoing need for investment in accessibility and equity in PSE. 107. Another UNICEF contribution was directed to the development of the web based EMIS for pre-school education. UNICEF support aimed at streamlining indicators, data collection tools and analytical methods. Document review and stakeholder interviews noted that the EMIS is helpful and allows for availability of PSE data disaggregated by gender, location and pre-school types, which is used in decision making processes. However, challenges were observed in terms of expansion and outreach of the EMIS in light of the internet accessibility and general skills to use the EMIS, outside of centres.
6.2.3. UNICEF’s contribution to enhancement of tools and skills to better formulate and implement programmes for inclusive and quality general secondary education for all children (revised output 2.3/original outputs 2.2; 2.5; 2.6)44
108. Another area of UNICEF’s focus was on development of the NCF outlining the competency-based approach to curriculum as well as a review of textbook policies in the country to ensure their alignment with the NCF and improving their quality. Besides, UNICEF supported the government in drafting the new Teacher Qualification Frameworks and Teacher Professional Standards. All these investments were aimed at improving quality of education, elaborating legislative basis into more practical tools for educational professionals and decision makers to improve delivery. Stakeholder interviews and feedback received through group discussions highlighted that the NCF as well as the standards bring important new standards and principles for quality of education. Besides, UNICEF conducted a textbook policy review, aiming to identify overall key system gaps in the process of organization of textbooks’ development, production, state procurement and dissemination. The review was conducted jointly by UNICEF and MOPE, with contributions of the UNICEF consultant who lead the process. Only recently (in November 2020 during finalization of this evaluation report), UNICEF initiated textbook reviews for grades 1 and 2, with involving of four (4) subject experts. In line with agreement with MOPE, textbooks for grades 3 and 4 will be reviewed in 2022.
6.2.4. UNICEF’s contribution to enhancing tools and skills to promote safe behaviour practices among children and communities before and during emergencies (revised Output 2.4/original output 2.7)45
109. To respond to Uzbekistan’s vulnerability to disasters, UNICEF planned to invest in enhancing the DRR preparedness of the educational system. During the implementation of the current CPD, the COVID – 19 pandemic re-emphasised the relevance and necessity to investment in enhancement of tools and skills to promote safe behaviour practices among children and communities before and during emergencies. UNICEF’s response to the
44 In response to EQ5 45 In response to EQ5
37 implications of the COVID-19 Box 1. Overview of UNICEF activities in pandemic on the education sector response to COVID-19 pandemic were very relevant and appropriate to the raising issues and the needs of the • Support to Republican Education sector as noted by interviewed Centre (REC) to organize Distance stakeholders (See section 6.1.1) and Learning. Support to the REC was provided were in line with its strategic to develop Distance Learning plan, which was positioning and mandate. UNICEF based on the Rapid Assessment of Distance assisted the government to devise and Learning Programme to inform the plan implement a range of measures, as development. In cooperation with DFID, presented in Adjacent Box. Notably, UNICEF also supported REC to 1) continue a UNICEF was at the forefront in variety of Distance Learning programmes development of the COVID-19 Response Plan of the education sector during summer vacation; 2) to prepare the and the design and implementation of Blended Learning guidelines, Remedial Distance Learning plan and related Education Guidelines and Assessment distance learning programmes early on mechanisms; 3) to prepare training packages when the restrictions were introduced for building capacity of teachers on these (in April 2020). National stakeholders important support measures praised these efforts particularly as the government did not have sufficient • Supporting PES distant learning. UNICEF experience in distance learning and MOPSE with DIFID funds developed a approaches, so the support was timely series of TV programmes for pre-school and relevant. It enabled the education children based on Curriculum and supported sector to organize and implement in development of educational platform for alternative education programmes in parents with children with disabilities. Besides, short period, creating little disruption UNICEF supported the development of Safe of the ongoing school year. The pre-school education institutions re-opening cooperation with UNICEF helped the guidelines by initiating and establishing cross Government of Uzbekistan to realize sectoral Working Group, providing technical the opportunities resulting from the and expert support. Finally, video trainings restrictions imposed by the COVID – were developed for Capacity building of all 19 pandemic. Like in many other situations, UNICEF was the primary pre-school staff. partner to be asked for assistance in • Awareness raising. UNICEF developed that situation. C4D and IEC Materials in support of Safe pre- 110. Following the need to learn school education institutions re-opening and from implementation of measures, COVID-19 response for children, teachers, UNICEF also conducted a COVID-19: parents and Communities. Besides, UNICEF Rapid Assessment on ECE and the conducted a series of Communication COVID-19: Rapid Assessment on Activities #LearningAtHome based on the Distance Learning. Findings from these regional and global initiatives assessments served as inputs for development of the Safe pre/schools • WASH. Analysis of schools and pre-school re-opening guidelines ahead of the new education institutions from WASH school term (2020/2021). Ahead of the perspective was done as evidence base for new school term, UNICEF supported policy making in light of school reopening the government to develop following the facilitation of restrictions. Communication for Development
38 (C4D) and IEC materials related to COVID-19, safe behaviour, information, protection and learning (for children, parents and families, staff, communities). 111. Finally, the UNICEF Education Section in close synergy with the Health Section developed a WASH analysis for schools and pre-school education institutions, which brought important, yet very bleak data on state of school premises across the country. The WASH study noted a range of challenges that schools, particularly in remote areas and outside capital and regional centers, face when it comes to access to water and sanitation, critical to overall health and wellbeing of children and in particular at times of COVID-19. The WASH analysis serves as advocacy and policy advisory basis for UNICEF’s work with the government. The study also significantly contributed to raising the Government’s awareness of the fact that hygienic conditions in educational institutions are a crucial factor in creating a proper teaching and learning environment. Particular issues pertaining hygienic conditions for girls were highlighted in the study. According to stakeholder feedback, this perspective of the study was very important and raised the issue of hygiene in schools in the public. 112. UNICEF supported the organization of a series of trainings for students on behaviour in emergency situations. UNICEF support has led to the institutionalization of safety trainings in schools. Training includes the design of emergency plans. In view of the fact that emergency situations are likely to put girls at particular risk, in the trainings particular attention was given to the role of girls in higher grades as role models for those in lower grades to make the latter more receptive to the risks they face and appropriate behaviour in emergency situations. The institutionalization was supported by the design of teaching and learning materials for safety trainings. A series of manuals were developed to support the integrated teaching of DRR-issues in 14 subjects from grades 1 to 11. The publication “Students’ personal safety” is available on the webpage of the Ministry of emergency situations. Stakeholders see the institutionalization of the teaching of DRR-issues as key to sustainability of the UNICEF programme. UNICEF support has also led to an increased cooperation between government agencies on DDR-related issues. This includes a joint action plan for increased security training. In addition to trainings and the development of emergency plans, UNICEF supported monitoring of the implementation of emergency plans.
6.2.5. Consideration of gender and vulnerability in programme planning and implementation
This section presents an analysis of UNICEF’s Education programme consideration of gender and vulnerability in response to the Evaluation Question 6: To what extent did the programme effectively mainstream gender? And Evaluation Question 7: To what extend the needs of vulnerable children were considered in the programme planning and implementation?
“’Gender’ is a concept frequently referred to. However, in 113. Gender equality and absence of one universally definition, it also tends to be attention to vulnerable population interpreted differently, depending on previous knowledge, was formally considered as an assumptions, expectations and the political and social context. A commonly agreed definition of the concept of gender and a integral part of UNICEF’s shared understanding of it by all partners is essential to upstream work with national achieving progress in this regard.” stakeholders. However, ensuring Key informant full gender equality and equity and successfully reaching vulnerable populations remains a challenge. UNICEF has increased its alignment with the gender action plan. While gender dimensions are generally included in the UNICEF
39 Education Programme, the interventions appear to be more formal than systematic. There are divergent perspectives from external stakeholders on the relative need for the prioritization of gender, with some pointing to the greater importance of attention to the most vulnerable groups (e.g. those children with disability, children affected by poverty, rural children, etc.). The Uzbekistan context is still faced with limited prioritization of gender mainstreaming by external actors, and constrained by general understanding that universal education access is ensured in the country, with high enrolment rates of girls and boys. However, there have been successes in integrating gender into interventions, including the review of regulations, curriculum reform and in particular studies and analytical reports prepared by UNICEF education team. Furthermore, sex- and age-disaggregated data (SADD) is usually reported in activities involving educational professionals and also in studies. 114. Stakeholders reported that gender has a traditionally limited focus in the country context, which has implications for programming design and adjustments, leading to relatively little gender analysis regarding programme interventions or models. Review of programme documentation and feedback received through group discussions stakeholder interviews note that, as a normative stance, gender and vulnerability are consistently reflected in the programming documentation and also advocated by UNICEF. UNICEF’s approach to integration of gender and equity principles in supported policy and legislative documents was to ensure that such documents include references to rights of girls and boys, and recognition of equity and gender equality dimensions and related vulnerabilities of girls and boys, particularly most vulnerable groups (e.g. children with disability). Review of new/revised legislative and policy documents developed with support of UNICEF shows that such notions are included, albeit with varying consistency and focus. From that perspective, it may be understood that UNICEF’s efforts were appropriate and ensure that there is a legal/policy foundation for fulfilment of rights of boys and girls, recognizing vulnerabilities and needs. However, document review, feedback received through group discussions and stakeholder interviews raise concern with remaining gaps in actual focus of the government on gender equality and responding to the needs of the most vulnerable children. 115. In particular, gaps are “One of the things that's been overlooked so far being noted in terms of responding to the services for children with disabilities” urgent need for Key informant deinstitutionalization of children in education, i.e. investment in inclusive education. Uzbekistan is still providing services of the so-called “special education schools’ for children with disability”, raising concern with social exclusion, segregation and breaching the rights of children. During this programme cycle, there was limited interaction and engagement between the UNICEF’s education and social and child protection sections on issues of social inclusion, especially focusing on inclusive education and wider issues of children with disability. Feedback received through group discussions and stakeholder interviews raised this as a weakness. Interviewed stakeholders also raised concern that formal advocacy for integration of notions of gender in policy/legislative making process is not sufficient and that UNICEF could have invested much more concrete effort in putting the issues of access to education for girls and boys on the agenda of the government. 116. Interlocutors generally affirm the consideration of gender issues and the needs of vulnerable children across reform measures. However, in the vast majority of cases, the Evaluation Team was unable to obtain concrete examples of addressing gender or vulnerability issues (DRR being a notable exception).
40 6.2.6. Unintended negative and positive outcomes and their management46
117. This section presents an analysis of unintended outcomes in response to the Evaluation Question 8: Were there any unintended negative or positive outcomes and, if so, were they appropriately managed? 118. Key informants and focus groups did not report any unintended negative outcomes. Improved communication and cooperation between state agencies was mentioned on several accounts as an unintended positive outcome. In particular UNICEF’s role as facilitator/convener in this regard was mentioned. Among UNICEF’s key contributions is ensuring that partners use the same terminology, have a proper understanding of procedures related to programme implementation, and respect timeframes so as to avoid delays due to one of several partners involved in an activity.
6.2.7. Factors contributing to and hindering the delivery of outcomes
119. The following section summarises the key evidence in relation to the internal and external contribution factors that have enabled and/or hindered the achievement of outcomes. Internal Factors 120. Strategic positioning of UNICEF has been a driver. Evidence was found of strategic positioning of UNICEF as a neutral and supportive partner of the government in the reform process. Longer-term support to the reform process has been instrumental in delivering progress towards the achievement of enhanced policy and institutional capacity in the reform of primary and general secondary education. UNICEF is seen as partner of choice for government institutions, which presents a strong opening for UNICEF interventions to maximise their potential when it comes to more complex reforms (e.g. curriculum reform, inclusive education). UNICEF’s neutral role, combined with the ability to tap into its team expertise and a pool of national and international expert advisors and technical specialists, is valued as a driver of effectiveness by interviewed stakeholders. 121. Ownership as a driver. The stability of UNICEF support and consultative approach by the UNICEF team to ensure buy-in and ownership over the interventions and outputs has been a key driver of results. This was achieved primarily through the involvement of relevant government institutions (and to some extent also development partners) at all stages of the planning and implementation of activities. Feedback received through group discussions and stakeholder interviews confirmed that the consultative and participatory approach increased alignment with needs and priorities and created ownership of UNICEF’s interventions. 122. Flexibility and applied learning. UNICEF showed a high level of flexibility to adapt to changing circumstances and to respond to arising needs within the sector (e.g. the political transformation, establishment of the Ministry for PSE and SISQE, etc.). UNICEF team redesigned its results framework during the Strategic Moment of Reflection in 2018 to ensure better targeting. 123. Risk management. In interviews with the UNICEF team it was noted that risks have been continuously assessed along with mitigation strategies, though a very limited paper trail exists on these measures. Thus far, no major risks emerged during project implementation that could have adversely impacted the achievement of results.
46 In response to EQ8
41 External factors 124. The most important external contribution factors across the education sector are the socio-economic context, political support for reforms, stakeholder capacity to engage and absorb assistance, and synergies with other related initiatives. 125. Socio-economic context. Strong demand for reforms of the education sector is considered to be an enabling factor. Document review, feedback received through group discussions and stakeholder interviews revealed a strong momentum for reforms education, as reflected in demands by civil society actors and citizens. In light of this, UNICEF provided a timely response to this momentum, by investing in supporting relevant ministries and education sector institutions towards the achievement of reform priorities. 126. Political support for reforms. Ministries of Education (MoPSE and MoPE) have been committed to the reform and open to absorb the support provided. Despite the frequent changes within the ministries, feedback received through group discussions and stakeholder interviews show that there is high interest and commitment to improving their internal operations, which was among the decisive factors ensuring the smooth implementation of UNICEF support interventions. However, concern was raised that slow change and a certain possibility of resistance to change at the regional and local level within educational structures may affect the level of sustainability. 127. Government / stakeholder capacity to engage. Absorption capacity of educational institutions to integrate new mechanisms within their structures is seen to be limited in some cases, particularly at subnational level. While central level institutions have been very responsive to UNICEF’s support interventions, the evaluation found a certain level of donor dependence and expectation that UNICEF would plug the gaps with their own capacities. This is a hindering factor particularly when it comes to sustainability of achieved results. 128. Staff turnover within the ministries and other central level educational institutions is a serious challenge. As a result of staff turnover, capacity and knowledge are lost to the institution and processes are slowed down. For instance, during the current programming cycle there have been frequent changes in top decision-makers at the ministerial level along with staff changes across the ministries. 129. Synergies with wider initiatives. Joining efforts with other donors contributes to coherence, effectiveness and value added of donor aid. UNICEF has participated and convened a number of donor coordination meetings. Overall, donor coordination is not lead by relevant national institutions, which represents a challenge with regards to full- fledged coordination and alignment. Further synergies and joint initiatives would have been beneficial in light of limited and decreasing UNICEF funding for education. 130. Perception of children with special educational needs. The perception of children with special educational needs in society is a hindering factor to their access to and inclusion in mainstream education. Discontinuing the tradition of separating children with special educational needs and their institutionalization in special educational institutions is a challenge which cannot be solved solely through education. In this regard, the collaboration between the UNICEF Country Offices’ sections is particularly important.
42 6.3 Efficiency
131. Three evaluation questions underpin the assessment of efficiency of the education programme, i.e. assessment of utility of UNICEF’s implementation strategies (EQ9); adequacy of financial, material and human resources (EQ10) and programme cost-efficiency (EQ11). The following is the presentation of main findings to these evaluation questions. The evaluation did not conduct a comparative cost analysis, due to time, scope and resource limitations.
6.3.1. Review of UNICEF’s implementation strategies (EQ 9)47
132. As noted in preceding chapters, UNICEF implemented a range of implementation strategies to contribute to reforms of education sector. The table 2 below presents an overview of these strategies and their utility.
TABLE 2. OVERVIEW OF THE FULFILMENT OF UNICEF ROLES (IMPLEMENTATION STRATEGIES)48
Key UNICEF Assessment of application of the roles roles49
(5-point rating scale: Highly successful, successful, moderately successful, limited, unsuccessful)
Advocacy Rating: Successful (the independent Strategic partnership with the government enabled UNICEF to utilise its voice) advocacy role successfully across all subjects of focus of UNICEF. Such advocacy efforts resulted in raising awareness and understanding of ways and means to translate the needs and priorities into viable policy and legislative documents (legislation, new policies, and mechanisms). In particular, strong advocacy role towards enhancing quality of PSE was highlighted by stakeholders as important.
“What we particularly appreciate is that UNICEF makes a serious effort to gain a thorough understanding of the circumstances before addressing our requests for assistance.”
Key informant UNICEF Country office could have done more to advocate strongly for issues of gender and equity, in particular of access to inclusive education for children with disability.
47 EQ 9: Which implementation strategies have worked and which have not worked? Why? 48 Includes all core roles except Horizontal cooperation beyond borders and Ensuring proper internal controls and risk management). Judgements made in this table are based on thorough triangulation and cross-checking between different data collection methods interviews with interlocutors, included extensive documentation review and analysis of data from monitoring reports. 49 UNICEF Regional Office, ‘Discussion Note on Core Roles and Achieving Results for Children in the CEE/CIS Region’, 2015.
43 Policy Rating: Highly successful dialogue and advice Across all stakeholder groups, key informants praised UNICEF’s technical expertise, flexibility and timely policy advice and convening power. UNICEF’s ability to tap on its own regional and international best practices and consultants was praised as a good investment in development of policies and legislation through technical assistance. As a result of its technical advisory functions and contributions to policy dialogue, UNICEF directly influenced the development of the normative
“The most valuable recommendations are being provided by UNICEF experts, and we are touched by the great contribution to achieving international standards. We know that, most importantly, these experts are developing a national education content that is the mix of international and local experience, they are not providing only their experience, but I can say that they are taking into account the needs of the Uzbekistan and Uzbekistan situation.” Key informant frameworks for education, regulations and standards, based on the application of best practices
Knowledge Rating: Highly successful generation and child UNICEF contributed significantly to establishment of the evidence base. rights Studies and reviews that UNICEF produced (See 6.1.1 above) helped inform monitoring decision-making, leading to a better understanding of unmet needs of children in education sector and bottlenecks and gaps. This is in particular when it comes to National Learning Achievements Study, the KAPs and SitAn on Children with Disability, which brought important data on state, outreach and gaps in education sector when it comes to fulfilment of the rights of children. UNICEF has made a major contribution to increasing the Government’s awareness of the need of reliable, disaggregated data as basis for informed decision-making.
Convening Rating: Highly Successful partnerships and UNICEF’s added value was seen to be its convening power and ability to leveraging bring national and international stakeholders together to discuss and define resources for priorities and responding education sector reform measures. Stakeholders children praised UNICEF’s contribution to convening education sector stakeholders to discuss and come to common definition of key terms and concepts within education, which have helped creation of better serving policies. Such example is the Education Sector Plan. UNICEF was also praised as a government partner whose reform “UNICEF played a key role in ensuring proper communication between the involved state agencies and ensuring that there were no delays caused by any of them.” Key informant contributions have helped the government to raise and leverage resources from other development partners. For instance, UNICEF helped the government to access 50 million USD of the World Bank funding stream for
44 general secondary school education and 10 million USD for pre-school education.
Capacity Rating: Successful development of UNICEF’s capacity development support for the MoPSE and SISQE and professionals other levels of the education system made a contribution to building the and capacity of professionals in the education system. UNICEF served as gap organizations plugging agent in cases where the MoPSE and MoPE did not have human resources or capacity to fulfil some reform objectives. Besides, UNICEF’s main training focus was to raise capacity of master trainers on ELDS and PSE, which has a scalability potential. However, education sector professionals and development partners criticised UNICEF for lack of focus on subnational level, where actual application of normative framework happens. Criticism was directed towards lack of UNICEF’s focus on preparing educational professionals and institutions to embrace reforms. While this criticism is well-founded, insight into UNICEF’s financial and human resources, but also the phase of the education sector reform shows that UNICEF, during this programme cycle was not able to stretch its resources on this task. Taking into account the size of the Uzbekistan’s PSE and general secondary education sector, such task is huge and requires significant human and financial resources, much of which is beyond the reach of UNICEF at the moment. Still, capacity development of professionals in the education system is key to sustainability and should continue to be high on the agenda throughout future programme implementation.
Modelling Rating: Successful and testing innovations Due to the fact that UNICEF’s primary focus was on upstream policy level, only some efforts were found when it comes to modelling, when it comes to PPP in PSE. Thus far, sharing of best practices and PPP models were promoted and introduced, but actual testing was delayed due to COVID-19 pandemic. However, supporting the government to model and implement distance learning models in light of COVID-19 pandemics, but also other COVID-19 response activities and measures of UNICEF have been very useful, appropriate and conducted in timely manner, as praised by interviewed stakeholders. This part of fulfilment of this UNICEF’s role was very successful.
6.3.2. Adequacy and cost-efficiency of financial, material and human resources to meet the programme objectives (EQ 1050 and EQ1151)
133. Review of UNICEF’s financial data shows significant discrepancy between initial financial allocations for the education programme, commitments and actual expenditures. As presented in Table 3 below, the initially plannedresources particularly for years 2016, 2017
50 EQ 10: Were the available financial, material and human resources adequate to meet the programme objectives? 51 EQ 11: To what extent is the programme cost-efficient? Are there any potential efficiency gains to make?
45 and 2018 did not materialize creating significant deficiencies in terms of budget for planned activities. This is due to the fact that allocation of funds at the beginning of CP was considered as funding estimation, which did not materialize during the CP implementation. Review of financial data shows that the education programme received app. a half of what was initially planned for this sector. Document review and interviews with UNICEF team show that this discrepancy in terms of budget was problematic as it came in the least opportune moment when UNICEF initiated a number of support interventions, affecting the extent to which UNICEF could follow its demand driven nature. Situation slightly improved in 2019 and then in 2020, particularly to reflect, inter alia, UNICEF’s increased support to the education sector in light of the COVID-19 pandemic response.
TABLE 3. FINANCIAL OVERVIEW 2016-2020 (AS OF DECEMBER 2020)
Planned Funded Commitment Actual Utilized Util % Year (a) (b) (c) (d) (c+d) (c+d)/b 2016 1.500.000 465.233 0 465.233 465.233 100% 2017 2.500.000 773.971 0 773.971 773.971 100% 2018 3.000.000 1.032.018 0 1.032.018 1.032.018 100% 2019 730.814 832.078 0 832.078 832.078 100% 2020 802.910 1.371.300 168.765 1.184.090 1.352.855 99% Total 2016- 8.533.724 4.474.600 168.765 4.287.390 4.456.155 99% 2020
134. The evaluation found that UNICEF’s education programme was generally efficient and adaptive to changing circumstances and in particular the new reform momentum brought by political transformation. Review of internal programming and reporting records shows that the team is currently stable, after some internal changes and turnover noted within the reference period of the evaluation. Interviews with the UNICEF Education team noted that there has been a significant shift in the composition of the education team, moving from more administrative tasks towards more essential programmatic contribution to reforms. The main driver, as noted in the interviews, was the aspiration of the team to bring more comprehensive content contribution to the education sector, a shift that was praised also by national stakeholders. This shift was also a necessity: UNICEF embarked to support a vast number of reform initiatives across PSE and general secondary education, that required expertise and concrete thematic technical assistance, and naturally budget resources. Due to the fact that the budget was limited but also there was aspiration within the team to take more proactive, substantial role in the reform, investment in further capacity strengthening of the education team was welcome. This investment filled in the gap in provision of outsourced expertise and helped build stronger relations with government counterparts. According to interviews with UNICEF team, education section had least number of outsourced experts in comparison to other UNICEF CO sections, and a relatively small team (5 persons) in comparison to for instance health (16 member-team) or social policy or child protection teams.
46
135. Deep engagement of the team in delivery of essential outputs is praised for its value in terms of saving financial resources that would have been spent on outsourcing expertise. However, concerns were raised that this approach also brought challenges in terms of significant overstretching of human resources, as confirmed by UNICEF interlocutors. Such challenges raise issues and weaknesses in terms of scoping of support (i.e. too many interventions were undertaken without much prioritization), which comes with the demand driven nature. The aspiration to engage and support reforms across various education areas is commendable, but too wide approach beyond the financial and human resource ability to provide stable and substantive support is not conducive as found by this evaluation. UNICEF’s demand driven nature also provided the government with a belief that UNICEF is (financially and in terms of human resources) able to respond to ever-raising needs and priorities in the education sector. Being viewed as a competent and trusted partner inevitably leads to increasing requests for assistance. Handling those requests and addressing them in a balanced manner both in terms of timing and the availability of human and financial resources is a challenge. Experience in this programme has shown that, despite limited resources, requests can be addressed in a manner that allows the national partner to conclude that requests are taken seriously and that an effort is made to address them. However, they did put a strain on the human and financial resources at UNICEF. 136. In light of evidence decrease of funds, UNICEF’s ability to fulfil the tasks that were taken with available funding and staff resources was commendable but is not viable in medium to long term. Hence, in short term, the evaluation found that cost-efficiency of UNICEF’s engagement was high, as most of the tasks were accomplished through efforts of the team, but it does not offer positive prospect medium to long term as such level of effort and energy cannot be sustained for longer period of time. These challenges call for consolidation and more realistic planning. 137. The evaluation did not find any significant delays at Education Programme level. Delays were noted in accomplishment of certain tasks and studies (e.g. drafting of survey reports based on data collected by national partner took a long time), but these did not affect the quality of the timeliness of final products. 138. The Programme results framework, particularly following its mainstreaming as a result of SMR is assessed as a good basis for UNICEF’s monitoring and evaluation efforts. The results framework is conceptually sound, yet rather ambitious in light of institutional capacities and contextual limitations for full achievement of results. 139. UNICEF monitors closely its activities and the delivery of its outputs. Evaluation found evidence that UNICEF Education section used the monitoring data (particularly at activity- output level but also assumptions and risks underpinning the portfolio) to inform decision making on next steps, and support interventions. Good example of this was the response to COVID-19 pandemic or decisions between outsourcing or using in-house expertise for studies and analytical papers. Feedback received through group discussions and stakeholder interviews pointed to UNICEF Education team’s efforts to also reflect internally but also with the government partners on achievement of results and obstacles. These venues are used also for decision making on next steps in support. However, an important weakness was is found in terms of reporting on overall implementation and results. This is an inherent weakness of UNICEF’s reporting system, which has undergone change in formats for annual reporting during the programme cycle. There is a general lack of deeper reflection of catalyst potential of delivered outputs to overall reforms in the country. This prevents understanding of current achievements and how they can in the future bring more long-term results for children in Uzbekistan. It is noted that the initiative of the UNICEF’s CO to embark in the
47
specific Education Programme level evaluation on top of the Country programme evaluation is an avenue to assess the contributions and catalyst potential of the delivered results, as well as offer learning potential.
6.4 Sustainability
140. This Section provides answers to Evaluation questions 1252, EQ 1353 and EQ 1454. 141. This evaluation found ample evidence of direct contribution of UNICEF to concrete changes in the legislative and policy frameworks, which now serve as strong foundation for fundamental reforms of education sector. Interlocutors representing Uzbekistan’s government institutions noted that support to changes in the normative framework guiding the education sector measures has been significant and a value add of UNICEF. Government views normative acts as drivers for action on any issue or measure, so investment in new legislation and amendments to existing laws is a strong prerequisite for sustainability and a basis for actual reform. Some of the normative acts adopted with the help of UNICEF also foresaw additional allocations from the national budget and other donor resources like the upcoming World Bank loan to the education sector. However, the adoption of the normative framework is just the beginning of actual reform. Uzbekistan now has strong normative framework but the capacity to implement the reform is weak. Operationalisation of the normative framework through raising capacity of regional and local education institutions is of critical importance for actual reform to take root across the country and to ensure access to most excluded and vulnerable children. 142. UNICEF’s contribution has also been strong to institutional capacity development of the MoPSE and SISQE, with direct contribution to strengthening the overall institutional structure for PSE. UNICEF closely collaborated with government stakeholders on all relevant policy documents and initiatives in the PSE area, while other types of capacity building events (workshops, trainings, on-line sessions) also were a direct investment in institutional capacity building. For instance, UNICEF convened a number of workshops, meetings and discussions sharing international expertise, best practices and through discussions and joint work enhanced capacity of national experts in support to the development of the new Law on Pre-school education but also in revision of Law on education UNICEF. Interviewed stakeholders from the government, education professionals and development partners noted that these investments bring an important framework for scaling up PSE across the country. However, one reservation was noted when it comes to UNICEF’s technical assistance to the institutions, i.e. that UNICEF expertise filled in the gap in institutional capacity. In the short term, this has proven relevant and effective to gear the reform forward. However, in the medium to long term, such practice should have been gradually accompanied by stronger investment in capacity development and training of national stakeholders to assist build internal capacity for reform. Such approach was not adopted as explicit UNICEF strategy thus far. One important challenge in this regard is the ongoing staff turnover within the ministries of education and other central education
52 EQ 12: Did the programme lead and/or contribute to improvement in the allocation and use of resources for the programme priorities? Were these improvements sustainable? 53 EQ 13: Did the programme interventions lead and/or contribute to improvements in legislative and policy frameworks in educator sector? 54 What are the opportunities for and risks to the sustainability of the programme in the short and long term?
48
institutions. This makes capacity investments fragile, due to the fact that institutional memory is weak as a result of frequent turnover. However, discussions with national stakeholders noted that the turnover is expected to be less pronounced now that the country stabilises slowly after the political transformation. National stakeholders also noted that institutional capacity building remains the most critical priority particularly when actual reforms need to be brought to the regional and local level. 143. UNICEF’s investment in building capacity of Master Trainers on ELDS and PSE has created a pool of motivated and capacitated educational professionals who are able to multiply the knowledge across the country. However, the actual scale up of trainings depends on government’s commitment and motivation to actually allow/organise such training opportunities for educational professionals across the country. Another important challenge is also linked to restrictions in response to COVID-19 pandemic which prevent any trainings to happen beyond online training opportunities. This makes it more difficult to plan and access such trainings particularly to those outside of more affluent centres and regions.
144. UNICEF has made advocacy efforts to “Social media have turned out to play a key role in integrate gender and issues of the ensuring sustainability with regard to training. Social most vulnerable groups (socially media channels remain open after training events have excluded children, children with been completed. Participants continue to stay in touch, disability). Still, work in this regard engage in discussions and attract additional interested needs to be continued and, preferably, persons. Training materials continue to be made available stepped up. Deinstitutionalisation of on social media channels.” children from special schools and their Key informant inclusion in mainstream education is an outstanding priority which was not addressed more explicitly UNICEF during this programming cycle. 145. Overall, UNICEF did not have sufficient opportunity (due to financial, human resource, but also reform focus limitations) to expand its reach outside the capital and some regional centres. This prevented stronger outreach to subnational level. Due to these reasons, UNICEF’s contribution to actual changes in the way education institutions organize educational activity have not materialized, as confirmed by both UNICEF and external stakeholders. 146. In the field of DRR, stakeholders interviewed within the scope of the evaluation noted that the institutionalization of safety training in 14 subjects was a good avenue to ensure sustainability of the investment. There is also an increased cooperation between state agencies in DRR, which is also a mechanism to ensure that the programme will be sustained as noted by interviewed stakeholders. 147. UNICEF was the Government’s primary partner in the situation resulting from the COVID- 19 pandemic. While the situation imposed serious limitations to teaching and learning, it also generated a need to develop alternative learning mechanisms. The Government’s recognition of the need to do so is likely to create a long-term awareness of a backup system for teaching and learning. It may also lead to the introduction of a teaching and learning system that combines face-to-face and online learning. The COVID-19 pandemic provides an important opportunity for further thinking of the government on how to reform education system to be more resilient to shocks such was the COVID-19 imposed halting of the education.
49
6.5 Impact
148. This section presents main findings in response to Evaluation Question 15: To what extent can the programme be considered to have made a contribution to an observed progress in education system? In the ToR, the relevant evaluation criterion was defined as ‘long term change’ and per request of UNICEF, this was changed to assessment of “impact’. 149. Review of UNICEF programming documents for the 2016-2020 cycle shows that the overall desired outcome of UNICEF investment in education was stated to be: “By 2020, girls and boys benefit equitably from improved pre-school education and enhanced quality of basic education including in emergency”.55There is ample evidence of UNICEF’s contributions to creation of prerequisites for educational institutions to provide for equitable and accessible PSE and quality basic education. The normative and institutional framework (at central level) is now largely in place to lead the actual implementation of norms and standards across the country. Particularly when it comes to PSE, strong basis for next steps in the reform have been taken through strengthening capacity of the MoPSE and SISQE. However, the reform is just generating speed, so it is too early to see actual changes in the way PSE operates beyond the capital and more affluent regional centres where PSE was anyway in a better shape to start with. Time will be needed for reforms to trickle down to the local level for the benefit of all children that are in need of PSE. 150. UNICEF’s investment in revisions of the national curriculum and textbooks has the potential to bring about long-term change in the identification of learning contents and the development of teaching and learning materials. Although the purpose and results of these revisions are not fully recognized by all stakeholders, the results of revisions of curricula and textbooks can, if properly interpreted, communicated and used for future planning, bring about a fundamental change in the contents of teaching and the materials used to communicate those contents to students. In particular changes in textbook design with a strong focus on student-centred teaching are likely to lead to significant long-term changes in approaches to teaching.
6.6 Coherence56
6.6.1. External coherence (EQ 1657)
151. There is evidence collected through document review and corroborated through feedback received through group discussions and stakeholder interviews across the spectrum of interviewed stakeholders of UNICEF’s efforts to create and cultivate good relationships across multiple levels of stakeholders. UNICEF works closely with relevant ministries– collaborating with central institutions with a particular emphasis on moving the reforms forward to advance rights of children. Stable relations with central government bodies help promote the reform agenda and gear reforms in the right directions, helping leverage resources and interventions. 152. The findings of this evaluation echo findings of the UNICEF Country Programme Evaluation, which found that UN Agencies see UNICEF as a good partner in terms of supporting the
55 UNICEF Country Programme Action Plan (CPAP) 2016-2020, p. 7 56 ToR, the relevant evaluation criterion was defined as ‘synergies’. Per request of UNICEF, this term was changed to assessment of “coherence”. 57 EQ 16: To what extent did UNICEF coordinate with development partners and other UN agencies to avoid overlaps, leverage contributions and catalyse joint work?
50
UNDAF process and also ensuring that potential overlaps are avoided, and synergies explored. In some sense, this task is relatively easy for the Country Office’s Education Section as not many UN actors engage in the education reform process. One exception is UNESCO, which focuses on vocational education and training as well as special and higher education. UNESCO also installed an open EMIS platform. 153. There are diverging views when it comes to UNICEF’s cooperation with other development partners. Most actors see UNICEF as a positive partner, contributing to the overall education reform process primarily serving as a bridge between the government and development partners. However, development partners noted the fact thatUNICEF operates somewhat independently of other development actors, in some cases leading to fragmentation of overall efforts. Overall coordination of development partners is mostly informal or lead by individual agencies or in some cases ministries, but it is not sufficient to ensure that coherence is strong. This creates challenges in terms of ensuring consistent assistance in selected priority areas and coordinated communication with national partners. The weak coordination mechanisms may serve as a potential area for future UNICEF technical support.
6.6.2. Internal coherence (EQ 1758)
154. There is a direct correlation between the way in which the UNICEF Country Office (CO) is structured and efforts to seek synergies. The CO is structured per thematic portfolios, so there is little incentive for cross-portfolio (cross-sector) integration and synergies. Document review and feedback received through group discussions and stakeholder interviews with UNICEF teams and national stakeholders point to good overall contributions but a lack of more comprehensive and holistic approach to protection and promotion of child rights (particularly rights and inclusion of children with disability) in education due to lack of synergies between UNICEF’s thematic portfolios (e.g. child protection and social policy section).Similar to external coordination with other development partners, key informants noted the fact thatthe Education Team operates rather independently within their own thematic area, while also raising issues of other sections individualist approaches. Overall, the lack of cooperation and a silo approach are justified for the specific focus and status of the reform (primary focus on normative framework during the current cycle) but the needs for more integration remain outstanding. Sector/thematic silos hinder the full achievement of outcome- and impact-level results when it comes to child rights and social inclusion in education sector and limit the catalytic effects of results within each sector individually as well. 155. The evaluation found a variety of instances of cooperation in which exchange occurred between sectors. Health and Education sections closely worked in developing pre/schools re-opening guidelines, based on the guidelines video-training was developed and filmed. In collaboration with Communication team series of activities for children and parents as well C4D/IEC materials for children, caretakers, teachers and communities were developed, broadcasted and distributed to each pre-school institution. In response to the COVID-19 pandemic, there has been good collaboration between the health, education and social policy sections on COVID-19 related issues and activities to ensure rapid response in terms of hygiene kits and other urgent educational needs. The need to tackle multidimensional needs of children and access to education requires a continuous collaborative approach between
58 EQ 17: To what extent the education programme contributed to cross-sectoral synergies with other sectoral programmes within UNICEF?
51
social policy, health, education and child rights monitoring components. More inter- sectoral integration across UNICEF-supported sectors should have been applied through strategic inter-linkages within UNICEF’s interventions.
52
7. Conclusions, lessons learned and recommendations
156. The following sections of the report present sets of cross-cutting conclusions, lessons learned, and recommendations that have derived from the findings and answers to the EQs.
7.1. Conclusions
7.1.1. Relevance
157. The relevance of the UNICEF programme is ensured, first and foremost, through its alignment with national priorities and the design of joint multi-annual work plans with the Government. UNICEF’s invests continuous efforts to address national priorities and align its regional and global priorities with national priorities. UNICEF has succeeded to maintain its relevance and strategic positioning in light of the political transformation and political changes in Uzbekistan. UNICEF closely consults with government in preparation and delivery of its technical assistance, which helps increase the relevance of delivered outputs. Openness to national priorities and potential changes in priorities is crucial for ensuring broad acceptance of assistance. It also shows that the development partner is genuinely interested in a country’s development efforts rather than his own priorities. 158. UNICEF was a driving force behind the development of national sector normative and policy framework, that is better tailored and responsive to the needs of children when it comes to access to quality pre-school and general secondary education. Through its own efforts and support measures, UNICEF produced relevant evidence through studies and reviews, which served as an evidence base to inform policy making processes. In light of the COVID-19 pandemic, UNICEF showed agility and responsiveness to the pandemic, offering timely and efficient assistance to organisation of alternative modes of learning through distant learning methods. Its persistent commitment to assisting policy and institution building, as well as the demand-driven nature of UNICEF’s engagement, has imprinted strong local ownership to UNICEF support. UNICEF proved to be good in leveraging partnerships with Uzbekistan’s educational Institutions. More could be done with academia and civil society sector as well as potential to leveraging private sector partnerships or South-South cooperation in the field of education. UNICEF’s positioning strengths lays in its capacity for coordination, convening, and leverage. This coordinating, leveraging, and convening potential could serve as a core foundation of all reform support initiatives of UNICE. In addition, UNICEF’s reputation as the evidence agency provides an opportunity for more systematic tracking of progress against both central- and local-level engagements in the next phase.
7.1.2. Effectiveness
159. UNICEF’s direct contribution to improvement of the normative framework for PSE and general secondary education has been provided through generating knowledge and evidence on the state of the education sector and child rights as well as direct support o policy making processes. Various UNICEF supported studies and reviews have helped the government to better understand the needs and priorities. The policy advisory and convening power of UNICEF contributed to drafting of relevant legislation and ESP organizing the education sector priorities and principles according to international standards. UNICEF’s engagement in capacity development was rather limited and focused primarily at central level institutions, which conforms with the current state of reform process. There is significant interest among stakeholders for deeper engagement of UNICEF in support to strengthening institutional and
53
individual technical capacity of educational institutions and professionals. UNICEF’s support to strengthening enabling environment has provided the foundation for continued work on individual technical capacity dimensions as well as further support to curriculum reform and equal access to education, which remains a priority for Uzbekistan. 160. However, while gender equality and attention to vulnerable populations are considered integral parts of UNICEF’s work at the national and sub-national levels, successfully reaching vulnerable populations remains a challenge. UNICEF could have done more to promote gender and equity issues, to ensure that much needed reforms in terms of deinstitutionalization and inclusive education are on the government agenda early on in initiation of reform process.
7.1.3. Efficiency
161. UNICEF’s engagement with the education sector reform in Uzbekistan was founded on the demand-driven approach to the prioritization and implementation of strategies. The responsiveness and flexibility to address raising needs of the government secured an excellent partnership with the government and responsiveness to priorities. UNICEF reacts quickly and efficiently to Government requests for assistance, which come in addition to the implementation of the programme. However, UNICEF encounters challenges with overstretching its portfolio, too wide scope of engagement which may threaten to lead to superficial engagement in some themes (e.g. review of textbooks). areas. There are observed limitations in the staffing structure vis-à-vis the nature and scope of the education portfolio, which affect the team to extended hours to fulfil all the undertaken obligations.
162. When it comes to financial management, UNICEF exercised diligence in programmatic and financial portfolio management. The budget was slightly decreasing over the last two years of implementation while the level of engagement increased, contributing to overstretching of resources. UNICEF could have done more to capture and report on its own contribution and leveraging power within the education sector. UNICEF CO reports (e.g. annual reports) present only a fraction of the actual engagement of UNICEF within various education reform initiatives, and rarely reflect and elaborate on outcomes associated with its interventions. Such outcome level analysis would have been beneficial to understand the full scope of UNICEF’s actual achievements to education sector reform.
7.1.4. Impact
163. UNICEF applied a demand driven approach to supporting the government during this programme cycle. This approach was viable taking into account the tectonic changes in the country’s political and education sector context and situation. The initial wave of reform has successfully resulted in improved legislative and normative framework for education sector, along with stronger institutional framework. Adoption and revisions of the education sector laws and establishment of the separate Ministry for PSE present strong foundations for further reforms. UNICEF’s efforts have brought notable contributions to these efforts, both in terms of building an enabling environment and institutional prerequisites for accessible and equitable education for all children. At the time of the evaluation, institutional capacities of government ministries and agencies have been strengthened to gear the reform towards full coverage. The impact of UNICEF’s assistance has thus far been limited, as the education reforms are just getting speed and depend on a number of factors, many of which are beyond UNICEF’s control. Education sector reform targets problems that are deeply rooted and require long- term efforts before measurable incremental results can be observed, particularly in terms of
54
operationalisation of the reform at local level, in more excluded communities. It is still early to conclude based on available evidence (which is limited) about the extent to which the short-term results achieved to date are sufficient to secure the continuation of reforms and its scaling down to local level. This is due to country contextual issues including the political and the socio-economic situation, financial prospects and the government’s absorption capacity and commitment to continue the reform process.
7.1.5. Sustainability
164. UNICEF’s considerable efforts to support the government to establish a stronger normative and institutional framework to lead the reforms have been effective and brought cumulative improvements at central level. However, sustainability of many interventions depends on government’s commitment and capacity to scale the reform down to the local level and ensure that full accessibility, quality and equity of the educational sector is achieved. Important reasons for reduced probability of sustainability include but are not limited to: staff turnover across central ministries and agencies; frequent leadership changes; system-wide absorption capacity for reform interventions, and too fast pace of reform which affects a lot of changes in reform priorities.
7.1.6. Coherence
165. UNICEF invested some efforts to maintain open partnerships with the government, though a rather individualistic approach was noted when it comes to other development actors. Its interventions are coherent with other types of interventions and there is sufficient diligence to avoid overlaps between development support interventions. Internally, education section but also UNICEF CO as a whole could have done more to ensure cross-sector and cross- portfolio synergies and joint interventions. This is a weakness of the current UNICEF programme structure composition.
7.2. Lessons learned
166. The evaluation results allow to draw several lessons about the planning and implementation of future UNICEF programmes. 1) Stable and reliable support to the government in situations of rapid reforms is critical to ensure main principles enshrined in CRC and those underpinning equality and equity in education are integrated in reform priorities. In response to fast-paced reforms that aim at overhauling education system in Uzbekistan, UNICEF provided unreserved support to building evidence base to inform policies and regulations, supporting institution and regulatory framework building, Reliability and commitment over a longer period of time enhanced government’s trust in UNICEF as a partner, providing also the space to UNICEF to raise critical issues from the perspective of a trusted friend. National partners underlined UNICEF’S calm and considerate approach to requests (including those labelled “urgent”). 2) Demand driven programmes need to come with a caveat of what is possible to provide within time and resource limitation. Possibility to be on the forefront of support to overhauling reforms in a country is attractive, as it offers opportunity to influence the direction, scope and breath of reform priorities and principles. However, demand driven nature provides a trap of doing too many things
55
at the same time, affecting overstretching of resources (human, financial), resulting in fragmented and ‘shallow’ support. Responding to the requests needs to be balanced to ensure that partnerships are maintained but are founded on realistic expectations about the extent to which assistance is feasible given resource limitations. In some cases, not addressing a request may be better that trying to cover too many issues at the same time. 3) Reform support initiatives including modelling of approaches or mechanisms need to integrate scale-up and institutionalisation mechanisms from the onset to ensure sustainability. The institutionalization of assistance is key to ensuring sustainability and bringing about long-term change. Institutionalization tends to face administrative and legal barriers, which need to be recognised and addressed in the risk-mitigation strategy which should underpin the implementation of the reform support initiative.
7.3. Recommendations
167. These recommendations derive from extensive consultations with key stakeholders across the evaluation process, analysis of documentation, and the findings and conclusions of this evaluation. The ET used interviews and GDs to generate ideas for improvement and for future UNICEF strategic directions in support to education sector reform priorities. Evaluation recommendations were further presented and validated through interactive discussions with UNICEF and key national stakeholders during the ERG meeting held in October 2020.The recommendations are structured to link the dimension of desired change, the strategic recommendations and thematic recommendations that could be considered possible options for achieving the desired change. The primary orientation of these recommendation is towards supporting increased linkages in the cascade from policy to the implementation of services to affected populations; and increasing shared understandings regarding sustainability and institutionalisation of best practices.
No. Recommendation Addressee Timeframe Strategic planning recommendations for UNICEF Education section 1 Consolidate the education programme to UNICEF Planning and ensure full support to deepening of reform Education implementation gains programme of the next in close programme cycle As noted in findings and conclusions sections, consultation 2021-2025 UNICEF has been a reliable partner to the with government in the reform process. UNICEF Country should use the momentum of the reform gains team (to and its strong strategic positioning to deepen the ensure reform through ensuring energy and plans are coherence) there to scale down the reform principles to the local level. Stronger and more focused education sector programmes should be designed with a selection of specific themes in which UNICEF has comparative advantage and which can be covered with UNICEF’s limited resources. In
56
No. Recommendation Addressee Timeframe particular, UNICEF should ensure that the educational policies and regulation that were supported are operationalised and implemented synchronously across the country. In this regard, further operationalisation of adopted regulations should be supported through advisory in development of bylaws, regulations, guides and related implementation manuals. Besides, further targeted institutional building support should be considered to strengthen capacities of central and regional institutions to implement educational policies. Interventions should be underpinned by consolidated programme intervention logic with clear baselines, particularly for capacity-building interventions.
Define a comprehensive monitoring plan to enable measuring outcomes of support in the next cycle. This will ensure that reporting covers all activities and their transformative potential. 2 Ensure cross-sector and cross portfolio UNICEF Planning and synergies are in place to provide for Country implementation holistic approach to child rights. team (to of the next ensure programme cycle Cross-sector and cross portfolio synergies can coherence) 2021-2025 have a significant impact on the results of support measures. UNICEF education section should invest continuous efforts to create synergies across sectors (especially with the social policy and child protection sections on issues of inclusive education, but also other section within their respective cross-sectional themes). These efforts should be periodically reviewed to ensure UNICEF Uzbekistan utilises its full potential in assisting the Government in the complex issue of education reform. Good communication and cooperation between the responsible managers are key to creating synergies which have the potential to add value to UNICEF’s contribution to education reform in the country. 3 UNICEF should invest in further UNICEF Planning and collaborative approaches with other Education implementation development partners programme of the next in close programme cycle UNICEF should continue investing in more consultation 2021-2025 collaborative approaches with other with national development partners to enhance its leverage partners and
57
No. Recommendation Addressee Timeframe when it comes to, in particular IFI investments in other education reform. development partners To do this, UNICEF should utilize its convener role by working closely with educational institutions (Ministries) to organize and convene formal donor coordination mechanisms. UNICEF’s reputation of being a ‘bridge’ between the government and development partners should serve as a driver to bring various actors together. Thematic recommendations 4 Continue supporting curriculum reform, UNICEF Planning and ensuring also that gender (as well as Education implementation disability) dimension of education is team in of the next integrated. close programme cycle partnership 2021-2025 Curriculum reforms are meant to address with the various education requirements, particularly national making education relevant and emphasize on government foundational, transactional, job-specific and partners digital skills. Nevertheless, the curriculum reform at its core need to ensure that adopted curriculum changes fully recognise equity and gender equality dimensions. UNICEF’s support to curriculum development cannot be wide due to human resource and financial constraints. A strong focus on curricula reform conceptual framework and the resulting implications for teaching and learning materials will allow UNICEF to achieve a concrete systemic impact on the education system. UNICEF should promote key principles and support the government to accomplish the result of curricula reform, i.e. to establish curricula meeting the following key requirements:
• They clearly identify the user of curricula (education administration, textbook authors, teachers, …) • They clearly specify learning outcomes in terms of subject-related knowledge and skills; • They clearly specify learning outcomes in terms of transversal skills; • They clearly specify learning outcomes in terms of students’ personal development
58
No. Recommendation Addressee Timeframe taking into account equity and gender equality dimensions; • They clearly specify the pedagogical approach to teaching taking into account equity and gender equality dimensions.
The result of the curricula and textbook policy review that has been conducted can be used to develop generic recommendations for curricula and textbook development. These recommendations will provide the authorities with clear and reliable guidance. If well designed, those recommendations can be used for the design of curricula and teaching and learning materials throughout all sectors and levels of the education of Uzbekistan. 5 Continue building local expertise through UNICEF Implementation investment in Training of Trainers (ToTs) Education of the next and their multiplication across the programme programme cycle country. 2021-2025
Now that the legislative and institutional foundations are in place, it is time to assist the education sector to reform itself. During its consolidation efforts, UNICEF should consider expanding its technical assistance to further build key institutions and educational professionals to embrace reform principles and measures at the subnational level through direct or cascaded trainings. This will allow the country to make significant progress toward both accessibility to and the quality and equity of education. Such measures are critical to ensure supported reform gains are sustained for benefit of each child. 6 Promote, advocate and offer technical UNICEF Planning and assistance to inclusive education and the Education implementation deinstitutionalization process team in of the next close programme cycle During the current programming cycle, partnership 2021-2025 UNICEF’s education programme did not focus with the on inclusive education extensively or the national deinstitutionalization advocacy. Current government government efforts to initiate interventions in partners this area should be strongly supported, towards advancement of the rights of the most socially excluded and vulnerable groups of children.
59
No. Recommendation Addressee Timeframe As the Government’s trusted partner, UNICEF is in a position that allows it to exemplify, through its programme, the mainstreaming of inclusive education in the process of education reform. This will require close cooperation and coordination of planning with the social policy and child protection sections. The role of UNICEF as advocate of rights of children in education should be used to drive the reform in this field. 7 Continue investing in modelling of UNICEF Implementation alternative educational services, Education of the next particularly for PSE team in programme cycle close 2021-2025 As the Government’s trusted partner, UNICEF partnership is well-positioned to propose alternative models with the for the provision of education. International best national practices, particularly models tested by other government UNICEF COs should be further explored and partners suggested to the Government. The introduction of alternative models will allow the Government:
• to further increase coverage of educational services (with the ultimate goal of 100% coverage in pre-school education); • to diversify educational services and thus prepare for situations in which traditional models of education can, for one or another reason, not be pursued due to various reasons (medical, environmental).
8 DRR and climate change resilience (also UNICEF Planning and COVID) Education implementation team in of the next The Government’s focus on DRR and its close programme cycle aspiration to promote DRR training through the partnership 2021-2025 education system enabled UNICEF to introduce with the concrete teaching and learning contents across national the curriculum. The institutionalization of DRR government through a variety of subjects is a good example partners of how sustainability of the UNICEF programme can be achieved. A similar approach can be taken toward teaching contents related to climate change, the danger it represents and the measures that can and need to be taken. DRR- related contents are also among those were the key role of girls can be underscored. Besides, while the situation resulting from the COVID-19
60
No. Recommendation Addressee Timeframe pandemic imposes serious limitations, it also opens up new opportunities and puts the government in a position where it is forced to examine and try alternative models of education.
61
8. List of Annexes
(Annexes are provided in a separate volume)
Annex 1: Evaluation Matrix Annex 2: Indicator overview Annex 3: List of Documents Annex 4: List of interviewed stakeholders Annex 5: Interview Guides Annex 6. Main legislative documents in the area of education in Uzbekistan Annex 7. Programme Theory of Change Annex 8. Ethical Review Clearance Letter Annex 9: Terms of Reference Annex 10. Informed Consent Forms Annex 11. Stakeholder map
62