Formative Evaluation of UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia EVALUATION REPORT NOVEMBER 2019 Formative Evaluation of UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia Evaluation Report Year of evaluation: 2019 Report submitted: November 2019 Commissioning organization: UNICEF Malaysia Authors: Lorraine Wapling and Nick Corby (Equal International); Wan Nuradiah (Ipsos Malaysia), Jessica Bruce (Ipsos MORI) Cover photo: UNICEF Malaysia/2013 © UNICEF Malaysia United Nations Children’s Fund Menara PJH Level 10, No. 2, Jalan Tun Abdul Razak Precinct 2, 62100 Putrajaya November 2019 The purpose of publishing evaluation reports produced by UNICEF Malaysia is to fulfil a corporate commitment to transparency through the publication of all completed evaluations. The reports are designed to stimulate a free exchange of ideas among those interested in the topic and to assure those supporting the work of UNICEF that it rigorously examines its strategies, results, and overall effectiveness. The contents of the report do not necessarily reflect the policies or views of UNICEF. The text has not been edited to official publication standards and UNICEF accepts no responsibility for error. The designations in this publication do not imply an opinion on the legal status of any country or territory, or of its authorities, or the delimitation of frontiers. The copyright for this report is held by the United Nations Children’s Fund. Permission is required to reprint/reproduce/photocopy or in any other way to cite or quote from this report in written form. UNICEF has a formal permission policy that requires a written request to be submitted. For non- commercial uses, the permission will normally be granted free of charge. Please write to UNICEF Malaysia at the address below to initiate a permission request. For further information, please contact: UNICEF Malaysia United Nations Children’s Fund Menara PJH Level 10, No. 2, Jalan Tun Abdul Razak Precinct 2, 62100 Putrajaya Email: [email protected] CONTENTS Acronyms ................................................................................................................................ iii Executive summary ............................................................................................................... iv Background ......................................................................................................................... iv Evaluation purpose, objectives and scope ........................................................................... v Methodology ........................................................................................................................ vi Main findings .......................................................................................................................vii Main recommendations ......................................................................................................viii 1. Introduction ..................................................................................................................... 1 1.1. Disability inclusion within UNICEF Malaysia .............................................................. 1 1.2. UNICEF Malaysia’s #ThisAbility Flagship Initiative within the national context ......... 2 1.2.1. Disability in Malaysia: Policy context ................................................................... 2 1.2.2. Perceptions of children with disabilities in Malaysia ............................................ 3 1.2.3. Inclusion in education for children with disabilities .............................................. 5 1.2.4. Official data on children with disabilities in Malaysia ........................................... 7 1.3. #ThisAbility Flagship Initiative strategic goals ............................................................ 9 2. Evaluation purpose, objectives and scope.................................................................11 2.1. Purpose .................................................................................................................... 11 2.2. Evaluation objectives ............................................................................................... 11 2.3. Scope ....................................................................................................................... 12 2.4. Key evaluation questions ......................................................................................... 13 3. Evaluation methodology ............................................................................................... 15 3.1. Design and methods ................................................................................................ 15 3.2. Sampling .................................................................................................................. 15 3.3. Data management and handling .............................................................................. 16 3.4. Approach to equity, gender and human rights ......................................................... 16 3.5. Ethical issues and considerations ............................................................................ 17 3.6. Methodological limitations ........................................................................................ 18 4. Findings ......................................................................................................................... 19 4.1. Effectiveness ............................................................................................................ 19 4.1.1. Satisfaction levels and public awareness .......................................................... 20 4.1.2. The effectiveness of Disability Equality Training ................................................ 25 4.1.3. The effectiveness of partnerships ...................................................................... 27 4.1.4. Impact of partnerships on empowerment and self-advocacy ............................ 28 4.1.5. The effectiveness of public advocacy campaigns .............................................. 29 4.1.6. Evidence of cross-sector work ........................................................................... 31 UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report i 4.1.7. The extent of gender inclusion ................................................................................ 32 4.2. Relevance ..................................................................................................................... 32 4.2.1. Alignment to Agenda 2030 ...................................................................................... 33 4.2.2. Alignment to Malaysia Vision 2020 ......................................................................... 33 4.2.3. Alignment to UNICEF’s Strategic Plan ................................................................... 34 4.2.4. Intersectionality and the voice of children with disabilities ...................................... 35 4.3. Efficiency ....................................................................................................................... 36 4.3.1. Human, technical and financial resourcing ............................................................. 36 4.3.2. The adequacy of partners’ capacity ........................................................................ 37 4.4. Sustainability ................................................................................................................. 38 4.4.1. Sustainability at national and sub-national level ..................................................... 38 4.4.2. Enabling and constraining factors for future work ................................................... 39 4.5. Utility of a Flagship Initiative.......................................................................................... 39 5. Conclusions ........................................................................................................................ 43 6. Lessons learned ................................................................................................................. 45 7. Recommendations ............................................................................................................. 47 Annex A: Evaluation Terms of Reference .................................................................................... 50 Annex B: Evaluation Matrix ......................................................................................................... 64 Annex C: Research Tools ............................................................................................................ 67 Annex D: Stakeholders interviewed............................................................................................. 78 Annex E: Document analysis map............................................................................................... 79 Annex F: Theory of Change ........................................................................................................ 92 UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report ii ACRONYMS BIM Borneo International Marathon C4D Communication for Development CPAP Country Programme Action Plan CPD Country Programme Document CPMP Country Programme Management Plan CRC UN Convention on the Rights of the Child CRPD Convention on the Rights of Persons with Disabilities DAC Development Assistance Committee DET Disability Equality Training