Formative Evaluation of UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia
EVALUATION REPORT NOVEMBER 2019 Formative Evaluation of UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia Evaluation Report
Year of evaluation: 2019 Report submitted: November 2019 Commissioning organization: UNICEF Malaysia Authors: Lorraine Wapling and Nick Corby (Equal International); Wan Nuradiah (Ipsos Malaysia), Jessica Bruce (Ipsos MORI) Cover photo: UNICEF Malaysia/2013
© UNICEF Malaysia United Nations Children’s Fund Menara PJH Level 10, No. 2, Jalan Tun Abdul Razak Precinct 2, 62100 Putrajaya
November 2019
The purpose of publishing evaluation reports produced by UNICEF Malaysia is to fulfil a corporate commitment to transparency through the publication of all completed evaluations. The reports are designed to stimulate a free exchange of ideas among those interested in the topic and to assure those supporting the work of UNICEF that it rigorously examines its strategies, results, and overall effectiveness.
The contents of the report do not necessarily reflect the policies or views of UNICEF.
The text has not been edited to official publication standards and UNICEF accepts no responsibility for error.
The designations in this publication do not imply an opinion on the legal status of any country or territory, or of its authorities, or the delimitation of frontiers.
The copyright for this report is held by the United Nations Children’s Fund. Permission is required to reprint/reproduce/photocopy or in any other way to cite or quote from this report in written form. UNICEF has a formal permission policy that requires a written request to be submitted. For non- commercial uses, the permission will normally be granted free of charge. Please write to UNICEF Malaysia at the address below to initiate a permission request.
For further information, please contact:
UNICEF Malaysia United Nations Children’s Fund Menara PJH Level 10, No. 2, Jalan Tun Abdul Razak Precinct 2, 62100 Putrajaya Email: [email protected] CONTENTS
Acronyms...... iii Executive summary...... iv Background...... iv Evaluation purpose, objectives and scope...... v Methodology...... vi Main findings...... vii Main recommendations...... viii 1. Introduction...... 1 1.1. Disability inclusion within UNICEF Malaysia...... 1 1.2. UNICEF Malaysia’s #ThisAbility Flagship Initiative within the national context...... 2 1.2.1. Disability in Malaysia: Policy context...... 2 1.2.2. Perceptions of children with disabilities in Malaysia...... 3 1.2.3. Inclusion in education for children with disabilities...... 5 1.2.4. Official data on children with disabilities in Malaysia...... 7 1.3. #ThisAbility Flagship Initiative strategic goals...... 9 2. Evaluation purpose, objectives and scope...... 11 2.1. Purpose...... 11 2.2. Evaluation objectives...... 11 2.3. Scope...... 12 2.4. Key evaluation questions...... 13 3. Evaluation methodology...... 15 3.1. Design and methods...... 15 3.2. Sampling...... 15 3.3. Data management and handling...... 16 3.4. Approach to equity, gender and human rights...... 16 3.5. Ethical issues and considerations...... 17 3.6. Methodological limitations...... 18 4. Findings...... 19 4.1. Effectiveness...... 19 4.1.1. Satisfaction levels and public awareness...... 20 4.1.2. The effectiveness of Disability Equality Training...... 25 4.1.3. The effectiveness of partnerships...... 27 4.1.4. Impact of partnerships on empowerment and self-advocacy...... 28 4.1.5. The effectiveness of public advocacy campaigns...... 29 4.1.6. Evidence of cross-sector work...... 31
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report i 4.1.7. The extent of gender inclusion...... 32 4.2. Relevance...... 32 4.2.1. Alignment to Agenda 2030...... 33 4.2.2. Alignment to Malaysia Vision 2020...... 33 4.2.3. Alignment to UNICEF’s Strategic Plan...... 34 4.2.4. Intersectionality and the voice of children with disabilities...... 35 4.3. Efficiency...... 36 4.3.1. Human, technical and financial resourcing...... 36 4.3.2. The adequacy of partners’ capacity...... 37 4.4. Sustainability...... 38 4.4.1. Sustainability at national and sub-national level...... 38 4.4.2. Enabling and constraining factors for future work...... 39 4.5. Utility of a Flagship Initiative...... 39 5. Conclusions...... 43 6. Lessons learned...... 45 7. Recommendations...... 47 Annex A: Evaluation Terms of Reference...... 50 Annex B: Evaluation Matrix...... 64 Annex C: Research Tools...... 67 Annex D: Stakeholders interviewed...... 78 Annex E: Document analysis map...... 79 Annex F: Theory of Change...... 92
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report ii ACRONYMS
BIM Borneo International Marathon C4D Communication for Development CPAP Country Programme Action Plan CPD Country Programme Document CPMP Country Programme Management Plan CRC UN Convention on the Rights of the Child CRPD Convention on the Rights of Persons with Disabilities DAC Development Assistance Committee DET Disability Equality Training FGD Focus group discussion IPG Integrated Play Group KAP Knowledge, Attitudes and Practices KII Key informant interview MoU Memorandum of Understanding NGO Non-governmental organization
OECD Organisation for Economic Co-operation and Development OKU Orang Kurang Upaya (less- abled person) SDGs Sustainable Development Goals ToC Theory of Change TOR Terms of Reference UN United Nations UNEG United Nations Evaluation Group UNICEF United Nations Children’s Fund WHO World Health Organization
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report iii EXECUTIVE SUMMARY
Background The World Health Organization (WHO) World Report on Disability estimates that globally over 1 billion people are living with disabilities, representing just over 15 per cent of the world’s population.1 Around one household in every four includes a person with disabilities, which means that over 2 billion people live with the impact of disability on a daily basis. Moreover, the prevalence of disability is growing due to ageing populations and global increases in chronic health conditions and non-communicable diseases.
The lives of children and young people with disabilities in Malaysia, like those with disabilities in many other countries, tend to be disadvantaged compared with their peers without disabilities.2 While the Government of Malaysia has shown commitment to improving the lives of persons with disabilities, including by ratifying the United Nations Convention on the Rights of Persons with Disabilities (CRPD) in 2010 and through its Disability Act (2008), gaps remain in the implementation of CRPD compliant policies and plans.3 There is a lack of data on the prevalence of disability among children and young people in Malaysia; their voices and lived experiences remain generally unheard, and the underlying drivers for stigma and discrimination still exist.
In 2013, the United Nations Children’s Fund (UNICEF) published a groundbreaking State of the World’s Children report, which focused specifically on children with disabilities. It highlighted the impact that disability can have on children’s life chances, even when their lives may already be difficult. It made a powerful case for demonstrating that children and young people with disabilities are at greater risk of being poor than their peers without disabilities, and they experience their poverty more acutely because of the additional barriers they face living with their impairments.
UNICEF Malaysia began planning its #ThisAbility Flagship Initiative in 2016. The goal was to make positive changes for children with disabilities in Malaysia so that they are free from discrimination and able to access services to support their growth, development and protection. The plan was to run the Flagship Initiative from 2017 to 2020, however, in early 2019, UNICEF Malaysia decided to no longer refer to its work on disability as a ‘flagship’, and committed to mainstreaming disability- inclusive programming.
This evaluation focused on activities carried out under the #ThisAbility Flagship Initiative between 2017 and 2019. UNICEF Malaysia did not define what a ‘flagship’ is but, by establishing a ‘flagship initiative’, it signalled intent to scale up its focus on disability, with opportunities to establish new partnerships, carry out more focused activities, and learn from the experience of implementing disability initiatives.
1 World Health Organization, World Report on Disability, WHO, Geneva, 2011.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report iv UNICEF Malaysia did not define a fixed budget, target number of beneficiaries or geographical coverage for the Initiative. At the time of the evaluation, UNICEF Malaysia did not appear to have plans to scale up or replicate the Initiative or its associated activities, but the activities it delivered will provide valuable learning for future disability-inclusion activities delivered by the Country Programme.
Evaluation purpose, objectives and scope UNICEF Malaysia commissioned this evaluation as a formative, forward-looking process. The specific objectives of the evaluation, as defined in the Terms of Reference (ToR) (see Annex A) were to: • analyse the strategies used to map out the trajectory of the #ThisAbility Flagship, and re- construct the Theory of Change (ToC) behind making positive changes for children with disabilities in Malaysia; • assess the extent to which the #ThisAbility Flagship has been appropriately designed, efficiently and effectively managed, and integrated across the Country Programme, including the incorporation of equity, gender equality and human rights considerations; • examine the evolution of UNICEF’s work on children with disabilities in Malaysia, and in particular the intermediate results achieved by UNICEF’s #ThisAbility Flagship, enabling and disabling factors, considering aspects of both reduction of discrimination as well as access to services – including advocacy and awareness-raising both at the national and sub-national level; and its likely sustainability; • analyse the utility of creating a flagship on specific focus issues to promote cross-sectoral work, including strengths and weaknesses.
This evaluation consequently looked at both the design and management of the initiative as well as assessing its programmatic effectiveness, relevance, efficiency and sustainability. The relatively nascent nature of the Initiative meant that the evaluation focused on finding evidence of outputs and early-stage outcomes, not on measuring its impact.
It was anticipated that the evaluation would identify areas of promising practice and lessons learned, and provide recommendations as to how UNICEF Malaysia might mainstream disability across its programme. These recommendations include a ToC that aims to assist UNICEF Malaysia to better meet its commitments towards championing the rights of children with disabilities.
The scope of this evaluation included all aspects of the #ThisAbility Flagship Initiative implemented since 2017. At the request of UNICEF’s Evaluation Specialist, the external evaluators paid particular attention to 10 select activities, including the Disability Equality Training (DET) and the Borneo International Marathon (BIM). The utility of the Initiative is also considered, but this was not the focus of the evaluation questions (see Figure 1).
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report v Figure 1: Evaluation questions
Organisation for Economic Cooperation and Development (OECD)/ Development Assistance Committee (DAC) criteria
Effectiveness of UNICEF’s strategy to children with disabilities and its implementation considering:
To what degree has UNICEF’s work on disability, and in particular through its #ThisAbility Flagship Initiative, contributed to the creation of positive conditions to reduce discrimination and enable access to services to support the growth, development and protection of children and adolescents with disabilities in Malaysia? Have there been any unintended results?
Relevance of the programme strategy and approach, considering:
How relevant is UNICEF’s work on disability and its #ThisAbility Flagship Initiative to the developmental challenges in Malaysia, and particularly attitudes and practices with regard to disability?
Efficiency of the programme implementation and management, considering:
To what extent and how has UNICEF mobilized and used its resources (human, technical and financial) and improved its cross-sectoral coordination to achieve its planned results for children with disabilities?
Sustainability in the programme, considering:
To what extent are the benefits and achievements of the #ThisAbility Flagship Initiative likely to continue after 2020 through national ownership and changes in terms of attitudes, knowledge and practices?
The primary users of this evaluation are expected to include UNICEF Malaysia, the Government of Malaysia and implementing partners. It is expected that the primary users will employ the conclusions and lessons learned to shape the forthcoming Country Programme Document (CPD) and to inform the development of new partnerships, programmes and initiatives. Secondary users are expected to include but not be limited to: other agencies working on disability-inclusive programmes for children and young people; UNICEF’s East Asia and Pacific Regional Office; UNICEF headquarters (especially the Disability and Child Protection Divisions); and the parents of and children with disabilities in Malaysia. It is envisaged that the secondary users of this evaluation will utilize the conclusions and lessons learned to strengthen disability inclusion in their work, identify potential synergies and/or identify opportunities for collaboration.
Methodology The methodological framework for this evaluation was based on the United Nations Evaluation Group (UNEG) Norms and Standards for Evaluation (2016) and informed by UNICEF guidelines. The evaluation followed a mixed-methods approach, non-experimental design to addressing the key evaluation questions. This evaluation drew upon three key strands of data collection: three focus group discussions (FGDs) with children and young people with disabilities and their parents/ carers; 42 key informant interviews (KIIs) with staff from within UNICEF, as well as those in partner
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report vi organizations, and a small number of government representatives; and a comprehensive desk review of internal UNICEF documents alongside available external materials. UNICEF provided a sample frame of participants for both the FGDs and the KIIs; the external evaluators used purposive sampling to ensure diverse perspectives on the #ThisAbility Flagship Initiative.
It is worth noting that a lack of outcome data and the mid-term nature of the evaluation meant that impact criteria were excluded. Furthermore, in the absence of financial data, value for money was not measured.
To fully examine the #ThisAbility Flagship Initiative, data was analysed against key barriers to inclusion, including those that were attitudinal, environmental and institutional. This analysis undertook to identify barriers experienced by children and young people with disabilities in the context of the Initiative; this was used to help determine whether the Flagship addressed key issues of relevance to programme participants.
Main findings In terms of effectiveness Most of those directly involved in activities implemented under the #ThisAbility Flagship Initiative found them to be interesting, exciting and of benefit to the children with disabilities who took part. However, since these activities were all one-off events, there was little opportunity for effecting lasting attitudinal change, and no clear evidence of effectiveness in reducing discrimination and enabling access to services by children and adolescents with disabilities. Due to the lack of an advocacy strategy, behavioural change activities, follow-on activities, and an absence of monitoring structures (including the collection of disability-disaggregated data), UNICEF Malaysia was unable to determine whether its interventions were having an impact on changing attitudes. There is a disconnect between the Initiative’s goals and objectives and the intent for the Initiative to promote mainstreaming of disability programming within the UNICEF Malaysia office, which has limited the effectiveness of this Initiative in all respects.
In terms of relevance UNICEF Malaysia has signalled a strong desire to work on disability inclusion within its CPD and country strategy documents. In turn, the Government of Malaysia has also made public its intentions to create more equal and inclusive services. Given that the Government has ratified the CRPD but has struggled with implementation, UNICEF’s intention to work on disability is highly relevant. UNICEF Malaysia has missed opportunities to better align its activities under the Initiative with broader influencing around the rights of persons with disabilities. This is because it has not made explicit use of international frameworks such as the CRPD in campaigning. The lack of an overall advocacy strategy has limited the success of individual events to have a broader impact.
In terms of efficiency Stakeholders generally observed the activities delivered under the Initiative as well resourced and well managed, with a high degree of technical support provided. Internally concerns were raised that the prioritization of resources, especially human resources, was insufficient for organization- wide change and that a more strategic approach to disability is required. The lack of systematic monitoring made it difficult for UNICEF Malaysia to assess whether its resources have been used in the most efficient way.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report vii In terms of sustainability Because the #ThisAbility Flagship Initiative was intended to be primarily an internal management device to increase disability work across the organization, it lacked the national ownership required for sustainability. While the Initiative has held many events and campaigns to change attitudes, knowledge and practices (as noted in previous sections), the lack of an advocacy strategy and any systematic monitoring data meant this evaluation could not determine their impact or sustainability.
Main recommendations Recommendation 1 To: Country Representative; Deputy Representative; Planning Monitoring and Evaluation Chief Priority: High Timeline: 6-months
UNICEF Malaysia needs to make a strategic commitment to disability inclusion by fully embedding disability into key planning documents with the expectation that children with disabilities will be included and monitored across all areas of programming. We encourage UNICEF Malaysia to start doing this in the short-term, beginning with the forthcoming strategic planning.
Recommendation 2 To: Country Representative; Deputy Representative; Communications Chief; Private Funding and Partnerships Chief; C4D Specialist Priority: Medium Timeline: 12-months
UNICEF Malaysia, starting immediately, needs to be much more explicitly rights-focused in its messaging around disability, and ensure its campaigns fully align with key international frameworks, including the CRPD.
Recommendation 3 To: Country Representative; Deputy Representative; Communications Chief; Programme Staff Priority: Medium Timeline: 12-months
UNICEF Malaysia, starting immediately, needs to provide more opportunities to meaning- fully engage children and young people (with and without disabilities) during the design of campaigns and messaging.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report viii Recommendation 4 To: Country Representative; Deputy Representative; Communications Chief; Programme Chiefs; Programme Staff Priority: High Timeline: 6-months
UNICEF Malaysia urgently needs to develop robust and rigorous safeguarding review processes for all activities involving children and young people with disabilities.
Recommendation 5 To: Country Representative; Deputy Representative; Gender Focal Point; Communications Chief; Programme Chiefs; Programme Staff Priority: Medium Timeline: 12-months
UNICEF Malaysia, starting as soon as possible, to routinely assess new and existing programmes to ensure gender and social inclusion norms and unequal power relations are considered in full.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report ix 1. INTRODUCTION
1.1. Disability inclusion within UNICEF Malaysia UNICEF Malaysia has placed itself at the forefront of UNICEF’s growing focus on disability. In 2013, the United Nations Children’s Fund (UNICEF) published a groundbreaking State of the World’s Children report, which focused specifically on children with disabilities. UNICEF Malaysia contributed to the report and supported its launch and dissemination. The report underlined the impact that disability can have on children’s life chances even when their lives may already be difficult. It emphasized that children and young people with disabilities are at greater risk of being poor than their peers without disabilities and experience their poverty more acutely because of the additional barriers they face living with their impairments.
In 2014, UNICEF Malaysia published a comprehensive study mapping the policies, programmes and interventions relevant to children with disabilities at that time. The study pointed to some significant systemic gaps around the provision of health, education, rehabilitation, protection and welfare services for children with disabilities and their families. UNICEF Malaysia led a Communication for Development campaign in 2014 (Disabled2Enable) and supported the establishment of the Malaysian Partnership on Children with Disabilities in 2015 and 2016, creating solid connections between UNICEF and the disability sector.
In 2016, UNICEF Malaysia commissioned a further study, this time focusing specifically on the experiences, challenges and aspirations of children and young people with disabilities. The resulting Knowledge, Attitudes and Practice (KAP) Survey on children with disabilities in Malaysia (2016)4 provided UNICEF Malaysia with a solid evidence base highlighting the lack of awareness of the rights of children with disabilities; the extent of the barriers to participation across services; and the impact that continued misconceptions around disability have on limiting the opportunities available for children and young people with disabilities to develop into confident, independent citizens.5
Building on these foundations, UNICEF Malaysia began to formulate its #ThisAbility Flagship Initiative in 2016, the goal being to effect positive change for children with disabilities across Malaysia. The intention was to run the Initiative from 2017 to 2020. However, in early 2019, UNICEF Malaysia decided to no longer refer to its work on disability as a’ flagship’ and committed to mainstreaming disability-inclusive programming. At the beginning of this evaluation, the Flagship Initiative was, therefore, at a midpoint in its intended implementation. By establishing a flagship initiative, UNICEF Malaysia committed to scaling up its focus on disability, with the opportunity to establish new partnerships, carry out more focused activities and learn from the experience of implementing disability programming. UNICEF Malaysia did not determine a budget, target number of beneficiaries or geographical coverage for the Initiative.
4 United Nations Children’s Fund (UNICEF), Childhood Disability in Malaysia: A Study of Knowledge Attitudes and Practices, UNICEF Malaysia, Kuala Lumpur, Malaysia, 2017
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 1 1.2. UNICEF Malaysia’s #ThisAbility Flagship Initiative within the national context Malaysia is a culturally diverse upper-middle-income country, home to around 30 million people of whom about 33 per cent are below 18 years of age.6 The Government aims for Malaysia to become a high-income country by 2020. Malaysia’s Vision 2020 plan promotes not just economic progress, but also commits to improving the quality of life for everyone through areas such as political stability and social justice. Malaysia’s National Transformation 2050 sets a target for Malaysia to become one of the top 20 nations in the world economically and socially. The Eleventh Malaysia Plan (2016–2020) also recognizes that as well as making economic progress, it is important to ensure that growth is inclusive and, as such, it needs to do better in being able to reach the most marginalized in society, including adults and children with disabilities.7
Within its Country Programme Document 2016–2020 (CPD), UNICEF commits to supporting Malaysia to make the transition to high-income status. UNICEF Malaysia’s Country Programme Action Plan (CPAP) and the Country Programme Management Plan (CPMP) recognize the Country Programme’s unique position by focusing on the most hard-to-reach children to ensure that no child is left behind. Disability is identified in both the CPD and the CPMP, in effect connecting national development targets with UNICEF’s priorities.
1.2.1. Disability in Malaysia: Policy context Several national social welfare and service policies help to define the rights of people with disabilities in Malaysia; however, as the most recent Universal Periodic Review report notes, Malaysia’s Constitution (1957) does not include any reference to children or adults with disabilities.8 The National Welfare Policy (1990) made provision for those with disabilities and was an early attempt to set out the Government’s intention to create a society that promotes ‘equalization of opportunities’. The National Social Policy (2003) specified that people with disabilities should ‘enjoy equal rights and full participation in society’ and focused on fifteen different areas where inclusion should be prioritized including, for example, rehabilitation, education, employment, social assistance and accessibility. Women and children with disabilities were also specifically mentioned as a target group.
The most significant step forward in recognizing the rights of people with disabilities camein 2008 with both the signing of the United Nations Convention on the Rights of Persons with Disabilities (CRPD) and the enactment of the Persons with Disabilities Act. This was the first time the Government formally acknowledged the need to reduce barriers to participation and adopted a rights-based approach to disability.9 Malaysia went on to ratify the CRPD in 2010 (albeit with reservations to Articles 15 and 18) which, given it has only ratified three of the ten core human rights treaties, at least demonstrates serious intent to promote inclusion.10
6 index mundi, Malaysia Demographics Profile 2019,
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 2 A key deficiency of the Persons with Disabilities Act however, is that it makes no specific provisions for children or young people with disabilities beyond ensuring their right to access education (Article 28) and their right to access recreation, leisure and sport (Article 32).11 In fact, there is no recognition of the impact of age or gender on disability at all. Another significant problem with the Act is the lack of monitoring and redress mechanisms should any party or the Government violate the rights of persons with disabilities. Similarly, Malaysia has yet to sign the Optional Protocol to the CRPD concerning procedures for individual communications and inquiry, which indicates the lack of legal mechanism in the event that the rights of persons with disabilities are violated. This is also reflected in the omissions within the Persons with DisabilitiesAct. 12
The National Council for Persons with Disabilities has the responsibility of overseeing national progress against plans and policies, but there are no penalties for non-compliance with the Disability Act. Malaysians with disabilities are unlikely to be able to bring a legal claim against the Government or service providers for violation of their rights.13 A recent paper on the effectiveness of the 2008 Act14 noted that in its nine years of enactment, no case has referenced it in relation to protecting the rights of persons with disabilities in registered court cases. Therefore, there have been calls for the Persons with Disabilities Act to be abolished, and proposals made for a Disability Discrimination Act as well as the establishment of a commission with enforcement powers to put in place legal measures for non-compliance.15
There is little direct protection of the rights of children with disabilities across the legislative framework. Malaysia enacted the Child Act in 2001 as part of its obligations under the UN Convention on the Rights of the Child (CRC). Although based on non-discrimination, the Federal Constitution and reservations in the CRC do not protect children with disabilities from discrimination.16 Since the Persons with Disabilities Act also fails to specifically reference children and young people with disabilities, there is a significant gap in the protection and promotion of their rights.
1.2.2. Perceptions of children with disabilities in Malaysia The KAP study, commissioned by UNICEF in 2016, revealed that the majority of participants (approximately 60 per cent) were not well informed about disability.17 The lack of awareness and understanding, as well as misconceptions of disability in Malaysia, appear to be key barriers restricting the opportunities available to children with disabilities in comparison with their peers without disabilities.
11 Persons with Disabilities Act 2008, Act 685, Government of Malaysia 12 Abdullah, Hanafi and Hamdi, ‘The rights of persons with disabilities in Malaysia’, 2017. 13 Section 41 in Part V of Persons with Disabilities Act 2008. 14 Abdullah, Hanafi and Hamdi, ‘The rights of persons with disabilities in Malaysia’, 2017. 15 Group: Do away with People with Disabilities Act https://www.thestar.com.my/news/nation/2018/07/02/group-do- away-with-disabilities-act-it-merely-serves- as-an-administrative-document-with-no-enforceme/ 16 Children with Disabilities in Malaysia: Mapping the Policies, Programmes, Interventions and Stakeholders https:// www.unicef.org/malaysia/UNICEF-Children_with_Disability_in_Malaysia_2014_lowres.pdf 17 United Nations Children’s Fund (UNICEF) Malaysia, Childhood Disability in Malaysia: A Study of Knowledge Atti- tudes and Practices, UNICEF Malaysia, Kuala Lumpur, Malaysia, 2017
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 3 Barriers at family level Attitudinal barriers have created anxiety among parents of children with disabilities, who hesitate or refuse to register their children as having a disability. Those who do attempt to access services for their children, such as enrolling them in mainstream schools, continue to face resistance or rejection.18 A recent study noted high levels of unmet needs especially among mothers of children with disabilities.This study found that parents of children with disabilities often find themselves quite isolated as they struggle to find appropriate information, financial and social support. Parents reported experiencing a wide range of challenges from lack of accessible transport, increased healthcare costs, inaccessible services and fragmentation of support, to poor coverage of specialist services, lack of experienced professionals and widespread negative attitudes.
Mothers, in particular, are expected to take responsibility for raising children with disabilities, adding considerable levels of psychological pressure to more general financial and logistical difficulties, which often changes the dynamics within families. The same study noted that it was mothers who were most often expected to take their children to hospital or habilitation/rehabilitation appointments or to accompany them to school. The mothers themselves reported feeling very unsupported by their families, communities and service providers, partly as a result of the negative attitudes prevalent towards disability, which they experienced as feelings of shame.19
Barriers at community level At the community level, disability is still subject to deep-rooted taboos. Strong religious beliefs that link disability to past misdemeanours can hamper people’s understanding of what causes impairments.20 While the majority of people recognize that disabilities have congenital and genetic causes or result from accidents or disease (77 per cent), a significant minority believe that disability is caused by the ‘will of God’ (10 per cent); the fault of parents (4 per cent); or fate/karma (2 per cent).21
Welfarist-based thinking is also pervasive and reflected in the language used to describe disability: terms such as cacat, a derogatory word meaning handicapped for example, is still widely used and is often synonymous with the official term Orang Kurang Upaya.22 There is still a strong assumption that having a disability implies a state of abnormality, dependency and a need for specialist provisioning. The provision of services implies an act of charity rather than being an entitlement and, as such, can be poorer in quality, regulation and availability.23
The level of discrimination in Malaysia varies for each type of disability. For example, UNICEF Malaysia (2016) found that children with psychosocial or behavioural disabilities, such as hyperactive disorders, face greater stigma compared with those with physical disabilities. There
18 Study shows that Malaysians don’t understand disability https://www.star2.com/people/2017/11/21/disability- prejudice-malaysia/ 19 Sukeri Surianti, Raishan Bakar, and Azizah Othman, ‘Barriers to unmet needs among mothers of children with disabilities in Kelantan, Malaysia: A qualitative study’. Journal of Taibah University Medical Sciences, 2017, vol 12(5), p.424-429
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 4 is a lack of social acceptance of children with these impairments, even if they manage to enrol in mainstreams schools.24
Nevertheless, there has been some transition in awareness and perception of children with disabilities. Use of the term ‘special’, which is typically used to refer to children with disabilities, is starting to be recognized as potentially segregating and contributing to the overall experience of exclusion.25
1.2.3. Inclusion in education for children with disabilities While the Ministry of Education uses the term ‘inclusive education’ (for example, in the Malaysia Education Blueprint 2013–2025) it does not actually follow an internationally recognized inclusive system. Instead, it operates a special education/integrated education model offering essentially three education pathways for children with disabilities:
• Special needs schools for specific impairments, such as visual and hearing impairments; • Special education integrated programme, which provides specialist classes within mainstream government or government-aided schools; • Inclusive education programme, where students with and without disabilities are assigned to the same classes within mainstream schools.
The Education Ministry’s latest target is to have at least 75 per cent of students with disabilities in the Inclusive Education Programme by 2023, with a zero rejection policy.26 However, even if this target is achieved, it still assumes almost a third of students with disabilities are not expected to be in mainstream education programmes and, given there are no disability-disaggregated statistics available for the prevalence of disability in school-age populations, it is a target that will be hard to track. Nevertheless, as the total investment allocated for education is high (RM42.9 billion) the Ministry has plans to carry out mid-term reviews of the Education Blueprint to identify areas for improvement.27
The Inclusive Education Programme is recognized as a model that demonstrates inclusive education is achievable. However, the challenges around implementation are significant, for example, the current shortage of qualified teachers and accompanying professionals such as therapists and psychologists; the lack of an adaptable curriculum; inadequate provision of assistive technology; and inaccessible school facilities. There have been proposals for case studies and data analysis to review existing practices to determine and evaluate any gaps identified. However, it would seem that the Inclusive Education Programme remains difficult to implement.28
24 Malaymail, Malaysians meaner towards children with mental disabilities, Unicef study shows, 16 October 2017
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 5 Barriers within the education system In addition to structural issues around implementation of the Inclusive Education Programme, there are also considerable attitudinal barriers to overcome in the education system. Parents themselves may be reluctant to enrol their child in a mainstream school, preferring instead the perceived security of a special school education. There is a strong sense that mainstream teachers would not be able to provide an appropriate learning environment, especially for children who have hearing or visual impairments. Moreover, when mainstream schools feel under pressure to meet government performance targets, they may refuse to enrol children with disabilities. Teachers themselves may be reluctant to include children with disabilities because of a lack of training, appropriate materials or classroom support.29
Most recently, to promote inclusive education, the Education Ministry produced a directive abolishing streaming in mainstream education (i.e. the placement of students in classrooms based on their academic performance). Theoretically, this should encourage schools to compete based on their level of inclusiveness instead of on academic achievements, but this requires further public awareness and information. The benefits to students both with and without disabilities need to be highlighted as many parents and teachers remain concerned that with the abolishment of streaming, fast learners could be restricted.30
It has been shown that inclusive education benefits students without disabilities, perhaps due to exposure to different ways of experiencing the world, but more awareness-raising is needed to break the deep-rooted stigma attached to disability.31 A systematic review of studies showed that, for the most part, children without disabilities did not experience any negative effects from being in the same classroom as their peers with a disability. In some cases, academic performance overall improved, although this only applied to classrooms that did not have multiple students with severe emotional and behavioural disabilities. Overall, studies suggest that it is the quality of instruction and the collaborative teaching of special and general education teachers that plays a more significant role in educational outcomes for children with disabilities than the type of classroom a student is placed in (inclusive or non-inclusive).32
Inclusion at preschool level There have been recent moves to make inclusive education mandatory at preschool level. The benefits of implementing inclusion at this level have been widely promoted, such as to enable early identification and intervention at a critical stage for children with disabilities to receive support and reduce challenges in later years. Early intervention could also be made affordable if inclusive preschool education is compulsory. The Government could subsidize the cost of special education, including private preschools. However, some challenges remain, such as the lack of special needs teachers to play a role as curriculum specialists by providing specialized instructions to supplement the basic curriculum in preschools. More special needs teachers must be trained, as well as training general education teachers.33 Nevertheless, plans are ongoing for
29 United Nations Children’s Fund (UNICEF) Malaysia, Childhood Disability in Malaysia: A Study of Knowledge Attitudes and Practices, UNICEF Malaysia, Kuala Lumpur, Malaysia, 2017
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 6 the Education Ministry’s special needs officers at the district level to train teachers at all public and private preschools.34
Various models of inclusive education have been put forward to raise awareness and demonstrate possibilities for successful implementation in Malaysia, particularly with the increasing number of children with autism. One example is the Integrated Play Group (IPG) model that was developed to encourage socialization and improve autistic children’s communication with other students in natural settings. IPG pilot programmes have shown positive outcomes, such as mutual friendships. The IPG requires individualization of education programmes tailored to each student’s needs and abilities.35
The bi-annual National Early Childhood Intervention Conference is another initiative to increase the awareness and acceptance of children with disabilities. The 2018 event (entitled: One Child, One Family, One Community) was attended by various stakeholders and provided guidance for current services for children with disabilities, which encompassed preschool and school inclusion, parent empowerment, training of professionals and balance to the private sector.36
1.2.4. Official data on children with disabilities in Malaysia Data on the numbers of children with disabilities in all contexts in Malaysia remain unreliable. The main estimates are still taken from the 2011 World Report on Disability, which suggests that 5.1 per cent of children (0–14 years) have a disability, with around 0.7 per cent of those children experiencing ‘severe disability’.37
Available data suggest that one in three children with disabilities is out of school, compared with one in seven of their peers without disabilities. Children with disabilities are likely to experience a range of barriers depending on factors such as degree and type of impairment, age, location, gender and ethnicity. For girls especially, having a disability can seriously impact on participation; globally girls with disabilities are less likely to be successful in education, to benefit from vocational training or to find employment, even compared with young men with disabilities or girls without disabilities.38 For example, estimates of primary school completion rates show just 42 per cent of girls with disabilities complete primary education compared with 51 per cent of boys with disabilities.39
Currently, Malaysia does not collect household-level data on disability, so there is no reliable measure of prevalence in children. Given a current 0–14 years population of around 9 million,40 global prevalence rates would suggest there could be around 440,000 children with disabilities in Malaysia,41 although rates of 10–16 per cent have also been reported.42
34 The Star, MoE vows to help special needs students thrive, 02 September 2018
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 7 It is worth noting that in 2018 UNICEF commissioned the translation of the Washington Group Child Functioning question set into Bahasa Malaysia to make it easier for the Department of Statistics to collect child disability data in its household survey. The Washington Group data sets have been designed specifically to enable data on disability to be sensitively collected within a rights-based framework; UNICEF has worked on producing the Child Functioning sets to ensure more accurate recording of disability in children.43
The latest statistics on the official website of the Department of Social Welfare report that there were 114,933 persons with disabilities under the age of 18 in 2016.44 The reason the rates appear so low is that this data reflects the numbers of children officially registered as having a disability as part of the OKU identity card system. OKU or Orang Kurang Upaya is the official terminology used to describe people with disabilities and roughly translates to ‘less-abled person’.45
Under the Persons with Disabilities Act (2008), a person is defined as disabled when they ‘.... have long-term physical, mental, intellectual or sensory impairments, which, in interaction with various barriers, may hinder their full and effective participation in society.’ However, for OKU registration, the Government is much more prescriptive and defines seven types of impairments that can be categorized as a disability: hearing (which it further refines by decibels of hearing loss); sight; speech difficulties (which in children have to be assessed at 5 years or above); physical; learning disabilities (which includes Down’s syndrome, autism and other learning difficulties); mental (which includes diagnosed psychiatric conditions); and multiple (where there is more than one significant impairment present).46
Since this is a voluntary process with minimal incentives, and one that can be difficult to navigate or even feel shameful, many parents choose not to register their children. Efforts are underway to encourage registration of children with disabilities47 and to highlight the importance of registering.48 These include initiatives to increase awareness of the benefits of registering such as financial assistance, subsidies, job attachments and places at education institutions.49 However, until there is a national survey using internationally recognized data collection tools (such as the Washington Group question sets) rates of disability in children will remain under-reported.
Consistent with a lack of prevalence data is a general lack of empirical research relating to the voice and lived experiences of children and young people with disabilities in Malaysia. In 2016, UNICEF Malaysia commissioned an important population-based KAP Survey to try to address this knowledge gap; this provides some key observations that are of relevance to this evaluation.50
43 See: http://www.washingtongroup-disability.com/ 44 Department of Social Welfare Statistics Report 2016 ,
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 8 This survey noted that the medical model is still the dominant approach to understanding disability in Malaysia, and has helped maintain a situation where children and young people with disabilities remain: ‘passive recipients of services’51 with no sense of them having individual agency. Generally, people do not focus on barriers to inclusion but instead tend towards viewing the impairment as being the barrier to participation. Therefore, while government policy could be said to align with a rights-based approach, this is not well understood at community level. The underlying drivers for stigma and discrimination remain essentially unchallenged.
1.3. #ThisAbility Flagship Initiative strategic goals Working within and responding to the national context outlined in the previous sections, the strategic direction for UNICEF Malaysia’s #ThisAbility Flagship Initiative is provided in a Working Paper and accompanying Monitoring Framework documents.52 The overall goal of the Initiative is ambitious and forward-looking and underpinned by six key objectives that focus on changing the lives of children with disabilities (see Figure 2).
Figure 2: Goals and objectives of the #ThisAbility Flagship Initiative
Goal
UNICEF commits to making positive changes for CwD [sic] in Malaysia so they are free from discrimination and able to access services to support their growth, development and protection.
Objectives
a) Children with disabilities have increased access to high quality services; b) Children with disabilities do not experience abuse and discrimination; c) Children and adolescents with disabilities are empowered to advocate for themselves; d) General public are aware of children with disabilities; e) Children with disabilities are socially included; f) Children with disabilities can contribute positively in society.
51 Ibid. p. 85 52 #ThisAbility Flagship working paper; #ThisAbility Flagship taskforce monitoring framework (2017–2020).
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 9 The #ThisAbility Flagship Initiative spans the period from 2017 to 2020, therefore this evaluation is akin to a mid-term review. The chronological order of key events is shown in Figure 3:
Figure 3: Summary of key events under the #ThisAbility Flagship Initiative 2017 2018 2019 • Launch of #ThisAbility • #ThisAbility truck • What’s so special about Flagship Initiative • Disability Equality Training special education call to action • Disability Actors Database • Public Advocacy compiled Campaign for Borneo • Public Advocacy International Marathon Campaign for Borneo • Sime Darby Properties – International Marathon Inclusion Champion • RAGE videos showcase • Inclusive Playground Paralympians inception – design • ManUnited campaign • We are all special launch in combination inclusive carnival with Borneo International • What’s so special Marathon international day of • Disability Equality Training persons with disability • Petrosains – Inclusion campaign Champion • What’s so special about • #ThisAbility Makeathon Ima? U-Report pick a path • Launch of KAP Survey story • NGO Impact Academy • National conference on inclusion at National Early • Empowering Teachers and Childhood Intervention Trainers training Council
Some key events were nationwide, but many were concentrated in Kota Kinabalu and Sabah. A financial budget for the Initiative was not available and, while supported across UNICEF Malaysia, the Initiative was primarily led by a Disability Consultant.
Part of the mandate for this evaluation was to assist UNICEF Malaysia to create a Theory of Change (ToC) for future disability inclusion activities undertaken by the Country Programme. The resulting ToC is presented in Annex F. There was not a results framework available for consideration by this evaluation and therefore the ToC developed by the external evaluators is the first attempt to articulate a logical results chain for UNICEF Malaysia’s disability inclusion work. UNICEF Malaysia will need to decide how to use the ToC.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 10 2. EVALUATION PURPOSE, OBJECTIVES AND SCOPE
2.1. Purpose The primary purpose of this formative, forward-looking evaluation was to draw lessons from a focused analysis of the #ThisAbility Flagship Initiative to identify how UNICEF Malaysia might transition from being an organization that undertakes disability-focused activities, to one that fully embeds disability in its work.53 The evaluation sought to identify and highlight knowledge generated by the Initiative to inform UNICEF Malaysia’s current and emerging programmes.
The relatively nascent nature of the Initiative meant that the evaluation focused on finding evidence of outputs and early-stage outcomes, not on measuring the impact of the Initiative.
The primary users of this evaluation are expected to include UNICEF Malaysia as well as the Government of Malaysia and implementing partners. It is expected that they will utilize the conclusions and recommendations to shape the forthcoming CPD and to inform the development of new partnerships, programmes and initiatives.
Secondary users are expected to include but not be limited to: other agencies working on disability- inclusive programmes for children and young people; UNICEF’s East Asia and Pacific Regional Office; UNICEF headquarters (most especially the Disability and Child Protection Divisions); and the parents of and children with disabilities in Malaysia. Secondary users of this evaluation could utilize the conclusions and lessons learned to strengthen their own disability-inclusion work, identify potential synergies and/or identify opportunities for collaboration.
This evaluation recognizes that the findings can be used to bring UNICEF Malaysia to the forefront of disability inclusion, setting an example for key partners in Malaysia and other countries, as well as providing fresh evidence and impetus to support UNICEF’s strategic focus on equity.
2.2. Evaluation objectives The four specific objectives of the evaluation, as defined in the Terms of Reference (ToR) (see Annex A) were to:
• analyse the strategies used to map out the trajectory of the #ThisAbility Flagship, and re- construct the Theory of Change (ToC) behind making positive changes for children with disabilities in Malaysia; • assess the extent to which the #ThisAbility Flagship has been appropriately designed, efficiently and effectively managed, and integrated across the Country Programme, including the incorporation of equity, gender equality and human rights considerations;
53 During the finalization of this report UNICEF staff emphasized that the #ThisAbility Flagship Initiative was “established as a platform to encourage sections to work collaboratively on disability-related initiatives. It was designed to test out innovative ways of approaching disability in UNICEF’s work. It also intended to build confidence of staff and partners to be disability-inclusive in their approach to designing and implementing initiatives and programmes”.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 11 • examine the evolution of UNICEF’s work on children with disabilities in Malaysia, and in particular the intermediate results achieved by UNICEF’s #ThisAbility Flagship, enabling and disabling factors, considering aspects of both reduction of discrimination as well as access to services – including advocacy and awareness-raising both at the national and sub-national level; and its likely sustainability; • analyse the utility of creating a flagship on specific focus issues to promote cross-sectoral work, including strengths and weaknesses.
2.3. Scope At the request of UNICEF’s Evaluation Specialist, the external evaluators paid particular attention to the following 10 activities (see Figure 4) when analysing the effectiveness, relevance, efficiency and sustainability of the aims and objectives in the Initiative Working Paper and when addressing the key evaluation questions (see Section 2.4). A discussion around the utility of the #ThisAbility Flagship Initiative is detailed in Section 4.5.
Figure 4: #ThisAbility Flagship Initiative events analysed by the evaluation team
Event Dates Location Stakeholders Disability Equality Training 2017/18 Kuala Lumpur and Included participants Kota Kinabalu from: Petrosains, UNICEF, Tandemic, Sime Darby Property and Sabah Art Gallery Inclusive Journalism 2017 Kuala Lumpur, Journalists/members of Workshop Kota Kinabalu and Malaysian Press Institute Kuching #ThisAbility Public Advocacy 2017/18 Nationwide General public Campaign Knowledge, Attitudes and 2017 Kuala Lumpur Children with disabilities Practices Survey launch/ dissemination #ThisAbility Makeathon and 2017 Nationwide, event Children with and without video hosted in Kuala disabilities Lumpur Non-governmental 2017 Nationwide, event Nationwide disability and organization (NGO) Impact hosted in Kuala women’s rights civil society Academy Lumpur organizations The Borneo International 2017 International event General public, children with Marathon #ThisAbility hosted in Kota and without disabilities themed Kinabalu Corporate Inclusion 2017/18 Kuala Lumpur and Petrosains and Sime Darby Champions Shah Alam Property We Are All Special: A Carnival 2018 Kota Kinabalu, Children with and without for All Sabah disabilities Inclusive Play, 2018/19 Shah Alam Children with and without disabilities
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 12 2.4. Key evaluation questions The key evaluation questions (see Figure 5) and the evaluation criteria were described in the original ToR (see Annex A) and are based on UNICEF Malaysia’s priorities. The questions were mapped to relevant data sources in the Evaluation Matrix (see Annex B).
Figure 5: Evaluation questions
Organisation for Economic Cooperation Sub-questions and Development (OECD)/ Development Assistance Committee (DAC) criteria
Effectiveness of UNICEF’s strategy • How satisfied are the children and their to children with disability and its caregivers with the quality of the services they implementation considering: have received through UNICEF’s partners? • To what degree has Disability Equality To what degree has UNICEF’s work on Training been effective in increasing disability, and in particular through its knowledge of children with disabilities #ThisAbility Flagship Initiative, contributed and changing understanding of disability to the creation of positive conditions to from an individual to a social/rights-based reduce discrimination and enable access to perspective? services to support the growth, development • How effective have partnerships with the and protection of children and adolescents private sector been at heightening their with disabilities in Malaysia? Have there awareness of inclusion and promoting their been any unintended results? adopting of strategies which model inclusion good practice? • How effective have partnerships with the private sector been in empowering children and adolescents with disabilities to advocate for themselves and in ensuring that they are socially included? • To what extent have public advocacy campaigns and behaviour change interventions been effectively designed and implemented to increase public awareness on children with disabilities? • To what extent has the #ThisAbility Flagship Initiative enabled cross-sectoral work to increase access to high quality services for children with disabilities? • Has gender equality been taken into account in the planning and implementation of the #ThisAbility Flagship Initiative?
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 13 Relevance of the programme strategy and • How relevant and appropriate has UNICEF’s approach, considering: work on disability been to the Agenda 2030, and should it be further adjusted to align with the How relevant is UNICEF’s work on disability Sustainable Development Goal (SDG) roadmap? and its #ThisAbility Flagship Initiative to the • How relevant and appropriate has UNICEF’s developmental challenges in Malaysia, and work on disability been to Malaysia Vision 2020 particularly attitudes and practices with regard to and National Transformation 2050? disability? • To what extent is #ThisAbility Flagship Initiative aligned with the new UNICEF’s Strategic Plan? • Has the design of the #ThisAbility Flagship Initiative taken into account the different needs of girls and boys, according to their age, gender, ethnicity, religion or faith, disability category, state and other social identities, especially the most vulnerable? • Have the voices of children and adolescents with disabilities taken into account throughout?
Efficiency of the programme implementation • To what degree have UNICEF resources and management, considering: (human, technical and financial) been sufficient in implementing the activities under the #ThisAbility To what extent and how has UNICEF mobilized Flagship Initiative? and used its resources (human, technical • How adequate are the capacities of UNICEF’s and financial) and improved its cross-sectoral organizational and individual partners to engage coordination to achieve its planned results for on the rights of children with disabilities? children with disabilities?
Sustainability in the programme, considering: • To what degree has the #ThisAbility Flagship Initiative thus far contributed to the generation of To what extent are the benefits and sustainable capacities at the national and sub- achievements of the #ThisAbility Flagship national levels? Initiative likely to continue after 2020 through • What are the enabling as well as constraining national ownership and changes in terms of factors that are likely to influence future cross- attitudes, knowledge and practices? sectoral work on focus issues?
During the inception phase, in which an initial round of key informant interviews (KIIs), internal document review, and an on-site visit took place, it became clear that the evaluation would not be able to assess efficiency as it relates to financial management. This was because UNICEF Malaysia did not hold financial or output monitoring information that could be used as the basis for such an assessment. Therefore, the evaluation process did not carry out a value for money assessment.
Moreover, the evaluation’s approach to assessing the effectiveness and sustainability of the Initiative was also relatively limited. This was partly due to the early-stage nature of the Initiative, but mostly because of the disconnect between the stated aims of the Initiative – as a mechanism for providing UNICEF Malaysia with a platform for developing its internal capacity for work on disability inclusion – and the goals and objectives as outlined in the #ThisAbility Flagship Initiative Working Paper (see Figure 2). This issue is explored in full in Section 4.
While being aware of the above constraints, the evaluation team made no changes to the evaluation objectives (or evaluation questions) following a request by UNICEF to use the same objectives and evaluation questions as those in the ToR to the fullest extent possible.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 14 3. EVALUATION METHODOLOGY
3.1. Design and methods The evaluation followed a mixed-methods approach, non-experimental design to addressing the key evaluation questions. This evaluation drew upon three key strands of data collection:
I. Three focus group discussions (FGDs) were conducted with 11 pairs of children and young people with disabilities and their parents/carers. The groups consisted of five male and six female children with disabilities (one participant aged 0–5 years, five participants aged 6–12 years and five participants aged 13–19 years). The participants included those with cognitive, mobility and hearing impairments. A key aim of this research strand was to provide insights into the experiences and barriers faced by children with disabilities and their families. Focus groups were conducted in Klang Valley, Sabah, and Sarawak, where UNICEF Malaysia activities have been most prominent, or where there is intention to do more. Each session took around 120 minutes and was based on a topic guide agreed with UNICEF Malaysia during the inception phase.
II. Forty-two key informant interviews (KIIs) were completed with staff of partner organizations (including mainstream organizations and disability organizations), UNICEF staff (including staff working on any disability-related programming, finance, and human resources) and three Government representatives. The interviews were conducted face- to-face, or via telephone or Skype; all interviews were guided by semi-structured interview questions that reflected the evaluation questions and were aligned with the evaluation matrix, utilizing a topic guide agreed with UNICEF Malaysia during the inception phase. Each interview was recorded, transcribed and translated to English (when necessary) to aid further analysis.
III. A desk review of internal (UNICEF) documents provided by UNICEF Malaysia enabled the external evaluators to familiarize themselves with all aspects of the #ThisAbility Flagship Initiative. A desk review of external documents (including national and international strategies) supplemented the review of internal documents, providing greater context around disability inclusion in Malaysia and the relevance of the #ThisAbility Flagship Initiative. The document review was guided by the evaluation matrix (see Annex B), which identified the potential use of each document for addressing the evaluation questions. In total, the evaluation team reviewed more than 180 documents.
The evaluation team agreed the above three strands of data collection with UNICEF through an inception report that went through several rounds of review and revision; this was agreed with UNICEF’s Evaluation Specialist before data collection.
3.2. Sampling Participants for the FGDs were selected from a sample frame of Initiative participants provided by UNICEF Malaysia. The external evaluators used purposive sampling of children and their parents/carers to ensure diverse perspectives on the Initiative, using pre-defined criteria agreed with UNICEF Malaysia. This ensured a mix of participants across gender, age and type of impairment. Participant children’s ages ranged from 5–19 years; overall 5 female and 6 male children participated; and types of children’s impairments included cerebral palsy, hearing
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 15 impairment, attention deficit hyperactivity disorder (ADHD), autism, and Down’s syndrome. All participants were living in, or close to, the respective market centres (Klang Valley, Kota Kinabalu and Kuching); UNICEF selected these areas because they were where the key activities that were the focus of this evaluation took place (see Section 2.3) and/or they were the three main areas of focus communicated by UNICEF.
UNICEF’s Evaluation Specialist and UNICEF Malaysia’s Disability Specialist provided a sampling frame for the KIIs. The external evaluators used purposive sampling to ensure different viewpoints were sufficiently captured from a range of programme implementers, partners and other stakeholders. The external evaluators sought to interview more Government stakeholders but they were not available, despite support from UNICEF Malaysia to contact them and an extension to the data collection period. The full list of interviewees is available in Annex D.
3.3. Data management and handling The external evaluators put a series of processes and systems in place with the specific intent of ensuring high quality data collection and analysis:
• Triangulation of information: In preparing this evaluation report, the external evaluators triangulated data collected. FGDs and KIIs were recorded and the transcribed data was analysed by using an evaluation matrix (see Annex B) derived from the main evaluation questions. The external evaluators also held regular team discussions to jointly discuss and triangulate information, validate findings, and identify areas for further exploration throughout the data collection process. • Quality assurance: Interviews and FGDs were undertaken by two team members and were immediately followed by debrief sessions. A Director from Equal International and from Ipsos MORI reviewed all evaluation tools and outputs. The evaluation approach and associated tools were finalized following multiple reviews by UNICEF’s Evaluation Specialist. UNICEF Malaysia also established a Reference Group consisting of selected UNICEF staff and external partners to support the delivery of the evaluation.
3.4. Approach to equity, gender and human rights The external evaluators followed the CRPD in promoting a human rights-based understanding of disability. Children and young people with disabilities are defined as those aged 18 years and under:
‘.... who have long-term physical, mental, intellectual or sensory impairments which in interaction with various barriers may hinder their full and effective participation in society on an equal basis with others.’
– (Article 1, CRPD).
This concept of disability moves away from the traditional individual, medical-based perspective characterised by a focus on physical deficits (impairments), to one that encompasses the attitudinal, environmental and institutional barriers that limit or exclude children and young people with impairments from participation. In the context of this evaluation process, it meant a rights- based approach to the analysis was taken, starting from the premise that children and young people with disabilities have the right to participate in and benefit from all development actions.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 16 To fully examine the #ThisAbility Flagship Initiative, data was analysed against key barriers to inclusion, those that are attitudinal, environmental and institutional. This analysis undertook to identify barriers experienced by children and young people with disabilities in the context of the Initiative, and was used to help determine whether or not the Initiative addressed issues of relevance to programme participants.
Taking a rights-based approach encompasses not just disability but also issues around gender, age and other contextual factors. The focus of this evaluation was to understand the situation in terms of barriers, rather than needs, and to look at the situation from the context of both the rights holders and the duty bearers. Conclusions and lessons learned stem from a focus on equity and equality rather than needs and problems and are set within the context outlined by the CRPD.
3.5. Ethical issues and considerations This evaluation was completed with reference to the United Nations Evaluation Group (UNEG) Norms and Standards (2016), with particular attention given to Norm 6 on ethics and Norm 9 on human rights and gender equality. Reference was also made throughout this evaluation to the Revised Evaluation Policy of UNICEF (2018); UNEG Ethical Guideline for Evaluation (2008); Ethical research Involving Children (2013); UNICEF Procedure for Ethical Standards and Research, Evaluation and Data Collection and Analysis (2015); UNICEF-adapted UNEG Evaluation Reports Standards (2017); the Global Evaluation Reports Oversight System Handbook (2017) and UNEG Guidance on Integrating Human Rights and Gender Equality in Evaluation (2014). Ipsos is also a member of the Market Research Society, which has a Code of Conduct governing the design and implementation of all its research. In drawing upon the above guidance documents, the external evaluation upheld the following key principles:
• Independence: The evaluation is free from bias. • Impartiality: The external evaluators have given a balanced view of both strengths and weaknesses observed. • Credibility: The evaluation is based upon reliable data and observations. • Conflicts of interest: The external evaluators avoided any conflicts of interest that threatened the credibility of the evaluation. • Accountability: External evaluators recognize their responsibilities and have accurately represented themselves and their data, and exercised prudence throughout the evaluation.
Informed consent protocols ensured that all participants received a comprehensive briefing on the evaluation aims and how the data collected would be used/stored, and they were then required to provide explicit consent to proceed. Information sheets and consent protocols, including a privacy note, were prepared in collaboration with UNICEF’s Evaluation Specialist before data collection.
The external evaluators ensured strict data confidentiality throughout the evaluation. For example, all KIIs were given a unique code and this final evaluation report contains only anonymized data that cannot be attributed to specific individuals. Personal data files held by the external evaluators will be destroyed upon formal completion of this evaluation. Primary data was collected by Ipsos Malaysia, and Ipsos MORI ensured compliance of all research and analysis processes with its Business Excellence System.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 17 3.6. Methodological limitations The completion of this evaluation was subject to a series of challenges and limitations that are important to note:
• Absence of strategic and monitoring information in programme documentation: There was an absence of baseline information, a lack of systematic monitoring data and no impact indicators related to the objectives. The internal documents provided by UNICEF for review gave no sense of scope or scale to the #ThisAbility Flagship Initiative, no geographical information or description of the target population and no disaggregation by intersectional markers (such as gender, ethnicity, geographical location or age). This made it challenging to fairly measure the progress of the Initiative against the recommended Development Assistance Committee (DAC) criteria of relevance, effectiveness, efficiency and sustainability, as stipulated in the ToR. In particular, the Initiative documentation included no overall programme-based outcome or impact reporting, nor baseline statistics against which this evaluation could track progress of the Initiative. Instead, the activity-level focus of the Monitoring Framework meant this evaluation had to focus on the delivery of individual activities. • Lack of financial data:Financial data, including an overall budget for the Initiative, were not available to measure efficiency or value for money. • Reliance on qualitative data collection: The absence of programme-based outcome or impact reporting meant that this evaluation relied upon the collection of primary data through KIIs and FGDs to assess progress. A reliance on research participant information for programmatic analysis can be restricting insofar as participants tend to have limited experiences. The external evaluators sought to mitigate this through saturation of potential participants and triangulating the data. A reliance on research participant information also hinges upon the availability of participants (as already stated many Government representatives were not available for interview despite multiple attempts by the external evaluators and UNICEF Malaysia) and a sufficient number of participants (resource constraints particularly limited the number of FGD participants). • Objectivity: At UNICEF Malaysia’s request, a member of UNICEF’s Evaluation Team joined the external evaluators in the FGDs. The proximity of the UNICEF staff member to a key strand of data collection increased the risk that participants’ responses would be biased and therefore objectivity could be compromised; this risk was discussed with UNICEF Malaysia. In ensuring participants’ informed consent, it was explained throughout recruitment protocols and during the FGDs that a representative from UNICEF Malaysia’s Evaluation Management Team was present as an observer. The UNICEF staff member did not participate in the discussion; they were there only to observe. The approach adopted by the external evaluators to triangulate information and validate findings did not include the member of UNICEF’s Evaluation Team.
While not a methodological limitation, it should also be noted that it is the firm belief of the external evaluators that UNICEF’s request for this evaluation to stick rigidly to the evaluation questions and criteria in the ToR has restricted its potential value. Recommendations made by the external evaluators at the end of the inception stage and informally accepted by UNICEF Malaysia at that time, were abandoned at the request of UNICEF’s Evaluation Specialist. Had the external evaluators been afforded the scope to conduct this evaluation as an evaluability assessment, they could have taken the iterative and loose construction of the Initiative into account more and focused analysis on providing strategic guidance for UNICEF Malaysia’s future work around disability inclusion, and an associated monitoring and evaluation strategy. As it is, the evaluation was constrained by the summative nature of the key evaluation questions, which required assessment of the Initiative against its intended long-term objectives.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 18 4. FINDINGS
4.1. Effectiveness Box 1: Effectiveness questions
How satisfied are the children and their caregivers with the quality of the services they have received through UNICEF’s partners?
To what degree has Disability Equality Training been effective in increasing knowledge of children with disabilities and changing understanding of disability from an individual to a social/rights-based perspective?
How effective have partnerships with the private sector been at heightening their awareness of inclusion and promoting their adopting of strategies which model inclusion good practice?
How effective have partnerships with the private sector been in empowering children and adolescents with disabilities to advocate for themselves and in ensuring that they are socially included?
To what extent have public advocacy campaigns and behaviour change interventions been effectively designed and implemented to increase public awareness on children with disabilities?
To what extent has the #ThisAbility Flagship Initiative enabled cross- sectoral work to increase access to high quality services for children with disabilities?
Has gender equality been taken into account in the planning and implementation of the #ThisAbility Flagship Initiative?
Summary: Most of those directly involved in activities implemented under the #ThisAbility Flagship Initiative found them to be interesting, exciting and of benefit to the children with disabilities who took part. Since the activities were all one-off events, however, there has so far been little opportunity for effecting lasting attitudinal change and no clear evidence of effectiveness in reducing discrimination and enabling access to services by children and adolescents with disabilities. The lack of an advocacy strategy, behavioural change activities, follow-on activities, and an absence of monitoring structures (including the collection of disability-disaggregated data), mean that UNICEF Malaysia is currently unable to determine whether its interventions are having an impact on changing attitudes. The disconnect between the #ThisAbility Flagship Initiative goals and objectives and the intent for the Initiative to promote mainstreaming of disability programming within the UNICEF Malaysia office has limited the effectiveness of this Initiative in all respects.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 19 This evaluation concluded that activities carried out under the Initiative cannot be fairly evaluated against effectiveness as anticipated by the evaluation questions. The evaluation questions require a focus on the Initiative’s anticipated results, which are to ‘reduce discrimination and enable access to services to support the growth, development and protection of children and adolescents with disabilities in Malaysia’. The objectives in the Working Paper54 are entirely focused on systemic- level outcomes requiring changes to the way services are designed and delivered.
Initiative activities were not focused on addressing service design and delivery and instead appeared to focus on a narrow range of attitudinal and behavioural changes. UNICEF had not collected data that would enable it to demonstrate how each of the activities contributed to any of these higher-level objectives. Staff and partner informants confirmed that individual activities were not monitored at outcome or impact level, with the absence of this level of reporting leading to some degree of frustration among staff respondents. A key point to emphasize here is that there was no baseline and no programme targets or milestones against which to measure the degree of effectiveness achieved by UNICEF Malaysia in reducing discrimination and access to services more broadly, as stated in the evaluation questions.
4.1.1. Satisfaction levels and public awareness Evaluation question: Summary of findings: How satisfied are the children and their • Events were fun, exciting and of personal benefit caregivers with the quality of the services they • Lack of follow through resulted in a loss of have received through UNICEF’s partners? momentum for effecting longer-term change
Because the children with disabilities and their families that participated in various Initiative activities were not in receipt of any direct services provided by UNICEF Malaysia or its partners, this part of the evaluation question could not be answered. Therefore, rather than ‘access to services provided by UNICEF’, this evaluation focused on satisfaction with the activities delivered under the Initiative, and identification of any outcomes resulting from these activities.
Across all regions, parents of children with disabilities were found to have some general awareness of Initiative activities. It has to be noted, however, that this is partly because the participant referral process was done through UNICEF Malaysia and doesn’t necessarily indicate awareness in general.
Sabah In Sabah, the Borneo International Marathon (BIM) and the We are all special Carnival were the activities that generated the most responses in FGDs. However, in FDGs in Kuala Lumpur and Kuching, these Sabah-based events were not well known, suggesting the impact was geographically limited.
The 2017 BIM was the first opportunity for UNICEF to launch a public awareness campaign under the #ThisAbility Flagship Initiative (see Box 2). Several key disability organizations in Sabah were involved, including Cheshire Homes Services, Special Olympics, Child Sabah and Sabah Society for the Deaf. Their engagement was primarily focused on promoting the participation of children with disabilities from within their constituencies. Several key informants noted that while they had been asked to provide inputs to the organising committee around positive messaging, their suggestions were not taken up.
54 #ThisAbility Flagship working paper, p.2
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 20 Box 2: Case study illustration of a public awareness event
Borneo International Marathon The Marathon first took place in 2008 and since then has grown in size and reputation. Around 10,000 people now take part in this annual event. Part of the mandate of the Marathon is to raise money for local charities, including those supporting disability. UNICEF has been linked to the Marathon since its inception, but it was not until 2017 that it formally sponsored the event with a #ThisAbility disability-inclusive theme. Primarily a public awareness campaign event, UNICEF also sought to showcase this as an inclusive event through the addition of a new 3km race for children with and without disabilities and their families. Between March and the event in May, mainstream and social media exposure activities helped promote inclusion and challenge biases against children with disabilities in sport. On the day, there were themed t-shirts for runners and speeches and medals for all the child participants. Several local disability organizations mobilized their stakeholders, resulting in around 300 children with disabilities taking part.
UNICEF/UN068401/Ho
Key informants from Sabah reported positive experiences of the BIM. Parents commented that the Marathon had made a difference to them personally, as they got to know other parents who also had children with disabilities. In general, people thought the event was fun, well organized and a good opportunity for children with disabilities to do something outdoors. Parents regarded a lack of outdoor accessible space as a significant barrier to inclusion for children with disabilities; in this regard, they saw the BIM as a positive opportunity for their children to get involved in outdoor activities.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 21 In terms of impact, however, the consensus among all groups of informants was that this was a one-off activity that had minimal impact on challenging deep-rooted attitudes. Parents expressed particular disappointment that there had been no ongoing attempts to engage them or to actively follow up on the BIM with any tangible programmes for children with disabilities. There was also concern that people with experience of disability (people with disabilities or the parents of children with disabilities, for example) were not actively engaged in the design or monitoring of the aware- ness-raising elements of this event. In commenting about the impact of the BIM, one focus group participant explained:
“Running is the only thing about this event. Other than that, there is nothing... we could not share the knowledge.”
Key informants were not convinced that the BIM had any impact on changing social attitudes beyond those individuals who participated in the event. One organization felt quite strongly that the event had not challenged stereotypes or removed barriers sufficiently for them to consider taking part again.
It is worth noting that the BIM itself has continued to offer the 3km race for children with disabilities and has in more recent years expanded this to include children with and without disabilities. Several accessibility audits have been carried out that have resulted in gradual improvements to the route and race facilities (for example, including provision for a toilet accessible to persons with disabilities). UNICEF continues to sponsor the event with new themes each year.
The 2018, the We are all special Carnival in Sabah was another awareness-raising event that brought together a wide range of partners to run various events (see Box 3). The Carnival included the Sabah State Library, Sabah Art Gallery, the Jesselton Philharmonic Orchestra and several local disability organizations. Given that working with children with disabilities was new to a number of these agencies, UNICEF provided staff representatives from Sabah State Library, Sabah Art Gallery, and the Jesselton Philharmonic Orchestra with a one-day Disability Equality Training (DET) session.
Box 3: Inclusion in the Sabah Carnival
We are all Special – Sabah Carnival In November 2018, Sabah Welfare Services Department, Sabah Community Services Council and UNICEF ran the first ‘Carnival for All’ in Sabah. Another key activity under the #ThisAbility Flagship Initiative, this one-day public event was an opportunity to promote social inclusion, showcase the abilities of children with and without disabilities, challenge negative attitudes and help increase levels of self-esteem and self-confidence among children with disabilities. While the Carnival itself was a regular annual event run by local government, until 2018 it had always been exclusively designed for children with disabilities. This time, the Carnival was purposively designed to include children with and without disabilities participating together in a variety of different activities. Throughout the day, there were fun activities for the children to take part in including arts and crafts workshops; music workshops; a reading corner; team games; a chess competition; and a selection of information stands.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 22 Feedback from informants in relation to the Carnival was positive, with the impact felt particularly by those who hosted events. In many instances, this was the first contact people had had with disability, and it led to some level of awareness-raising, especially of improving physical accessibility and in realizing the potential of children with disabilities.
Parents provided positive feedback that their children had the opportunity to mix with their peers without disabilities. However, while parents were very enthusiastic about all the different activities their children could take part in, because things were taking place in different locations (unlike the BIM), they felt they didn’t have the same opportunity to meet and mix with other parents. One significant barrier all parent focus groups raised repeatedly was the lack of support services for them as key carers of children with disabilities. When UNICEF runs inclusive events or activities, usually parents get to meet each other, providing a type of support key carers say they lack; parents commented that UNICEF events have proved beneficial in helping them to gain peer support, learn more about their children’s disabilities and feel slightly less isolated. Unfortunately, because events have not been systematically followed-up with ongoing engagement activities, opportunities to assist parents of children with disabilities have been missed.
Opinions were also mixed as to the Carnival’s impact on improving the social inclusion of children with disabilities more broadly. While it had some effect on raising awareness among the public who attended, it appears to have done little to change perceptions more broadly. Since many people who had been invited already had children with disabilities, the impact on changing wider social attitudes was agreed to have been limited. Parents suggested greater impact could be gained by hosting events in more public spaces, such as in shopping malls, where people without children with disabilities can be more easily reached.
There was also a call for more activities in which children with and without disabilities participate equally. One example raised was the possibility for more outdoor activities like camping or hiking, where there was a genuine effort to be inclusive of all children and where their children’s specific requirements could be accommodated and not treated separately.
In summary, what parents expressed gaining the most from these two key events was being able to come together and share experiences and to see their children taking part in community events which were relatively barrier-free. Given the isolation that families with children with disabilities can face, the value of community participation should not be underestimated. However, these were not the primary stated aims of the activities themselves; rather, these represented positive unintended outcomes.
Kuala Lumpur The Initiative activities most frequently encountered by parents in Kuala Lumpur were the 2017 launch of the KAP Survey, the Makeathon (see Box 4), and participation in the design of an inclusive playground with Sime Darby Property. On the whole, parents had received fewer opportunities to engage directly with the Initiative in Kuala Lumpur because there had been limited events.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 23 Box 4: Disability-inclusive Makeathon
The Makeathon In 2017, UNICEF Malaysia trialled a new activity called a ‘Makeathon’, which was aimed at all children and young people. This was a nationwide competition calling for innovative ideas to assist children with disabilities to participate more fully in their schools, homes and public spaces. Around 20 teams including children with and without disabilities, parents and others submitted ideas that were shortlisted, so that eventually eight teams were invited to participate in the Makeathon event. Petrosains, a well-known national science and education centre in Kuala Lumpur hosted the event, with each team being paired up with private sector experts (including engineers, product designers, 3D modellers and occupational therapists). A range of ideas, from a remote-controlled wheelchair-mounted 360-degree camera, to a wheelchair accessory bag with integrated table-top and a portable computer keyboard underwent prototype design during the Makeathon event itself; the most successful was then developed over a further two-month period. The eventual winning design (a finger-grip device) has since been tested for use by occupational therapists in hospitals and playgroups, although only around 10 products have so far been produced.
As an event, the Makeathon was a popular activity among those who were involved (either as organizer, host, technical specialist or participant). Parents whose children had participated welcomed the activity because they felt it really engaged their children’s creative skills, while at the same time it raised awareness of some of the challenges their children face. Key informants who had been part of the design and delivery of the activity also expressed how much they had learned from engaging directly with young people with disabilities; overall it was viewed as an exciting experience.
The extent to which this activity has been successful in increasing public awareness and challenging negative attitudes, however, was questioned by all groups of informants. The one-off nature of the activity was a key issue in limiting its impact. With no subsequent campaign or follow-up series of events, the momentum both the Makeathon and the launch of the KAP study generated was considered to be lost.
Overall, informants felt that the Makeathon had real potential, but that its effectiveness was diminished due to its one-off nature. Parents, in particular, expressed that while activities like these are exciting, unless there is sustained follow up, there is little chance they will have any effect on changing perceptions or making life more inclusive for their children.
A broad range of key informants and parents also perceived a lack of communication and transparency from UNICEF, with questions raised about how decisions had been made during the Makeathon process. Some of those who participated reported being left wondering what, if anything, might happen next. It was clear from discussions with a range of different stakeholders that UNICEF Malaysia’s communication strategies were not enabling stakeholders to feel engaged.
Informants also reported some frustration with communication regarding UNICEF Malaysia’s work with Sime Darby Property. The Sime Darby Property partnership was a much broader Initiative activity aimed at influencing a significant property development company to build more disability- inclusive spaces (including playgrounds). UNICEF took advantage of Sime Darby’s openness to explore ideas around inclusive spaces and the fact they were willing to set aside an area of land within one of their new developments to trial an inclusive playground. As part of this process, UNICEF invited several disability stakeholders and children with disabilities to take part in a design
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 24 workshop. This resulted in the submission of plans which could, if approved by local government, become the blueprint for Malaysia’s first inclusive playground. While a potentially significant achievement, informants expressed disappointment that they did not receive feedback and were not kept informed of the process, which made them think that nothing had resulted from their discussions, given the playground had not been built at the time of this evaluation.
Some informants had taken part in discussions and dissemination events in relation to the KAP Survey and gave the study itself a high rating. Once again, the informants raised concerns about what had happened since it was launched. There was a sense among informants that, while the messages and recommendations were excellent; nothing seemed to have been done to follow them up at time of the evaluation.
Sarawak UNICEF Malaysia didn’t undertake any Initiative activities in Kuching following a strategic decision in 2019 to focus its interventions in one sub-national geographical location. The intention was to maximize the impact of limited resources and to create evidence that could support changes at both national and sub-national levels across the country.
Parents were aware of Initiative activities that had taken place in other parts of the country, but they pointed to the limited geographic reach of the Initiative and any spillover benefits.
“I don’t see any programme by UNICEF in Kuching other than donation drive.... In our national support group, we know that they have a lot of programme in Kuala Lumpur.”
Parent groups in the area have organized activities for children with disabilities themselves. They described coming together to run an early intervention programme for children under the age of six and a youth group for young people, which includes an element of ‘job empowerment’. Parents in Kuching expressed a degree of dissatisfaction and were somewhat resentful that UNICEF Malaysia had not yet made any commitments to support the area under the Initiative. They nevertheless described UNICEF Malaysia as a good organization that is committed to helping to improve the situation for children (information they had gleaned from fundraising material). There was agreement that UNICEF Malaysia had the power to effect change, but surprise that it took the organization until 2016 to start talking about children with disabilities (suggesting no awareness of UNICEF Malaysia’s work regarding disability outside of the Flagship Initiative). As one participant highlighted:
“Another thing I want to know is why UNICEF just decided to start focusing on disabilities in 2016? There is disabilities many years back, but why in 2016? Your organization is so big, but why you started so late?”
4.1.2. The effectiveness of Disability Equality Training Evaluation question: Summary of findings: To what degree has Disability Equality • Good quality delivery, appropriately targeted Training been effective in increasing • Lack of focus on children with disabilities resulted knowledge of children with disabilities in a missed opportunity for increasing knowledge and changing understanding of disability around children’s rights from an individual to a social/rights-based perspective?
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 25 An expert disability trainer with a disability provided DET. Key informant interviews concurred that the quality of the training was high, and it was well received. Records showed that 103 certificates were awarded to participants in 2017, and 74 awarded in 2018; indicating some 177 people have benefitted from DET. Certificates were awarded to personnel in Petrosains, UNICEF, Tandemic, and Sime Darby Property. In addition, over 90 teachers in Sabah took part in a two-day workshop on ‘Empowering teachers to teach children with disabilities’ in November 2017, during which DET was provided; and KIIs revealed DET had taken place at the Sabah Art Gallery. Over 250 individuals have therefore likely benefitted from DET provided under the Initiative since 2017.
In general, staff and partners found the DET to be very useful in helping them to understand disability from a different perspective. People felt more comfortable and confident in how to manage, collaborate and interact with people with disabilities. There were some who felt that without any follow-up activities the impact of the training was limited. Some also felt that it could more usefully be broadened to take place in schools so that children can learn more about disability.
Some also sensed that the DET was too focused on the needs of persons with physical disabilities, while others noted that it was at quite a basic level, and therefore may not be appropriate for all stakeholders. From the DET slides, the focus of the training appears to have been on disability in general; it was advertised as being especially useful to employers who might want to make their workplace more accessible for employees with disabilities.55
Participants also felt the training was not specifically tailored towards children, with the lead trainer having no previous experience with children with disabilities. This means that while DET was focused on moving people towards a rights-based understanding of disability in general, its role in raising awareness or increasing knowledge of children with disabilities has been limited. This seems to have been a missed opportunity for UNICEF.
This evaluation considers that UNICEF has been ambitious in its expectations around the effect of one-off training on attitudinal change, since this type of activity works best as part of a wider strategy with more targeted messaging and support on how to implement inclusion. It is a good approach, but it needs to be implemented as part of a wider programme where behaviour change expectations are mapped out and monitored. UNICEF Malaysia has not articulated clearly enough how these initial training activities will be used to promote longer-term attitudinal and behaviour change.
In their engagement with different stakeholders, the evaluation team noted the extent to which the language they used was often still charity/medical focused. It was reasonably common for participants to talk in terms of ‘disabled children’ and ‘normal children’ (rather than children with or without disabilities), or to consistently refer to children with disabilities as ‘special’ (rather than equal). Other phrases included use of ‘the handicapped’ and ‘the disabled’ or even ‘PWDs’ (to mean People with Disabilities). These stakeholders are the ones who have had the most direct contact with UNICEF’s messaging, suggesting that there is still considerable work to be done to promote a rights-based approach to disability.
In summary, the DET appears to have been an effective way to introduce people to a social-model approach. However, since the Initiative did not specify a target group for influencing knowledge and understanding of a rights-based approach, nor establish a baseline measure, it is not possible for this evaluation to comment on the degree to which the training has been more broadly effective.
55 DET only flyer for UNICEF 2017.pdf.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 26 4.1.3. The effectiveness of partnerships Evaluation question: Summary of findings: How effective have partnerships with • Creation of corporate inclusion champions was a well- the private sector been at heightening conceived strategy with plenty of future potential their awareness of inclusion and • Limited evidence (at this point in the partnerships) of promoting their adopting of strategies the partnerships resulting in the adoption of inclusive which model inclusion good practice? strategies
Partnerships are regarded as being ‘at the heart of the UNICEF mandate’ and are defined as ‘voluntary and collaborative relationships between various parties, both public and non-public, in which all participants agree to work together to achieve a common purpose or undertake a specific task...’.56 UNICEF Malaysia refers to partnerships in different ways. One activity under the Initiative was to establish partnerships with two corporate ‘inclusion champions’, with the objective of ensuring children with disabilities are socially included.57 UNICEF signed a Memorandum of Understanding (MoU) with Petrosains in 2017 and another with Sime Darby Property in July 2018, which formalized their relationship as corporate inclusion champions. Both of the agreements were two years in duration and were non-financial.
While not formalized relationships, UNICEF Malaysia also refers to organizations such as Sabah Art Gallery, Sabah State Library, Jesselton Philharmonic Orchestra and the various disability- focused NGOs they have worked with as ‘partners’. Such organizations described their own relationship with UNICEF as ‘a partnership’, even though no formal relationship was established. In addition, Tandemic is a corporate organization contracted as an implementation partner under a Partnership Cooperation Agreement and has been linked with UNICEF since 2016. Its partnership was based on a set of key deliverables linked to the Makeathon event with a fee attached. This section only focuses on the #ThisAbility Flagship Initiative corporate inclusion champions.
The MoU with Petrosains included DET for Petrosains staff; support for the launch of the KAP Survey report; hosting of a Makeathon event; use of the innovations lab and pilot-testing of inclusive tours and communications materials.58 This partnership was considered to have undergone quite a long ‘courtship’ process, but both sides were clearly in agreement once the relationship was formalized with an MoU in 2017. Since becoming a corporate inclusion champion, Petrosains reported gaining moral and technical support (in the form of DET) from UNICEF and felt UNICEF had been a good ‘sounding board’.
The Sime Darby Property MoU was signed in July 2018 and included DET for Sime Darby Property staff; joint work on an inclusive playground; and a joint awareness campaign focused onthe importance of inclusion, with the UNICEF-Sime Darby Property partnership utilized as an example of good business practice.59
The Petrosains partnership has resulted in increased staff awareness around disability with a reported ‘change of mindset’, but there is less confidence over the extent to which this has changed the way Petrosains engages with the public more generally. Internally, Petrosains has plans for some of the new science space to be used to help raise the profile of disability; it is
56 E/ICEF/2012/18 Report on the implementation of the strategic framework for partnerships and collaborative relationships, UNICEF, 2012, 57 Disability Flagship Working Paper (n.d.) p.2 58 Memorandum of Understanding between Petrosains SDN BHD and UNICEF, Sept 2017, p.2. 59 Memorandum of Understanding between Sime Darby Property Berhad UNICEF, July 2018, p.2
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 27 exploring the development of a sign language app to provide signs in real time, which will be tested at Petrosains once the prototype becomes available. Overall, there is a sense that the MoU with UNICEF has helped raise the profile of disability, but that disability inclusion still needs to be institutionalized, including changes to organizational policies, which will inevitably take time and require a much broader buy-in from across the company.
The longer-term impact on the ability of Petrosains to be more disability-inclusive will inevitably take more time, although with the positive relationship that UNICEF Malaysia has established, things should continue to improve. Petrosains is currently working on making changes to improve its overall accessibility, but at this stage it is too early to assess the partnership’s effect on improving their ability to model inclusive good practice.
When UNICEF signed an MoU with Sime Darby Property, the latter was already developing a relatively inclusive new township. Sime Darby Property referred to this development as ‘multi- generational’; houses were being designed with step-free access, extra wide doors on the ground floor, and some extra-large bathrooms, making the houses wheelchair friendly. UNICEF was able to influence the design of the township further by suggesting Sime Darby designs and builds an inclusive playground space. Through the consultation process, UNICEF was able to guide the design so that it became genuinely inclusive (rather than just accessible) drawing on the ideas and interests of young people with disabilities themselves.
Currently, there is not enough information on the partnership with Sime Darby Property to assess whether the inclusive playground work will affect disability-inclusive practices more broadly. Given Sime Darby Property is already implementing elements of Universal Design into their housing developments, there is a good chance the inclusive playground design will be utilized in the future.
4.1.4. Impact of partnerships on empowerment and self-advocacy Evaluation question: Summary of findings: How effective have partnerships with • Some evidence to suggest interactions between children the private sector been in empowering with disabilities and corporate inclusion champions have children and adolescents with increased disabilities to advocate for themselves • No evidence to show the partnerships have resulted in the and in ensuring that they are socially empowerment of children and adolescents to undertake included? self-advocacy
This evaluation found no evidence that partnerships had been effective in empowering children and adolescents with disabilities to advocate for themselves and in ensuring they are socially included. Partly this is due to the nature of the partnerships: the MoUs were not designed specifically to encourage or enable corporate inclusion champions to work with children and young people with disabilities on advocacy issues. There is no clear linkage between empowerment and self-advocacy results and the partnership activities. Moreover, UNICEF Malaysia itself does not yet work in a way that promotes self-advocacy by children and young people with disabilities and, therefore, the activities that it currently supports are not yet providing the kind of platform that could act as a model for corporate inclusion champions.
Nevertheless, what these partnerships have achieved via the key activities (the Makeathon and the inclusive playground) is to enable consultations to happen between children and young people with disabilities and the corporate partners, which has helped the children to express their needs more directly. These events have provided a very small number of children and young people
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 28 with disabilities the opportunity to engage in activities where their views, experiences and ideas were given prominence, and some of the children were supported to express themselves in media articles and short films.
4.1.5. The effectiveness of public advocacy campaigns Evaluation question: Summary of findings: To what extent have public advocacy • Public awareness-raising has been central to the work of campaigns and behaviour change the Initiative with good use made of existing public events interventions been effectively designed • The one-off nature of events and the lack of UNICEF’s and implemented to increase public own monitoring data means their effectiveness at awareness on children with disabilities? increasing public awareness cannot be evaluated
Public awareness-raising campaigns and behaviour change activities (rather than advocacy) have been a core component of UNICEF’s #ThisAbility Flagship Initiative. All the activities carried out under this Initiative were focused on raising awareness, with the assumption that by promoting positive messages and imaging of children and young people with disabilities, the attitudes and behaviour of persons without disabilities would change. Specific events such as the BIM, the Sabah Carnival and the Makeathon were used as vehicles around which to build public campaigns that helped to raise the profile of children with disabilities and challenge assumptions by showcasing what children and young people with disabilities can do.
Considerable efforts were put into events such as the Sabah Carnival, which involved many different sections within UNICEF Malaysia working together to create a positive experience for children and young people with disabilities, as well as promoting the inclusion and visibility messaging of the #ThisAbility Flagship Initiative (see Box 3).
The Man United awareness campaign provides a key example of a well-designed messaging platform. This was commissioned by UNICEF and completed before the #ThisAbility Flagship Initiative, but was used quite extensively throughout 2017 and 2018 to support UNICEF’s messaging around inclusion. This two-minute film uses sport as the entry point, promoting a shared interest in football among young people with and without disabilities. Children and young people with disabilities were a key part of the design of the video, and were identified through members of the Malaysia Partnership for Children with Disabilities (established with the support of UNICEF pre- 2017).
As many key informant groups highlighted, however, UNICEF Malaysia lacks systematic monitoring of awareness-raising events. Without pre- and post- event data it is not possible to monitor the impact of campaigns on the public’s awareness of children with disabilities. The KAP Survey Report actually provides UNICEF Malaysia with a baseline from which it could design a behaviour change strategy, and there are three key indicators that the Report suggests could be used to monitor the impact of interventions (see Box 5).60 However, while the Disability Flagship Working Paper references the KAP Survey Report, it does so only in terms of identifying a broad set of key messages, ‘...designed to fill the knowledge gap outlined in the KAP Survey Report findings and recommendations.’61 These key messages do not reference the indicators suggested by the KAP Survey Report, which would seem like a missed opportunity for UNICEF Malaysia.
60 Moore K., and Bedford, J., Childhood disability in Malaysia: a study of knowledge, attitudes and practices. UNICEF Malaysia, Kuala Lumpur, (n.d), p.82. 61 Disability Flagship Working Paper,( n.d.), p.3.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 29 Box 5: Suggested KAP indicators for UNICEF Malaysia to monitor
Baseline indicators Q.26: ‘How well informed are you about children with disabilities?’ 15.9% of the population surveyed reported that they were ‘Not at all informed’ about children with disabilities. The least informed answer options (i.e. answer options one, two and three on the Likert scale) account for 58.4% of the population surveyed.
Q.34: ‘Is a child with disabilities equal to a child without disabilities?’ 49.7% of the population surveyed reported that a child with disabilities is not equal to a child without disabilities.
Q.44: ‘ Are children with disabilities more likely to be abused?’ 59.7% of the population surveyed reported that children with disabilities are more likely to be abused.
In general, focus group participants and key informants had varying levels of awareness of specific campaigns. However, there was not a great deal of support for the campaigns overall, as one participant commented:
“...because it is always joined by the same group of people.”
Most external key informants sensed that the campaigns had not had much impact on people who did not already have an interest in children with disabilities and that they had limited effects on mainstream audiences. Informants had more support for seeking opportunities to target and change the mindsets of young people, especially through school-based events or campaigns. They felt that this is the target group where the most impact could be had, since attitudes were still relatively flexible.
Responses from parent informants suggested that, while big events like the BIM and the Carnival did offer the opportunity to affect behaviour, their lives had not changed as a result. They felt their children were still quite isolated and that significant barriers remained; where there were specific events that focused on inclusion their children could participate, but beyond that little had changed. All key participant groups repeatedly mentioned the lack of follow through: events happened in isolation with little collaboration; decisions were made with no subsequent actions or activities to help reinforce messaging. UNICEF staff also noted the lack of follow through, with one commenting that events are run and there is a focus on disability, but afterwards:
“…we go back to our daily routine until another event gains our attention.”
Feedback regarding the Carnival event also suggested to the evaluation team that the messaging around disability was not as rights-based as it could have been. One informant who was involved (and who had undergone DET), talked about the impact of the event in terms of eliciting people’s sympathy:
“Well I think everybody was quite touched with it. Of course, the families, but also the public who were there because we noticed the response when they performed.”
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 30 One of the issues seemed to be that UNICEF had not developed a specific behaviour change model, and was therefore not sure what effect its awareness-raising messages might be having. The sense from staff informants within UNICEF Malaysia was that there had not been enough direct engagement with communities and people with disabilities, and it was therefore not possible to understand exactly what efforts might be most successful in effecting behavioural changes.
The Man United campaign was the only example where young people with disabilities had been part of the development and implementation of an awareness-raising activity. Unfortunately, this happened prior to the #ThisAbility Flagship Initiative and, while it was used extensively as a resource, the process was not repeated. It is also notable that UNICEF does not appear to have a relationship with the disability movement in Malaysia. The organizations UNICEF has worked through are not Disabled People’s Organizations and there have not been any active efforts to link the Initiative’s activities with disability-focused advocacy priorities such as implementation of the CRPD or updating of the Disability Act (2008).
4.1.6. Evidence of cross-sector work Evaluation question: Summary of findings: To what extent has the #ThisAbility • The Initiative successfully promoted collaborative working Flagship Initiative enabled cross- in different ways sectoral work to increase access to • No evidence was found that this collaborative work has led high quality services for children with to an increase in high quality services disabilities?
On the basis of FGDs and KIIs, it was evident that many of the individual activities have been collaborative. For example, the Makeathon was run as an active collaboration between Tandemic, technical experts, UNICEF and Petrosains. This resulted in a well-planned event that was enjoyed by those who took part. Likewise, the Sabah Carnival was the result of multiple stakeholders (public, private and UNICEF) collaborating actively to provide the venue, funding, expertise, and other material inputs to make the event possible. The effectiveness of the collaborations directly contributed to the success of key activities, which played a role in helping to increase access and inclusion for children with disabilities and their families (albeit on a one-off basis).
Within the UNICEF office too, a key feature of the #ThisAbility Flagship Initiative taskforce, which is made up of representatives from several different departments, is that it has created a collaborative feeling among staff working on disability.
Beyond the UNICEF office and one-off events, however, this evaluation did not find any examples of where cross-sector work has led to increased access to high quality services for children with disabilities. The partnership with Sime Darby Property has the potential to lead to changes in the way playgrounds are designed in the future, however. But given the first inclusive playground has yet to be built (despite obviously very positive progress) it is too early to say whether or not this will prove to be effective.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 31 4.1.7. The extent of gender inclusion Evaluation question: Summary of findings: Has gender equality been taken • There is no evidence to indicate that gender equality has into account in the planning and been taken into consideration in the design, planning or implementation of the #ThisAbility implementation of activities Flagship Initiative?
Despite providing all key informants with opportunities to talk about gender and other intersectionality issues, this evaluation could find no evidence that gender equality considerations have played any role in the planning or implementation of #ThisAbility Flagship Initiative activities. UNICEF Malaysia does not collect disability-specific monitoring data that is disaggregated by gender or any other metric. The Working Paper and Monitoring Framework do not reference gender markers or gender equity activities. Children with disabilities are treated as a homogenous group in all programmatic references with no considerations given to any intersectional identities (age, gender, ethnicity, migration status, socio-economic status, location, impairment etc.). Anecdotal evidence suggests efforts were made to try and balance gender and impairment representation in events, but this falls short of anything that could be described as transformative from a gender and social inclusion perspective.
4.2. Relevance Box 6: Relevance questions
How relevant and appropriate has UNICEF’s work on disability been to the Agenda 2030, and should it be further adjusted to align with the Sustainable Development Goal (SDG) roadmap?
How relevant and appropriate has UNICEF’s work on disability been to Malaysia Vision 2020 and National Transformation 2050?
To what extent is #ThisAbility Flagship Initiative aligned with the new UNICEF’s Strategic Plan?
Has the design of the #ThisAbility Flagship Initiative taken into account the different needs of girls and boys, according to their age, gender, ethnicity, religion or faith, disability category, state and other social identities, especially the most vulnerable?
Have the voices of children and adolescents with disabilities been taken into account throughout?
Summary: UNICEF Malaysia has signalled a strong desire to work on disability inclusion within its CPD and country strategy documents. In turn, the Government has also made public its intentions to create more equal and inclusive services. Given that the Government of Malaysia has ratified the CRPD but has struggled with implementation, UNICEF’s intention to work on disability is highly relevant. UNICEF Malaysia has missed opportunities to better align its activities under the #ThisAbility Flagship Initiative with broader influencing around the rights of persons with
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 32 disabilities, because it has not made explicit use of international frameworks such as the CRPD in its campaigning. The lack of an overall advocacy strategy has limited the success of individual events to have a broader impact.
4.2.1. Alignment to Agenda 2030 Evaluation question: Summary of findings: How relevant and appropriate has • Intrinsically, the work on disability is well aligned to Agenda UNICEF’s work on disability been to 2030 and the SDGs the Agenda 2030, and should it be • No explicit link has been made by the Initiative to Agenda further adjusted to align with the SDG 2030 or any international development frameworks roadmap?
Currently, the #ThisAbility Flagship Initiative makes no mention of Agenda 2030 and it is not immediately obvious how the Initiative is assisting the Malaysian Government to realize its full responsibilities around Agenda 2030 and the SDGs. Key informants were generally in agreement that the Initiative is indirectly contributing towards Agenda 2030 given its focus on inclusion and its visibility.
It is a key issue that while the messaging has indirectly supported the SDGs, the lack of overt rights-based campaign messaging on Agenda 2030 has meant UNICEF Malaysia has missed opportunities to link up its activities with wider development issues.
4.2.2. Alignment to Malaysia Vision 2020 Evaluation question: Summary of findings: How relevant and appropriate has • The work on disability is relevant to the Malaysian UNICEF’s work on disability been to Government’s longer-term development plans but the Malaysia Vision 2020 and National Initiative does not align itself to these plans Transformation 2050?
Within its CPD, UNICEF commits to supporting Malaysia to make the transition to high-income status (a focus of Malaysia’s Vision 2020 plan) through three cross-sector focal areas: enhanced child and adolescent well-being; social inclusion and disparity reduction; and enhanced engagement and partnerships for child rights. These focus areas are reflected in more detail as ‘outcomes’ in the CPAP and the CPMP.
Within the CPD, disability features in ‘enhanced child and adolescent well-being’ in relation to early learning opportunities; and in ‘social inclusion and disparity reduction’. This indicates that at the country programme level, UNICEF Malaysia has the intention of using its relationship with the Government to help promote inclusive growth.
Clearly, disability is an issue that has been flagged within the Government’s development plans and at country programme level, and UNICEF Malaysia is responding to this within its own country agreement. Overall, the #ThisAbility Flagship Initiative can be said to be of relevance since it is an attempt by UNICEF Malaysia to operationalize its commitment to support inclusion.
The main issue is that it is difficult to make any strategic link between the Initiative, the CPD and the Government’s development plans. Neither the Initiative Working Paper nor the Monitoring Framework (2017–2020) mentions Malaysia’s Vision 2020 plan. Malaysia’s National Transformation
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 33 2050 is also absent from the Working Paper and Monitoring Framework, perhaps because the preparation phase for Malaysia’s National Transformation 2050 only began in 2017. It is also hard to link the Initiative directly to the CPD. There is not an obvious link between the focus of the Initiative’s activities and the Government’s commitments towards improving childcare facilities, education and employment opportunities, although awareness-raising activities will undoubtedly help reduce the high levels of stigma and discrimination that are in part leading to exclusion.
4.2.3. Alignment to UNICEF’s Strategic Plan Evaluation question: Summary of findings: To what extent is #ThisAbility Flagship • There is implicit alignment to UNICEF’s Strategic Initiative aligned with the new UNICEF Plan however, there is no clear results chain guiding Strategic Plan? implementation of disability inclusion work that obviously links it to the CPMP (or the CPD)
The UNICEF Malaysia CPMP states that disability is a key focus area for UNICEF Malaysia, particularly in relation to education, although this has not been the subject of any significant activities implemented under the Initiative. UNICEF Malaysia has nevertheless restructured its Programme Section to ensure the Education Officer position reports directly to the Head of Programme/Deputy Representative with a view to improving cross-sectional collaboration. This was also designed in part so that a ‘…focus on children with disabilities will be strengthened through increased coordination of the work done by both education and child protection.’62 There is recognition here too that the Communications Section requires a strong element of communication for development (C4D) in order to address the deep-rooted stigma associated with disability, among other issues which should be brought out across the programme.
The CPD Results Framework includes two disability-related indicators: • Output 1.1 – percentage of population surveyed with positive perceptions and attitudes towards children with disabilities, with a target increase of 30 per cent, based on baseline data gathered from a disability KAP Survey commissioned by UNICEF in 2016. • Output 1.4 – existence of Ministry of Education improved strategy for inclusive education for children with disabilities, based on modelling, with measurement of this reliant on reporting by Ministry of Education.
What is notable about the disability indicators themselves is that they do not relate directly to the experiences of children with disabilities which forms the main focus of the #ThisAbility Flagship Initiative objectives. The indicators are also not reported against on an annual basis. For example, UNICEF does not intend to carry out another KAP Survey until 2025, which is some way beyond the current CPMP.63
The introduction of the #ThisAbility Flagship Initiative in 2017 can be seen as a determined effort to find a mechanism through which to operationalize disability work. Key UNICEF informants suggested that the benefits of having a Flagship Initiative included the ability to work more proac- tively on disability despite the lack of concrete direction within the CPMP. The Initiative therefore provides a helpful point of reference around which Sections can engage. It appears, however, that UNICEF Malaysia does not yet have the mechanisms in place to monitor progress or measure success at this level.
62 UNICEF Malaysia 2016–2020 Country Programme Management Plan, p6. 63 Personal correspondence, August 2019.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 34 4.2.4. Intersectionality and the voice of children with disabilities Evaluation questions: Summary of findings: Has the design of the #ThisAbility • No explicit attempts have been made to address Flagship Initiative taken into account intersectional issues in the design of the Initiative, with the different needs of girls and boys, children and young people with disabilities being treated according to their age, gender, ethnicity, as a homogenous group religion or faith, disability category, state • Impairment needs were generally well accommodated and other social identities, especially the during activities most vulnerable? Have the voices of children and • Some children and young people with disabilities have had adolescents with disabilities been taken opportunities to actively participate in the design of some into account throughout? events but there is no evidence of systematic engagement
This evaluation could find no evidence that the diversity of children’s experiences had been taken into consideration during the design of the #ThisAbility Flagship Initiative. KIIs suggested there was no explicit mechanism through which children and young people could feed into the design of the Initiative. In fact, there was a general sense among informants that UNICEF Malaysia had not done very well in meaningfully engaging children and young people with disabilities in any of its planning processes.
KIIs and especially FGDs revealed a mixed reaction in relation to whether or not people felt UNICEF had considered diversity issues. In the main, participants agreed that both girls and boys were freely able to join activities, but there was no sense that things had been designed in ways that could challenge norms.
Some participants noted that it would be good to have higher participation levels from girls with disabilities because they are generally more isolated. Most participants believed that neither ethnicity nor other social identities would be a barrier to participation, although a few respondents highlighted an absence of participation by some ethnic groups. Some participants felt that there could be financial barriers to participation:
“... if you don’t have a wheelchair, you can’t take your child out; if you don’t have a car, it is almost impossible to bring them...”
Overall, UNICEF Malaysia appeared to have tried to consider a wide range of impairment needs in the design of individual activities. Many of the non-disability organizations tasked with implementing activities during the Sabah Carnival revealed how much UNICEF had supported them to think about different accessibility issues. Several staff informants reported that they appreciated being prompted by the Disability Consultant to think about accessibility, with the inclusive events checklist proving especially helpful.
Several informants referred to the inclusive playground consultation as being especially inclusive. One informant commented that the proposed design had even considered how to make the playground a safe place for deaf children with the use of visual prompts to help communication between deaf and hearing children.
The main challenge in evaluating the extent to which intersectionality has been considered in the programme is the fact that the Initiative does not collect or collate data that can be disaggregated by any diversity indicator. For UNICEF Malaysia to be confident it is creating inclusive activities that take account of the diverse range of identities among children with disabilities, it is necessary to monitor participation.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 35 4.3. Efficiency Box 7: Efficiency questions
To what degree have UNICEF resources (human, technical and financial) been sufficient in implementing the activities under the #ThisAbility Flagship Initiative?
How adequate are the capacities of UNICEF’s organizational and individual partners to engage on the rights of children with disabilities?
Summary: The activities delivered under the #ThisAbility Flagship Initiative were generally observed by stakeholders to have been well resourced and well managed with a high degree of technical support provided. Internally, concerns were raised that the prioritization of resources, especially human resources, was insufficient to effect organization-wide change and that a more strategic approach to disability is required. The lack of systematic monitoring has made it difficult for UNICEF Malaysia to assess whether its resources have been used in the most efficient way (see Section 4.4 for more details).
4.3.1. Human, technical and financial resourcing Evaluation question: Summary of findings: To what degree have UNICEF • Resourcing of individual activities appears to have been resources (human, technical sufficient and financial) been sufficient in • implementing the activities under the The stand-alone nature of activities means UNICEF #ThisAbility Flagship Initiative? Malaysia cannot measure their efficiency overall
This evaluation was not able to assess efficiency to any great extent because the evaluation team were not provided with a #ThisAbility Flagship Initiative programme budget; there is no accountability process against which to measure dispersal of resources. KIIs with staff and partners suggested that resources were sufficient for the effective implementation of planned activities. Indeed, several partners commented that UNICEF had been a good organization to work with in relation to resources. Informants expressed some concern that the Initiative had fallen short in terms of how funding had been prioritized. Informants suggested that the #ThisAbility Flagship Initiative had not been given enough of a priority status in resourcing terms rather than a lack of funds per se:
“I think there could have been more that could’ve been done to support us as a flag- ship and the person driving that forward.”
Informants raised specific concerns about whether the #ThisAbility Flagship Initiative was sufficiently staffed to achieve its ambitious objectives. Given this was a Flagship, some staff were surprised it was being led by a consultant (rather than a full-time member of staff). However, the consultant provided both excellent technical support and a much-needed driving force, for example simply being available to attend meetings, address forums or to follow up with different sections on their disability activities. Without that centralised support role, participants were sure less progress would have been made in actually implementing such a range of successful activities. Informants expressed caution that this does not necessarily equate to a working model that is efficient or sustainable
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 36 Several informants also suggested that contributing to the #ThisAbility Flagship Initiative was not perceived as mandatory or something that all parts of the organization must do. While teams contributed to the Initiative and engaged in related activities, several informants perceived the Initiative to be driven by one person and not as the collective responsibility of more UNICEF Malaysia team members.
“The weakness is that it is only by one person...”
This reinforced the point that the #ThisAbility Flagship Initiative Working Paper and Indicators Framework are insufficiently strategic to be able to influence and embed disability work into the Country Programme more broadly.
4.3.2. The adequacy of partners’ capacity
Evaluation question: Summary of findings: How adequate are the capacities of • UNICEF Malaysia has a reputation for delivering good UNICEF’s organizational and individual technical support to partners partners to engage on the rights of • An overreliance on organizations that work for and children with disabilities? on behalf of people with disabilities rather than with organizations of people with disabilities is limiting opportunities for UNICEF Malaysia to advance a rights agenda
Mainstream partners generally commented that having UNICEF on board in matters relating to children with disabilities did give them more confidence and ‘mileage’ to engage with disability issues. Mainstream organizations were especially keen to highlight the value of UNICEF support in helping them understand how best to meet the needs of children with disabilities. For example, all partners commended the DET arranged by UNICEF because it had given them a better understanding of how to work with people with disabilities.
In terms of events, all the organizations UNICEF Malaysia engaged have done well to accommodate the specific requirements of children with disabilities and, as noted extensively in this evaluation, feedback from those who participated in activities was generally very positive. Beyond the activities, however, it is clear from KIIs and FGDs that most partners have not made progress in taking up issues on the rights of children with disabilities as a result of the Initiative. In fact, what has been striking across all key informant groups is the lack of reference to the rights of children with disabilities in the language used to describe the Initiative.
As noted in Section 4.1.4, it is also notable that UNICEF Malaysia has not so far engaged directly with the disability movement in Malaysia.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 37 4.4. Sustainability Box 8: Sustainability questions
To what degree has the #ThisAbility Flagship Initiative thus far contributed to the generation of sustainable capacities at the national and sub-national levels?
What are the enabling as well as constraining factors that are likely to influence future cross-sectoral work on focus issues?
Summary: Due to the fact the #ThisAbility Flagship Initiative is intended to be primarily an internal management device to increase disability work across the organization, it currently lacks the national ownership required for sustainability. While many events and campaigns have been run with a view to changing attitudes, knowledge and practices, as noted in previous sections, the lack of an advocacy strategy and any systematic monitoring data meant this evaluation could not determine their impact or sustainability.
4.4.1. Sustainability at national and sub-national level Evaluation question: Summary of findings: To what degree has the #ThisAbility • The design of the #ThisAbility Flagship Initiative does not Flagship Initiative thus far contributed currently enable UNICEF Malaysia to have an impact at to the generation of sustainable national and sub-national levels capacities at the national and sub- national levels?
The evaluation was unable to address the question of the #ThisAbility Flagship Initiative’s sustainability at national and sub-national levels because its activities were not designed to effect changes in capacity at national and sub-national level.
Most activities were one-off events with limited geographical reach (e.g. Klang Valley and Sabah), therefore it is challenging to gauge the benefit of the Initiative on a wider scale.
Without a baseline or any systematic monitoring of outcomes and impact, the evidence available to assess sustainability at this level was also insufficient. Efforts to engage Government stakeholders (referred to the evaluation team by UNICEF Malaysia) in the evaluation process proved extremely difficult, with most of the potential Government key informants declining to be interviewed. Of the officials who did take part, there was limited direct experience with UNICEF Malaysia on issues related to children with disabilities, again despite UNICEF Malaysia suggesting these officials to the external evaluators. The most important benefits that Government informants reported in relation to disability was the willingness to fund inclusive events and the quality of UNICEF’s press releases and communications materials.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 38 4.4.2. Enabling and constraining factors for future work Evaluation question: Summary of findings: What are the enabling as well as • The design of the #ThisAbility Flagship Initiative does not constraining factors that are likely to allow for this question to be answered influence future cross-sectoral work on focus issues?
The evaluation was also unable to address the question of the enabling and constraining factors likely to influence future cross-sectoral work because of the current design of the #ThisAbility Flagship Initiative activities. The one-off nature of events with limited geographical reach meant such factors could not be identified.
4.5. Utility of a Flagship Initiative In the absence of a formal definition of what constitutes a ‘flagship initiative’ within UNICEF Malaysia, this evaluation has taken the views expressed by staff and partners to assume that it means a programme or set of activities which are somewhat ‘off-plan’ and cross-cutting; effectively, activities created during an existing planned period to provide UNICEF Malaysia with a platform for developing innovative working relationships across sections, in a new area of work; in this case, disability.
The absence of an agreed definition of a ‘flagship initiative’ inevitably means it is difficult for people to evaluate how successful it has been and whether or not it is a mechanism they would seek to utilize again. This point was highlighted in a 2016 Evaluability Assessment in which it was noted that the programme had been running two flagships, on Child Marriage and Out of School Children, but that: ‘(I)t is not clear what a ‘flagship’ denotes’.64
In summary, evidence collected through this review suggests there are three key strengths to having a flagship initiative:
• Enables cross-sectional working to grow, creating new synergies Many informants expressed that having the opportunity to work together on activities was a key strength, including within UNICEF; between UNICEF and partners; and in some cases, between partners. For example, civil society organizations in Sabah commented that the opportunity to work on events like the BIM and the Sabah Carnival was instrumental in helping them to engage with each other in ways they had not done in the past. Within UNICEF, the working relationship between C4D, Innovations, CSR and the Disability Consultant seemed particularly productive (see Box 9).
64 Vine Management Consultanting (2016) Evaluability Assessment: UNICEF Malaysia Country Programme 2016- 2020. P.11
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 39 Box 9: Case study illustration on positive inter-office collaboration
‘We are all special’ Carnival This event involved a wide range of UNICEF Sections. C4D worked closely with the Disability Consultant around partnership building and advocacy; Education provided some advice and messaging around inclusive education; Private Fundraising and Partnerships mobilized street fundraisers to use face-to-face encounters to raise awareness and get people to sign up to the Innovations Section U-Report mechanism to take an online attitudes survey; Communications created a press release, event speeches, engaged the media and filmed numerous short videos.
The digital assets created at this event were used the following month to underpin a mini-awareness campaign – ‘What’s so special?’ – coinciding with International Day of Persons with Disabilities 2018. Opportunities were taken to showcase videos, disseminate information, education and communication materials and display billboards in Kuala Lumpur and Selengor.
• Helps raise the visibility of disability Within UNICEF, informants were clear that by running events under the #ThisAbility Flagship Initiative, awareness of disability had increased. During 2017 and 2018, around 26 UNICEF staff and 25 FACERS undertook a one-day DET session and it’s likely that more were involved in sessions provided to partners. The session itself helped to raise awareness but, with the ongoing series of activities, staff and partners were asked to focus on children with disabilities more explicitly than had happened in the past. Several staff informants described how their own knowledge and experience of engaging directly with children with disabilities through these events had changed the way they approach disability. They felt more confident around children with disabilities and no longer saw the need to treat them differently or ‘in a special way’. Among the non-disability focused partner organizations, this was one of the benefits most often cited of working with UNICEF on Initiative activities.
• Good quality technical guidance Many staff and partner informants commented on the quality of the support provided through the Initiative. Of particular note was the Disability Consultant’s skill in bringing people together, motivating and prompting where necessary. Staff informants noted the Consultants’ rich experience in disability, which was readily shared in ways that encouraged participation. From the perspective of partners, many informants described UNICEF as providing good technical support during the planning and implementation of activities, which particularly helped non-disability organizations to become more familiar with requirements around accessibility and accommodation for children and young people with a variety of different impairments. The work around the Inclusive Playground Initiative is a good illustration of how appropriate and timely technical support resulted in what promises to be a good quality inclusive design (see Box 10).
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 40 Box 10: Case study illustration on good use of technical expertise
Inclusive Playground Initiative During the latter part of 2018, UNICEF engaged Sime Darby Property as an Inclusive Champion. One of the key areas of collaboration was the design of what could be Malaysia’s first Inclusive Playground for children. As a property development company, Sime Darby was a good stakeholder to influence and was especially open to the idea of setting aside an area of land in a new development. Given the innovative nature of the activity and the fact that UNICEF was keen that end-users had the opportunity to input into the design, an architectural company (Jeavons Landscape Architects), with experience in inclusive playgrounds, was brought in to lead the design process. A key part of the design process was a half-day workshop hosted by Sime Darby in which 45 end-user participants took part (including children and young people with disabilities, parents and grandparents). At this workshop, UNICEF used interactive play-based sessions to help children express their ideas of what they would want from a playground. This helped the architects to appreciate not just accessibility, but also what children with disabilities were interested in doing in an outdoor space. While the playground has yet to be constructed, the plans are now undergoing the approval process with the local municipal council.
In terms of weaknesses of the Flagship approach, three key areas of concern emerged, which are summarized below:
• Driven by one person Most staff informants and some partners noted that the work seemed to be driven by one person – an external consultant – and not a permanent member of staff, which many perceived as indicating that the initiative might not have had full support across the organization.65
While all informants were keen to acknowledge the passion and expertise provided by the Consultant, they were nevertheless concerned that the Initiative was not a sustainable model and had not helped them to mainstream disability into their work. Several informants highlighted this and noted that, since disability was not reflected in key performance indicators, it sometimes made it hard for people to provide the time required. Moreover, it had the effect that this was seen as ‘additional work’, leaving the consultant having to ask for support.
Others also sounded a note of caution, saying that simply working together on events and attending joint meetings did not necessarily mean that cross-section engagement was happening. Often the motivation for coming together was provided by the Disability Consultant rather than something that was driven by a common objective.
• No clear strategy with a lack of follow up or follow through While the #ThisAbility Flagship Initiative did have a Working Paper and Monitoring Framework outlining a strategy, UNICEF staff reported experiencing the Flagship as a series of unconnected activities or events. In fact, many informants from all stakeholder groups noted that there seemed to be an over reliance on large-scale one-off events that were not seen to be part of awider narrative.
65 It must be noted that the consultancy position was made into a staff position during the period of the evaluation. As a result, this may have weakened the perception that the initiative might not have total support across the organization because it was ‘driven’ by an external consultant.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 41 Most staff informants and many partners perceived the main weakness of the Initiative to be a lack of strategic direction, with a lack of a narrative or journey. Several informants feared that disability could be an issue that, although it had now been raised, would soon be superseded by something else as priorities shifted. Without an overarching narrative to which all staff are making a contribution, some informants were concerned that disability could be ‘dropped’. To address this, many expressed considerable support for a disability-focused ToC or long-term strategy, with targets for each section that are regularly monitored; in this way, disability could be more effectively mainstreamed across the organization.
• No outcome or impact reporting Staff informants expressed some level of frustration about the lack of formal analysis of events or activities. Several noted that while they were in receipt of anecdotal evidence for what people thought of events, there was often no actual reporting. As one staff informant noted:
“...I spoke to parents at the events as well they are also very happy (sic) but I can’t really identify what are you really happy about what does your child think...”
There was some concern that as an organization, UNICEF was not properly tracking what was going well or what was not working because there were no regular evaluations happening. In the words of one staff informant:
“...sometimes it can be a bit all over the place...”
Staff informants suggested that this distanced staff from the work; taking away some of the agency and ownership because without feedback on the impact of the work it became difficult for them to build upon it in future plans.
To a large extent, this view was echoed during FGDs and partner interviews, when informants who had participated in events noted that they had not been asked to provide feedback on the events or activities. The lack of formal monitoring has not only deprived UNICEF of important data on whether its strategies are having an impact (intended or unintended), but it also has the effect of closing off information-sharing opportunities and accountability to partners and other stakeholders. Focus group participants particularly felt they were unsure what had happened since events like the Makeathon and the Inclusive Playground, despite having invested time in supporting them.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 42 5. CONCLUSIONS
Based on the analysis outlined in each section of this report, the evaluation team makes the following conclusions:
In terms of effectiveness: Conclusion 1 On the whole, parents and children who participated in the various events implemented under the #ThisAbility Flagship Initiative found them to be fun, exciting and of personal benefit. The lack of direct engagement with parents, children and young people with disabilities in the design and evaluation of Initiative activities has been a missed opportunity for UNICEF Malaysia. The lack of follow up on Initiative activities with participants has also presented missed opportunities for UNICEF Malaysia.
Conclusion 2 The DET was of good quality and has played a role in challenging participants’ preconceptions around disability: in many cases helping them to be more confident in engaging with people with disabilities. However, the DET lacked a focus on the rights of children with disabilities and the specific barriers they face.
Conclusion 3 Establishing the two formal partnerships with corporate ‘inclusion champions’ was a well-conceived strategy that has had a degree of impact on promoting inclusive practice. However, there is no evidence to suggest these partnerships have empowered children and adolescents with disabilities to undertake self-advocacy; nor is there evidence of the potential of the partnerships to do so based on the objectives and design of the partnerships.
Conclusion 4 Public awareness-raising has been central to the work of the #ThisAbility Flagship Initiative. UNICEF Malaysia did well to utilize existing public events and to influence them to become more inclusive. The effectiveness of the public awareness-raising could not be analysed because of the lack of monitoring data and the nature of these one-off activities. The medical/charity model approach to disability appears to still be widely used.
Conclusion 5 The #ThisAbility Flagship Initiative successfully promoted collaborative working. Through participation in events, civil society organizations, corporate agencies, local government and media agents gained experience in working together to promote disability inclusion. Within UNICEF Malaysia too, individuals within different Sections worked well together to implement Initiative activities. There is no evidence yet of this collaboration leading to better quality services for children and young people with disabilities, although the inclusive playground initiative with Sime Darby Property could lead to significant improvements in the way outdoor spaces like this are designed in future.
Conclusion 6 The evaluation could find no evidence that gender equality, or any other intersectional issue had been taken into consideration in the design, planning or implementation of #ThisAbility Flagship Initiative activities. The Initiative appeared to treat children and young people with disabilities as a homogenous group. A transformative approach to gender and social inclusion had not been taken,
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 43 although attempts have been made to balance gender and impairment representation in activities. This is a serious gap that UNICEF Malaysia will need to address in its future work.
In terms of relevance: Conclusion 7 Intrinsically, the #ThisAbility Flagship Initiative is well aligned to Agenda 2030 and the fulfilment of the SDGs. However, the Initiative strategy makes no explicit link to Agenda 2030 and there is a lack of reference to international frameworks in UNICEF Malaysia’s messaging.
Conclusion 8 UNICEF Malaysia is not actively aligning the #ThisAbility Flagship Initiative to the Malaysian Government’s longer-term development plans, despite the link having been identified in the CPD. Until a clearer strategic plan for disability is in place and embedded within the reporting structures of the CPD, the Flagship nature of the initiative will mean it remains an internal mechanism for promoting collaborative work, rather than a strategy for promoting disability- inclusive development.
Conclusion 9 While the ideas about the need to include children and young people with disabilities are positive, there is not a clear results chain guiding implementation of disability inclusion work that obviously links the #ThisAbility Flagship Initiative to the CPMP (or the CPD). The lack of a clear framework for disability-inclusive work evidences a gap at this strategic level that has not so far been alleviated by the Initiative.
In terms of efficiency: Conclusion 10 UNICEF Malaysia seems to have provided enough resources to implement activities to a good standard. The standalone nature of #ThisAbility Flagship Initiative events made it difficult to assess efficiency overall, as does the lack of systematic monitoring.
Conclusion 11 UNICEF Malaysia has a reputation for delivering good technical support to its partners. However, an overreliance on engaging with organizations that work for and on behalf of people with disabilities, rather than with organizations of people with disabilities, is limiting opportunities for UNICEF Malaysia to advance a rights agenda.
In terms of sustainability: Conclusion 12 The evaluation was unable to determine the extent to which the Initiative’s campaigns are having an impact on generating sustainable changes at national and sub-national levels.
In terms of the utility of a Flagship Initiative: Conclusion 13 This Flagship Initiative enabled cross-sectional collaboration, raised the visibility of disability inclusion, and strengthened UNICEF staff’s access to good quality technical guidance. To further strengthen flagship initiatives, there is a need for UNICEF Malaysia to be explicit in providing a clear definition of what constitutes a ‘flagship initiative’ before developing such mechanisms in the future.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 44 6. LESSONS LEARNED
Lesson learned 1 – Deep-rooted attitudes towards disability require strong, coordinated rights-based messaging to effect change. Evidence gathered from the desk review, KIIs and FGDs indicates that UNICEF staff and key external partners focused primarily on promoting inclusion and increasing visibility of children with disabilities, to the detriment of promoting the rights of children with disabilities. By not explicitly promoting the rights of children with disabilities or linking their exclusion and lack of visibility to the barriers that exist within communities and systems, it is possible for disability to continue to be considered an individual’s problem, undermining efforts to achieve systemic and sustainable change.
Developing messaging that is explicit about the rights of children with disabilities, and framed in terms of reducing some of the barriers within communities (such as communities becoming more welcoming, so that the children and parents feel more respected) would be more compliant with the CRPD. Disability inclusion would be better framed as the result of removing barriers and upholding rights. This approach would also support mainstreaming because whatever child rights issues were being raised, messaging around children with disabilities could always be incorporated.
Lesson learned 2 – A Theory of Change and strong monitoring systems are required to help mainstream disability at an organizational-level. An objective of the evaluation was to assess the extent to which the #ThisAbility Flagship has been appropriately designed. Observations throughout the evaluation indicated that the system for monitoring the Initiative was creating uncertainty and a lack of cohesion. The lack of a ToC has inhibited monitoring at all levels. Each UNICEF Section appears guided by its own set of actions, for which it has output indicators. If these do not include disability-related outputs, then they are unlikely to measure the impact of their activities on children with disabilities, and there is no disaggregation of data by disability at impact level for any Section-based intervention. Furthermore, while a Flagship Taskforce was created, key informants explained that its main purpose was to update everyone on progress towards upcoming events, rather than analysing the outcomes and progress against the strategy. These observations underscore the importance of some key components that, in the external evaluators’ experience, are essential to disability-inclusion programming:
a. Explicit organizational commitment – the organization as a whole makes it clear why it is committed (its overall values); and what it seeks to achieve through this commitment (its overall purpose). b. Inclusive programmes and services – the organization will ensure that all its direct services and programmes are delivered in inclusive, equitable and non-discriminatory ways and do not create barriers or reinforce negative stereotypes. This means ensuring all programmes are accessible to people with disabilities, regardless of whether or not people with disabilities are an intended target. c. Disaggregated data – all programme data is disaggregated by disability using an international standard definition of disability (for example the Washington Group Question sets - www.washingtongroup-disability.com).
Lesson learned 3 – A short-term mindset can present safeguarding concerns and possible challenges for future engagement. Several of the assistive tools that were created during the Makeathon event were broken by the time of this evaluation, leaving children who participated in the event without their new gadgets. While this was a predictable outcome, the children and their families may not have been fully cognisant of this when they agreed to participate. For the We are
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 45 all special Carnival in Sabah, the Jesselton Philharmonic Orchestra supported a young deaf girl to play the violin and perform on the day of the Carnival. The Orchestra’s engagement with the girl reportedly stopped at the end of the Carnival, requiring the girl to hand the instrument back with no follow-up actions. The disappointment and frustration that participants reported regarding these two examples raises serious questions around the coordination of one-off events where children have participated in something potentially life-enhancing, but are left without the products they helped to create or continued opportunities to develop their new skills. UNICEF Malaysia may have inadvertently crossed the ‘do no harm’ threshold with these activities because they were relatively high profile but did not adequately consider or prepare for the experiences of the child participants whose expectations were raised and then not fulfilled. Safeguarding issues also arise from such activities; it is necessary to adequately consider the well-being of vulnerable young children when exposing them to such experiences without an obvious plan for follow up.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 46 7. RECOMMENDATIONS
The following recommendations are based upon the key findings, the conclusions reached by the evaluation team, and the lessons learned; they have been developed to reflect the views of key informants and with the input of key stakeholders that formed a Reference Group for this evaluation.
Recommendation 1 To whom: Country Representative, Deputy Representative, Planning Monitoring and Evaluation Chief Priority: High Timeline: 6-months
UNICEF Malaysia needs to make a strategic commitment to disability inclusion by fully embedding disability into key planning documents with the expectation that children with disabilities will be included and monitored across all areas of programming. We encourage UNICEF Malaysia to start doing this in the short-term, beginning with the forthcoming strategic planning.
Considerable support has been generated within UNICEF Malaysia for working on disability. The high-quality technical support provided during the initial phase of the #ThisAbility Flagship Initiative and the opportunities for staff within sections such as C4D, communications and innovations to work together has increased capacity and generated positive momentum within the office. For sustained progress to be made from the Initiative, this work needs to be mainstreamed across all Sections. The ToC constructed as part of this evaluation (see Annex F) provides a single narrative that can frame UNICEF Malaysia’s future disability inclusion work. Embedding the ToC (or one similar) in strategic plans and establishing a monitoring system that can track progress toward the strategic goal (including the capture of disability- disaggregated data) at all monitoring points will significantly strengthen UNICEF Malaysia’s work in this area.
Recommendation 2 To whom: Country Representative, Deputy Representative, Communications Chief, Private Funding and Partnerships Chief, C4D Specialist Priority: Medium Timeline: 12-months
UNICEF Malaysia, starting immediately, needs to be much more explicitly rights focused in its messaging around disability and ensure its campaigns fully align with key international frameworks, including the CRPD.
Any future public engagement needs to promote the rights of children with disabilities, with a greater emphasis placed on removing barriers to achieve equality and inclusion. This evaluation recommends that UNICEF Malaysia develops an advocacy strategy that incorporates behaviour change targets that can be used to track the effectiveness of its messaging. More explicit reference to and use of the CRPD is highly recommended.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 47 Recommendation 3 To whom: Country Representative, Deputy Representative, Communications Chief, Programme Staff Priority: Medium Timeline: 12-months
UNICEF Malaysia, starting immediately, needs to provide more opportunities to meaningfully engage children and young people (with and without disabilities) during the design of campaigns and messaging.
While the #ThisAbility Flagship Initiative has provided some limited opportunities for children and young people with disabilities to voice their needs, it has not attempted to systematically engage them in discussions around how to shape advocacy priorities and activities. It is also evident that UNICEF Malaysia does not yet have a particularly strong relationship with the disability movement, with a tendency to rely on working with and through organizations that work on behalf of people with disabilities. This means that it is missing opportunities to support capacity building of the disability movement and to support a new generation of disability advocates. There are some precedents that UNICEF Malaysia might usefully take advantage of including a previously highly successful youth advocacy programme called ‘Young Voices’ which ran in Sabah with support from Cheshire Services.66 While this programme is no longer running, there is continued interest in using its experiences to develop more opportunities for young people with disabilities to come together to work on rights-promotion activities.
Recommendation 4 To whom: Country Representative, Deputy Representative, Communications Chief, Programme Chiefs and Programme Staff Priority: High Timeline: 6-months
UNICEF Malaysia urgently needs to develop robust and rigorous safeguarding review processes for all activities involving children and young people with disabilities.
As a matter of priority, UNICEF Malaysia should initiate measures to ensure that all activities involving children and young people with disabilities undergo a safeguarding assessment to ensure that the well-being of children will not be put at risk during or after events. Consulting with parents and older children with disabilities would help UNICEF Malaysia gain a better understanding of what mitigating actions could be put in place to ensure activities are fun and safe as well as protecting long-term well-being.
66 Sabah Cheshire Home,
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 48 Recommendation 5 To whom: Country Representative, Deputy Representative, Gender Focal Point, Communications Chief, Programme Chiefs and Programme Staff Priority: Medium Timeline: 12-months
UNICEF Malaysia needs, starting as soon as possible, to routinely assess new and existing programmes and activities to ensure gender and social inclusion norms and unequal power relations are considered in full.
A key finding from this evaluation was the absence of any consideration of gender and social inclusion norms and unequal power relations in the design and delivery of activities. Whilst there has been a conscious effort towards ensuring environmental accessibility during events for participants with disabilities, which has resulted in improved awareness of the need to provide reasonable accommodation, the #ThisAbility Flagship Initiative has not yet reached a level which could be assessed as being either gender and social inclusion (GESI) accommodating or transformative. There is considerable scope now for moving beyond the assumption that children and young people with disabilities can be treated as a homogenous group with just their individual impairment’s needs accommodated. As UNICEF Malaysia develops its disability-inclusive ToC it would be useful for it to undergo a Gender, Equality and Social Inclusion (GESI) assessment process to determine the extent to which all new programmes and activities can help move them towards becoming GESI transformative.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 49 ANNEX A: EVALUATION TERMS OF REFERENCE
1. Introduction This Terms of Reference (ToR) document outlines the purpose and scope of a formative, forward- looking and learning-oriented Evaluation of UNICEF’s #ThisAbility Flagship Initiative in Malaysia. Further to the UNICEF’s State of the World’s Children on Children with Disability in 2013 that pointed to the need to reduce the vulnerability of children with disabilities, UNICEF Malaysia launched a number of initiatives to bring disability into the political and social discourse. Through partnerships with organizations and individuals UNICEF Malaysia created momentum to engage in dialogue on the rights of children with disabilities, and in 2016 #ThisAbility became a flagship priority within the new UNICEF Country Programme 2016-2020. During the flagship period UNICEF committed to make positive changes for children with disabilities in Malaysia so that they are free from discrimination and able to access services to support their growth, development and protection, in line with the United Nations (UN) Agenda 2030 principle of “leaving no one behind”.
With this backdrop, this evaluation seeks to examine #ThisAbility Flagship Initiative from 2016 until its current implementation, but it will also look at the evolution of UNICEF’s work on disability in Malaysia since 2013. The ToR present thus methodological options and operational modalities for an institutional contract of a team of two evaluation consultants (one international and one national). This independent evaluation is commissioned by UNICEF Malaysia; hence, UNICEF is looking for institutions with deep commitment to, and strong background in, the evaluation of social programmes, and particularly disability. Findings and recommendations from this evaluation will inform the direction #ThisAbility should take in the course of the remaining part of the Country Programme, including future evaluation of the flagship. Lessons from the evaluation may inform further cross-sectoral work and integration of disability across all UNICEF programmes, policy- advocacy for children with disability and the application of innovations. The evaluation is expected to being October 2018 and to be completed by January 2019.
2. Background and Rationale 2.1. UNICEF Malaysia and its Focus on Disability UNICEF’s work in middle-income countries such as Malaysia has transitioned in recent decades to focus on upstream policy and advocacy interventions, capacity development and evidence generation. Malaysia ratified the Convention on the Rights of the Child (CRC) in 1995 and the Convention on the Rights of Persons with Disabilities (CRPD) in 2010, albeit with reservations, and implemented a Persons with Disabilities Act in 2008. Whilst progress has been made in improving the lives of children with disabilities (right holders), critical issues remain including equitable access to services and participation and overcoming widespread negative perceptions towards persons with disabilities. Such discriminatory views have a profound effect at all levels, from the development and life-course of children with disabilities, through to the shaping of laws and policies at the national level. In their concluding observations to Malaysia’s first periodic report, the Committee on the Rights of the Child made three key recommendations: that the country combats stigma and discrimination of children with disabilities; collects adequate statistical data on children with disabilities; and ensures that such data is used in the development of policies and programmes for children with disabilities.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 50 A key component of UNICEF Malaysia’s 2016-2020 Country Programme67 is to support the country to realise the rights of all children, reduce disparities and foster social inclusion. This is in line with the UN General Assembly High Level Meeting on Disability and Development (2013) which urged increased understanding, knowledge and greater social awareness about persons with disabilities to eliminate discriminatory social and attitudinal barriers so that they participate fully in society, and the UN Agenda 2030 that promotes a world where “no one is left behind”.
UNICEF has an important role to play in supporting national and sub-national authorities and other partners to create an enabling environment for children with disabilities in Malaysia, and an environment in which they are included into society and have the right and ability to participate without fear of discrimination or marginalisation. UNICEF’s Malaysia recent knowledge, attitudes and practices (KAP) study68 noted, for instance, that respondents have limited knowledge about the actual causes of disability, attitudes are poor and practices are sporadic for children with disabilities. In addition, children with disabilities, their parents and families face stigma and discrimination at home, in schools and many other public environments. Having said that, children and adolescents are more tolerant and accepting than the general adult population, acting thus as effective agents of change by taking a stand against discrimination, promoting diversity and equal opportunities, and helping to ensure the social inclusion and participation of children and adolescents with disability.
In 2016 disability hence became a flagship priority around which the whole new Country Programme can gather. UNICEF’s work on disability is spread across four programme outputs as follows: a) Output 1.1: Duty bearers have increased knowledge and demonstrate positive attitude towards protecting all children from harmful practices and discrimination (incl. disability);
b) Output 1.4: Improved equal learning and development opportunities for vulnerable children;
c) Output 2.4: Government ministries and research institutions have strengthened institutional capacities for undertaking research-based social and behavioural change communication (on issue related to disability, among others); and
d) Output 3.1: The Malaysian private sector has increased knowledge and commitment to its responsibility to respect children’s rights, and can access quality technical support and capacity for implementing the CRBP (incl. children with disabilities).
In addition, UNICEF developed a #ThisAbility Flagship Working Paper and Monitoring Framework that outlines the objectives, activities and measures that UNICEF committed to implement until 2020 to make positive changes for children with disabilities in Malaysia so that they are free from discrimination and able to access services to support their growth, development and protection. These objectives include: • Children with disabilities have increased access to high quality services; • Children with disabilities do not experience abuse and discrimination; • Children and adolescents with disabilities are empowered to advocate for themselves; • General public are aware of children with disabilities; • Children with disabilities are socially included; and • Children with disabilities can contribute positively in society.
67 https://www.unicef.org/about/execboard/files/2015-PL24-Malaysia_CPD-ODS-EN.pdf 68 https://www.unicef.org/malaysia/Final_File_Childhood_Disability_in_Malaysia_Book_(Interactive_PDF). compressed_3_Oct_2017.pdf
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 51 #ThisAbility Flagship Change Commitments and Monitoring Framework are presented in Annex I and II.
2.2. Rationale This evaluation is expected to be formative, forward-looking and learning-oriented in nature and to produce credible, reliable and useful evidence on the results achieved to date by UNICEF’s #ThisAbility Flagship in Malaysia – including through the cross-sectoral coordination. The evaluation will also help identify strengths and weaknesses to inform the future direction of UNICEF’s work on children with disabilities in Malaysia, and it will be followed by an outcome evaluation by 2020. It will be shared with UNICEF’s Regional Office for East Asia and the Pacific (EAPRO) and other offices to promote learning on the utility of creating a flagship on disability.
3. Purpose, Objectives and Scope of Work 3.1. Purpose The main purpose of this independent, formative evaluation is to foster learning and improvement within the #ThisAbility Flagship initiative. The evaluation will document the evolution of UNICEF’s work on children with disability in Malaysia and assess both programme design and management, and assess its programmatic relevance, appropriateness, effectiveness, efficiency and sustainability against regional and international benchmarks. The evaluation will also assess the utility of creating a flagship on specific focus issues to mobilize cross-sectoral work among programme components in Malaysia.
The evaluation will be used to inform the future direction UNICEF’s work on children with disability should take in the remaining part of the Country Programme. It will help identify good practices, lessons learned, as well as inform cross-sectoral work and integration of disability across all UNICEF programmes, policy-advocacy for children with disability and the application of innovations. The evaluation will also make recommendations as to the type of evaluation that should be conducted at the end of the Country Programme.
The primary users of the evaluation include UNICEF Malaysia, the government and implementing partners (duty bearers). Secondary users include other agencies that work on children with disabilities in Malaysia, EAPRO, UNICEF Headquarters (i.e., Disability and Child Protection Divisions) (secondary duty bearers) and children with disabilities themselves (right holders).
3.2. Objectives and Scope of Work The objectives of the evaluation include the following: • Analyse the strategies used to map out the trajectory of the #ThisAbility Flagship and re- construct the theory of change behind to make positive changes for children with disabilities in Malaysia; • Assess the extent to which the #ThisAbility Flagship has been appropriately designed, efficiently and effectively managed, and integrated across the Country Programme, including the incorporation of equity, gender equality and human rights considerations; • Examine the evolution of UNICEF’s work on children with disability in Malaysia, and in particular the intermediate results achieved by UNICEF’s #ThisAbility Flagship, enabling and disabling factors, considering aspects of both reduction of discrimination as well as access to services – including advocacy and awareness raising both at the national and sub-national level; and its likely sustainability; and
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 52 • Analyse the utility of creating a flagship on specific focus issues to promote cross-sectoral work, including strengths and weaknesses.
The evaluation is expected to provide reasonable conclusions based on the findings and substantiated by evidence, and clear, specific and actionable recommendations for informing the future direction of UNICEF’s work on children with disability in Malaysia. It should also identify good practices, lessons learned and innovations throughout the evaluation process.
The evaluation will not be an impact evaluation, but it will cover the evolution of UNICEF’s work on disability in Malaysia since 2013, with a primary focus on the #ThisAbility Flagship initiative since 2016 to date. Data collection will focus on states where UNICEF is operating through partners, notably Kuala Lumpur in Peninsular Malaysia and Sabah and Sarawak in East Malaysia. To the extent possible, the evaluation should be participatory and include the views of children and adolescents with disabilities that participated in the Flagship as well as those of their caregivers.
Evaluation evidence will be judged using modified Organisation for Economic Co-operation and Development (OECD) / Development Assistance Committee (DAC) criteria of relevance, efficiency, effectiveness and sustainability, as well as equity, gender equality and human rights considerations. Key evaluation questions (and sub-questions) include the following:
Effectiveness of the UNICEF’s strategy to children with disability and its implementation considering: To what degree has UNICEF’s work on disability, and in particular through its #ThisAbility Flagship initiative, contributed to the creation of positive conditions to reduce discrimination and enable access to services to support the growth, development and protection of children and adolescents with disabilities in Malaysia? Have there been any unintended results?
Sub-questions: • How satisfied are the children and their caregivers with the quality of the services they have received through UNICEF’s partners? • To what degree has Disability Equality Training been effective in increasing knowledge of children with disabilities and changing understanding of disability from an individual to a social/rights based perspective? • How effective have partnerships with the private sector been at heightening their awareness of inclusion and promoting their adopting of strategies which model inclusion good practice? • How effective have partnerships with the private sector been in empowering children and adolescents with disabilities to advocate for themselves and in ensuring that they are socially included? • To what extent have public advocacy campaigns and behaviour change interventions been effectively designed and implemented to increase public awareness on children with disabilities? • To what extent has the #ThisAbility Flagship initiative enabled cross-sectoral work to increase access to high quality services for children with disabilities? • Has gender equality been taken into account in the planning and implementation of the #ThisAbility Flagship?
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 53 Relevance of the programme strategy and approach, considering: How relevant is UNICEF’s work on disability and its #ThisAbility Flagship initiative to the developmental challenges in Malaysia, and particularly attitudes and practices with regard to disability?
Sub-questions: • How relevant and appropriate has UNICEF’s work on disability been to the Agenda 2030, and should it be further adjusted to align with the SDG road-map? • How relevant and appropriate has UNICEF’s work on disability been to Malaysia Vision 2020 and National Transformation 2050? • To what extent is #ThisAbility Flagship initiative aligned with the new UNICEF’s Strategic Plan? • Has the design of the #ThisAbility Flagship initiative taken into account the different needs of girls and boys, according to their age, gender, ethnicity, religion or faith, disability category, state and other social identities, especially the most vulnerable? • Have the voices of children and adolescents with disabilities taken into account throughout?
Efficiency of the programme implementation and management, considering: To what extent and how has UNICEF mobilized and used its resources (human, technical and financial) and improved its cross-sectoral coordination to achieve its planned results for children with disabilities?
Sub-questions: • To what degree have UNICEF resources (human, technical and financial) been sufficient in implementing the activities under the #ThisAbility Flagship initiative? • How adequate are the capacities of UNICEF’s organizational and individual partners to engage on the rights of children with disabilities?
Sustainability in the programme, considering: To what extent are the benefits and achievements of the #ThisAbility Flagship initiative likely to continue after 2020 through national ownership and changes in terms of attitudes, knowledge and practices?
Sub-questions: • To what degree has the #ThisAbility Flagship initiative thus far contributed to the generation of sustainable capacities at the national and sub-national levels? • What are the enabling as well as containing factors that are likely to influence future cross-sectoral work on focus issues? One of the key tasks to be initiated at the proposal stage will be to interrogate these questions and criteria and determine if all key issues have been given due prominence. Bidders are required to propose appropriate evaluation criteria (e.g., OECD/DAC criteria for evaluating development programmes, including sub-criteria such as equity, gender equality, human rights). Improvements
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 54 and/or refinements to the draft questions may be offered at the proposal stage; however, the expectation is that the inception process will yield the final set of questions.69
4. Evaluation Approach and Methodology Based on the objectives of the evaluation, this section indicates a possible approach, methods, and processes for the evaluation.70 Methodological rigor will be given significant consideration in the assessment of proposals. Hence bidders are invited to interrogate the approach and methodology proffered in the ToR and improve on it, or propose an approach they deem more appropriate. In their proposal, bidder should clear refer to triangulation, sampling plan and methodological limitations and mitigation measures. Bidders are encouraged to also demonstrate methodological expertise in evaluating initiatives related to social inclusion and in particular the rights of children with disabilities.
It is expected that the evaluation will employ both a theory-based and a mixed methods approach drawing on key background documents and the monitoring framework for guidance. The timing of the evaluation is such that it will take mainly a formative approach, identify and assessing the effectiveness of UNICEF’s work on children with disabilities and to the extent possible children’s satisfaction to date to inform the future of the #ThisAbility Flagship initiative. To this end the evaluation will provide continuous and rapid feedback to primary users in the course of the evaluation process. The evaluation should consider throughout issues of equity, gender equality and human rights, in line with UNICEF’s Evaluation Policy (2013) and the UN Evaluation Group (UNEG) Norms and Standards (2016). Given the nature of the programme, equity, gender equality and human rights-based approach to development will be central in the conduct of the evaluation.
At minimum, the evaluation will draw on the following methods: • Desk review of background documents and other relevant data, including review and analysis of secondary quantitative data; • Key Informant Interviews (KIIs) with disability actors in Malaysia (existing and potential), non-governmental organizations participating in the NGO Impact Academy, journalists, other stakeholders such as teachers, medical practitioners, community-based rehabilitation workers, inclusion champions and UNICEF staff; • Focus Group Discussions (FGDs) with children and adolescents with disabilities and their caregivers; • Case studies of children and adolescents participating in the #ThisAbility Flagship initiative; • Cost-effectiveness analysis of the #ThisAbility Flagship initiative; and • Surveys.
69 The actual final decisions on the detailed questions will be taken in the inception phase, based on the following principles: 1) Importance and priority: the information should be of a high level of importance for the various intended audiences of the evaluation; 2) Usefulness and timeliness: the answer to the questions should not be already well known or obvious, additional evidence is needed for decision; 3) Answerability and realism: all the questions can be answered using available resources (budget, personnel) and within the appropriate timeframe; data and key informants are available and accessible, and performance standards or benchmarks exist to answer the questions; and 4) Actionability: the questions will provide information which can lead to recommendations that be acted upon to make improvements. 70 The proposed methodology is just indicative, and based on internal experience in conducting similar evaluations. The will be a need to develop a detailed design, analytical methods and tools during the inception phase based on additional literature review and in consultation with UNICEF Malaysia.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 55 The data collected should be disaggregated by age, gender, ethnicities, religion or faith, disability category, state, etc. where relevant and focus on the #ThisAbility Flagship activities both at the national and sub-national level. Sampling of KIIs and FGDs should be done in consultation with UNICEF. Baseline data will be provided based on the KAP survey conducted in 2016. Additionally, secondary data sources can be used. Other secondary data specific to the #ThisAbility Flagship will be provided by UNICEF, such as monitoring reports.
Likewise, conventional ethical guidelines are to be followed during the evaluation. Specific reference is made to the UNEG Norms and Standards and Ethical Guidelines, as well as to the UNICEF’s Evaluation Policy, the UNEG Guidance on Integrating Human Rights and Gender Equality in Evaluation, the UN SWAP Evaluation Performance Indicator,71 and the UNICEF Procedure for Ethical Standards in Research, Evaluation and Data Collection and Analysis and UNICEF’s Evaluation Reporting Standards.72 Good practices not covered therein are also to be followed. Any sensitive issues or concerns should be raised with the Evaluation Manager as soon as they are identified.
5. Management and Conduct of the Evaluation 5.1. Evaluation Management Structure The evaluation will be conducted by an external evaluation team to be recruited by UNICEF Malaysia. The evaluation team will operate under the supervision of the Evaluation Specialist at UNICEF Malaysia that will act as evaluation manager and therefore be responsible for the day- to-day oversight and management of the evaluation and for the management of the evaluation budget. The evaluation manager will assure the quality and independence of the evaluation and guarantee its alignment with UNEG Norms and Standards and Ethical Guidelines and other relevant procedures, provide quality assurance checking that the evaluation findings and conclusions are relevant and recommendations are implementable, and contribute to the dissemination of the evaluation findings and follow-up on the management response. Additional quality assurance will be provided by the Regional Evaluation Adviser. The final report will also be approved by the Country Representative at UNICEF Malaysia.
A reference group will be established, bringing together the Chief of Child Protection at UNICEF Malaysia, the Disability Consultant, the Chief of Communication for Development, the Innovation Specialist, the Education Specialist and representatives from UNICEF EAPRO (Regional Disability Focal Point), UNICEF Headquarters (Disability and Child Protection Divisions), JKM (Department for Persons with Disabilities/National Council for Persons with Disabilities), civil-society organizations, Disabled People’s Organisations/Disability Activist and Academia (Malaysian University academics/Study Centres that are specialists on Disability Studies). The reference group will have the following role: contribute to the preparation and design of the evaluation, including providing feedback and comments on the inception report and on the technical quality of the work of the consultants; provide comments and substantive feedback to ensure the quality – from a technical point of view – of the draft and final evaluation reports; assist in identifying internal and external stakeholders to be consulted during the evaluation process; participate in review meetings organized by the evaluation management team and with the evaluation team as required; play a key role in learning and knowledge sharing from the evaluation results, contributing to disseminating the findings of the evaluation and follow-up on the implementation of the management response.
71 Please refer to: http://www.unicef.org/evaluation 72 https://www.unicef.org/supply/files/ATTACHMENT_IV-UNICEF_Procedure_for_Ethical_Standards.PDF
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 56 5.2. Evaluation Team Profile The evaluation will be conducted by engaging an institution. The proposed team consists of one (1) international senior-level consultant (Team Leader) to conduct the evaluation that will be supported by at least one (1) national consultant (Team Member/Technical Expert).
The Team Leader should bring the following competences: • Having extensive evaluation experience (at least 15 years) with an excellent understanding of evaluation principles and methodologies, including capacity in an array of qualitative and quantitative evaluation methods, and UNEG Norms and Standards. • Having extensive experience on social inclusion interventions – planning, implementing, managing or monitoring and evaluation. • Holding an advanced university degree (Masters or higher) in international development, public policy, public health, sociology, anthropology or similar, including sound knowledge of policy and systemic aspects; familiarity with the rights of children with disabilities. • Bringing a strong commitment to delivering timely and high-quality results, i.e., credible evaluations that are used for improving strategic decisions. • Having in-depth knowledge of the UN’s human rights, gender equality and equity agendas. • Having a strong team leadership and management track record, as well as excellent interpersonal and communication skills to help ensure that the evaluation is understood and used. • Specific evaluation experience of disability programming is strongly desired, but is secondary to a strong mixed-method evaluation background, so long as the disability expertise of the other team member (see below) is harnessed to ensure the team’s collective understanding of issues relating to disability programming from a UN or NGO perspective. • Previous experience of working in an East Asian context is desirable, together with understanding of Malaysia context and cultural dynamics. • The Team Leader must be committed and willing to work independently, with limited regular supervision; s/he must demonstrate adaptability and flexibility, client orientation, proven ethical practice, initiative, concern for accuracy and quality. • S/he must have the ability to concisely and clearly express ideas and concepts in written and oral form as well as the ability to communicate with various stakeholders in English. The Team Leader will be responsible for undertaking the evaluation from start to finish, for managing the evaluation, for the bulk of data collection, analysis and consultations, as well as for report drafting in English and communication of the evaluation results.
One (1) national Team Member/Technical Expert: • Holding advanced university degrees (Masters-level) in international development, public policy or similar. • Hands-on experience in collecting and analysing quantitative and qualitative data, but this is secondary to solid expertise in social inclusion interventions. • Strong expertise in equity, gender equality and human rights based approaches to evaluation and expertise in data presentation and visualisation. • Be committed and willing to work in a complex environment and able to produce quality work under limited guidance and supervision.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 57 • Having good communication, advocacy and people skills and the ability to communicate with various stakeholders and to express concisely and clearly ideas and concepts in written and oral form. • Excellent Bahasa Malaysia and English communication and report writing skills. The Team Member will play a major role in data collection, analysis and presentation, and preparation of the debriefings and will make significant contributions to the writing of the main evaluation report.
The evaluation team is expected to be balanced with respect to gender to ensure accessibility of both male and female informants during the data collection process. Back-office support assisting the team with logistics and other administrative matters is also expected. It is vital that the same individuals that develop the methodology for the RFPS will be involved in conducting the evaluation. In the review of the RFPS, while adequate consideration will be given to the technical methodology, significant weighting will be given to the quality, experience (CV’s and written samples of previous evaluations) and relevance of individuals who will be involved in the evaluation.
5.3. Evaluation Deliverables Evaluation products expected for this exercise are: 1) An inception report (in English), including a summary note in preparation for data collection (in both English and Bahasa Malaysia); 2) A report of the initial evaluation findings from primary data collection (in English), including a desk review analysis and a PowerPoint presentation to facilitate a stakeholder consultation exercise; 3) A draft and final report (in English and Bahasa Malaysia) that will be revised until approved (incl. a complete first draft to be reviewed by the evaluation management team and UNICEF; a second draft to be reviewed by the reference group and Regional Evaluation Adviser within UNICEF EAPRO, and a penultimate draft). The final report will need to be developed in several media to maximize its utility and accessibility including in text and audio versions; 4) A PowerPoint presentation (in both English and Bahasa Malaysia) to be used to share findings with the reference group and for use in subsequent dissemination events; and 5) A four-page executive summary (in both English and Bahasa Malaysia) that is distinct from the executive summary in the evaluation report and it is intended for a broader, non- technical and non-UNICEF audience. The executive summary should also be produced both in text and audio versions. Other interim products are: • Minutes of key meetings with the evaluation management team and the reference group; and • Presentation materials for the meetings with the evaluation management team and the reference group. These may include PowerPoint summaries of work progress and conclusions to that point. Outlines and descriptions of each evaluation products are meant to be indicatives, and include: • Inception report: The inception report will be key in confirming a common understanding of what is to be evaluated, including additional insights into executing the evaluation. At this stage, evaluators will refine and confirm evaluation questions, confirm the scope
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 58 of the evaluation, further improve on the methodology proposed in the ToR and their own evaluation proposal to improve its rigor, as well as develop and validate evaluation instruments. The report will include, among other elements: i) evaluation purpose and scope, confirmation of objectives and the main themes of the evaluation; ii) evaluation criteria and questions, final set of evaluation questions, and evaluation criteria for assessing performance; iii) evaluation methodology (i.e., sampling criteria), a description of data collection methods and data sources (incl. a rationale for their selection), draft data collection instruments, for example questionnaires, with a data collection toolkit as an annex, an evaluation matrix that identifies descriptive and normative questions and criteria for evaluating evidence, a data analysis plan, a discussion on how to enhance the reliability and validity of evaluation conclusions, the field visit approach, a description of the quality review process and a discussion on the limitations of the methodology; iv) proposed structure of the final report; v) evaluation work plan and timeline, including a revised work and travel plan; vi) resources requirements (i.e., detailed budget allocations, tied to evaluation activities, work plan) deliverables; v) annexes (i.e., organizing matrix for evaluation questions, data collection toolkit, data analysis framework); and vi) an evaluation briefing note for external communication purposes. The inception report will be 15-20 pages in length (excluding annexes), or approximately 15,000 words, and will be presented at a formal meeting of the reference group. • Initial evaluation findings: This report will present the initial evaluation findings from primary data collection, comprising the desk-based document review and analysis of the disability inclusion flagship programme. The report developed prior to the first drafts of the final report should be 10 pages, or about 8,000 words in length (excluding annexes, if any), and should be accompanied by a PowerPoint presentation that can be used for validation with key stakeholders. • Final evaluation report: The report will not exceed 40 pages, or 25,000 words, excluding the executive summary and annexes; it will be produced both in text and audio versions.73 • PowerPoint presentation: Initially prepared and used by the evaluation team in their presentation to the reference group, a standalone PowerPoint will be submitted to the evaluation management team as part of the evaluation deliverables. • An evaluation briefing note, data and a four-page executive summary for external users will be submitted to the evaluation management team as part of the evaluation deliverables, in both text and audio versions. • Reports will be prepared according to the UNICEF Style Guide and UNICEF Brand Toolkit (to be shared with the winning bidder) and UNICEF standards for evaluation reports as per GEROS guidelines (referenced before). All deliverables must be in professional level standard English and they must be language-edited/proof-read by a native speaker. • The first draft of the final report will be received by the Evaluation Manager and UNICEF who will work with the team leader on necessary revisions. The second draft will be sent to the reference group for comments. The Evaluation Manager will consolidate all comments on a response matrix, and request the evaluation team to indicate actions taken against each comment in the production of the penultimate draft.
73 UNICEF has instituted the Global Evaluation Report Oversight System (GEROS), a system where final evaluation reports are quality assessed by an external company against UNICEF/UNEG Norms and Standards for evaluation reports. The evaluation team is expected to reflect on and conform to these standards as they write their report. The team may choose to share a self-assessment based on the GEROS with the evaluation manager.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 59 Bidders are invited to reflect on each outline and effect the necessary modification to enhance their coverage and clarity. Having said so, products are expected to conform to the stipulated number of pages where that applies.
An estimated budget has been allocated for this evaluation. As reflected in Table 1, the evaluation has a timeline of four months from October 2018 to January 2019. Adequate effort should be allocated to the evaluation to ensure timely submission of all deliverables, approximately 12 weeks on the part of the evaluation team.
Table 1: Proposed Evaluation Timeline74
RESPONSIBLE ACTIVITY DELIVERABLE TIME ESTIMATE PARTY 1. INCEPTION, DOCUMENT 3 weeks, concurrent REVIEW AND ANALYSIS (Oct, 2018) 1. Inception meeting by Skype with Meeting minutes Week 1 Evaluation team, the Evaluation Specialist, Chief, evaluation manager Child Protection and Disability Consultant 2. Inception visit (incl. initial data Draft inception Week 1 Evaluation team collection and desk review; report development of evaluation matrix, methodology and work plan, data collection material, drafting of the inception report) 3. Present draft inception report to PowerPoint Week 2 Evaluation team, the reference group presentation evaluation manager, reference group
4. Receive inception report and Evaluation Week 3 Evaluation feedback to evaluation team commenting matrix manager, reference group 5. Present inception report, confirm Final inception Week 3 Evaluation team, planning for field visit report evaluation manager, reference group 5 weeks, 2. DATA COLLECTION consecutive (Oct to Nov, 2018) 1. Pilot data collection tools - Weeks 4-7 Evaluation team and conduct field-based data collection 2. Prepare initial evaluation findings Initial evaluation Week 8 Evaluation team, report and prepare presentation findings report evaluation manager, for validation workshop to (incl. desk review), reference group validate data collection results PowerPoint presentation, meeting minutes
74 Please note that the timing of the data collection may change depending on the possibility of carrying out KIIs and FGDs and other contextual factors.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 60 4 weeks, 3. REPORTING AND consecutive COMMUNICATION OF (Dec, 2018 – Jan RESULTS 2019) 1. Prepare and submit first draft of Draft report Week 9 Evaluation team evaluation report 2. Receive first draft and feedback Evaluation Week 10 Evaluation manager to evaluation team commenting matrix 3. Prepare and submit second draft Draft report Week 11 Evaluation team of evaluation report 4. Receive second draft and Evaluation Weeks 12-13 Evaluation manager, feedback to evaluation team commenting matrix reference group 5. Prepare and submit penultimate Draft report Week 14 Evaluation team draft of evaluation report 6. Submit and present final report Final report, Week 15 Evaluation team, to reference group and prepare executive summary, evaluation manager, presentation and other materials PowerPoint reference group presentation, meeting minutes
6. Payment Schedule Unless the proposers propose an alternative payment schedule, payments will be as follows: • Approved inception report: 25% of the contractual amount; • Approved initial evaluation findings report: 30% of the contractual amount; • Approved final report: 30%; and • Approved final presentation and other materials: 15%.
7. Application Process Each proposal will be assessed first on its technical merits and subsequently on its price. In making the final decision, UNICEF considers both technical and financial proposals. The Evaluation Team first reviews the Technical Proposals followed by review of the Financial Proposals of the technically compliant firms. The proposal obtaining the highest overall score after adding the scores for the technical and financial proposals together, that offers the best value for money, will be recommended for award of the contract.
The Technical Proposal should include but not be limited to the following: a) Request for Proposals for Services Form (provided above). b) Presentation of the Bidding Institution or institutions if a consortium (maximum two institutions will be accepted as part of the consortium), including: • Name of the institution; • Date and country of registration/incorporation; • Summary of corporate structure and business areas;
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 61 • Corporate directions and experience; • Location of offices or agents relevant to this proposal; • Number and type of employees; • In case of a consortium of institutions, the above listed elements shall be provided for each consortium members in addition to the signed consortium agreement; and • In case of a consortium, one only must be identified as the organization lead in dealing with UNICEF. c) Narrative Description of the Bidding Institution’s Experience and Capacity in the following areas: • Evaluation of social inclusion interventions; • Process evaluation of social inclusion interventions, ideally implemented by government institutions; • Previous assignments in developing countries in general, and related to social inclusion programmes, preferably in East Asia; and • Previous and current assignments using UNEG Norms and Standards for evaluation. d) Relevant References of the proposer (past and on-going assignments) in the past five years. UNICEF may contact references persons for feedback on services provided by the proposers. e) Samples or Links to Samples of Previous Relevant Work listed as reference of the proposer (at least three), on which the proposed key personnel directly and actively contributed or authored. f) Methodology. It should minimize repeating what is stated in the ToR. There is no minimum or maximum length. If in doubt, ensure sufficient detail. g) Work Plan, which will include as a minimum requirement the following: • General work plan based on the one proposed in the ToR, with comments and proposed adjustments, if any; and • Detailed timetable by activity (it must be consistent with the general work plan and the financial proposal). h) Evaluation Team: • Summary presentation of proposed experts; • Description of support staff (number and profile of research and administrative assistants etc.); • Level of effort of proposed experts by activity (it must be consistent with the financial proposal); and • CV of each expert proposed to carry out the evaluation.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 62 The Technical Proposal will be submitted in hard copy and electronic (PDF) format.
Please note that the duration of the assignment will be from October 2018 to January 2019, and it is foreseen that the Team Leader and the Team Member will devote roughly half of their time to the evaluation. The presence of a conflict of interest of any kind (e.g., having worked for or partnered with UNICEF on the disability inclusion flagship programme in the design or implementation phase will automatically disqualify prospective candidates from consideration).
The Financial Proposal should include but not be limited to the following: Bidders are expected to submit a lump sum financial proposal to complete the entire assignment based on the different sample sizes (as per instruction given under Section 8 below). In order to arrive at the lump-sum offer for a given sample size, the firm may work out the budget detail as below: a) Resource Costs: Daily rate multiplied by number of days of the experts involved in the evaluation. b) Conference or Workshop Costs (if any): Indicate nature and breakdown if possible. c) Travel Costs: All travel costs should be included as a lump sum fixed cost. For all travel costs, UNICEF will pay as per the lump sum fixed costs provided in the proposal.A breakdown of the lump sum travel costs should be provided in the financial proposal. d) Any Other Costs (if any): Indicate nature and breakdown. e) Recent Financial Audit Report: Report should have been carried out in the past 2 years and be certified by a reputable audit organization.
Bidders are required to estimate travel costs in the Financial Proposal. Please note that: i) travel costs shall be calculated based on economy class fare regardless of the length of travel; and ii) costs for accommodation, meals and incidentals.
The financial proposal must be fully separated from the Technical Proposal. The financial proposal will be submitted in hard copy and electronic (PDF) format. Costs will be formulated in US$ and free of all taxes.
8. Evaluation Weighting Criteria The proposals will be evaluated against the two elements: technical and financial. The ratio between the technical and financial criteria depends on the relative importance of one component to the other. Cumulative Analysis will be used to evaluate and award proposals. The evaluation criteria associated with this ToR is split between technical and financial as follows:
• Weightage for Technical Proposal = 70% • Weightage for Financial Proposal = 30% • Total Score = 100% a. Technical Proposal: The Technical Proposal should address all aspects and criteria outlined in this Request for Proposal.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 63 Table 2: Evaluation of Technical Proposal
The Technical Proposals will be evaluated against the following: REF CATEGORY POINTS 1 Overall response: • Completeness of response 2 • Overall concord between RFP requirements and proposal 3
2 Company/key personnel/Individual Consultant: • Range and depth of experience with similar projects 8 • Samples of previous work 5 • References 5 • Key personnel: relevant experience and qualifications of the 14 proposed team for the assignment
3 Proposed methodology and approach: • Detailed proposal with main tasks, including sound methodology to 20 achieve key outputs • Proposal presents a realistic implementation timeline 13
Total Technical 70 Only proposals which receive a minimum of 55 points will be considered further. b. Financial Proposal The total amount of points allocated for the price component is 30. The maximum number of points will be allotted to the lowest price proposal that is opened and compared among those invited firms/institutions which obtain the threshold points in the evaluation of the technical component.
All other price proposals will receive points in inverse proportion to the lowest price, e.g., Max. score for price proposal * Price of lowest priced proposal Score for price proposal X = ------Price of proposal
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 64 ANNEX B: EVALUATION MATRIX
Sources of Main informants Overarching questions Sub-questions Indicators Key documents information (qual)
Percent satisfied How satisfied are the children and their Internal document Disabled young people caregivers with the quality of the services Feedback forms/MI - if available from UNICEF or Average level of satisfaction review they have received through UNICEF’s partners (TBC) Parents of disabled FGOs children partners? Qualitative evidence of satisfaction
DET documentation To what degree has Disability Equality Evidence of training materials incoprorating a Internal document Peter Tan - UNICEF DET 2017 Disabled young people Training been effective in increasing social/rights-based perspective review knowledge of children with disabilities Worksheet DET UNICEF 2017 Parents of disabled FGOs children and changing understanding of disability Percent reporting change in understanding Outcomes-Outputs and Indicator Status by Regional from an individual to a social/rights based / Business Area seep.9.29 Qualitative evidence of change in understanding KIIs UNICEF staff perspective? Feedback forms/MI - if available (TBC)
Activities matrix Partnership agreements How effective have partnerships with the Internal document Qualitative evidence of change in awareness review Company documentation private sector been at heightening their UNICEF staff awareness of inclusion and promoting Documented evidence of changes in strategies External document 2016 COAR Disability Summary Analysis review Partners Effectiveness of the their adopting of strategies which model UNICEF Rolling Workplan 2018 Final inclusion good practice? Qualitative evidence of change in strategies UNICEF’s strategy to KIIs Thisability activity matrix 2017-2019 (project 3,10) children with disability and its Company documents including websites implementation considering: Activities matrix Disabled young people To what degree has UNICEF’s How effective have partnerships with Interviewee assessments of outcomes of private Internal document the private sector been in empowering sector activities review Parents of disabled Partnership agreements children work on disability, and children and adolescents with disabilities FGOs Private sector interviewee assessments of role of Child Rights in Business Principles in particular through its to advocate for themselves and in UNICEF staff ensuring that they are socially included? UNICEF in fostering partnerships KIIs MI - if available #ThisAbility Flagship initiative, Partners contributed to the creation of MI (including social media) Alignment of UNICEF Malaysia’s strategy with positive conditions to reduce _thisability Flagship Monitoring barriers prioritized by children/YP or evidenced in discrimination and enable literature Disability KAP Internal document Disabled young people access to services to support To what extent have public advocacy Alignment of UNICEF Malaysia’s strategy with review Country Programme Action Plan Parents of disabled the growth, development and campaigns and behaviour change international best practice/rights-based approach External document Country Programme Document children interventions been effectively designed review UNICEF staff protection of children and and implemented to increase public Interviewee perspectives on appropriateness of Country Programme Management Plan FGOs adolescents with disabilities awareness on children with disabilities? strategy and inclusion of relevant stakeholders in RAM Partners programme design KIIs in Malaysia? Have there been EN_X-Sectoral Programme Guidance Disability Final Government Quantitative evidence of increased public Strategic Plan any unintended results? awareness UNSPFR
Internal document Task Force meeting minutes Disabled young people To what extent has the #ThisAbility Interviewee assessments of outcomes of cross- review Flagship initiative enabled cross-sectoral sectoral work RAM Reports Annual Reports Parents of disabled work to increase access to high quality Documented evidence of changes in FGOs Disability Flagship meeting minutes children services for children with disabilities? programming in all sections KIIs MI information UNICEF staff
Evidence of gender parity in disaggregated output EN_Annex A_Intervention by Focus Area Disabled young people and outcome indicators (by gender) Has gender equality been taken Internal document Parents of disabled review _thisability Flagship Monitoring Framework children into account in the planning and Interviewee perspectives on inclusion of gender- #thisability Flagship Working Paper implementation of the #ThisAbility sensitive approaches FGOs UNICEF staff Flagship? Annex B_Key Messaging Disability Evidence of gender mainstreaming approaches in KIIs Partners strategy documents Final File Childhood Disability in Malaysia Alignment of UNICEF Malaysia’s strategy/TOC Disability KAP with barriers prioritized by children/YP and their Its_About_Ability Final Disabled young people UNICEF strategy UNICEF Children_with_ Disability How relevant and appropriate has families docs Parents of disabled UNICEF Childhood Disability in Malaysia children UNICEF’s work on disability been to the Interviewee perspectives on appropriateness of UNICEF reports SWCR2013_ENG_ Lo_res24April Agenda 2030, and should it be further strategy and inclusion of relevant stakeholders in UNICEF staff SDG documentation Country Programme Action Plan adjusted to align with the SDG road-map? programme design Partners External documents Country Programme Document Disaggregation of evidence by subgroup (gender, Country Programme Government age, region, impairment, etc.) SDG documents
Disability KAP Its_About_Ability Final Internal document Disabled young people Alignment of UNICEF Malaysia’s strategy/ review UNICEF Children_with_ Disability How relevant and appropriate has TOC with Malaysia Vision 2020 and National Parents of disabled External document UNICEF Childhood Disability in Malaysia children UNICEF’s work on disability been to Transformation 2050 SWCR2013_ENG_ Lo_res24April Malaysia Vision 2020 and National review UNICEF staff Country Programme Action Plan Transformation 2050? Interviewee perspectives on alignment of FGOs Partners Relevance of the programme UNICEF’s work with national goals and activities Country Programme Document KIIs strategy and approach, Country Programme Government considering: How relevant is Government documents UNICEF’s work on disability and its #ThisAbility Flagship _thisabilty Flagship Monitoring Framework #thisabilty Flagship Woring Paper initiative to the developmental Alignment of UNICEF Malaysia’s strategy/ Internal document Annex B_Key Messaging Disability Disabled young people challenges in Malaysia, To what extent is #ThisAbility Flagship TOC with Malaysia Vision 2020 and National review Transformation 2050 Final File Childhood Disability in Malaysia Parents of disabled and particularly attitudes initiative aligned with the new UNICEF’s FGOs children Strategic Plan? Annex 4-2018-03-16 PRIME 2018-2019 and practices with regard to Interviewee perspectives on alignment of KIIs UNICEF staff UNICEF’s work with national goals and activities Annual Management Plan 2018 disability? MCO-CPAP 2016-2020 Signed FINAL CPMP 11052015 MLY
Has the design of the #ThisAbility Evidence of programme strategies and EN_Annex A_Intervention by Focus Area Disabled young people Flagship initiative taken into account documents disaggregated by sub-group Internal document the different needs of girls and boys, review _thisabilty Flagship Monitoring Framework Parents of disabled Interviewee perspectives on inclusiveness of #thisabilty Flagship Woring Paper children according to their age, gender, ethnicity, UNICEF’s approach FGOs religion or faith, disability category, state UNICEF staff KIIs Annex B_Key Messaging Disability and other social identities, especially the Children/family perspectives on alignment of Partners most vulnerable? UNICEF’s approach with needs Final File Childhood Disability in Malaysia Evidence of processes for engagement with UNICEF staff children being in place Internal document Have the voices of children and review Final File Childhood Disability in Malaysia Disability stakeholders Interviewee perspectives on the effectiveness of 2017-2018-Final Result Framework-activities 4,5,11 adolescents with disabilities been taken processes FGOs Disabled young people into account throughout? KIIs Thisability activities matrix 2017-2019 Parents of disabled Alignment of strategy with priorities and needs children expressed by children and their families
Interviewee perspectives on capacity to Efficiency of the programme implement activities in TOC implementation and Annual Management Plan To what degree have UNICEF resources Documented evidence of budget and staff UNICEF Rolling Workplan management, considering: resources to carry out activities in TOC Internal document UNICEF staff (human, technical and financial) been review RAM sufficient in implementing the activities To what extent and how Evidence of effective governance structures/ KIIs Partners under the #ThisAbility Flagship initiative? procedures Organigram has UNICEF mobilized and Evaluations of other UNICEF flagships used its resources (human, Benchmarking of resources against comparator programmes technical and financial) and improved its cross-sectoral Number of FTE UNICEF and partner staff Disabled young people How adequate are the capacities of Internal document coordination to achieve its working on the rights of children with disabilities review Parents of disabled UNICEF’s organizational and individual (disaggregated by years of experience in Data request to UNICEF and partners children planned results for children partners to engage on the rights of FGOs disability) Org chart UNICEF staff children with disabilities? KIIs with disabilities? Interviewee perspectives on capacity Partners
Disabled young people Sustainability in the Evidence of capacity developed programme, considering: To what degree has the #ThisAbility Parents of disabled Interviewee perspectives on risks to sustainability FGOs children Flagship initiative thus far contributed to and likely sustainability To what extent are the the generation of sustainable capacities KIIs UNICEF staff benefits and achievements at the national and sub-national levels? Evidence of mitigation strategies for risks Partners identified of the #ThisAbility Flagship Government initiative likely to continue after 2020 through national What are the enabling as well as Interviewee perspectives on key factors ownership and changes containing factors that are likely to KIIs UNICEF staff influence future cross-sectoral work on Lessons learnt from existing activities in terms of attitudes, focus issues? knowledge and practices? ANNEX C: RESEARCH TOOLS This annex contains the research tools for the key informant interviews (KIIs) and focus group discussions (FGDs). These tools were developed in line with the evaluation matrix. Each KII had a duration of approximately 120 minutes.
Tool 1: Key Informant Interview – Topic Guide Note to Interviewer These consultations aim to inform the development of a Theory of Change (ToC) for the UNICEF Malaysia #ThisAbility Flagship Initiative, as well as to address the specific questions in the evaluation framework. Equal International and Ipsos will complete consultations with representatives from UNICEF, its partners, and government stakeholders. This topic guide asks stakeholders about: • their experiences of the Initiative; • inputs, activities and outputs of the Initiative, intended outcomes, long-term impacts and key risks and assumptions underpinning the ToC; • wider contextual factors to be aware of; • views on the strategy of the Initiative and its alignment with the needs of its intended beneficiaries. Consultations are semi-structured; interviewers should be aware that not all consultees will be able to provide responses to all questions, and should therefore tailor interviews accordingly by selecting relevant questions, probes and according to previous involvement in the programme.
Introduction (All) Interviewer Note: Interviewers should give the following introduction before the start of each interview (although this should be tailored for each consultee):
My name is … and I am from Ipsos, the market and social research company. Your contact details were provided to my organization by UNICEF. As you know, Ipsos and our partners Equal International have been commissioned by UNICEF Malaysia to complete an evaluation of the #ThisAbility Flagship Initiative. The evaluation aims to assess the implementation of the Initiative to date and its future strategy.
The purpose of this consultation is to support our understanding of the Initiative. This includes discussing: • the rationale for the Initiative, its objectives and anticipated results; • your experience of the Initiative; • the wider context for the Initiative.
Information collected in consultations will be used to evaluate the relevance, effectiveness, efficiency, impact and sustainability of the #ThisAbility Flagship initiative. Personal Data shall only be processed for the purposes of UNICEF’s #ThisAbility Flagship Initiative Evaluation.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 67 The study team will not attribute quotes to any consultees. However, given the specific roles of consultees, complete anonymity cannot be guaranteed. Please be assured that you can request the interview to be stopped at any time. You are not obliged to provide responses to any question during the interview.
Interviewer Note: Please obtain the following consultee consent prior to starting the interview: • Obtain verbal consent from the consultee that they are happy to complete the interview. Note their response. • If a recorder is used, request permission to record the interview prior to starting the recording. Additionally, obtain consent from the consultee to complete the interview once the recording has started. State that the recording will be securely destroyed upon completion of the evaluation. • Close the interview if consent is not provided by the consultee. • Check that the consultee has any questions about the consultation before moving on.
Overview (All) - Please could you give me a brief overview of your role?
o Is your role disability-focussed? If not, to what extent does your work relate to disability? o How long have you been in this role? - For government and partners:
o Please describe your organization and its key activities. o Is the organization’s remit regional or national? o What is the nature of the relationship of your organization with UNICEF? - For disability partners:
o Please describe to what extent and in what ways your organization works on behalf of children and young people with disabilities.
o To what extent are your organization’s activities informed by: . Agenda 2030/the SDG agenda? . Malaysia Vision 2020, National Transformation 2050, and any other national goals or activities? . The needs or priorities identified by children/young people with disabilities and/or their families? • Have you undertaken any direct consultations with children and young people with disabilities as part of your activity planning?
o Do you work in collaboration with any other organizations/agencies in relation to disability?
o What services does your organization provide that are inclusive for children and young people with disabilities?
o Are there any specific gender issues that you take into consideration when providing these services? Are there any other ways in which you take into account the different needs of children, especially the most vulnerable?
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 68 - For mainstream partners:
o Prior to becoming involved in the #ThisAbility Flagship Initiative, did your organization have a disability inclusion strategy (whether formally or informally)? - How have you been involved in the UNICEF #ThisAbility Flagship Initiative?
o When did you become involved? o Probe on any involvement in: design, implementation, or management of the initiative. When, how and why did you become involved?
o Probe on involvement in/awareness of specific activities (see boxA).
Box A: Specific activities delivered by the #ThisAbility Flagship Initiative
Programme design related activities, for example: (1) Inclusive Playground design collaboration (2) ‘Inclusion Champions’ – identification of private sector partners Capacity building activities, e.g.: (3) NGO Impact Academy – 15-week course for NGO partners (4) Disability Equality Training (DET) – workshops for UNICEF staff, partners, and government (5) Media Disability Training – training for journalists (6) Disability-Inclusion Teacher Training – pilot project in Sabah (7) Design and development of Inclusive Ethos training for Petrosains Activities with partners, e.g.: (8) National launch of the Childhood Disability: KAP Study – with Petrosains (9) #ThisAbility Calendar – with MPH Book Store (10) Disability Theory of Change Exercise with MCO Public activities, for example: (11) #ThisAbility Makeathon – team-based activities for children (12) #ThisAbility public advocacy campaigns – awareness-raising campaigns, including Borneo International Marathon, ASEAN Para Games, Manchester United PSA campaign, What’s So Special? Campaign (13) National launch of the Childhood Disability: KAP Study (14) National Conference on Inclusion ‘One Child, One Family, One Community’ (15) Disability Inclusion Communication for Development (C4D) Ground Activation – Carnival in 2018, and planned activity in 2019 (16) #ThisAbility Truck – branding of truck with messaging
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 69 Programme design (UNICEF staff, disability partners, and government)
Rationale for the Initiative - In your own words, how would you describe the aims of the #ThisAbility Flagship Initiative? What is the rationale for the Initiative? - In your own words, how would you describe the overall strategy for the Initiative? How does the Initiative intend to achieve its expected aims?
o How was this strategy developed? o Could you please describe how UNICEF Malaysia first began to work on disability issues? What kind of things were happening at that time?
o In your view, what prompted this work to start? - In your view, how relevant and appropriate has the Initiative been to the following goals and strategies, and (if you are aware) how were these considered in the design of the programme:
o Agenda 2030/the SDG agenda? o Malaysia Vision 2020, National Transformation 2050, and any other national goals or activities?
o UNICEF’s Strategic Plan? o Are there any other goals or strategies that are relevant to the Initiative? - What do you consider are the main barriers facing children and young people in Malaysia in the sector that you work in? - In what ways do you see disability as having an impact in your area of work? - In your view, how relevant and appropriate has the Initiative been to the following:
o The barriers prioritized by children/young people and their families? o The different needs of boys and girls? o The different needs of children of different ages? o The different needs of children with different ethnicities, religions or faiths? o The different needs of children across disability categories? o The different needs of children with different legal status (e.g. migrants) or other social identities (such as working status, education level and socio-economic status)?
o The most vulnerable children? - Who would you consider the key stakeholders for the Initiative? To what extent were relevant stakeholders involved in the design of the Initiative?
o To what extent have the voices of children and adolescents with disabilities been taken account in the design and implementation of the programme?
o Have you undertaken any direct consultations with children and young people with disabilities as part of programme planning? If yes, can you please describe what kind of accommodations were made for children of different ages/gender/impairments.
o What processes are in place for engaging with children throughout implementation of the programme, and do you consider these to be effective?
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 70 Implementation of the Initiative - How do you think the way UNICEF Malaysia approaches disability has changed during your time working with/at UNICEF? - In your view, what are the strengths and weaknesses of the UNICEF team working on the #ThisAbility Flagship Initiative?
o Probe on: knowledge of disability, design of the Initiative, management of the Initiative, cross-sectoral engagement, financial management
o Does the UNICEF team have sufficient human, technical and financial resources to achieve its mission?
Implementation and management (UNICEF staff) UNICEF processes - From your perspective, how is UNICEF Malaysia’s work on disability inclusion coordinated? What guidelines or guidance have you been following? Do you have a strategy that you are using? (Please provide any documents you might have that outline your strategic-level approach to disability.) - To what extent has the #ThisAbility Flagship Initiative enabled cross-sectoral work? In what ways have you collaborated with other sectors in the development and implementation of disability-inclusive activities/programmes? Can you provide any specific examples where you have collaborated? - How do you currently measure the inclusion and participation of children and young people with disabilities in your programmes? Please provide details including the extent to which this information is further disaggregated by age, gender and other vulnerability variables.
Internal results - To what extent has the Initiative changed your knowledge of children with disabilities?
o What was your level of knowledge prior to being engaged in the Initiative? o What were your key learnings? o What do you understand are the differences between taking a charity/medical- based approach to disability inclusion and a human rights-based approach? - Has cross-sectoral work resulted in changes in programming across sections, or in the way programmes have been delivered? If so, how was this achieved? - Has cross-sectoral work resulted in increased access to high quality services for children with disabilities? If so, how? Why was the cross-sectoral collaboration important? - What factors were important for success? What internal or external factors may have constrained success? - To what extent are these results likely to be sustained over time? Why (or why not) might the results be sustained? How could sustainability of results be improved?
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 71 Capacity building activities (Respondents who have been involved in capacity building activities) Overview - You said that you have been involved in capacity-building activities under the #ThisAbility Flagship Initiative. In your own words, what is the rationale for these capacity building activities? - How effective do you consider the capacity building activities to have been in achieving their intended aims? Why/why not?
Results - For UNICEF staff:
o To what extent have capacity building activities (especially Disability Equality Training) been effective in increasing knowledge of children with disabilities? What evidence of this result exists? How was this achieved?
o To what extent have capacity building activities (especially Disability Equality Training) been effective in changing understanding of disability from an individual to a social/rights based-perspective? What evidence of this result exists? How was this achieved? - For all others:
o Please describe in what ways you believe your organization has benefitted? . Probe on: increased awareness, new programmes or services, new partnerships.
o To what extent have capacity building activities changed your knowledge of children with disabilities? What was your level of knowledge prior to completing the activity/ activities? What were your key learnings?
o What do you understand are the differences between taking a charity/medical-based approach to disability inclusion and a human rights-based approach? - All:
o What factors were important for success? What internal or external factors may have constrained success?
o To what extent are these results likely to be sustained over time? Why (or why not) might the results be sustained? How could sustainability of results be improved?
Activities with partners (Partners and other respondents who have been involved in partnerships)
Overview (Partners) - Please describe to what extent and in what ways your organization works to include children and young people with disabilities.
o What services do you provide that are inclusive for children and young people with disabilities?
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 72 o Are there any specific gender issues that you take into consideration when providing these services?
o When did your organization first begin to work on disability inclusion? Do you know what prompted that? - Does your organization follow any guidelines or guidance in regards to your partnership with UNICEF on disability? Do you have a strategy that you are using or have any overarching aims/objectives? Would you be able to share any documents you have which outline your approach to disability? - Has UNICEF provided your organization with any training or support in the area of disability rights/disability inclusion? If yes, please describe the training/support you have received. What were the benefits? (Probe: increased awareness, new programmes or services, new partnerships.)
Rationale/Aims - From your perspective, what do you think UNICEF Malaysia’s overall objective is when it comes to developing partnerships (with organizations like yours)? - How effective do you consider the partnership/partnership activities to have been in achieving the intended aims? Why/why not? - For partners:
o Has your organization undertaken any direct consultations with children and young people with disabilities as part of your activity planning?
Results - For partners:
o How does your organization currently measure the inclusion and participation of children and young people with disabilities in your programmes/activities/services? Are any of these measures further disaggregated by gender or any other vulnerability variables?
o Do you have an organizational policy on the inclusion of people with disabilities in your workforce? Has this been influenced in any way by your organization’s involvement with UNICEF? - All:
o To what extent have partnerships resulted in empowerment of children and adolescents? Has empowerment resulted in children advocating for themselves to ensure they are socially included? - For partners:
o To what extent has your participation in the Initiative changed your knowledge of children with disabilities? What was your level of knowledge prior to being engaged by UNICEF? What were your key learnings?
o What do you understand are the differences between taking a charity/medical-based approach to disability inclusion and a human rights-based approach?
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 73 - For UNICEF staff: o What, in your experience, is the general response from partners towards the inclusion of children and young people with disabilities? - All: o What factors were important for success of partnership activities? What internal or external factors may have constrained success?
o To what extent are these results likely to be sustained over time? Why (or why not) might the results be sustained? How could sustainability of results be improved?
Public activities (Partners and other respondents who have been involved in public activities)
Rationale/Aims - You said that you have been involved in campaigns or other public activities under the #ThisAbility Flagship Initiative. In your own words, what is the rationale for these activities? - To what extent were relevant stakeholders involved in the design of public activities?
o To what extent does the strategy align with wider strategies? With the needs prioritized by children or their advocates?
o Was a rights-based approach incorporated in the design of public activities? How? - How effective do you consider the public activities to have been in achieving their intended aims? Why/why not?
Results - To what extent have these activities resulted in increased public awareness of children with disabilities? What evidence is available to demonstrate this? - What factors were important for success? What internal or external factors may have constrained success? - To what extent are these results likely to be sustained over time? Why (or why not) might the results be sustained? How could sustainability of results be improved?
Other (All) - Have there been any unintended results from the Initiative, whether positive or negative? How did these results come about? - In your view, what types of activities would you like to see UNICEF undertake in the future? Are there any activities that UNICEF is currently undertaking through the #ThisAbility Flagship Initiative that are less valuable? - Do you have any other comments or suggestions for the #ThisAbility Flagship Initiative which this evaluation should consider?
Contextual factors (Disability partners) - What, in your experience, is the general response from the public towards the inclusion of children and young people with disabilities? Does this vary when you take factors like age, gender or impairment into consideration?
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 74 - How prepared do you believe government is to incorporate the rights of children with disabilities into the delivery of services and programmes? - In your experience, what are the key needs or priorities for children with disabilities or their families? How are these needs currently being met by your organization, the government, UNICEF, and others? - What are the greatest challenges your organization faces in working inclusively with children and young people with disabilities? Does this vary when you take factors like age, gender or impairment into consideration? - What have been the biggest benefits gained from working inclusively with children and young people with disabilities?
Close (All) Interview Note: To finish the interview, check through the list of activities below: - Request permission to re-contact interviewee to clarify information - Check consent to use the consultation to support the evaluation - Reminder that participation is anonymous - Thank and close.
Tool 2: Focus Group Discussion Overview
Below is a list of topics for inclusion in focus group discussions with children with disabilities and their parents/carers. These topics have been selected in line with the evaluation matrix. Appropriate tools and any necessary stimuli will be developed as part of the research phase following confirmation of the Inception Report. Each FGD will have a duration of 90–120 minutes. Prior to the FGDs, a consent form will be collected from each respondent (sample in Annex G)
Discussion topics - Experiences and needs of children with disabilities
o Consideration of different needs according to: . Gender . Age . Ethnicity . Religion . Disability category . Other social identities75 (such as working status, education level and socio- economic status)
o Priorities for services and support
75 Social identity is defined as a person’s sense of who they are based on their group membership(s). In this regard, we will be capturing information such as their working status, education level and socio-economic status, which is represented by their income level. This information is obtained from a set of questions administered during the recruitment period.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 75 - Experiences of participation in UNICEF programming
o Activities participated in o Perceptions of programme . Overall perceptions . Relevance of the programme to children’s needs . Satisfaction with quality of services provided/activities . Consideration of different needs according to: • Gender • Age • Ethnicity • Religion • Disability category • Other social identities (such as working status, education level and socio- economic status) . Satisfaction with staff, including knowledge, skills, and responsiveness to needs . Satisfaction with opportunities to provide feedback
o Effects of the programme on: . Increasing knowledge of children/parents . Changing understanding of children/parents to a rights-based perspective . Access to services (and quality of services) . Changing behaviour of children/parents
o Likelihood of sustainability of effects - Experiences of participation in UNICEF partner programming
o Activities participated in o Perceptions of programming . Overall perceptions . Relevance of the programme to children’s needs . Satisfaction with quality of services provided/activities . Consideration of different needs according to: • Gender • Age • Ethnicity • Religion • Disability category • Other social identities (such as working status, education level and socio- economic status)
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 76 . Satisfaction with staff, including knowledge, skills, and responsiveness to needs . Satisfaction with opportunities to provide feedback
o Effects on: . Feelings of empowerment . Changes in children’s behaviour: advocating for themselves . Social inclusion of children with disabilities . Access to services (and quality of services)
o Likelihood of sustainability of effects - Public advocacy campaigns
o Awareness of campaigns o Perceptions of campaigns . Relevance of campaigns to children’s needs
o Effects on public awareness of children with disabilities o Likelihood of sustainability of effects - Future strategy
o Types of activities children/carers would like to see UNICEF carry out o Availability of other services, and UNICEF’s unique role
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 77 ANNEX D: STAKEHOLDERS INTERVIEWED
In total, 42 interviews were conducted; five interviewees were interviewed twice. 1. Manager of Cheshire Homes Services 2. Race Director, Secretariat of UNICEF Borneo Marathon 3. Coordinator, Secretariat of UNICEF Borneo Marathon 4. Director of Programmes and Services at Seri Mengasih Centre 5. Manager at Sabah Art Gallery 6. Co-Founder of Persatuan Child Sabah 7. Co-Founder of Persatuan Child Sabah 8. Principal at Sabah Society for the Deaf 9. Director of Sabah State Library 10. Manager of Special Olympics Sabah 11. President of Special Olympics Sabah 12. Executive Council for Sabah Council of Social Services 13. Associate #1, Tandemic 14. Associate #2, Tandemic 15. Director of Research and Innovation, Petrosains 16. U Report lead; UNICEF consultant 17. Partnerships and Programming Consultant in Sabah, UNICEF 18. CSR Officer, UNICEF 19. Innovation Specialist, UNICEF 20. Communications for development specialist, UNICEF 21. Chief of Communication, UNICEF 22. Country Representative/ Head of Office, UNICEF 23. Deputy Country Representative, UNICEF 24. Chief of Child Protection, UNICEF 25. Disability Consultant, UNICEF 26. Director, Malaysian CARE 27. President, National Early Childhood Intervention Council 28. Chair, Special Olympics 29. President, Malaysian Federation of the Deaf 30. Founder, Jesselton Philharmonic Orchestra 31. Co-founder, WeCare Journey 32. Trainer, Disability Equality Training 33. Operations Manager, UNICEF 34. Communications Officer, UNICEF 35. Education Officer, UNICEF 36. C4D Deputy Officer, UNICEF 37. Principal Assistant Secretary, Ministry of Women, Family and Community Development
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 78 ANNEX E: DOCUMENT ANALYSIS MAP
The following table is intended to provide an overview of the documents UNICEF shared with the Evaluation Team for the desk review. The table is designed to reflect the structure of the file sharing system, to help track the documents included to date. Folders names are bolded.
Document Name Date Owner Description Date Finalized Added
00 Essential Documents 01 Documentations #ThisAbility Makeathon Judging Kit Judging Criteria and Overview of team invention on creating solution for their everyday challenges Makeathon Info Sheet Aina July,2017 UNICEF Aina’s Team made a wheel 20 Nov 2018 Malaysia organizer and back pouch Makeathon Info Sheet Amir July,2017 UNICEF Amir’s team made a series 20 Nov 2018 Malaysia of communication boards Makeathon Info Sheet July,2017 UNICEF Braden’s team developed a 20 Nov 2018 Braden Malaysia grip- assistive device Makeathon Info Sheet Ethan July,2017 UNICEF Ethan’s team developed an 20 Nov 2018 Malaysia activity pack Makeathon Info Sheet Janna July,2017 UNICEF Janna’s team made a multi- 20 Nov 2018 Malaysia purpose rear mounting frame for wheelchair Makeathon Info Sheet July,2017 UNICEF Junsen’s team made a 20 Nov 2018 Junsen Malaysia prosthesis and belt holder Makeathon Info Sheet Nikhil July,2017 UNICEF Nikhil’s team made 20 Nov 2018 Malaysia camera/iPad remote control assistive device Winner-Branden Lim Magic Holder Makeathon Funds Release May,2018 UNICEF Fund release letter to the 21 Nov 2018 Letter Malaysia winners of the #ThisAbility Makeathon 2017 Braden Lim July,2017 UNICEF Image of the winner with 21 Nov 2018 Malaysia his invention tool FB_Image1 July,2017 UNICEF Closer Image of Winner 21 Nov 2018 Malaysia with his invention tool Institutional Contract July,2017 UNICEF Overview on performance 21 Nov 2018 Evaluation Form Kaki DIY Malaysia evaluation institution contractor 2017 Magic Holder July,2017 UNICEF Image of winner’s invention 21 Nov 2018 Malaysia – a gripping assistive device
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 79 Makeathon Device Plan_V2 July,2017 UNICEF Overview Excel 21 Nov 2018 Malaysia spreadsheet on device plan Re_UNICEF Makeathon July,2017 UNICEF 21 Nov 2018 MagicHolder Malaysia Winner Image 1 July,2017 UNICEF Image of Winner- Braden 21 Nov 2018 Malaysia Lim Winner Image 10 July,2017 UNICEF Image of Braden Lim with 21 Nov 2018 Malaysia his family Winner Image 11 July,2017 UNICEF Image of Braden 21 Nov 2018 Malaysia Winner Image 12 July,2017 UNICEF Image with Braden with his 21 Nov 2018 Malaysia brother Winner Image 13 July,2017 UNICEF Image of Braden with 21 Nov 2018 Malaysia volunteers from PetroSains Winner Image 2 July,2017 UNICEF Image of tools used 21 Nov 2018 Malaysia to make the devise by Braden’s team Winner Image 3 July,2017 UNICEF Image on one of the 21 Nov 2018 Malaysia members from technical support Winner Image 4 July,2017 UNICEF Image of a representative 21 Nov 2018 Malaysia from technical support team Winner Image 5 July,2017 UNICEF Image of a representative 21 Nov 2018 Malaysia from technical support team Winner Image 6 July,2017 UNICEF Image of a representative 21 Nov 2018 Malaysia from technical support team Winner Image 7 July,2017 UNICEF Image of Braden with 21 Nov 2018 Malaysia physicians Winner Image 8 July,2017 UNICEF Image of Braden painting 21 Nov 2018 Malaysia Winner Image 9 July,2017 UNICEF Image of Braden 21 Nov 2018 Malaysia presenting his painting Expert Onboarding Kit July,2017 UNICEF An overview handbook 21 Nov 2018 Malaysia listing team and programme information on #ThisAbility Makeathon event Makeathon Compilation of July,2017 UNICEF PowerPoint slides on 21 Nov 2018 Participants Ideas During Malaysia Makeathon the Making- Part 1 Makeathon Overview_CP July,2017 UNICEF Overview 21 Nov 2018 Innovation Malaysia recommendations to support Children with Disabilities to develop innovative solutions to the challenges they face
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 80 Makeathon User Handbook July, 2017 UNICEF Overview introduction 21 Nov 2018 Malaysia report of the event Participant Kit Agenda + July,2017 UNICEF General guideline to the 21 Nov 2018 Map + Team Malaysia participants, volunteers and partners on the event NGO Impact Academy Challenges Young Inclusive July,2017 UNICEF General introduction essay 20 Nov 2018 Journalist Project (CYIJP) Malaysia to the event NGO Impact Academy_ April,2017 UNICEF A list of recommendations 20 Nov 2018 Overview_CP Innovation Malaysia by CP Innovation ThisAbility programs Parent Manual - 25218 July,2017 Inclusive An overview proposal of a 20 Nov 2018 Space community project called Services “The Inclusive Outdoor Classroom” Programme Brief_NGO July,2017 Tandemic A programme brief on NGO 20 Nov 2018 Impact Impact Academy ThisAbility Flagship July,2017 UNICEF ThisAbility Flagship 15 Nov 2018 Monitoring Malaysia Monitoring Framework of 2017-2020 DET Only Flyer for UNICEF 2017 UNICEF Event flyer of DET 14 Nov 2018 2017 Malaysia programme under Peter Tan Training Peter Tan – UNICEF DET 2017 UNICEF PowerPoint Slides layout of 14 Nov 2018 2017 Malaysia DET for UNICEF Malaysia ThisAbility Activities Matrix Nov,2018 UNICEF A list of conceptual notes 22 Nov 2018 2019-2019 Malaysia and resource materials of all the ThisAbility activities Worksheet DET UNICEF 2017 UNICEF Overview worksheet of 14 Nov 2018 2017 Malaysia DET UNICEF on attitudes and awareness 02 Meeting Minutes _thisabilityflagshipmtg_ July,2018 UNICEF Minutes of ThisAbility 20 Nov 2018 minutes-040718 Malaysia Taskforce Meeting follow- up _thisabilityflagshipmtg_ May,2018 UNICEF Minutes of ThisAbility 20 Nov 2018 minutes-230518 Malaysia Taskforce Meeting _thisabilityflagshipmtg_ June,2017 UNICEF Minutes of ThisAbility 20 Nov 2018 minutes-290617 Malaysia internal meeting _thisabilityflagshipmtg_ Sept,2017 UNICEF Minutes of ThisAbility 20 Nov 2018 minutes-060918 Malaysia internal taskforce meeting _thisabilityflagshipmtg_ Oct,2017 UNICEF Minutes of ThisAbility 20 Nov 2018 minutes-23102018 Malaysia internal taskforce meeting 01. Evaluation Designs and Stakeholders 1. Terms of Reference
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 81 - Annex C ToR Evaluation November, UNICEF An overview of Formative 2 November 2018 Malaysia Evaluation of UNICEF’s 2018 #ThisAbility Flagship Initiative in Malaysia 2. Technical Proposal - Technical Proposal_ June, 2018 Equal An overview of 1 November LRPS20189139940 International Procurement of 2018 consultancy services to undertake a formative evaluation of UNICEF’s #ThisAbility Flagship Initiative - UNICEF Malaysia – November, Equal A list of CVs of IPSOS 1 November IPSOS CVS 2018 International members 2018
3. Evaluation Design - 2019 Official UN November, UNICEF A list Public Holidays in UN 13 November Holidays 2018 Malaysia Malaysia 2018 - 2018 Official UN November, UNICEF A list of Public Holidays in 16 November Holidays Malaysia 2017 Malaysia Malaysia 2017 - Key Informant-Contact October, 2018 UNICEF Spreadsheet list of 31 October List Malaysia Informant Contact List 2018 - 2018-10-25-Agenda November, UNICEF Meeting Agenda with Equal 1 November Meeting with Equal 2018 Malaysia 2018
- 2018-11-02- Agenda November, UNICEF Meeting Agenda with Equal 1 November Meeting with Equal 2018 Malaysia 2018 - 2018-11-02 Key November, UNICEF Spreadsheet list of 1 November Informant Contact List 2018 Malaysia Informant Contact List 2018 - 2018-11-02 Members November, UNICEF A list of Members 2 November Mgmt-Reference Group 2018 Malaysia Management Reference 2018 Group 4. Evaluation Guidelines and Standards - 4.1 UNEG UNEG Code of Conduct March, 2008 UNEG Overview of Code of 27 June 2018 Conduct for Evaluation in the UN system UNEG Ethical Guidelines for March, 2008 UNEG Overview on ethical 27 June 2018 Evaluation guidelines for evaluation UNEG Human Rights and March, 20111 UNEG Overview on Human Rights 27 June 2018 Gender Quality and Gender Equality guidance UNEG Norms and June, 2016 UNEG PowerPoint slides brief on 27 June 2018 Standards 2 norms and standards for evaluation UNEG Norms and June, 2016 UNEG Overview on norms and 27 June 2018 Standards standards for evaluation
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 82 - 4.2 UNICEF Brand Book UNICEF August, 2017 UNICEF Brand manual version 27 June 2018 1.0 on brand strategy in compliance with the Global Regulatory Framework Gender Responsive 2015 UN Women Highlighting evaluation 27 June 2018 Evaluation Handbook process on gender equality, women’s empowerment and human rights GEROS Feedback Matrices UNICEF Feedback form on 15 October evaluation quality 2018 assurance tool with grades and explanations GEROS Final Report UNICEF Feedback form on 14 October Template evaluation quality 2018 assurance tool with grades and explanations GEROS Handbook UNICEF Summary UNICEF staff 27 June 2018 handbook orient to GEROS tool and GEROS system more generally
GEROS Inception Report UNICEF Evaluation Inception 15 October Template Report assessment form 2018 How to Design Equity- 2011 UNICEF Guideline on strategic 27 June 2018 Focused Evaluations evaluation findings, innovative approaches and methodologies UNICEF Guidance on Ethics April, 2015 UNICEF Overview on UNICEF 27 June 2018 in procedure for ethical Research standards in research, evaluation, data collection and analysis UNICEF Revised Evaluation June, 2018 UNICEF Outline the evaluation 16 July 2018 Policy principles and procedures, developed accordance with Executive Board decision UNICEF Style Book August, 2017 UNICEF Guideline on writing style in 27 June2018 consistent terminology UNICEF – Adapted UNEG June, 2017 UNICEF Guideline on report quality 27 June 2018 Evaluation Reports assessment tool Standards - 4.3 Other Documents De UNICEF 01.18.18 January, 2018 USAID Presentation slides on 18 January developmental evaluation 2018 overview Write-up LL De UNICEF Lesson learned in 7 November conducting developmental 2016 evaluation within UNICEF Myanmar
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 83 02. Desk Review
1- UNICEF Documents - UNICEF Malaysia Documents
o ThisAbility Flag- ship Documents . Essential Documentation
Documentations _thisability Flagship Moni- 2017 UNICEF Overview monitoring 14 Nov 2018 toring Malaysia framework on new activities of 2017-2018 and 2018-2019 under thisability flagship
DET Only Flyer for UNICEF 2017 UNICEF Event flyer of DET 14 Nov 2018 2017 Malaysia programme under Peter Tan Training DET_UNICEF-LIBRARY_ 2017 UNICEF Inaccessible 14 Nov 2018 SAG_PRO Malaysia Peter Tan – UNICEF DET 2017 UNICEF PowerPoint Slides layout of 14 Nov 2018 2017 Malaysia DET for UNICEF Malaysia Worksheet DET UNICEF 2017 UNICEF Overview worksheet of 14 Nov 2018 2017 Malaysia DET UNICEF on attitudes and awareness CwD MY KAP Study UNICEF Overview of Children with 10 November Dissemination Disabilities in Malaysia 2018 Knowledge, Attitudes and Practice Study Dissemina- tion Plan 2017-2019 Disability KAP November, UNICEF PowerPoint slides on 10 November 2016 childhood disabilities in 2018 Malaysia Disability Newsletter 2018 June, 2018 UNICEF Overview on UNICEF 10 November Children with Disabilities 2018 Newsletter in Word version Final File Childhood September, UNICEF A study report of knowl- 10 November Disability in Malaysia 2017 edge, attitudes and practic- 2018 es on childhood disabilities Its_About_Ability Final April, 2008 UNICEF An overview explanation 10 November of the Convention on the 2018 Rights of Persons with Disabilities SWCR2013_ENG_Lo_ May, 2013 UNICEF Overview of the State 10 November res24April of the World’s Children 2018 highlighting children with disability
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 84 UNICEF Childhood Disability September, UNICEF PowerPoint slides on 10 November in Malaysia 2017 childhood disabilities in 2018 Malaysia by Dr Amjad Rabi UNICEF Children_with_ May, 2014 UNICEF Overview of mapping 10 November Disability the policies, programme, 2018 interventions and stakeholders Useful Links for CwD Data UNICEF List of useful links for CwD 10 November or Resources or Resources 2018 o Situation of Children in Malaysia 2017 NHMS Infographic April 2018 Ministry Infographic booklet of 16 October of Health key findings from the 2018 Malaysia adolescent health and nutrition surveys 2013-002 Profile of Children November UNICEF Overview on 20 April 2015 in Malaysia 2013 implementation children’s rights with equality EIU Country Report December, EIU Overview on the forecast 17 December Malaysia 2015 and highlights of Malaysia 2015 MDG 2015 Summary UNICEF Presentation slides on 17 February the summary report of 2016 Millennium Development Goals Report UNDP_MDG Report 2015 UN Millennium Development 28 February Malaysia Goal Report on Malaysia 2016 UNICEF Children in Sabah 2014 UNICEF Situational Analysis 31 May 2015 150507 Finalized of children in Sabah Handbook o Country Programme Annual Management Plan Annex 1- Table of Delegation March, 2018 UNICEF A list of members of old 11 October of Old Functions Malaysia functions from Country 2018 Office Annex 2- MCO Organogram March, 2018 UNICEF Organization chart of 11 October 2016-2020 Malaysia Malaysia Country Office 2018 2016-2020 Annex 3- Draft Revised March, 2018 UNICEF A list of Focal Points in 11 October Malaysia CO Malaysia Country Office 2017 2018 Annex 3- Revised Malaysia March, 2018 UNICEF Overview of Working 11 October CO Working Group / Team Malaysia Divisions in the Country 2018 Office Annex 4- 2018-03-16 PRIME March 2018 UNICEF Overview on 2018-2019 11 October 2018-2019 Malaysia PRIME 2018 Annex 6- 2018HACT March, 2018 UNICEF Overview of HACT 11 October Assurance Malaysia Micro Assessment and 2018 Assurance Plan 2018
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 85 Annex 8- BCP ICT DRR March, 2018 UNICEF Overview of Business 11 October Plan Malaysia Continuity Plan of Country 2018 Office Annex 9- Consolidate MCO March, 2018 UNICEF Excel version of 11 October 2018 Malaysia Consolidate 2018 2018 Annual Management Plan March, 2018 UNICEF Finalised overview of 11 October 2018 Malaysia annual management plan 2018 in country office Country Programme Action Plan Annex01- December, UNICEF Overview Table on Results 07 December UNICEFMALAYSIA-GOM- 2015 Malaysia and Resources Framework 2015 CPAP-IRRF Annex02- December, UNICEF Overview Table on 07 December UNICEFMALAYSIA-GOM- 2015 Malaysia Integrated Monitor and 2015 CPAP-IMEP Evaluation Plan MCO-CPAP 2016-2020 December, UNIEF Overview Report on 23 March Signed 2015 Malaysia Country Programme Action 2016 Plan MCO CPAP 2016-2020 December, UNIEF Word version of Result and 26 June 2016 Standard Indicator template 2015 Malaysia Resources Framework with consolidated - Final programme components 16June2016 UNICEF Malaysia-CPAP December, UNIEF Overview Report on 13 December 2016-2020 Final Version 2015 Malaysia Country Programme Action 2015 Plan -final draft Country Programme Document 2015-PL24-Malaysia-CPD- August, 2015 UNICEF Overview of country 13 December ODS Malaysia programme 2015 Country Programme Management FINAL CPMP 11052015 April 2015 UNICEF Overview of Country 10 May 2015 MLY Malaysia Programme Management Plan 2016-2020 Malaysia- Costed Evaluation April, 2015 UNICEF Highlights the Costed 29 April, 2015 Plan Malaysia Evaluation Plan – Malaysia UNICEF country programme of cooperation, 2016–2020 Malaysia Country Office April, 2015 UNICEF Organization chart of 10 May 2015 Malaysia Malaysia country office UNICEF Malaysia Final CPD April, 2015 UNICEF Draft of country programme 29 April 2015 Malaysia document Rolling Workplan
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 86 Final Consolidated UNICEF- UNICEF Excel version of final 16 October Govt Malaysia 2016-2017 Malaysia Consolidated UNICEF- 2018 Govt Malaysia-Rolling Workplan Milestones UNICEF Rolling Workplan UNICEF Work Plan for CPAP 2016- 09 October 2018 Final Malaysia 2020 for period 2018-2019 2018 UNICEF Rolling Workplan UNICEF Overview of latest Excel 16 October 2016-2017 Malaysia version Updated 11 May 2018 2016 UNICEF Rolling Workplan UNICEF Overview of latest Excel 16 October 2016-2017 Version June Malaysia Updated 1 June 2016 2018 UNICEF Rolling Workplan UNICEF Sample Work Plan for 16 October Template Malaysia CPAP 2016-2020 for period 2018 2018-2019 Theory of Change Annex Result Structure – 27 UNICEF Overview of Result 3 May 2017 April Malaysia Framework Visual for period 2016-2020 Result Structure Visual 11 UNICEF Overview of Result 14 March March Malaysia Framework Visual for 2017 period 2016-2020- Draft Results Matrix CPD 2016- UNICEF Excel version of results and 3 May 2017 2020 Malaysia resources framework TOC Narrative UNICEF May, 2017 UNICEF UNICEF Malaysia country 3 May 2017 Malaysia Malaysia programme report theory of change 2016-2020 Monitoring Data Annual Reports Malaysia COAR 2016 UNICEF 2016 Annual Report of 12 October Malaysia UNICEF Malaysia 2018 Malaysia COAR 2017 UNICEF 2017 Annual Report of 12 October Malaysia UNICEF Malaysia 2018 RAM Outcomes-Outputs and October, 2018 UNICEF General overview reported 12 October Indicator Status by Regional Malaysia 2018 2018 / Business Area Output Performance and Fi- October, 2018 UNICEF General overview reported 12 October nancial Summary by Results Malaysia 2018 2018 Area Organogram MCO CPAP 2016-2020 October, 2018 UNICEF Latest organization chart 11 October (Current) Malaysia of Malaysia Country Office 2018 2016-2020 Research and Evaluation Documents 2013-001 Children with June, 2013 UNICEF Overview report on 11 October Disability mapping the policies, 2018 programmes, interventions and stakeholders
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 87 2013-002 Profile of Children November UNICEF Overview on 11 October in Malaysia 2013 implementation children’s 2018 rights with equality Anthrologica Malaysia UNICEF Overview study of 11 October Disability knowledge, attitudes and 2018 practise of children with disability in Malaysia Child Marriage in Malaysia June 2017 UKM A final study report of child 11 October Final marriage in Malaysia 2018 Children Without Version- February, DM Overview study on 11 October English 2018 Analytics & urban child poverty and 2018 UNICEF deprivation in low cost flats in Kuala Lumpur EA Malaysia Final Report 31 August, 2016 UNICEF Overview on Evaluability 11 October Aug 2016 Assessment UNICEF 2018 Malaysia Country Programme 2016-2020 EMR Equity Evaluation June 2015 UNICEF Overview on evaluation 11 October Malaysia management response 2018 document highlighting equity focus in country programme
EMR Document 2016-034 June 2017 UNICEF Overview on evaluation 11 October management response 2018 document highlighting capacities development initiated by UNICEF Malaysia
Evaluation of Capacity October, 2016 IDEA & Overview report on 13 October Development UNICEF evaluation of capacity 2018 development initiated by UNICEF Malaysia 2011- 2015
Final Report on CRC March, 2017 UNICEF Overview report on the 13 October Reservation study lifting Malaysia’s 2018 remaining reservations to the Convention on the Rights of The Child (CRC)
Mapping Alternative November, UNICEF Overview on mapping 12 October Learning in Malaysia 2015 alternative learning 2018 approaches, programmes, and stakeholders in Malaysia
UNICEF Malaysia Report August 2017 UNICEF Overview on Child 12 October 2017 Deprivation 2018
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 88 UNICEF Management 2016 UNICEF 6 August Response Plan 2016 2018
UNICEF Digital Landscape November UNICEF Overview report on 12 October in Malaysia 2014 access and use of digital 2018 technologies by children and adolescents UNICEF Children in 2015 UNICEF State of Planning Unit 12 October Sabah 2015 Sabah & UNICEF 2018 Situation Analysis handbook on children in Sabah UNICEF Profile of Children 2017 UNICEF Overview report on status 12 October in Sarawak 2017 children in Sarawak 2018 - UNICEF EAERPRO Documents o Global Guidance Documents 2013-Synergies Paper V6 2013 UNICEF Overview Analysis Report 13 October WebRevised Indonesia on Using Human Rights 2018 Framework to promote the rights of children with disability 2013- General Suggestions 2013 UNICEF UNICEF Guideline for 13 October for Disability Disability Situation Analysis 2018 2016 COAR Disability May, 2017 UNICEF Overview summary 10 October Summary Analysis analysis of 128 Country 2018 Offices Annual Reports globally on Children with Disability 2017 COAR Disability Full May 2018 UNICEF Overview summary 10 October Summary Final analysis of 128 Country 2018 Offices Annual Reports globally on Children with Disability - 2017 EARPRO Disability SiTan November, UNICEF PowerPoint slides on desk 10 October 2014 review findings on the 2018 situation of children and young people with disability in East Asia and the Pacific regions EN_Annex A_Intervention by UNICEF Excel List of possible 10 October Focus Area actions / interventions to 2018 Children Disability EN_ Annex B_List of UNICEF A list of resources and 10 October Resources and Websites websites related to children 2018 disability EN_X-Sectoral Programme April, 2007 UNICEF Overview report on 10 October Guidance Disability Final ending discrimination and 2018 promoting participation, development and inclusion
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 89 Malaysia Action Plan for UNICEF PowerPoint slides on the 10 October CwD action plan for Children 2018 with Disability in Malaysia Proposal UNICEF-DFAT May, 2015 UNICEF Partnership proposal to 10 October Partnership accelerate responses to 2018 the rights of children with disabilities, focusing East Asia and Pacific region Report of CwD Consultation December, UNICEF Overview report on East 10 October 2017 Asia-Pacific regional 2018 consultation to the needs of children with disabilities 2018-EAP Regional 2018 UNICEF PowerPoint slides on 28 Nov 2018 Headlines Standard drafted regional headlines Presentation Latest 2018-Growing Up Urban 2018 UNICEF Overview report of 28 Nov 2018 conversation with Mayors on making cities safe and sustainable for children 2018- UNICEF Programme 2018 UNICEF Overview guidance report 28 Nov 2018 Guidance for Second Decade 2018- UNICEF EAPRO 2018 UNICEF Overview report contains 28 Nov 2018 Headline 1 description of each of the 18 results areas and key programme strategies on early moments of children 2018-UNICEF EAPRO 2018 UNICEF Overview report contains 28 Nov 2018 Headline 2 description of each of the 18 results areas and key programme strategies on children grow in a sustainable and safe environment 2018-UNICEF EAPRO 2018 UNICEF Overview report contains 28 Nov 2018 Headline3 description of each of the 18 results areas and key programme strategies on adolescent potential unleashed - UNICEF Headquarters Documents o Strategic Plan 2017-17-Rev1 Strategic August, 2017 UNICEF Overview report on 14 January Plan 2018-2021 UNICEF strategic plan 2018 2018-2021 2017-18-Final Result July, 2017 UNICEF Overview report on final 14 January Framework result framework of the 2018 UNICEF Strategic Plan 2018-2021
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 90 2017-Annex Common July, 2017 UNICEF Annex of the common 14 January Chapter chapter implementing 2018 2030 Agenda with UNDP, UNFPA, UNICEF, and UN- Women strategic plans for 2018-2021 2017-EB11-Theory of July, 2017 UNICEF Overview paper of theory 14 January Change of change 2018 Presentation-SP for 22 Aug August, 2017 UNICEF Informal brief on the 14 January Informal Briefing strategic plan 2018-2021 2018 SP Brochure 2018-2 English January 2018 UNICEF Overview report of UNICEF 25 February New York Strategic Plan 2018-2021 2018 UNICEF_Strategic Plan October, 2017 UNICEF Orientation report on 14 January UNICEF Strategic Plan 2018 2013-Take Us Seriously! June, 2013 UNICEF Summary Report of 13 Nov 2018 New York Engaging Children with Disability in Decisions Affecting Their Lives 2018-UNICEF Programme October, 2018 UNICEF UNICEF General Guidance 13 Nov 2018 Guidance for The Second Book on Programming with Decade and for adolescents 2- Government Documents Eleventh Malaysia Plan May, 2015 EPU Overview report on the 2 February 2016-2020 milestones of Malaysia’s 2016 history and polices Malaysian Vision 2020 EPU Brief on the nation 16 Dec 2015 objective by Prime Minister, Tun Dr Mahathir
Tenth Malaysia Plan 2011- 2010 EPU Overview report on the 21 January 2015 milestones of Malaysia’s 2016 history and polices 3- UN Documents Draft 2 UNSPFR Agencies October, 2015 UNICEF Outline document of UN 13 Dec 2015 Comment Strategic Partnership Framework on “Government of Malaysia and the UN Country Team” UN Report on Contributing January, 2015 UN Overview report on 28 February Po Malaysia “Continuing Inclusive 2016 Dialogues On Implementation of The Post 2015 Development Agenda- Malaysia UNSPF Outline Document February, UN Draft on UNSPF 2016- 6 February 2014 Malaysia 2020 (for discussion 2015 purpose only)
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 91 ANNEX F: THEORY OF CHANGE
Evaluation findings suggest that the strategies used to shape the trajectory of the #ThisAbility Flagship Initiative are not clearly defined; resulting in a Flagship that appears more reactive and opportunistic than strategic. The Terms of Reference for this evaluation called for the reconstruction of the Theory of Change behind the #ThisAbility Flagship Initiative; this was not possible given the apparent nature of Flagship activities. An externally focused forward-looking Theory of Change has been produced instead (in agreement with UNICEF) that is intended to provide strategic direction and an overall goal for UNICEF Malaysia’s future disability work (see below).
This Theory of Change has been informed by a barrier analysis conducted with selected external partners, developed in consultation with UNICEF staff and sharpened using field data collected during the evaluation. The resulting Theory of Change reflects the current status of disability inclusion in Malaysia and the steps considered necessary by informants to effect positive change. It is important to note that the Theory of Change is very ambitious. UNICEF Malaysia’s convening power and strong profile provides a firm basis upon which the Programme can build to deliver the Theory of Change, but as this evaluation report highlights UNICEF Malaysia requires a significant investment in its capacity around disability inclusion. Not least, the Theory of Change requires UNICEF Malaysia to shift the balance of its disability inclusion activities (as suggested by the evaluated #ThisAbility Flagship Initiative activities) from one-off activities to a more strategic and concerted approach to changing public attitudes and policy.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 92
Box B: Accessible summary of Theory of Change
The Theory of Change (above) assumes that if UNICEF Malaysia i) mobilizes and amplifies networks and partnerships, ii) generates and disseminates evidence to inform action and iii) strengthens selected systems and/ or builds capacity, the following short- term change will be achieved: • Reduced stigma and discrimination toward children with disabilities at household- and community-level. • Increased coverage of disability-inclusive models that enable all children to learn and play together. • Increased use of disability data and disability-inclusive practices. • Enabling environment strengthened regarding inclusion of children with disabilities. These short-term changes are expected to in turn achieve: • Increased equality between children with and without disabilities in access to services and the fulfillment of their rights, leading to equality in full. • Improved attitudes and behaviours towards children with disabilities as regards inclusion. • A renewed, coherent and coordinated national approach to disability inclusion established that is centered on equality. The final impact of these changes is expected to be: All children with disabilities in Malaysia access high-quality services and enjoy the fulfillment of their rights on an equal basis with children without disabilities.
The impact sought in the Theory of Change is for all children with disabilities in Malaysia to access high-quality services and enjoy the fulfillment of their rights on an equal basis with children without disabilities. The focus on equality is deliberate; it aligns with language adopted by the Regional Office and UNICEF’s Disability Unit in New York as well as the CRPD and the SDGs. Underlying the chosen wording is also an emphasis on the quality of services for and the delivery of rights to all children; disability inclusion is not at the expense of other marginalized groups and does not preclude investment in better child-friendly services or activities. The intended impact subsequently requires not just a focus on inequalities between children with and without disabilities; it also requires the bar to be lifted for all children through inclusive and accessible strategies and/ or models. Three medium-term outcomes are set in the Theory of Change as stepping-stones towards achieving the intended impact:
• Increased equality between children with and without disabilities in access to services and the fulfillment of their rights, leading to equality in full. • Improved attitudes and behaviours towards children with disabilities as regards inclusion. • A renewed, coherent and coordinated national approach to disability inclusion established that is centered on equality.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 94 While it is not explicitly included in the Theory of Change, the medium-term outcomes are intended to be delivered by 2025. The three medium-term outcomes are intended to feed directly into the intended impact. Tangibly increasing equality will require proactive steps to adapt existing services or activities to include children with disabilities as well as designing and/ or replicating disability- specific interventions. It is important to remember that implicit in this is the need to improve quality for all children, not to seek equality in services or the fulfillment of rights that fall short of Malaysia’s obligations.
Positive attitudes are a key foundation of equality; children with disabilities will not utilize available services or enjoy their rights if attitudes to disability inclusion remain poor. UNICEF Malaysia’s KAP Survey provides a valuable baseline that can be repeated in 2025 to measure success. Delivering change that goes beyond UNICEF’s interventions to nationwide success and sustaining it will require clear commitment from the Government and an approach supported by all stakeholders. UNICEF’s convening power will be especially critical to achieving this last medium-term outcome as well as strong, ongoing collaboration with external partners to influence Government and to foster ownership across key stakeholders.
Four short-term outcomes are included in the Theory of Change that will be indicators of success toward achieving the medium-term outcomes:
• Reduced stigma and discrimination towards children with disabilities at household and community level. • Increased coverage of disability-inclusive models that enable all children to learn and play together. • Increased use of disability data and disability-inclusive practice. • Enabling environment strengthened regarding inclusion of children with disabilities.
Reducing stigma and discrimination will require changes to underlying attitudes and building awareness of disability. Disability awareness training and a countrywide behavior change campaign to transform negative social norms are essential; the behavior change campaign would ideally be a distinct part of a broader advocacy strategy. Examples of campaigns that seek to change social norms regarding marginalized groups are available from the HIV and other sectors. A campaign delivered by M&C Saatchi, for example, successfully reduced stigma and discrimination as a key barrier to HIV service access through the development and dissemination of messages co-created with LGBT and sex worker organizations. The M&C Saatchi campaign worked to humanise key HIV populations, generate positive narratives and role model a changed social norm. Adapting existing methodologies for disability inclusion will require some innovation from UNICEF Malaysia but success will resonate beyond Malaysia. Evidence of proven disability-inclusive models is sparse meaning that UNICEF will need to not only secure a broad commitment to disability inclusion but will also have to innovate and document results to show what works and can be replicated.
Measuring and tracking change will require the routine collection of disaggregated disability data. Critically, as demonstrated in other countries, it is also necessary to ensure decision-makers and key advocates (e.g. Disabled People’s Organizations) can use the data and evidence collected to inform and design effective policies and/ or services. Furthermore, a coordinated national approach will only work with political and financial buy-in (specifically CRPD compliant budgeting) from the Government.
UNICEF’s #ThisAbility Flagship Initiative 2017–2019 in Malaysia: Evaluation Report 95 Multiple strategies (or inputs) are identified in the Theory of Change that are required to deliver these outcomes. Strategies are grouped under three sub-headings:
• Mobilizing and amplifying networks and partnerships • Generating and disseminating evidence to inform action • System strengthening and capacity building
It is important to underline that it is not UNICEF Malaysia’s lone responsibility to deliver this Theory of Change; UNICEF Malaysia will require strategic and reliable partners. The first of the three sub- headings focus on galvanizing support for this Theory of Change outside of UNICEF Malaysia and mobilizing action. UNICEF Malaysia will need to be the strong lead for strategies under this first sub-heading, successful delivery of these strategies will have a pivotal impact on the successful delivery of other strategies. UNICEF Malaysia will have a key role to play in the strategies grouped under the other two sub-headings, but it is expected that external partners will also play a more significant role.
UNICEF Malaysia’s Sections have already started to identify how they will contribute to the strategies identified. The Nutrition Section recognizes the need to build the capacity of health service providers and health agencies to include people with disabilities. The Education Section recognizes the need to include persons with disabilities in national education polices and to develop an Inclusive Education roadmap. The Innovation Section identify the importance of designing and testing new inclusive tools and models, the Social Policy Section are capable of supporting capacity building and working to mainstream disability inclusion across Malaysia’s statistical systems, policy processes and legislation, while C4D has underscored its role in changing and tracking attitudes to disability inclusion. It will be important for each Section to more sharply define their specific inputs and to revise workplans annually.
Finally, the Theory of Change identifies a series of assumptions that may affect the delivery. UNICEF Malaysia will need to review and revise these assumptions as work to achieve the outcomes and the final impact gets underway. At this early stage, perhaps the most critical assumption is that UNICEF Malaysia will sufficiently prioritize disability inclusion in the CPD process to adequately resource delivery against this Theory of Change. UNICEF Malaysia is strongly encouraged to review and recalibrate the Theory of Change after the CPD process. For further information, please contact: UNICEF Malaysia United Nations Children’s Fund Menara PJH Level 10, No. 2, Jalan Tun Abdul Razak Precinct 2, 62100 Putrajaya Email: [email protected]
© UNICEF Malaysia, November 2019