Formative Evaluation of the Timor-Leste Youth Parliament Programme 2010 – 2018

Investing in future leaders Formative Evaluation of the Timor-Leste Youth Parliament Programme 2010 – 2018

A participatory approach to understanding the roles of adolescents and youth in a post-conflict nation toward SDG 16 Peace and Justice November 2018 An independent formative evaluation of the Government of Timor-Leste’s Timor-Leste Youth Parliament (Parlamentu Foinsa’e Nian), commissioned by UNICEF Timor-Leste based on the request from the Secretary of State of Youth and Sport (SSYS) of the Government of Timor-Leste. Timeframe: November 2017 to November 2018 Report published January 2019

Cover photos: Activities involving current and former Timor-Leste Youth Parliamentarians (© UNICEF/APFTL)

Our grateful thanks to all who have contributed to this report from government, TLYP current and former members, implementing NGOs and local community representatives from schools and administrative posts, as well as to UNICEF staff who supported this review throughout its different phases.

The statements in this report are the views of the author and do not necessarily reflect the policies or views of the Government of Timor-Leste or UNICEF.

List of acronyms APFTL Alumni Parlamentu Foinsa'e Timor-Leste (‘the Alumni Association’) ASRH Adolescent Sexual and Reproductive Health CPAP Country Programme Action Plan CNJTL Conselho Naseional da Juventude de Timor-Leste/(National Council of Youth) GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit GOTL Government of Timor-Leste I/NGO International Non-Government Organisation IPU Inter-Parliamentary Union MSSI Ministry of Social Solidarity and Inclusion NYC National Youth Council (note, English acronym; same organisation as CNJTL) OECD-DAC Organisation for Economic Cooperation and Development (OECD) Development Assistance Committee PFN Parlamentu Foinsa’e Nian (Timor-Leste Youth Parliament) SDGs Sustainable Development Goals SRH Sexual and Reproductive Health SSYS Secretary of State for Youth and Sports TOR Terms of Reference TLYP Timor-Leste Youth Parliament UNDAF UN Development Assistance Framework

Contents

1. Context and background ...... 12

Youth participation and the Sustainable Development Goals in Timor-Leste...... 13

Timor-Leste’s ‘youth bulge’ and associated challenges ...... 13

Timor-Leste Youth Parliament History...... 15

TLYP principles ...... 16

The role of UNICEF, and others, in Timor-Leste Youth Parliament ...... 18

Process and progress: Results Matrix and the TLYP Theory of Change ...... 19

2. Evaluation background...... 22

Evaluation Purpose, Objectives and Audience ...... 22

Evaluation scope ...... 23

3. Methodology ...... 26

Academic rationale: selecting theory-based evaluation ...... 26

Data collection sources, sampling and analysis methods ...... 27

Diversity and inclusion; ethics and risks ...... 28

Limitations and mitigative measures ...... 29

4. Findings ...... 31

4.1 Relevance ...... 31

4.2 Efficiency ...... 38

4.3 Effectiveness ...... 44

4.4 Impact ...... 51

4.5 Human rights and equity ...... 57

4.6 Sustainability ...... 63

6. Conclusions ...... 68

6.1 Conclusions by OECD-DAC criteria ...... 68

6.2 General conclusions ...... 69

7. Lessons Learned ...... 71

8. Recommendations ...... 73

9. References and further reading ...... 75

10. Annexes ...... 76

Annex 1: List of documents consulted ...... 76 4

Annex 2: Terms of Reference, TLYP Evaluation ...... 77

Annex 3: Evaluator biodata and suitability ...... 81

Annex 4: Timor-Leste Youth Parliament Strategic Plan ...... 82

Annex 5: Online survey demographics ...... 83

Annex 6: List of people interviewed; site visit schedules ...... 84

Annex 7: TLYP member survey tool ...... 85

Annex 8: Follow-up Telephone Validation interview tool ...... 88

Annex 9: TLYP participating schools analysis ...... 89

Annex 10: Ethics, privacy and informed consent ...... 90

Annex 11: TOR of Reference Group ...... 92

Annex 12: Revised timeline...... 93

Annex 13: Reflections on Youth Parliament models ...... 94

Annex 14: Analysis of TLYP Plenary Recommendations ...... 96

Annex 15: Survey results, detailed ...... 98

Annex 16: Case studies ...... 104

Annex 17: Evaluation Matrix (Evaluation and Evidence Framework) ...... 110

Annex 18: Outcome-level framework: The Theory of Change ...... 112

Annex 19: Output-level framework: the Results Matrix ...... 113

Annex 20: Summary of youth providers and their services ...... 114

List of figures and tables

Figure 1: Summary timeline, Timor-Leste Youth Parliament ...... 16 Figure 2: Objectives for the role of TLYP members ...... 18 Figure 3: Timeline of UNICEF support to TLYP ...... 19 Figure 4: Process map for theory-based analysis/validation ...... 27 Figure 5: Crossover of child, adolescent and youth definitions, Timor-Leste context ...... 33 Figure 6: Survey self-rated interest in civic and youth issues, before / after TLYP ...... 34 Figure 7: % survey respondents using skills in their community ...... 35 Figure 8: TLYP budget, proportional to overall SSYS budget, 2015-17 ...... 39 Figure 9: Flowchart of activities, three year cycle, showing outgoing transition to APFTL ...... 40 Figure 10: Five channels for youth extra-curricular activities / participation ...... 41 Figure 11: 'Multiplier' return on investment scenario ...... 43 Figure 12: TLYP plenary recommendation themes, three-year comparison ...... 47 Figure 13: Average rating, TLYP activities and trainings (minimum 1, maximum 3; note, no scores fell under 2) ...... 49 Figure 14: Survey self-reporting on how training was used ...... 49 Figure 15: Proportion of survey respondents NOT using their training at all ...... 50 Figure 16: Average score for support from home, school, community; minimum 1, maximum 3; no score fell under 2...... 50 Figure 17: Survey self-reporting, % respondents delivering community activities...... 52 Figure 18: Survey responses, % at university / school / other ...... 55 Figure 19: School level breakdown, current TLYP members ...... 58 Figure 20: Survey self-reporting: changes in peer attributes and interest ...... 60 Figure 21: Retrospective Theory of Change (Consultant, in collaboration with government, UNICEF, APFTL) ...... 112 ______

Table 1: Do objectives and activities remain relevant? ……………………………… 33 Table 2: Where is TLYP’s youth-led change evident? ………………………………. 46

Executive summary

“Young people are the architects and agents responding TLYP outcome objectives: to the challenges they face… it is crucial to consider  Influence: a primary function to bring children’s youth participation in democratic processes, in their voices into parliamentary process. Members meet in communities and in decision-making processes for annual parliamentary plenary to raise policy national development.” recommendations to appropriate ministries Timor-Leste National Youth Policy 2016  Empower: building lifelong capacity and confidence Timor-Leste is a young country, with independence in young people through training, civic leadership and restored in 2002. Its population is proportionally ‘life skills’ among the youngest in the world, and the wellbeing  Increase engagement and participation: as TLYP and participation of children and youth are widely members share knowledge and interest in youth recognised as vital to Timor-Leste’s future as a nation, policy with other young people including a pivotal role in achieving SDG16 Peace and Justice and other relevant goals and national  Lead: for multiplied reach and benefits, as TLYP strategies. Since independence, Timor-Leste has members take up roles as leaders, influencers and made significant progress in the lives of children and trusted role models among their peers youth. However, contextual economic, geographic and Stakeholders and roles, TLYP implementation cultural barriers continue to limit their full participation. Secretary of State for Youth and Sport (SSYS) Within this context, the Timor-Leste Youth  Accountable for sustained, effective TLYP operation Parliament (TLYP) stands out as one of few  Oversees TLYP connections to other youth initiatives for young people with nationwide reach movements under a broader youth strategy and influence. TLYP Secretariat: In 2018, TLYP is about to enter its fourth mandate  Installed as a coordinating resource for TLYP in 2010; period. Over 390 children aged between 12 and 17 reports to SSYS have taken part in TLYP since 2009. As the  Administers budget and manages activities including Government of Timor-Leste’s Secretariat of State for elections, training and annual plenary in Dili Youth and Sports (SSYS) commenced planning for UNICEF: this next cycle, UNICEF partnered with SSYS on a  A founding partner in TLYP since 2008 formative evaluation of TLYP progress and impact to  Phased out financial support in 2015; continues to date. provide technical support and some training Key facts, Timor-Leste Youth Parliament: APFTL: (the Alumni Association of TLYP)  Endorsed as a government-owned institution  The alumni organisation of TLYP and a formal youth- through Parliamentary Decree (2009) focused NGO; self-formed in 2015  First national sitting 2010  Trains and supports TLYP under contract to SSYS,  UNICEF supported planning and implementation, as well as training and awareness in schools and 2007 - 2015 communities across Timor-Leste  Built on inclusion and rights principles: One girl, Other UN, and Development partners and NGOs: one boy (12 – 17), from each electorate of Timor- Support various project-based initiatives involving Leste TLYP members, further expanding voice and action of  Three-year mandate; after this, retiring TLYP young people members join the self-starter alumni organisation Alumni Parlamentu Foinsa'e Timor-Leste (APFTL)

About the evaluation Findings by criteria: RELEVANCE Overall purpose: Documentation and stakeholder viewpoints showed  Generate TLYP knowledge/evidence in Timor-Leste strong alignment with original objectives (2009) over time. context for national development policies and Youth empowerment remains an urgent need in Timor- strategic planning; guide UNICEF CPAP 2015-2019 Leste. TLYP also shows strong alignment to national and Outcome 3: Child Protection and Participation. global strategies for youth inclusion and empowerment. Audience: TLYP is strategically relevant to the current UNICEF  Primary: Government of Timor-Leste and UNICEF Timor-Leste Country Programme Action Plan (CPAP) Timor-Leste; TLYP secretariat, APFTL under adolescent participation, and to other UN agencies  Secondary: Local and international partners working working in children’s and youth issues in Timor-Leste. in child and youth civic participation Youth-driven agendas remain core to TLYP plenary Scope: process, as do inclusion principles that ensure gender  TLYP inputs and outcomes 2010 – 2018; among these, APFTL outcomes balance and geographical representation from across  Measured using UNICEF/OECD-DAC evaluation Timor-Leste. Some concern was voiced that an original framework of relevance, efficiency, effectiveness, disability quota principle has not been seen as relevant. sustainability, impact, human rights and equity  Sources experiences and observations from TLYP Findings by criteria: EFFICIENCY members (current: aged 12 – 17; former: aged 18 – Overall process and schedule have been adhered to, 25), from all 65 administrative posts of Timor-Leste though some activities ran late, with risk of disrupting Methodology: school attendance, or were cancelled. The third cycle Mixed method, based on the following principles: (2016 – 2018) was worst affected, due to a government  Formative spending lockdown in 2018. Budget has scaled back  Theory-based compared to the first UNICEF-supported cycle (2010 –  Participatory 2012). While the program is not less effective as a result,  Utilisation focus TLYP members voiced dissatisfaction about reduced Data sources: resources. Efficiency of outreach work was enhanced by  Document review and literature review formation of APFTL, as school leavers took up  TLYP member survey (63 respondents) opportunities for leadership and training, at home and in  Semi-structured interviews (45 respondents) other locations. TLYP reaches fewer children directly than  Focus groups (10 groups) many other Timorese youth associations but offers nation-  Validation consultation (chaired by TLYP with wide engagement with a focus on civic participation that government, UNICEF, APFTL and other NGOs in is unique in Timor-Leste. Government fully supports TLYP attendance) on an annual budget of around $US200,000, representing $4,615 investment in each TLYP member over three Children and youth were primary participants in years. Using conservative estimates of the number of this evaluation design, highly capable young members conducting outreach training (one-third) and the people who took part enthusiastically and number they could reach (235, the average reported in the professionally in setting questions, preparing and survey), the scenario found potential to reach over testing tools, hosting field visits and advising on 10,000 indirect beneficiaries per cycle. This data analysis. Current TLYP members chaired the represents good value for money, though it is based on final validation workshop and endorsed the caveat that the training must be taking place, which recommendations. needs to have further studies and monitoring.

Findings by criteria: EFFECTIVENESS Findings by criteria: HUMAN RIGHTS / EQUITY Targets for direct involvement of young people have Inclusion principles of geographic representation and been met, including targets for gender balance and gender balance were formally mandated and closely geographical representation (though not disability followed. Two other principles were not followed: inclusion). Interviews with TLYP members showed they additional non-geographic membership of two young were aware of their responsibilities to represent other people with disability; and representation of children out young people. Survey respondents rated the of school. Interviews with stakeholders implied that while effectiveness of component inputs highly (activities, the concepts were sound and supported, the practicalities meetings, trainings). Survey data analysis showed were overly challenging given current networks and strong correlation between skills remembered/rated resources. Documentation and stakeholder reflection highly and skills subsequently used: for instance, public confirm UNICEF as the primary contributor to inclusion of speaking, leadership and life skills training. At this stage, child and adolescent rights and participation principles in causal links from TLYP process to national policy for TLYP. This could be further strengthened through a improvements are not apparent, with the exception of curriculum review. Mechanisms for safeguarding children the youth consultation leading to the 2016 National from harm are lacking in the TLYP model; reporting and Youth Policy, a highlight achievement for youth voice in response systems need to be formalised as a priority. Timor-Leste, and the first major event to be led by While there have been no incidents reported, children are APFTL. TLYP has been a pivotal program for the taking part in the program in ways that place them at risk. Government and UNICEF within Timor-Leste’s UN Development Assistance Framework (UNDAF). Findings by criteria: SUSTAINABILITY UNICEF influence can be seen in design elements such TLYP basis in law (Government Resolution No.23/2009) as gender balance, age group (12 – 17), disability has generally assured sustainability of the TLYP, inclusion, and training content, which includes child weathering changes of government leadership and the rights and gender. phase-out of UNICEF’s dedicated project support. Sustainability of quality outcomes for direct participants is now evident, though with some dependencies on human Findings by criteria: IMPACT resource commitments long-term. Interviews found Multiple, triangulated sources confirmed outcomes for strong support and internal advocacy from SSYS to personal empowerment of TLYP members. A causal resource TLYP, in line with accountability to the 2009 connection between TLYP participation and community- government resolution. Outcomes for child rights and based outcomes was partially evident. Current and protection, a UNICEF core mandate, are not fully former members reported they felt responsible and guaranteed. Analysis of TLYP policy platforms reveals an confident to return on the trust and time invested in them; emphasis on adult youth policy. TLYP members receive however, indirect impact for other young people less exposure and training on children’s issues so, with throughout Timor-Leste could be proven in only a small the exception of adolescent sexual and reproductive number of cases and is largely presumed / projected health, they do not usually discuss or make rather than measured. Interviews showed a preference recommendations for children. While government (SSYS, to work in groups when conducting community training and TLYP secretariat) continues as the driver of TLYP process, increasingly APFTL can be seen as a driver of or other events, an important observation for shaping sustainable TLYP outcomes, expanding community future guidelines and support for TLYP/APFTL outreach. outreach as a channel and network for planning and

supporting initiatives that benefit youth.

Highlight data from the TLYP member survey: Key lessons learned  92% believed they had influenced other youth The participatory approach involving children and youth was highly valued on all sides. This was not only  73% reported campaigning on youth issues and a great opportunity for young people to join and lead 54% had made a community proposal. research planning and application but also to contribute  On average, respondents rated their employability to the quality of evaluation by setting questions in line 89% higher than before TLYP with utilization principles and their rights and interests as  82% of school leavers went on to university (national stakeholders, collaborating on site visits and other data average is approximately 9%. collection, and contributing to analysis. The level and success of beneficiary participation was a strength of Conclusions by OECD-DAC criteria UNICEF’s methodology in this study and supports the UNICEF Strategic Plan 2018-2021 on the importance of RELEVANCE: The model and objectives of TLYP working with adolescence, recognising the critical second remain highly relevant to the agenda for HIGHLY decade in life as an emerging issue. youth in Timor-Leste and should RELE- Training is valued most highly when there are continue. VANT opportunities to use the skills directly and

immediately. The under-utilisation of theoretical EFFICIENCY: Good efficiency and performance are knowledge (gender, child rights, proposal writing, evident in managing TLYLP process, but journalism) compared to directly applied skills (public GOOD with opportunities for greater innovation EFFIC- speaking, debate, life skills) has limited some results key IENCY and quality programming under a more to UNICEF’s global strategy for children and adolescents, flexible budget. especially the most vulnerable. Implementers of similar EFFECTIVENESS: Effective practices throughout the models should therefore consider partnerships and TLYP cycle contributed to achievement pathways beyond training modules to put rights-based MAINLY skills into action. EFFEC- of process objectives, though objectives Up to date, consistent information is an ongoing need TIVE for social change are currently in such programs, to ensure schools, friends and unreached or unmeasurable. parents are supportive of children’s commitment and IMPACT: Partial success is evident against impact choices. Support and full understanding at home and in intention, with conclusions limited by the community is crucial to positive outcomes for SOME lack of tracking for impact. Note: as members. It should not be presumed that the privilege and IMPACT intended impact is long-term, this is a responsibility of participating is understood throughout a formative evaluation. child’s social sphere, and implementers of similar models HUMAN RIGHTS AND EQUITY: TLYP strongly reflects should monitor and meet community information needs. most of its envisaged human rights and More needs to be done to challenge existing attitudes GOOD inclusion principles, and has potential to and stereotypes if inclusion principles for the most RIGHTS BASIS guide human rights outcomes for marginalised are to be realised. Tightly held attitudes children and youth in Timor-Leste. and stereotypes existed in children’s homes, schools and SUSTAINABILITY: Sustainability has already been communities. If no-barriers inclusion remains the goal tested and is assured; TLYP is an then these attitudes must be understood and addressed, HIGHLY independent government-funded with full participation of decision makers associated with SUSTAIN the program. Realistically, such changes take time, and institution for children’s civic -ABLE may require technical support from inclusion advocates to participation. mainstream rights and inclusion in all activities. 10

Child protection principles and child-safe Recommendations for Government organisational processes are specialist areas; it should Within six months never be presumed that they are in place. Any initiative  Formalise the shift from two to three years for involving children – especially when that involvement takes TLYP mandate cycle, aiming for consistent them away from their homes – needs stringent child government budget commitment; ensure election protection protocols in place. This may not always be within costs can be covered under this budget. the scope or capacity of implementing partners to recognise. A key role for UNICEF is to ensure all their Six months to two years associated programs are aware of and applying child protection mandates.  Work with gender and inclusion specialists to increase overall curriculum emphasis on equality, Recommendations addressing gender stereotypes relevant to Timor- The following list of recommendations, based on Leste; and to identify missing or under- lessons and priority findings for stakeholders, was represented themes. developed and refined in consultation with government, UNICEF, TLYP and APFTL representatives. These  Establish a formal platform for regular recommendations are intended to be implemented in consultation with Members of National partnership with other agencies including UN where Parliaments and TLYP on child issues, including relevant. Only highlight recommendations are included connecting to the SDG Roadmap working group. here, with a more detailed list available in the main report. Recommendations for other partners, including Recommendations for UNICEF APFTL and other youth NGOs Within six months Within six months  Support SSYS in partnering with appropriate technical  Consider a network and programme for expertise to review and revise the current curriculum shortlisted / substitute TLYP election candidates, design for TLYP. potentially implemented through municipal alumni networks. Six months to two years  Support TLYP Secretariat to build and implement an Six months to two years achievable monitoring framework including TLYP within  Set up a mentoring system, using administrative broader National Youth Policy goals. Consider Most post / municipal youth networks to support TLYP Significant Change1 and other youth-friendly monitoring members on first-time community and chefe suco techniques, as well as quantitative reporting of policy engagement. community outreach by TLYP/APFTL.

1 A highly participatory story-based monitoring tool for understanding whether, and how, community-level change is occurring: see https://www.betterevaluation.org/en/plan/approach/most_significant_change 11

1. Context and background

“Young people’s involvement is … key if the call for participation, inclusion, accountability and revitalized global engagement embedded in Goals 16 (peaceful, just and inclusive societies) and 17 (partnerships and implementation) is to be achieved.”

UNDP, Youth as Partners for the Implementation of the SDGs.

“Young people are the architects and agents responding to the challenges they face. Thus, it is crucial to consider youth participation in democratic processes, in their communities and in decision-making processes for national development. This principle delineates the need for consulting with and listening to young Young people lead Children’s Day marches under the people, considering and/or involving young theme “Safe Learning Spaces for All”, Dili, Timor-Leste people when decisions on development 2018 activities that directly affect their lives are Image © UNICEF being made.”

Timor-Leste National Youth Policy, 2016

Youth participation and the Sustainable Development Goals in Timor- Leste Investment in young people’s active citizenship – encouraging their contribution to social and community advancement – is beneficial not only for young people and their personal empowerment, but also to SDG achievement over time. Of 169 targets set within the SDGs, over one-third reference young people, implicitly or explicitly, across Goal 2 (hunger), Goal 4 (education), Goal 5 (gender equality), Goal 8 (decent work), Goal 10 (inequality) and Goal 13 (climate change)2. UNDP highlights the value of youth participation more broadly:

Timor-Leste’s Roadmap for the Implementation of the 2030 Agenda and the SDGs (UNDP 20164) identifies the inclusion and participation of young people as a path to achieving many other goals for development. It notes that the rapid growth of the under-30 age group “if used effectively, could be a major opportunity, but if not handled successfully, could lead to serious social problems.”

The social problems referenced are linked to existing low employment levels, amid projections that around 18,000 new jobs will be needed each year to ensure future adult productivity. The Roadmap also highlights the importance of awareness of development priorities, including among youth, so that demand and accountability for health, education, inclusion and equality are driven by communities and their priorities for young people. Timor-Leste’s ‘youth bulge’ and associated challenges The wellbeing and participation of children (under 18), adolescents (15 – 19) and youth (20 – 24) is widely recognised as vital to Timor-Leste’s future as a nation. Timor-Leste’s population is proportionally among the youngest in the world. The 2015 Population and Housing Census found that 46% of population is under 18 years old and over one quarter aged between 6 and 15. The Government of Timor-Leste’s National Youth Policy (2016) articulates clear intentions to support and promote rights and participation of adolescents and youth, emphasising the importance of a secure and empowering transition ‘from life as a child into life as a youth and initial stage of adulthood” (National Youth Policy, p.3).

2 http://www.undp.org/content/undp/en/home/librarypage/results/fast_facts/fast-facts--youth-as-partners-for-the-implementation-of-the- sdgs.html 3 Ibid. 4 The 2015 UN Development Assistance Framework (UNDAF) for Timor-Leste envisions youth in development under the Social Sector outcome, with results co-owned between UNICEF, UNDP and as a sector co-owned by UNDP and UNICEF under the Social Sector outcome: ‘1.5 In an enabling environment, young men and young women make informed choices for a healthier and more productive life as citizens who contribute actively to the peace and development of their country.’

Economic, geographic and cultural barriers are all contextual factors that can hamper this transition in Timor-Leste, for instance5:  Inequalities in education access and standards across municipalities: though overall education levels have risen sharply in the last decade, 15% of 15-19 year-olds have never attended school; a further 24% have left by the age of 15.  Literacy challenges: despite strong progress in primary and secondary attendance rates, around one in five youth (15-24) remain illiterate, mainly in rural areas.  Employment challenges, along with a rural-urban migration trend for young people: Youth represent around 33% of all working age Timorese, but have a 14% share of available employment. The youth unemployment rate is 12.3%: doubled in urban areas, as young people migrate to Dili and Baucau in search of paid work.  Gender inequities and the burden of reproductive health: Almost one in five women marry before the age of 18; one in four has a child before 20. Health and maternal clinics do not have different programmes and services for adolescent mothers, and they face strong cultural stigma in returning to school or taking up civic roles.  A low level of adolescents’ and youth participation in the political, social and economic life of Timor-Leste, which has been linked to violent outbreaks in the past and continues to be of concern to long-term social cohesion.

Recognising risk to the nation’s future if more is not done to include and empower young people, the Government of Timor-Leste, UNICEF and several other in-country civil society/NGO partners have been working towards a solid strategy for youth protection and participation for over a decade. Policy framework achievements include:  Acknowledgement of the value of young people in nation-building in the 2002 constitution  The first National Youth Policy, 2007  Discussion of the rights, responsibilities and needs of Timorese adolescents and youth in the 2011-2030 Strategic Development Plan  The revised National Youth Policy, 2016

The National Youth Policy in 2016 was a significant step for acknowledging current and future needs of the growing youth population. The process of writing policy drew heavily on consultation with young people and technical partners including UNICEF. In a systematic approach across Timor-Leste, youth identified the most pressing problems faced in their daily lives, which were then grouped, analysed and categorised into five thematic visions for the nation:

5 All statistics in this list are sourced from the Timor-Leste National Youth Policy, 2016: https://timor-leste.unfpa.org/en/publications/national-youth-policy-nyp-2016 14

 “Education: Young people are educated and civic-minded citizens of Timor-Leste, who live a long and productive life which allows them to participate in the economic, social and political development process.  Healthy lifestyles: Timor-Leste’s youth live healthy lifestyles and are able to access youth-friendly and quality health services. Young people themselves become promoters of healthy living.  Employment and employability: Young men and women, including youth with disabilities have equal opportunities to increase their employment skills, are able to employ themselves, have employment opportunities and are able to demonstrate good performance in the work place.  Civic participation: The young people of Timor-Leste, guided by a spirit of solidarity, participate actively and constructively in the development process at all levels, and value their culture.  Violence and crime: The young people of Timor-Leste possess a strong spirit, a sense of responsibility and social sensitivity, and are able to create a peaceful environment through the strengthening and application of democratic principles6.”

Though the National Youth Policy has strong potential to deliver results, the actual supply of services for young people remains limited. Most municipalities have a youth centre connected to the national Secretariat of State for Youth and Sports, offering sports and other cultural or creative activities, as well as some vocational training. Community-based organisations or NGOs also conduct special projects to promote youth inclusion, employment and civic participation. However, the quantity and quality of programs is not well mapped or monitored and their centralised location puts them out of reach of most young people living outside municipal capitals.

Timor-Leste Youth Parliament History Within this context, the Timor-Leste Youth Parliament (TLYP) stands out as one of few initiatives for young people with nationwide reach and influence. First sitting in 2010, TLYP has now undergone three cycles of two to three years’ youth parliamentary membership, mirroring the Government’s National Parliament by representing all 65 administrative post (sub-municipal) electorates.

The model of Youth Parliament in Timor-Leste was formally recognised by Parliament through Government Resolution No.23/2009 (Youth Parliament) in 2009. It aimed to build legitimacy and capacity to address young people’s exclusion from decisions affecting them and the future of their

6 National Youth Policy 2016, Part 5, pp.21-49 15

country; policies and practices that did not sufficiently consider the particular needs, challenges and rights of children and youth; and disenfranchisement and negative influences on young people in a rapidly changing context.

Several partners including UNICEF worked together with government on models and governance for the Timor-Leste Youth Parliament (TLYP) between 2008 and 2010, and government formally inaugurated TLYP in August 2010. Overseen by dedicated staffing through the TLYP Secretariat, three youth parliament cycles have taken place since 2010. UNICEF has given substantial support to the process over this time, but in line with parliamentary commitment, it is structured now as a permanent institution under SSYS.

Between 2010 and 2016, 392 young people were appointed as youth members of the TLYP. In the first cycle, youth members also took on the responsibility of drafting constitutions and resolutions to guide their purpose and interactions. In 2015, former members from both cycles proposed and largely self-formed an alumni association that could build on previous relationships and skills and continue to work on social issues affecting children and youth. The resulting Alumni of Youth Parliament Association has been a self-sustaining group with strong networks nationally and internationally, demonstrating an ongoing and effective commitment to youth advocacy platforms and civic initiatives. Figure 1 below gives a summary timeline history of TLYP.

Figure 1: Summary timeline, Timor-Leste Youth Parliament

TLYP principles The TLYP programme, from the Government Resolution onwards, intended to reach all municipalities in Timor-Leste, bringing equal opportunity to participate in civic and community processes for all children. While rights-based, it also contained pragmatic ‘business case’ elements, expecting that investment in improved leadership for a small number of young people would have a ‘multiplier effect’ as those young people began to work with peers, local authorities and other youth organisations to share their knowledge and skills. Refining

16 how this would work during the design phase (2009 – 2010), implementation stakeholders envisaged a systematic, inclusive, empowering process based on the following principles:

 FORMAL ELECTION: One boy and one girl, aged 12-17, from each of the 65 administrative post (sub-districts) across Timor-Leste, elected by peers as s TLYP members.  QUOTA FOR DISABILITY: In addition, two young people living with disabilities join TLYP.  ACCOUNTABLE LONGTERM MEMBERSHIP: Members hold office for three years7, with TLYP plenary sessions taking place annually within this cycle.  CAPACITY AND SKILLS: Members receive a wide range of training from different providers covering civic education, personal development and rights issues.  NETWORKING AND EXPOSURE: TLYP members attend relevant workshops on youth and adolescent issues, and connect with international youth movement branches.  POSITIVE PEER INFLUENCE: TLYP members hold community-level events focused on youth empowerment, and try to act as ‘role-models’ among their peers.  PRIMARY INSIGHTS: TLYP members conduct research on issues affecting peers: for instance, education, health, early marriage, unemployment, security and violence.  DIRECT ACTION: TLYP member research informs community initiatives and municipal- level discussion on youth needs.  NATIONAL POLICY ANALYSIS AND ADVICE: TLYP member research informs dialogue at plenary sessions. Recommendations from these sessions are submitted to relevant line ministries within Government for action.

While no formal, timebound objectives were set for the TLYP, the 2012 strategy identified four core objectives for the role of TLYP members, as follow:

 To influence: TLYP had a primary function to bring children’s voices into parliamentary process, informed by nationwide consultation. Members met in plenary sessions once a year to raise municipal-level priorities to national consensus and deliver recommendations to appropriate ministries.  To empower: TLYP was intended to build skills and confidence in young people for lifelong leadership benefits. TLYP training included public speaking and debate, journalism, civic education, human rights and gender equality. A ‘life skills’ course covered areas of self- esteem, positive thinking and healthy choices.  To increase engagement and participation: Young people, especially in remote areas, were disconnected from the process of civic participation and accountable governance. By increasing knowledge and interest of TLYP members in youth policy and its

7 The original design was for a two-year mandate, but the second two cycles took up three years’ worth of activities. 17

implementation, it was intended that other young people in their community would learn how to participate in a well-functioning civil society.  To lead: Only a small proportion of Timor-Leste’s children could take part in TLYP. It was anticipated that the model would result in multiplied reach and benefits, as TLYP members took up roles as leaders, influencers and trusted role models among their peers.

These interlinking objectives are shown more simply in Figure 2 below.

Figure 2: Objectives for the role of TLYP members

The role of UNICEF, and others, in Timor-Leste Youth Parliament UNICEF has worked closely with the Government of Timor-Leste on TLYP since 2007, when a small pilot Youth Parliament took place in advance of Mr Ramos-Horta’s UN General Assembly statement. This included technical support to drafting and passing the 2009 Decree, followed by intensive planning and collaboration to recruit, orient and train the first Youth Parliament in 2010. At that time, the Government established the affiliated but independent Secretariat of Youth Parliament, with a small team of staff and volunteers responsible for coordination of TLYP activities.

TLYP has been a pivotal program for UNICEF as a cornerstone emphasising youth in Timor- Leste’s UN Development Assistance Framework (UNDAF). While other UN agencies, especially UNDP, have engaged alongside UNICEF on government youth initiatives, UNICEF has been the main UN focal point for TLYP, providing dedicated human and financial resource to the process across three cycles. For broader youth-related initiatives beyond TLYP, United Nations in Timor- Leste has a Youth Results Group, chaired by UNFPA, and UNICEF is part of the results group.

Other NGO partners have stepped in and out with special projects to enhance aspects of training, networking or field research for youth members, or to enhance links between youth policy and youth-led peacebuilding movements in Timor-Leste. As a long-term contributor to peacebuilding initiatives, the German government’s GIZ was highlighted by UNICEF staff as an important partner in this regard. The umbrella organisation for youth groups in Timor-Leste, Conselho Naseional da Juventude de Timor-Leste (CNJTL), has also been closely involved in promoting and raising interest in TLYP.

From 2015 onwards, UNICEF has stepped back significantly from the processes of TLYP, for two main reasons: in line with internal restructure and reduced resources available for adolescent partnerships; and to transition TLYP to its permanent, fully budgeted structure under government. The transition has been successful. The Secretariat now takes full responsibility for coordination and budget of TLYP. Most recently, the role of the TLYP Alumni Association (APFTL) is expanding to strengthen connections between old and new members, with the APFTL and SSYS signing a Memorandum of Understanding to become a formal partner in TLYP plenary process.

The capacity of the Secretariat was last assessed in 2011, with some adjustments made at that time. It continues to provide crucial services to fair election process, media coverage and general promotion of TLYP, as well as facilitating all training for TLYP members.

In line with its NGO status, APFTL also tenders for community and school-based programs, and manages its own donor base, including national and international NGOs working with adolescents, youth and peacebuilding.

Figure 3 below shows the different phases of UNICEF support to TLYP in the last ten years.

Figure 3: Timeline of UNICEF support to TLYP

Process and progress: Results Matrix and the TLYP Theory of Change It is usual for development evaluations to include summarised monitoring data of results against core indicators for projects and programmes. In the case of TLYP this is problematic. TLYP did not implement under a project framework, but as an ongoing government programme; because of this, timebound indicators were not set or monitored. However, the purpose of TLYP, to prepare young people to be future leaders in Timor-Leste, has been relatively clear from its early planning phases. Tangible and immediate benefits were to be delivered through leadership courses and participation in parliamentary process, intended to be empowering for lifelong personal capabilities. Beyond

19 personal benefits, stakeholders in planning assumed that current and former youth members would return on investment in their skills and confidence through increased peer and civil society engagement, helping to identify and deliver positive change for young people’s participation and opportunities across the nation.

Objectives contributing to this aspiration were stated with best clarity in the 2012 TLYP Strategic Plan (Annex 4); TLYP members were expected to:  Influence through advocacy and discussion  Empower through education, training and practical experience  Increase engagement and participation for young people in decision making  Lead by example

By doing so, they would trigger broader civic, social and economic participation for the next generation of Timor-Leste.

These objectives and their long-term impact for children and youth were never formally articulated through a mapped Theory of Change. In its absence, TLYP results measurements have focused largely on process – whether activities happen in a timely way, involving the appropriate people - rather than the broader social transformation envisaged when young people became equipped and empowered to participate in community and national development.

It is nonetheless important for the evaluation to commence with validation of this activity-level data. A framework for doing so was eventually found in the 2012 TLYP Strategic Plan. The indicator table drafted in this document builds towards the four outward-looking objectives for TLYP youth- led social change listed above, largely using activity-level indicators. While it was never used as a monitoring tool, it can now be retrospectively filled to provide a Results Matrix for validating process achievements.

The Results Matrix is included as Annex 19 to this document. Initial document review provided information for most of the indicators, followed by targeted requests for further documented information from the TLYP Secretariat, APFTL or UNICEF where necessary. Lastly, questions in semi-structured interviews tested the general substance and validity of indicator measures with appropriate stakeholders. Because data limitations exist for proportional indicators, a column for confidence of findings has been added, and reminders as to this limitation have been used in the body of the report where relevant

The results matrix is relevant across many of the questions in the evaluation; it confirms that something has taken place, and gives space for the analysis to focus on why it happened: building and testing theories of change.

However, the program also lacked an articulated theory of change that could support ‘cause and effect’ analysis. As a starting point to setting scope of research and lines of enquiry, therefore, the consultant retrospectively constructed a Theory of Change. The map, built in consultation and agreement with government stakeholders and APFTL in November 2017, pulled together information from strategic documents to summarise TLYP purpose and desired outcomes, including:

 The core intent, expressed by President of Timor-Leste Ramos-Horta in 2007;  Original proposal documents;  Implementation strategies;  Subsequent strategic planning from Youth Members in 2012 (see Annex 4), and;  Vitally (though aspirationally at this stage) the five outcome goals of the National Youth Policy.

This Theory of Change is included as Annex 18 to this document. It suggests there are two levels of social outcomes beyond the direct sphere of TLYP inputs and outputs. Firstly, it predicts outcomes for the TLYP as an institution, and for its members and peers in terms of personal development. Secondly, the revised and strengthened National Policy for Youth (2016) now provides clear outcome goals for young people, and these have been used as the final impact level of change sought. Risks and assumptions are also shown where relevant. The figure shows the complexity and ambition of social change outcomes stemming from TLYP cycles, and it is unreasonable to expect that this change will happen rapidly.

The TLYP evaluation draws heavily on the theoretical causal links, assumptions and risks mapped in the retrospective Theory of Change in order to define its scope for research.

2. Evaluation background Evaluation Purpose, Objectives and Audience In 2017/18, the Government of Timor-Leste agreed with UNICEF to conduct a formative evaluation on TLYP under UNICEF Country Programme Action Plan (CPAP) 2015-2019 (extended until 2020) signed between the GOTL and UNICEF, to learn from nearly 10 years of preparation and implementation and to take decisions on future direction for TLYP. The relevance of the evaluation was underscored during the state budget discussion on TLYP in National Parliament, 2017. UNICEF considers this research to be an important partnership contribution to strategies for youth in Timor-Leste long-term.

The evaluation’s main purpose is to generate knowledge and evidence on the TLYP in context of Timor-Leste to inform national development policies and strategic planning processes, and to guide UNICEF CPAP 2015-2019 (extended till 2020) Outcome 3: Child Protection and Participation.

The evaluation was intended to address the following needs8:  A lack of TLYP documentation to inform evidence-based policy actions and share valuable lessons learned with other countries  The need to develop a standard training/orientation package for TLYP based on the lessons learned  An opportunity to revisit TLYP strategic plan, and update it according to current context of the country including National Youth Policy (2016) and 2030 Agenda on Sustainable Development Goals (SDGs)

Specific evaluation objectives, stated in the Terms of Reference (Annex 2), are:  To evaluate outcomes under the OECD-DAC framework of principles of programme quality: effectiveness, efficiency, relevance, impact, sustainability, and equity of the Timor-Leste Youth Parliament (TLYP) programme.  To draw recommendations on how the TLYP Programme can be further enhanced towards contributing to achieving results for children in Timor-Leste.  To review and provide recommendations on the content of the standard training/ orientation package for TLYP.  To document key findings and lessons learned of TLYP for local and international community.  To provide strategic guidance to the Government of Timor-Leste and UNICEF in determining its focus areas of support to adolescent and youth participation.

8 Paraphrased from the UNICEF Terms of Reference for TLYP evaluation, Annex 3. 22

 To identify synergy areas with other initiatives related to youth and adolescents at national and sub-national levels (municipality).

Primary audiences are the Government of Timor-Leste and UNICEF Timor-Leste. As stakeholders with ongoing responsibilities, the TLYP Secretariat and APFTL are also key audiences for analysing and acting on results. It is also envisaged that the research will be of international interest as evidence of young people’s value in policy negotiations.

Evaluation scope Initial scope for evaluation was set by UNICEF in consultation with government representatives, with questions in the Terms of Reference (annex 2) according to UNICEF guidelines and OECD/DAC principles for evaluating programmes.

Scope of data collection, consultation and survey did not have a geographic limit as TLYP has been a nationwide youth structure. While many stakeholders and partners were located in Dili, the scope of enquiry included other municipalities as relevant.

Scope of participation emphasised the experiences of children and young people, as well as actors in government, NGO and UN agencies who were working on adolescent and youth issues in Timor-Leste.

Questions related to:  Relevance: the programme’s suitability to priorities and policies of country, target group, end users, partners and donors  Efficiency: the programme’s outputs in relation to its inputs, and whether a reasonable return on investment is evident  Effectiveness: the programme’s achievements against its targets and objectives, and factors supporting these achievements  Impact: The positive and negative changes produced by the programme, directly or indirectly, intended or unintended.  Human rights and equity: the programme’s performance against cross-cutting goals of UNICEF’s country programme to build landscapes of opportunity for all, regardless of gender, disability, ethnicity or cultural background  Sustainability: the likelihood that the programme benefits will continue beyond the end of UNICEF support

During the inception phase, stakeholders were asked to consider and adjust the evaluation questions in the Terms of Reference, a participatory approach to framing the scope of enquiry. This included not only government and UNICEF as partners in the evaluation planning, but also 23

organisers and participants of the program, and in particular the young people who have taken part (volume of consultation appears at Annex 6). They raised additional areas of operational interest to be included in the evaluation framework as follow:

 Components and comparisons: The quality and sustainability of training, and comparison of different approaches to exposure visits for TLYP members over the three cycles;  Local influence: The extent to which former youth members are influencing peers and local governance (chefe suco/aldeia; chief of village/sub-village);  Compliance to original vision: Why certain principles such as democratic election or disability inclusion had not been upheld, and what might be done about it;  APFTL: The effectiveness and potential future roles of the Alumni of Youth Parliament;  Child protection: Given the highly politicised context of Timor-Leste, how TLYP is protecting children from recruitment to political factions.

As a result, deeper or new enquiry was added to the draft questions in the Terms of Reference. The final scope of enquiry appears below. Also see Annex 17, Evaluation Matrix (Evaluation and Evidence Framework). Questions added after the Terms of Reference, as a result of UNICEF or other stakeholders’ specific interest in learning, appear in bold font. While the set of questions appears substantial, many of them are simple to answer and validate. The methodology for doing so is described in the next section.

1. Relevance 1.1 To what extent are objectives still valid; and activities consistent with these? 1.2 How well is the programme aligned to UNICEF’s priorities/policies/ reform agendas (including 2030 SDG agenda), and with government’s national youth policy including Govt. Resolution No.23/2009 (Youth Parliament)? 1.3 How visible and relevant are original principles of democratic election and youth- driven agendas? 1.4 To what extent do the role model and multiplier aspects envisaged through TLYP remain relevant? 1.5 Are outcomes and measures identified in the 2012 TLYP strategy still valid? 2. Efficiency 2.1 Were activities cost-efficient in achieving the program’s intended goal? 2.2 Were objectives achieved on time through the inputs? 2.3 How well does the programme connect with and complement other youth programs and initiatives in Timor-Leste (and can this be done better?) 2.4 Can a value-for-money proposition be made, that the results evident for young people at community level (indirect beneficiaries) justify investment in youth parliamentarians (direct beneficiaries)? 3. Effective- 3.1 To what extent have the planned results been achieved (quantitative and ness qualitative)?

3.2 How effective was UNICEF’s support to TLYP goals and process? 3.3 To what extent did TLYP advocacy influence government decisions concerning youth and adolescent issues? 3.4 What components (training modules, orientation, exposure trips) are of most value to young people for a/ personal empowerment and b/ helping others? 4. Impact 4.1 What has happened as a result of the TLYP programme, and which activities have contributed most to this impact? 4.2 What unexpected, including negative, effects are evident for youth parliamentarians? 4.3 What progress can be seen as a result of TLYP community engagement towards the five strategic outcomes of the 2016 National Youth Policy? 4.4 Is there a correlation between TLYP community engagement and social cohesion/peacebuilding outcomes? If so, how can this result be replicated and scaled? 5. Human 5.1 To what extent has TLYP reflected its original human rights and equality principles: rights and gender, socio-economic background, geographical disparities (urban/rural), child and equity youth participation, and participation of children and young people with disabilities? 5.2 What UNICEF strategies/ approaches have been most efficient in influencing improvements to child and adolescents’ participation rights? 5.3 How did the youth parliament programme benefit other youth who are not TLYP members, in terms of a/ local policies and behaviours and b/ personal capacity and resilience? 5.4 How successful was the programme in supporting the most vulnerable children and adolescents in communities? 5.5 To what extent, and how, has independence and protection of youth members been assured (do no harm safeguards)? 5.6 Are young people’s voices and opinions being heard and respected more than previously? To what extent is this linked to TLYP? 6. Sustain- 6.1 To what extent did the benefits continue after donor funding ceased? ability 6.2 What were the major factors which influenced the achievement or non- achievement of sustainability of the programme or project? 6.3 What is the trend of state budget allocated for TLYP programme in the past, and expected trend in the future? 6.4 Has the programme been successful in leveraging governments’ political will and financial resources for child and adolescent protection? 6.5 What is the significance of the Alumni Association to the sustainability of Youth Parliament? 6.6 What is the best role for others (NGO partners, local institutions) to play in Youth Parliament and Alumni activities?

3. Methodology Academic rationale: selecting theory-based evaluation The background section to this report explains that TLYP is an ongoing initiative, intended to be a sustainable institution. As such it cannot ever be ‘finished’, and an evaluation should consider progress and process, as well as outcomes: a formative9 evaluation with a focus on effective practices, rather than a summative ‘impact’ evaluation which looks primarily at end results.

Given this, theory-based evaluation10 provides a solid solution to analysing TLYP results. Theory- based evaluation involves examination of assumed relationships between cause and effect: firstly, identifying whether and to what degree the intended change took place, secondly, identifying whether sufficient correlation between programmatic inputs and change outputs/outcomes exists to prove the necessity of the program in change observed. A theory-based approach is merited in this case for many reasons, namely:  Ability to measure attitude and behaviour change  Ability to connect measured change with confidence to causal factors, including but not limited to TLYP; other influences for youth in Timor-Leste can be considered  Ability to understand progress towards goals, even where goals have not yet been met, which is important considering the small fraction of direct beneficiaries (TLYP members) compared to Timor-Leste’s overall youth population

The evaluation was designed to be participative11; programme participants and coordinators guided and took part in agenda setting, data collection and results analysis. It aligned in particular to the principle of youth participation, and aimed to involve current and former youth parliamentarians at every stage from refining evaluation scope through to analysis and validation of research questions and prioritising recommendations. An important principle for application of results by government and other stakeholders was utilisation, which approaches enquiry and setting of questions with audience needs in mind, to maximise application of results, lessons and recommendations. 12 In terms of data collection and analysis, this was a mixed method evaluation. Theory-based evaluation has an emphasis on qualitative data to identify whether and why change has occurred in individuals. To understand the extent and trends of such changes,

9 Formative evaluation does not consider outcomes to be final or results to be closed, and allows for ‘learning while doing’ adjustments to phased programs; emphasises implementation, process and component evaluation; https://www.socialresearchmethods.net/kb/intreval.php 10 Theory-based research considers connections between cause and effect to answer not only whether something happened, but also why: results reflect context and diverse experiences http://www.3ieimpact.org/media/filer_public/2012/05/07/Working_Paper_3.pdf 11 “An overarching term for any evaluation approach that involves program staff or participants actively participating in decision making and other activities related to the planning and implementation of evaluation studies” King p. 291: http://devinfolive.info/impact_evaluation/img/downloads/Participatory_Approaches_ENG.pdf 12 See Michael Quinn Patton: https://www.betterevaluation.org/en/plan/approach/utilization_focused_evaluation 26 quantitative data was also required, sourced mainly from direct programme beneficiaries (TLYP members).

Children and youth were primary participants in this evaluation design. TLYP and APFTL introduced the evaluation team to highly capable young people who took part enthusiastically and professionally in setting questions, preparing and testing tools, hosting field visits and advising on data analysis. TLYP members chaired the final validation workshop and endorsed recommendations.

Data collection sources, sampling and analysis methods Sampling of primary sources was guided by initial theory setting (see Theory of Change, Section 1.6), followed by stakeholder interviews on which outcomes were most likely to be evident within that theoretical framework. Stakeholders expressed low confidence as to visibility of the final level of change aligned to the National Youth Policy framework, and also thought that the limited number of direct beneficiaries at this stage would not have resulted in the type of reach and peer influence needed for effective random sampling. As is often the case with formative, theory-based evaluations, therefore, the full range of change articulated was not selected for evaluation; it was agreed that many of the final outcome statements would not link with confidence to TLYP process and results. No control group was required. Primary data providers were selected purposively because of their interactions with TLYP (or the alumni organisation APFTL).

An online survey for current and former TLYP members was the core quantitative tool. To put results of the survey into perspective, as well as to allow space for issues of quality, compliance and feedback to emerge, the evaluation team conducted a range of interviews and focus groups with targeted stakeholders, aiming to prove or disprove that inputs in the TLYP Theory (either theoretical or validated through the Results Framework analysis) led to outputs for young people. Questions were tailored relevant to interviewee experience, while a semi- structured approach also allowed participants to raise their own observations beyond those reported in stakeholder interviews. A process map for this analysis Figure 4: Process map for theory-based analysis/validation appears as Figure 4.

In line with the need for purposive sampling, three field visit locations were selected in consultation with APFTL representatives. Criteria for selection were that a current TLYP member was in the

27 suco (village), that APFTL had recently conducted events or training in schools of the suco. A third criterion of a community initiative or event coordinated by a current or former TLYP member in location was initially desired, but after lengthy consultation and poor response to this criterion, it was removed (see also Limitations). Specifically, data sources were:

Quantitative:  Online survey (including multiple choice and text fields), of current and former TLYP members; aimed for 30% opt-in of 392 possible respondents; achieved 15%, or 63 respondents. (Annex 5). Triangulation of perspectives was important (interview, stakeholder discussion) to mitigate potential quantitative confidence bias of the low number of respondents (see Section 3.4. limitation and mitigation measures).  Desk review of geographic representation, age, school location and type, of current TLYP members (Annex 9) Qualitative:  Document review, with emphasis on strategy planning, lessons and revisions 2008 – 2017 (see List of Information sources, Annex 1)  Policy analysis, with emphasis on youth policy 2012-2017  Semi-structured interviews in three locations, including: o Current and former Youth Parliament members (22) o Government partners and Ministry representatives (4) o Representatives from NGOs, UN and other development partners (3) o UNICEF staff (6) o Community leaders; chefe suco, teachers and principals (10) o Focus groups with TLYP participants (2 groups, included in numbers above) o Focus groups with students (8 groups, 64 people)

Validation of results by at least two different sources, preferably three (triangulation), has been a principle of the analysis phase. As well as using primary and secondary data, the evaluator wove validation questions into semi-structured interviews where relevant to the respondent: for instance, by taking initial results of the TLYP member survey to SSYS representatives to shape interview content. The analysis phase concluded with a full day validation workshop, chaired by Dili-based TLYP members and attended by government, APFTL, UNICEF and other NGO representatives.

Diversity and inclusion; ethics and risks Children and youth made up a strong majority of data contributors, through survey or direct interview. Semi-structured interviews and focus groups aimed for a balance of male and female participants and included questions on gender observations and attitudes. Though efforts were made to locate and interview TLYP members from disability intakes, it was not possible to do so.

Also, as TLYP had not worked with disability organisation or advocates, it was not practical to include disability partners as purposive key informants.

The evaluation followed UNICEF guidelines for ethical standards including rights and gender considerations13. Principles of informed consent, confidentiality and voluntarism were applied, including written information to each participant or focus group leader with the option to withdraw at any stage. A technical reference group comprising UNICEF, government and university partners provided a quality assurance mechanism including ethical review, between the submission of the evaluation plan and the start of consultation with children and communities. More information on handling ethical considerations is available at Annex 10, and the terms of reference for the technical reference group at Annex 11.

Limitations and mitigative measures Limitation Risk Mitigated by…?

Inconsistent vision Stakeholders did not share definitions of The retrospective ‘Theory of Change’ and on TLYP core TLYP purpose, targets and indicators. associated indicators, written in consultation purpose and desired Outcome / change indicators were not with stakeholders along with agreement on outcomes; no stable circulated or used. Without outcome using TLYP behaviour outcomes from the outcome indicators targets, questions about meeting targets 2012 Strategy: influence, empower, engage, were unable to be answered. lead. Change sought is This is a long-term programme, the social The theory-based formative methodology inter- generational; change it seeks to bring for young measuring progress rather than end results. more time is needed. people’s civic participation is complex and Broad community perspectives were not has not yet happened. surveyed as no change was likely. Sampling does not Time / cost constraints limited site visits to Triangulation of data from site visits and fully represent all four out of 65 administrative posts; the surveys through SSIs and the final workshop Timor-Leste online survey gave all TLYP members with TLYP members, government, APFTL, viewpoints. opportunity to respond, but not all did. UNICEF and other NGO representatives. Other youth Results in young people’s confidence, A theory-based approach, which considers programmes also empowerment and employability may be ‘cause and effect’ of changes identified. The contribute to youth associated with other initiatives rather evaluator also applied counterfactual empowerment. than TLYP. questions and logic in interviews to identify other potential influences.

13 UNICEF Procedure for Ethical Standards in Research, Evaluation, Data Collection and Analysis, available at: https://www.unicef.org/supply/files/ATTACHMENT_IV-UNICEF_Procedure_for_Ethical_Standards.PDF ; UNEG Guidance on integrating Human Rights and Gender Equality in Evaluation, available at: http://www.uneval.org/document/detail/1616; UN-SWAP Evaluation Performance Indicators, available at: http://www.uneval.org/document/detail/1452. 29

Participation bias Working with APFTL on case studies and Lessons/ recommendations were built interviewees meant most taking part had collaboratively and transparently, with good engaged well with TLYP, creating a opportunities for reflection and honesty positive bias to results. among participants. Rescheduling of Due to political instability in 2018, the An additional stakeholder workshop to evaluation field evaluation’s field enquiry phase moved refresh government and APFTL on inception enquiry from January 2018 to September 2018. phase findings and to relaunch This affected project momentum, implementation in early September 2018. including survey take-up. Online-only survey Change to evaluation schedule meant the Asking survey questions face-to-face where dissemination survey could not be handed out at the possible (for instance in TLYP member TLYP National Plenary as planned. It interviews during site visits); triangulation of became an online-only tool, making it survey responses through interview and difficult for children without internet to take consultation. part. Survey questions A small number of APFTL members Final analysis used proportional views of changed. completed the survey in 2017 before results against the number of responses per additional questions were added post- question, rather than total survey workshop in September 2018. participants. Limited local-level The evaluator could not identify any case Not mitigated: decision made to look at influence evident. studies of children leading local initiatives, output validation – mainly training or despite prompting over several months community consultation processes from through multiple sources14. APFTL – rather than measurable policy or peer influence outcomes. Linked to the above, A lack of monitoring for outreach activities The value for money calculations in this discrepancies and results, inability to identify examples report presume a conservative proportion of between TLYP self- of youth-led change, and low recall of 33% TLYP members conducting training, reported levels of training among peers and classmates, rather than the 92% reported in the survey. training and outreach made it necessary to estimate rather than to other young validate TLYP indirect beneficiary reach. people.

14 As the body of this report explains further, this did not mean such work had not taken place. Improved monitoring and reporting of TLYP community interactions is a priority recommendation for more intuitive identification of youth-led influence. 30

4. Findings

4.1 Relevance

The programme’s suitability to priorities and policies of country, target group, end users, partners and donors.

Summary of Findings

Documentation and stakeholder viewpoints showed Questions: strong alignment with original objectives (2009) as (bold indicates a question added through goals and processes were refined over time. Youth stakeholder consultation at inception phase) empowerment remains an urgent need in Timor-Leste. 4.1.1 To what extent are programme TLYP also shows strong alignment to national and objectives still valid, and activities global strategies for youth inclusion and empowerment. consistent with these? TLYP is strategically relevant to the current UNICEF 4.1.2 How well is the programme Timor-Leste CPAP under adolescent participation, and aligned to UNICEF’s priorities/policies/ to the goals of several other UN agencies working in reform agendas (including 2030 SDG children’s and youth issues in Timor-Leste. Youth do agenda), and with government’s not currently play a formal consultative role in the SDG national youth policy including Govt. roadmap planning for Timor-Leste, and TLYP are well Resolution No.23/2009 (Youth placed to step into the role. Parliament)? 4.1.3 How visible and relevant are Youth-driven agendas remain core to TLYP plenary original principles of democratic election process, as do inclusion principles that ensure gender and youth-driven agendas? balance and representation of all administrative posts 4.1.4 To what extent do the role in Timor-Leste. Some concern was voiced that a model and multiplier aspects disability quota principle introduced in the first cycle has envisaged through TLYP remain not been successful. These stakeholders would like to relevant? see a renewal of the principle or an alternative that 4.1.5 Are outcomes and measures equally assures youth disability perspectives are identified in the 2012 TLYP strategy incorporated into TLYP. still valid?

Relevance of the model to outcomes for direct beneficiaries (TLYP members) is consistently validated through various data sources. Connections to outcomes for youth policy and peer influence are visible to a degree but the evaluation has made recommendations for enhancement and re-targeting in this area.

4.1.1 To what extent are programme objectives still valid, and activities consistent with these? In general, programme objectives remain valid. The proportion of youth population in Timor-Leste is increasing (as described in the background to this document), and the government of Timor- Leste has committed to working with youth as a priority. At this stage TLYP is the most consistent programme for young people and the only government-funded programme with a civic responsibility component. TLYP has potential to trigger substantial reductions in corruption tolerance, and to emphasise social issues and advantages of public policy.

Specific to articulated TLYP objectives, the evaluation considered the list of four objectives set in the 2012 TLYP Strategic Plan, and found that current TLYP members remain aware and committed to all four (Table 3). Interviews consistently described the personal value of membership in increasing capacities for leadership, training and information sharing.

Among the four objectives, youth leadership has become the predominant intention of TLYP, for members and government organisers alike. A phrase used by almost all interviewees in this evaluation, without prompting, has been that TLYP ‘prepares young people to be future leaders.’ This does not necessarily indicate political leadership, though many interviewees showed interest in entering politics later in their lives. The actions and voluntarism of AFPTL members, as validated through interviews with different stakeholder groups, showed the majority of former parliamentarians were deferring political partisanship and continuing to work as intended in areas of community development, emphasising youth issues.

TABLE 1: Do objectives and activities remain relevant?

Still aligned with TLYP TLYP members – still government priorities? members – committed and active? still aware? Objective 1: Influence through Yes Yes Mostly - Current TLYP advocacy and discussion members are less likely to engage with local authorities than those from previous cycles. Objective 2. Empower through Yes Yes Yes education, training and practical experience Objective 3. Increase Mostly. Connections to Yes Yes engagement and participation policy making were for young people in decision strongest around the time making of drafting the National Youth Policy, 2015/16. Objective 4. Lead by example Yes Yes Yes

4.1.2 How well is the programme aligned to UNICEF’s priorities/policies /reform agendas (inc. 2030 SDG agenda), and with government national youth policy (inc. Govt. Resolution No.23/2009 (Youth Parliament)? Converting Timor-Leste’s ‘youth bulge’ from risk to resource was a primary motivator behind the Government Resolution for Youth Parliament in 2009. Alignment to this resolution remains strong, especially evident in government stakeholders perspectives; they consistently see it as the core strategic purpose. Literature review of other youth parliament models (see Annex 13) found Timor- Leste’s Youth Parliament is unusual in recognising the role of youth in nation-building and social cohesion among its objectives. TLYP and APFTL partnered on youth consultation for the 2016 National Youth Policy which also includes an end to violence among its five thematic priorities.

Another unusual attribute of TLYP compared to other youth parliaments was the age group, involving only children (aged 12 to 17) rather than youth (varying definitions but often from 15 to 25). Figure 5 below shows different definitions of youth applied in this situation, including the Timorese ‘foinsa’e nian’, which translates loosely to ‘almost grown’ and captures the essence of the TLYP and Alumni age group. The crossover needs of age groups are met well by TLYP, with policy platforms including sexual and reproductive health access, employability and youth employment markets and police accountability (see Analysis of TLYP Plenary Recommendations, Annex 14).

Figure 5: Crossover of child, adolescent and youth definitions, Timor-Leste context

TLYP aligns well with UNICEF’s mission for children. Working with youth, whether adolescent or early adulthood, is of increasing importance to UNICEF, who recognise the critical second decade in life as an emerging issue in the UNICEF Strategic Plan 2018-202115. The current UNICEF Timor- Leste Country Programme has adolescent participation components linked directly to TLYP target group (under 18). UNICEF remains the primary UN stakeholder for TLYP and its child members, and contributes to planning with other UN agencies, coordinated through the Youth Working Group.

15 https://www.unicef.org/publications/files/UNICEF_Strategic_Plan_2018-2021.pdf 33

4.1.3 How visible and relevant are original principles of democratic election and youth-driven agendas? Documents and stakeholder interviews show that the original intention of a formal democratic ballot across Timor-Leste was never used. After refinement in each cycle, the selection process now includes three phases: written application, public speaking and panel interview. While young people vote for candidates at the public speaking competition, they do not vote on the final decision. Discussions with TLYP members, government and UNICEF staff all indicated that this has not disadvantaged the principles of socio-economic inclusion or led to nepotism. However, in interviews some confusion and disappointment was noted among unsuccessful candidates with regard to final selection. A formal vote would increase transparency of results, but consideration must be given to costs in a nationwide voting process (see also Section 4.2: Efficiency).

Inclusion principles of geographic representation and gender balance were formally mandated and closely followed. Two other principles of inclusion were not followed: additional non-geographic membership of two young people with disability; and representation of children out of school. Interviews with stakeholders implied that while the concepts were sound and supported, the practicalities were overly challenging given current networks and resources. This is discussed in more detail in Section 4.5: Human Rights and Equity.

Interviews with TLYP members showed they were aware of their responsibilities to represent other young people. For some this was a major motivator for applying. On average, survey respondents reported a 92% increase in their interest in youth issues and doubled their interest in civic participation / policy (see Figure 6).

Figure 6: Survey self-rated interest in civic and youth issues, before / after TLYP

To meet this interest, TLYP is intended to encourage youth participation on issues relevant to them, from suco level through to municipal consultation and national plenary. At local level, though TLYP member interviews and survey results indicated local consultation took place to inform national policy, this was not strongly validated through recall by school students. The process became more visible from municipal level upwards, with policy content clearly reflecting contextual issues. More than half of survey respondents also reported they had proposed or campaigned on youth programs and policy (see Figure 7); again, this was not strongly supported by interviews with chefes or students.

4.1.4 To what extent do the role model and multiplier aspects envisaged through TLYP Figure 7: % survey respondents using skills in remain relevant? their community Multiplying results for young people through school and community networks has not been an automatic outcome of TLYP as hoped, but has required additional inputs. All interviewed TLYP members, and the majority of survey respondents, said they had shared information from their national or study trip visits, but focus groups with classmates did not show strong recall of these discussions. While usually aware that someone at their school was a TLYP member, school representatives struggled to describe what TLYP did or to name any direct training from TLYP. TLYP was never mentioned, even with prompting, as a training provider or youth event coordinator.

APFTL, however, was acknowledged in schools and communities as a known youth organisation. Their role now reflects a core intent of TLYP influence outcomes, and has proven essential to the multiplier effect, though it was not originally part of programme design. APFTL also contributes greatly to sustainability of youth empowerment and participation, by creating a cycle of renewal and exponential membership that expands reach and regularity of youth training. It is guaranteed to grow; as each mandate concludes, new alumni will engage with APFTL post-TLYP, while existing members are clear on their desire to stay involved long-term. One respondent described APFTL as ‘for life, like a tattoo.’ This long-

35 term commitment is contextually appropriate, with many other youth organisations in Timor-Leste also blurring the upper limits of the youth age group and youth work16.

4.1.5 Are outcomes and measures in the 2012 TLYP strategy still valid? Outcomes and measures in the 2012 TLYP strategy could be a starting point for a more robust monitoring framework, but are not well suited to measuring progress in their current form. As noted in the background section of this document, outcomes and measures have never been agreed or used for TLYP monitoring. The original concept paper (2008) proposed indicators to measure policy knowledge and participation of children in and out of school, not only those taking part in youth parliament but also across society more broadly. These indicators were not measured at the time or since, and government census data (2010) is also not framed to demonstrate shifts of this nature; as well, there was consensus among stakeholders that this broad change has not yet happened. The most practical way forward is for TLYP is to introduce monitoring systems that young people can manage for themselves, and that include qualitative elements of policy change and consequent improvements to opportunity, especially at the local level.

Interviews revealed strong understanding of the value of quality monitoring so that outcomes of TLYP could be better understood and reported. It remains unclear who might take on this monitoring, and how it might connect with broader monitoring of National Youth Policy progress, a duty formally assigned to umbrella youth organisation CNJTL.

In the meantime, this evaluation has used outcome indicators from the 2012 strategy to support its Results Matrix (Annex 18) and further analysis of the extent and cause of direct benefits for TLYP members. The National Youth Policy at this stage offers strong outcome statements for adolescents and youth, which were not available at the time of drafting outcomes in 2012. The Theory of Change built for this evaluation (Annex 17) shows how TLYP outcomes, based loosely on the 2012 TLYP Strategy and in particular the role-based outcomes for TLYP members (influence, empower, engage, lead), may be reframed to demonstrate pathways and contributions to progress the National Youth Policy.

16 For instance, according to interviews with CNJTL and chefe sucos, many government youth centre managers have been in the role long-term and are aged over 25; likewise, electorally appointed youth representatives who work in a voluntary capacity alongside the chefe suco do not have an upper age limit, so long as they are willing to focus on youth needs. 36

4.2 Efficiency

The programme’s outputs in relation to its Questions inputs, and whether a reasonable return on (bold indicates a question added through stakeholder consultation at inception investment is evident phase) Summary of Findings 4.2.1 Were activities cost-efficient in Overall process and schedule has been adhered to, though achieving the program’s intended some activities ran late, with risk of disrupting school goal? attendance, and some were cancelled. The third cycle has 4.2.2 Were objectives achieved on been particularly affected by this, due to a government time through the inputs? spending lockdown in 2018. Comparison of the three 4.2.3 How well does the programme cycles of TLYP reveals that the first cycle benefited from a connect with and complement other more substantial budget, allowing outsourcing of training youth programs and initiatives in provision and curriculum development, study tours Timor-Leste (and can this be done according to TLYP member interest, and strong media and better?) messaging during plenary activities. This cycle was 4.2.4 Can a value-for-money supported and subsidised by UNICEF. Spending has proposition be made, that the scaled back since that time. While the programme is not results evident for young people at less effective as a result, TLYP members voiced some community level (indirect dissatisfaction about reduced resources. Since the draft beneficiaries) justify investment in strategy plan in 2012, implementing stakeholders have not youth parliamentarians (direct documented changes to the model or reasons behind them. beneficiaries)? Formal strategy, measures and guidelines are now required.

Efficiency of outreach work has been greatly enhanced with the formation of APFTL. Still at school, TLYP members had many demands on their time and a primary responsibility to completing their studies. APFTL made it possible for school leavers to take up opportunities for leadership and training in communities (at home and in other locations). Other national and international organisations also implement youth programmes through schools and government youth centres, as well as using their own systems (for instance, Red Cross, Scouting Movement). TLYP reaches fewer young people directly than other initiatives but offers depth of training and nation-wide engagement unique in Timor-Leste; it was found to be the only initiative with a focus on civic participation of young people.

The government currently fully supports the TLYP on a budget of around $US200,000 per annum. This section includes a return-on-investment calculation demonstrating that significant reach (and therefore, value for money) is possible: however, with the caveat that figures are estimated and the impact of that reach currently unmeasured.

4.2.1 Were activities cost-efficient in achieving intended goals? Figure 8 shows overall expenditure as a proportion of SSYS overall over the last three years, showing spending fluctuations. Election years have high cost implications because of advertising, events and travel costs across 65 administrative posts. SSYS overall budget has reduced each year. Other factors may also be relevant: for instance, challenges to national budget release in 2017, which affected SSYS expenditure and meant fewer activities than usual. The first and second cycles (not shown here) included investment from other partners, predominantly UNICEF. Especially for the first cycle, these could be considered start-up costs, such as the costs of outsourcing training materials and provision or promoting the initiative.

Local resources have been low-cost solutions for implementation Figure 8: TLYP budget, proportional to and promotion. The first cycle received substantial coordination overall SSYS budget, 2015-17 support from partner agency CNJTL, a voluntary youth movement with strong visibility and reach across Timor-Leste. Existing local networks (chefe suco/village leader, along with elected gender representatives and youth representatives) were also called upon to support the nomination and application process.

The TLYP programme has fixed input costs such as transportation, accommodation, training and materials. All inputs demonstrated cost-efficiency considering the need to conduct activities in 65 different locations upwards to national. However, if these costs are considered by direct beneficiary (130, over two to three years), they appear high: another reason to encourage effective monitoring of the outreach or ‘multiplier’ component of TLYP. Section 4.2.4 below suggests a value for money calculation, which could be further strengthened with more specific reach and impact data.

The TLYP budget is dependent on government decisions and will change annually, but care must be taken that minimum costs of operation are fixed and supplied. Cost-cutting can have detrimental effects on results: for instance, training manuals have not been updated or converted to file for easy transfer; very few activities took place at municipal level (where members could meet more regularly and easily than at national level); training modules were short (usually one day) without refresher options; TLYP members wanting to create community mobilisation needed to apply to community development funds if they wanted to provide snacks and water. Interviews showed that this final point could be a deterrent for some TLYP members in their community work; they were not confident putting proposals forward to local authorities and would have preferred to apply to, or through, APFTL or the TLYP Secretariat.

4.2.2 Were objectives achieved on time through the inputs? Initial schedules for activities spanned two years. However, a key lesson reported from implementers of the first cycle (2010-2012) was to allow time for recruitment, to reach candidates from different locations and backgrounds, and for orientation and skill-building of new members. It is now understood (though not formally documented) that the cycle requires three years, with the first year focused on recruitment and orientation before community and policy activities start. This means one year in three where no plenary takes place. Figure 9 shows how the changeover of cycles and transition to alumni membership works.

Even with the expanded timeline, the current cycle (2016-18) has not implemented according to schedule. Delays were attributed to budget uncertainty and to the 2017-18 freeze of political process in Timor-Leste. TLYP member interaction is intended to take place largely in school holidays, which challenges rescheduling. This limitation has been the main reason for inability to provide study visits during the current cycle, a gap in activities which was raised consistently by current TLYP members in interviews as unfair.

Figure 9: Flowchart of activities, three year cycle, showing outgoing transition to APFTL

4.2.3 How well does the programme complement other youth programs and initiatives in Timor-Leste?

Visits to schools in three municipalities found five main types of program:  schools directly, where principals or teachers took initiative to run a non-curricular activity (though this was not common);  schools indirectly, where after-school or weekend courses were available, or where trainers came to schools and worked with class representatives; this was usually how APFTL conducted training 40

 government youth centres, where programs usually mirrored the skills and experiences of local volunteers; APFTL partnered with the youth centres on awareness work, for instance on civic participation  community, including consultations and awareness raising focused on youth issues; usually using community buildings and involving chefe suco or similar local leadership  TLYP training, held for members only and usually at the national level, but with expectation that key messages would be shared with other young people at home.

Figure 10 shows the extra-curricular learning focus of each provider type, demonstrating the advantages for young people in and out of school who can access one or more of these channels.

Other initiatives currently reach many more children; for instance, the scouting movement or Red Cross Figure 10: Five channels for youth extra-curricular activities / participation (a list of identified youth partners appears at Annex 19). TLYP members are very likely to be participating in multiple groups and events; in fact, interviews with TLYP members showed a common thread of active involvement in other youth organisations as a criteria for candidates’ success.

Focus groups revealed a shortfall between training resources and needs, with only slight duplication between programs, plus many gaps in access to life skills, vocational and rights training. Within this, APFTL training topics were well targeted for young people in and out of school. However, limited resources of all partners, including APFTL and government youth centres, challenges the reach and depth of learning provided. Schools did not proactively coordinate external trainers, and courses were rarely offered on a consistent annual basis.

Connections between youth groups and TLYP members have been important; for instance (as mentioned above) an advantage for candidates was previous association with local youth groups or chapters; government youth centres promoted TLYP as an opportunity and helped young people prepare their application. In the locations visited, school students and teachers did not use or

41 promote youth centre training, which was geared to vocational or out-of-school clients and did not strongly appeal to school-aged children; however, it should be reiterated that this is a small sample from three out of 13 municipalities.

Coordination within UN on youth programs is conducted through the Youth Results Group, chaired by UNFPA. This helps other UN agencies besides UNICEF, especially UNFPA and UNDP, to engage with government on youth. However, knowledge among students of the role of UNICEF or other UN agencies in youth participation was low. In the schools visited, nobody knew of programs from UNICEF or UN partners.

4.2.4 Can a value-for-money proposition be made, that the results evident for young people at community level (indirect beneficiaries) justify investment in youth parliamentarians (direct beneficiaries)? Though spending has fluctuated (see Section 4.2.1), government stakeholders estimated an annual budget of around $200,000 was required for consistent, standardised TLYP operation. Based on this figure, over three years, direct investment per TLYP member is $4,615. This provides 10 modules of training, coverage of municipal and national travel expenses, core staffing and resourcing costs, study tours and other activities. Equal inclusion of girls enhances this value through gender empowerment outcomes, and inclusion of participants from remote areas of Timor- Leste contributes to reduced geographic inequities in civic participation.

Drawing on survey responses, a return on investment scenario found that members had on average trained or presented to 235 other people17 (Figure 11, next page). Building this further, but using conservative estimates of the number of TLYP members conducting outreach training (33% of each cycle, rather than the unverified 92% reported in the survey), there is potential to reach over 10,000 indirect beneficiaries18. This represents good value for money.

If one-third of each cycle (43 members) becomes an active mobiliser during or after TLYP, and they reach this average of 235 people each, this represents potential to reach around 10,105 people. Thus, investment in 130 direct beneficiaries can be projected to have positive outcomes for over 10,000 indirect beneficiaries if the training takes place, which represents good value for money.

17 Figure 9 shows the actual survey amounts. From a sample of 48 respondents, 92% reported they had trained in their school or community; 33% gave specific numbers of events and attendees; the average number of training events was 4.9 and the average attendance 48. Multiplying 48 x 4.9 gives a total average outreach of 235 people. 18 Per cycle: 43 members (33%) x 235 indirect beneficiaries gives a total reach potential of 10,105. 42

Confidence in this calculation would be greatly strengthened if specific monitoring records were kept. Quality of training and retention of knowledge are also areas to consider before making final conclusions on this value-for-money scenario.

Figure 11: 'Multiplier' return on investment scenario

4.3 Effectiveness The program’s achievements against its targets and objectives, and factors supporting these achievements

Summary of findings Questions Targets for direct involvement of young people have been (bold indicates a question added met, including targets for gender balance and geographical through stakeholder consultation at representation; however, targets for disability inclusion were inception phase) 3.1 To what extent have the not met. Activities aiming for policy influence and youth voice planned results been achieved took place as expected. As there were no targets set for (quantitative and qualitative)? these, it is not valid to assess quantitatively whether they 3.2 How effective was UNICEF’s were reached. Interviews with TLYP members showed they support to TLYP goals and were aware of their responsibilities to represent other young process? people. For some this was a major motivator for applying. On 3.3 To what extent did TLYP average, survey respondents reported a 92% increase in advocacy influence government their interest in youth issues and doubled their interest in civic decisions concerning youth and participation / policy. adolescent issues?

3.4 What components (training Survey respondents rated the effectiveness of component modules, orientation, exposure inputs highly, though with variations between the different trips) are of most value to young courses and activities, as well as by year. A strong data trend people for a/ personal from the survey was correlation between skills empowerment and b/ helping remembered/rated highly and skills subsequently used: for others? instance, public speaking, leadership and life skills training. The length of training also affected survey ratings, with longer courses receiving higher scores for effectiveness.

At this stage, causal links from TLYP process to national policy improvements are not apparent, with the exception of the once-off youth consultation prior to the 2016 National Youth Policy. UNICEF and partners (government and NGO) reported this as a highlight achievement for youth participation and civic voice in Timor-Leste. It was also the first major event to be led by APFTL.

TLYP has been a pivotal program for the Government and UNICEF as a cornerstone of the emphasis on youth in Timor-Leste’s UN Development Assistance Framework (UNDAF). Interviews confirmed the essential nature of early UNICEF support to planning and implementation of TLYP. UNICEF influence can be seen in elements of design such as gender balance, age group (12 – 17), disability inclusion, and training content, which includes child rights and gender. UNICEF staffing allocation to TLYP has been a constant and appreciated resource since 2009.

4.3.1 To what extent have the planned results been achieved (quantitative and qualitative)? The Results Matrix (Annex 18) shows basic results for the program, measured in October 2018, but does not consider more nuanced aspects of progress towards change for children and youth. TLYP was designed to bring change for young people across Timor-Leste at three levels: nationally through policy engagement; locally through community influence and action; personally through peer support and sharing skills and knowledge. Table 4 below shows the effectiveness of the program model to date across these three levels. It uses the objectives set in the 2012 Strategy19 and draws from interview data, as well as feedback given in the survey, to summarise where qualitative results have met expectations of implementers and participants, and what more might be done to extend results in the future.

Table 4 shows that civic education has been beneficial to young people’s interest and engagement in Timor-Leste’s democratic process: also that TLYP members have received and used substantial benefits from their association with the program. A strong voice in youth policy is partially achieved, but requires more formal ministerial acknowledgement of national plenary recommendations. The positive personal results emerging from TLYP training in leadership, civic education, life skills and public speaking have potential to be shared more effectively with other young people. Interviews illustrate that many TLYP members lacked guidance and appropriate networks for engaging in local-level policy process or community event planning.

Measuring results of policy influence can be challenging, and it is unclear whether this has been a strong expectation for TLYP to date. Though young people’s active contribution to policy-making was intended, no specific platform was set. The children raised local and timely issues which changed from year to year, rather than advocating with consistency on specific policy requirements. However, personal conviction built over this time was shown to lead to advocacy on specific change at a later date: for instance, the group working on re-entry for teenage mothers in schools (see case study, Annex 16).

19 Objectives from the 2012 Strategic Plan (Annex 4), and the third level of the reconstructed Theory of Change; two outcome levels appear beyond these achievements. 45

TABLE 2: Where is TLYP’s youth-led change evident? Key:  Strongly evident Nationally: Locally: Personally:  Moderately evident Improved policy, youth Improved services, Change in youth  Slightly evident advancement through stereotypes, attitudes empowerment and  Not evident policy affecting youth participation

Objective 1:    Influence through There is a need for more TLYP members need help to TLYP members have skills advocacy and formal follow up on build stronger connections to influence others, but discussion recommendations to to chefe suco and local law. need pathways for doing national parliament. so to be fully effective. Objective 2. Empower through    education, training TLYP and APFTL are very Lack of monitoring limits Personal empowerment and practical active in and around Dili. insight into reach and depth outcomes for TLYP experience More support to remote of training achievements. members are clear. Formal areas will distribute guidelines for empowering services more fairly and other children are needed. consistently. Objective 3. Increase engagement and    participation for Policy participation through Some community events The model delivers young people the TLYP plenary is a core involve TLYP members and personal participation annual process recognised other young activists. Again, outcomes for TLYP by parliament. more monitoring is needed. members. Objective 4. Lead by example    TLYP and APFTL are APFTL members are more TLYP members are recognised and actively likely than TLYP members to excellent role models but promoted as role models be locally vocal on youth need guidance on how to for youth. They may benefit issues. extend this to others. from support to manage their own media and social media profiles.

4.3.2 How effective was UNICEF’s support to TLYP goals and process? Details of UNICEF’s different phases of support have been given in the background section to this document. Interview data gives consistent evident that UNICEF has been pivotal and valued for multiple roles throughout these phases. Current and former TLYP members spoke of the value of personal support, attendance and motivation at plenary sessions, and invitations to other youth participation opportunities (predominantly for Dili-based members. TLYP and APFTL Secretariat confirmed the value of training support, and expressed hope that UNICEF could support a curriculum review in the near future. SSYS government representatives appreciated UNICEF support to government over several years in terms of stabilising the institution of TLYP and promoting its worth within UNICEF government relations more broadly.

It was noted generally in interviews that the first cycle (2010-12), supported by UNICEF as a project, received the most attention and resources. This set standards that were not necessarily practical for government to reproduce: for instance, in the diversity and quality of external training providers and in opportunities for international study visits. While the theory and model of TLYP has not changed significantly since that time, some cost-cutting has been necessary which has led to a reduced training package and fewer opportunities for TLYP members to travel.

UNICEF has supported reflection and learning for TLYP stakeholders, particularly in Cycles 1 and 2. However, the handover of process to SSYS from 2013 onwards did not include reporting and learning components. The strategy produced in 2012 has not been used, and changes to the model since that time have not been formally documented: for instance, changes to the annual schedule under a three-year cycle. A recurring recommendation from stakeholders has been to create guidelines that will give clarity on current processes: another area where UNICEF could provide effective technical guidance to TLYP Secretariat.

5.3.4 To what extent did TLYP advocacy influence government decisions concerning youth and adolescent issues? There is insufficient evidence to suggest that TLYP has influenced national policy, with the exception of their involvement in the National Youth Policy drafting in 2015.

Aggregating recommendation themes from plenary sessions (2014, 2015, 2017, as provided by TLYP Secretariat), it can be seen that some issues recur strongly each year: for instance, early marriage/sexual and reproductive health; school and health infrastructure; vocational opportunities for youth. Others appear specific to young people’s exposure and experiences that year; for instance, after study tour visits to overland border towns, youth parliamentarians became vocal on labour migration laws, a theme that was not raised in other years. Figure 13 shows a highly summary analysis of TLYP recommendations, while Annex 14 shows more detail.

Formal parliamentary Figure 12: TLYP plenary recommendation themes, three-year comparison acknowledgement is not part of the existing process, which makes it currently impossible to determine whether TLYP recommendations have been considered in national policy making. While it was confirmed by 47 several stakeholders that recommendations were publicly submitted (including some media interest in the submission), national ministries were not accountable to respond, nor were follow- up advocacy actions taken.

TLYP recommendations in 2017 followed the National Youth Policy accountable framework of five thematic areas, making it simpler in theory to monitor both the National Youth Policy and TLYP contributions to it. However, though CNJTL is assigned responsibility for local monitoring off both, interviews with members of the organisation indicate that resources and guidance to do so are currently lacking.

APFTL provides an ideal opportunity to track TLYP member influence once they become alumni (and young adults). APFTL representatives reported around 100 former TLYP members (38%) remain engaged; the survey found a similar proportion with 42% of former members saying they regularly joined APFTL initiatives and a further 10% saying they stayed in contact. Many sources were able to confirm that APFTL members are locally active and seek opportunities for youth and community work. Again, the effect of this on policy decisions, at local and municipal levels of governance, is not monitored at this stage.

4.3.5 What components are of most value to young people for a/ personal empowerment and b/ helping others? Understanding the value of different components was a primary focus of data collection. To build the list of components for the survey, evaluators asked stakeholders at the inception phase what they remembered doing. In total, 12 training courses and six types of activities were identified, though not all were available across all cycles. Youth members were then asked, through the online survey, to rate their opinion of these trainings and activities20, ranging from ‘not useful’ (one point) to ‘very useful/highlight’ (three points). Figure 14 shows the results of this question, with a maximum score of three, based on 49 responses21. The results were then tested through telephone interviews with a sample of respondents, and presented for validation at a final workshop involving government, TLYP, APFTL and NGO partners. A full report on the survey results is available as Annex 15.

The most valued elements, public debate, public speaking and the plenary sessions, had similarities in that they built presentation confidence and skills; they allowed individuals to be noticed, which interviews confirmed was important in this culture. Leadership and civic education training were also highly valued, and interviews confirmed these courses were not usually available

20 The survey also allowed for ‘don’t remember / did not take part’ and ‘other’ 21 While 63 TLYP members took part in the online survey, not all completed all questions; see limitations section and Annex 13 for more detail on working with the survey data. 48

elsewhere. Shorter courses tended to receive lower ratings; for instance, journalism and anti- corruption were half day modules, while leadership and life skills were two days.

Figure 13: Average rating, TLYP activities and trainings (minimum 1, maximum 3; note, no scores fell under 2)

A follow-up question asked how each identified training was used: personally, or to help others. Helping others encompassed training young people, taking part in community work, and advocating for youth issues. The results are shown in Figure 1522. Here too, the data indicates that the civic education and leadership were valued highly, and proportionally were the most likely to be used to help others. While public speaking and debate skills were popular and widely used, TLYP members were more likely to use these for personal achievements.

Figure 14: Survey self-reporting on how training was used

The survey indicated journalism, anti-corruption/governance, social analysis and gender / child rights courses were substantially under-utilised (Figure 16). It should be noted, however,

22 Again, see Annex 13 for more detail. 49 that the figures have been averaged, and a small proportion nominated journalism, gender equality and human/child rights as highlights.

Homes, schools and community environments were often raised in interviews as intrinsic components for empowerment and confidence outcomes. Survey respondents were asked to rate the support they received from parents, schools, peers and the TLYP Secretariat (Figure 17). They gave consistently high scores, indicating they had received strong levels of support and flexibility to meet their TLYP Figure 15: Proportion of survey respondents NOT using their training at all responsibilities. Though still high, peer support and school support were a little lower than other categories. Interviews explored this trend and revealed that some participants had found it difficult to explain at school what they were doing or to gain an audience for their intended training or debriefing. Some also reported that students were jealous of the opportunities they received, though this was not a consistent finding. More support and information from Secretariat or APFTL to schools with current TLYP members – targeting students as well as teachers and principals - may help to reduce these problems.

Figure 16: Average score for support from home, school, community; minimum 1, maximum 3; no score fell under 2.

4.4 Impact . The positive and negative changes produced by the program, directly or indirectly, intended or unintended.

Questions: Summary of Findings 4.4.1 What has happened as a result of the Multiple, triangulated sources confirmed TLYP programme, and which activities have outcomes for personal empowerment of TLYP contributed most to this impact? members, directly related to their association 4.4.2 What unexpected, including negative, with the institution. Members are significantly effects are evident for youth parliamentarians? more likely to attend university despite initial 4.4.3 What progress can be seen as a result socio-economic disparities, and living in Dili of TLYP community engagement towards while attending university also supports the five strategic outcomes of the 2016 ongoing participation in youth activities National Youth Policy? through APFTL’s central office. A Dili-based 4.4.4 Is there a correlation between TLYP location was also advantageous for current community engagement and social members, who were more likely than their cohesion/peacebuilding outcomes? If so, rurally based counterparts to be attending how can this result be replicated and regular events and employing their public scaled? speaking skills.

A causal connection between TLYP participation and community-based outcomes was partially evident. Current and former members reported they felt responsible to return on the trust and time invested in them and were confident in finding ways to do so. However, indirect impact for other young people throughout Timor-Leste could be proven in only a small number of cases and is largely presumed / projected rather than measured. Interviews found a preference to work in groups rather than individually when conducting community training or other events, an important observation for shaping future guidelines and support for TLYP/APFTL outreach.

4.4.1 What has happened as a result of the TLYP program, and which activities have contributed most to this impact? Strong, triangulated evidence existed to show that TLYP directly empowered its members as intended (see Results Matrix (Annex 18) and Table 4: What changes are evident?). These results and changes sit within the second and third tier in the evaluation’s retrospective Theory of Change (Annex 18), with general consensus from implementation stakeholders that the final two tiers require more time and more deliberate planning and measurement.

Figure 18 illustrates self-reported survey results on multiple ways that TLYP members have used their skills to benefit young people in Timor-Leste. The extent of influencing and direct training reported by survey respondents reflects leadership, a core goal of the program, while interviews confirm a causal link between these activities and the inputs of TLYP.

Figure 17: Survey self- reporting, % respondents delivering community activities.

It should be reiterated within this conclusion that the survey did not include all TLYP members. In interviews, TLYP members observed some of their colleagues were more civically active than others, and on average estimated the proportion of TLYP members who took up civic responsibilities at two-thirds. Two survey respondents made recommendations for stricter minimum requirements for community work for TLYP members within their mandate. However, considering that TLYP members are children, and many are still in school at the end of their three- year service, pressure of enforced obligation to targets may have detrimental effects on some participants.

Another caveat to these results is limited evidence from other young people or local authorities that TLYP community actions were memorable or impactful. It proved more challenging than expected to locate a school where students identified TLYP members as a source of training or advice.

APFTL training was validated as taking place, but current students could not recall information if it had been delivered in previous years.

The ideal scenario for focus group discussion was found in just one school, where students had sufficient awareness of APFTL training, TLYP and training from other providers. Here, the evaluation team was able to confirm not only that training was available and remembered, but that it had been relevant, timely and used (see boxed text; also Annex 16).

Student focus group, Eskola Sekundária Públika Nino Koni Santana Image © UNICEF / Chalk Student council perspectives, Eskola Sekundária Públika Nino Koni Santana What extra-curricular  Public debating, public speaking (Ministry of Education competitions) activities do you recall,  Sexual and reproductive health; HIV/AIDS prevention (3 providers) and who provided them?  Gender equality and gender-based violence (2 providers)  Youth peacebuilding (2 providers)  Youth rights and the law (1 provider)  Disability inclusion (1 provider)  Environment, agriculture, tree planting (1 provider)  First aid (1 provider)  Online safety (1 provider: APFTL)  Democracy and civic participation (1 provider: APFTL) Out of these, what activities have you valued the most? Girls: Boys:  Public speaking: “It helped me gain confidence and  Debate, public speaking: “These have been it took me outside the school to meet other people.” opportunities for me to gain experience and prepare “We got a lot from going to the public debate for new responsibilities.” “Making speeches in competition in Dili.” public is very important in our culture, it helps us  Health and adolescent health / HIV: “Being able to stand out.” look after yourself is the basis for success in  Health: “Because if you don’t have that you can’t do everything else.” anything else.”  Democracy and civic participation: “I was a first-  Democracy and civic participation: “The chance time voter and the information from APFTL was the to learn about youth rights in law, what kind of only information I received that was targeted to my government policies we can turn to, is very useful age group and information needs.” information for us to know.”

The dominant change discussed in TLYP member interviews has been at personal level, and the obvious advantages of personal development. An important consideration throughout analysis has been the counterfactual: without TLYP, would these children have developed so rapidly into confident and intelligent leaders? The selection process favours children who are already confident, engaged in youth initiatives, self-starting and academically strong. While this is a head start for leadership qualities, data offered good evidence that TLYP enhanced skills and empowerment further, for instance:

 In the survey, the value accorded to specific training on public speaking and debating skills, training which was not available elsewhere  Evidence of the TLYP mandate in action, to represent and help others, especially in alumni/APFTL  Strengths in ability to organise, budget and implement public events, alone or in a group with shared responsibilities  University attendance; well above national average (82%, compared to national average of 9%23) and with a number of interviewees saying their school or family had not expected them to continue their studies  Deep understanding of Timor-Leste as a nation, and the role of Timorese youth, shown in interviews and attributed to networked exposure and learning

4.4.2 What unexpected, including negative, effects are evident for youth parliamentarians? A significant positive outcome of TLYP has been the formation of APFTL (see in particular Section 4.6.5, but also referred to throughout this report). Interviews and documents confirmed that APFTL members continued to meet responsibilities for community and advocacy work; as with TLYP membership, while specific skills training was important to this outcome (and these skills continue to be used), so were associated personal outcomes of leadership ability and confidence. Another key benefit of APFTL formation has been strong networks and friendships forged through relatively few, but greatly treasured, meetings. TLYP and APFTL both preferred to organise in groups if possible, meaning larger events and sharing of workload and responsibilities. Staying in touch, planning and participating has been personally rewarding. It was easier for young people online, who used WhatsApp/Facebook for communication. However, strong offline communication was also noted and remains powerful for planning at municipal level.

Personal opportunities have been greatly valued by TLYP members, many of whom said they would not otherwise have travelled, met students from different backgrounds or learnt leadership

23 Timor-Leste Population and Housing Census Analytical Report on Education, UNFPA 2017, p.23 https://timor- leste.unfpa.org/sites/default/files/pub-pdf/Education%20Monograph%20250418_0.pdf, 54 and life skills (see Martinho’s story, Annex 16). This has equated to greater opportunities for university education, which had not previously been a consideration for some interviewees. Just 7% of survey respondents were not in school, work or university (Figure 19). Of these, most had finished university and were looking for work. Telephone interview follow-up with these respondents found they were confident they would find a good job soon.

A geographic bias has become evident and may be challenging to resolve. Despite representation from every administrative post in Timor-Leste, practicalities of travel have created different workloads for those in and outside the capital Dili. Dili members (12 of 130) have been kept busy by UN agencies and NGOs as a first contact for youth voice and representation. Most current members outside Dili, however, reported they had only taken part in TLYP activities and were not usually called to contribute to formal community processes locally; they were far less likely to be Figure 18: Survey responses, % at university / school / other representing their peers in events organised by Government, UN, NGO, and other development partners. Travel costs, safety and inability to take time off from school all contributed to this trend. Within this observation, it should be noted that APFTL ensure current members are invited to community events and training when they travel, treating them as part of their municipal network.

Initial consultation with stakeholders to refine questions identified two concerns connected to mobilising children under 18: firstly, political coercion, secondly, protection from violence and sexual abuse. Some examples of political coercion were found, but with effective personal protection mitigation also noted. Regarding protection from violence, though no incidents were reported, safeguarding needs strengthening. These concerns evolved into specific questions under Human Rights and Equity and are discussed in greater detail in Section 4.5.4.

4.4.3 What progress can be seen as a result of TLYP community engagement towards the five strategic outcomes of the 2016 National Youth Policy? (Is there also a correlation between TLYP community engagement and social cohesion/ peacebuilding outcomes?)24

The evaluation’s ‘positive enquiry’ approach to understanding TLYP community engagement was challenged when stakeholders did not identify any examples of current TLYP members driving

24 These were originally two questions in the inception report. However, as data was lacking to report against either question, it seemed sensible to merge them into one. 55 community activities, as already noted in the Limitations section to this report. National-level progress towards the NYP was also not systematically mapped. As a result, it is not yet possible to answer questions about TLYP contribution to NYP outcomes.

APFTL member provided some local activity examples, with delivery of grant-based activities in peacebuilding, democratic process and youth centre vocation partnership (emphasising promotion of agricultural careers). It was possible to estimate reach and relevance of these activities, but unfortunately, again, not impact.

The gap in local impact data has been noted as a limitation to this study, and a key recommendation is to learn cost-effective ways to track and report TLYP and APFTL activities and their outcomes.

At national level, 2017 TLYP plenary session recommendations were arranged so as to align with and support achievement of all five NYP outcomes. These were based on priorities identified in TLYP consultation at municipal level. At this stage, as the take-up of these recommendations is not evident, their contribution to NYP outcomes cannot be assessed.

The final part of this question, specific to social cohesion and peacebuilding outcomes, was also not possible to answer, for the same reasons, that TLYP community engagement in this area was not recorded, and inputs could not be examined. As this was a question of great interest to NGO stakeholders seeking a replicable approach in the context of Timor-Leste, another attempt to measure this should be considered once monitoring of community work strengthens.

Most value (boys) • Leadership, debate, public speaking, helps us prepare • Youth perspectives on democracy • Health, HIV and how to protect ourselves • Learning about the law and our rights within it

4.5 Human rights and equity

The program’s performance against cross-cutting goals of UNICEF’s country program to build landscapes of opportunity for all, regardless of gender, disability, ethnicity or cultural background.

Summary of findings: Questions: Inclusion principles of geographic representation and 4.5.1 To what extent has TLYP reflected gender balance were formally mandated and closely its original human rights and equality followed. Two other principles of inclusion were not principles: gender, socio-economic followed: additional non-geographic membership of background, geographical disparities two young people with disability; and representation of (urban/rural), child and youth children out of school. Interviews with stakeholders participation, and participation of children implied that while the concepts were sound and and young people with disabilities? supported, the practicalities were overly challenging 4.5.2 What UNICEF strategies/ given current networks and resources. approaches have been most efficient in influencing improvements to child and Upholding rights of the most vulnerable has not been adolescents’ participation rights? a strength to date. Some interviewees reported they 4.5.3 How did the youth parliament had worked in this way, but did not easily define ‘most programme benefit other youth who are vulnerable’. Others, who were more aware of rights- not TLYP members, in terms of a/ local based challenges in their context, said they had not policies and behaviours and b/ personal done enough in this regard. capacity and resilience? 4.5.4 How successful was the Documentation and stakeholder reflection confirm programme in supporting the most UNICEF as the primary contributor to inclusion of child vulnerable children and adolescents in and adolescent rights and participation principles in communities? the overall strategy for TLYP. Their ongoing training 4.5.5 To what extent, and how, has has helped to maintain this inclusion, though it could independence and protection of youth be further strengthened through a curriculum review. members been assured (do no harm safeguards)? Mechanisms for safeguarding children from harm are 4.5.6 Are young people’s voices and lacking in the TLYP model; reporting and response systemsopinions need bein to gbe heard formalised and respected as a priority. more While there have been no reports of violence or sexualthan abuse, previously? children To are what taking extent part in is the this program in ways that could place them at risk. A small numbelinkedr ofto childrenTLYP? have been approached by political factions while serving with TLYP.

4.5.1 To what extent has TLYP reflected its original human rights and equality principles: gender, socio-economic background, geographical disparities (urban/rural), child and youth participation, and participation of children and young people with disabilities?

TLYP remains aligned with a human rights-based approach, which can be seen in the design’s inclusion principles, with a core emphasis on child participation and voice.

Throughout the cycle, gender balance and equal representation of administrative posts was assured through the selection quotas. Figure 20 shows the age breakdown of current TLYP members; confirming that the process has also been inclusive of different age groups. Members under 15 are slightly more likely to be female, and over 15 to be male.

Interviews confirmed that TLYP was a friendly, supportive and inclusive space for its members. Younger participants were treated equally, and both girls and boys said they saw no gender divide. Observation of current and former TLYP interactions supported this, with good camaraderie including between girls and boys; however, it was also noted that staff and authorities for TLYP were all male, and that AFPTL members taking part in gender advocacy attributed their abilities at least in part to association with specialist gender organisations (UN Women, Alola Foundation).

Figure 19: School level breakdown, current TLYP members It was originally intended that two children with disability would take part in addition to the 130 geographically selected representatives. According to implementing partners, it proved difficult to identify and work with disability organisations to find potential candidates. By the third cycle, this principle had been modified so that children with disability were welcome, but not specifically targeted, to apply. Criteria for selection, however, were geared towards active, confident and academically strong candidates, which children already facing challenges of disability inclusion in schools could find daunting. The principle of disability quotas should be reinstated. Interviews with current TLYP members as well as comments in the survey indicated that an absence of interaction with young people with disability is negatively affecting their awareness and action on inclusion.

No out-of-school children became members, which indicates that promotion of the opportunity was not sufficiently inclusive in reaching or raising interest of these children. While the evaluation team saw posters advertising TLYP elections in community centres, stakeholders did not remember any expressions of interest from out-of-school children, and did not work to locate and mentor these children to compete alongside their school-based counterparts.

Comments in government and APFTL interviews also showed an implicit exclusion attitude to allowing mothers under 18 to apply, or to remain in TLYP if they fell pregnant. This is despite other members of APFTL prioritising inclusion rights for pregnant adolescents. The attitude indicates ingrained gender beliefs and structures in Timor-Leste culture. Though acknowledging girls and young women are disadvantaged, the National Youth Policy does not include an outcome for gender equality. If the expectation is that shifts in gender empowerment are part of all development strategies, nonetheless these shifts need investment and planning. Institutions such as TLYP are needed to drive gender awareness and challenge dominant attitudes, and more should be done to enhance TLYP’s internal gender policies.

4.5.2 What UNICEF strategies/approaches have been most efficient in influencing improvements to child and adolescents’ participation rights? At the time of the first cycle, UNICEF Timor-Leste maintained a specific program for adolescents and youth, which enhanced its technical capacity to support training needs. UNICEF supported the design of the training program for TLYP. A key contribution was the Life Skills Basic Education (LSBE) course, based on international good practice for adolescent transition, and containing strong, effective guidance on making healthy, positive choices. Survey results and interviews show this course remains popular though not as highly valued as leadership and civic education.

UNICEF’s support to training ensured that child rights, gender equality and human rights were represented. Training responsibility was outsourced to appropriately qualified training providers where necessary. This rights focus has been an important inclusion to the overall curriculum; however, as previously discussed, it has not resonated to the same degree as other areas of training. Young women who have worked with other gender organisations since leaving TLYP now describe the human rights/gender rights courses as ‘quite basic’.

4.5.3 How did the youth parliament program benefit other youth who are not TLYP members, in terms of a/ local policies and behaviours and b/ personal capacity and resilience? Over 90% of survey respondents believed they had passed on knowledge in some way that made a difference to the behaviour of others around them. Figure 21 shows how certain peer attributes 59 and interests were scored before and after TLYP influence, noting a solid increase across all measures.

Other areas of this report have already discussed the challenge of validating this self-reporting, and this result is presented with low confidence due to lack of triangulation. Depth of engagement with other young people, especially outside Dili, was not obvious. Interviews with chefe sucos gave mixed answers on whether TLYP members were connected to local policy processes. Figure 20: Survey self-reporting: changes in peer attributes and interest Resolving this inconsistency through guidelines, as well as direct mentoring and encouragement, will support more intuitive local-level interactions on youth policy.

4.5.4 How successful was the program in supporting the most vulnerable children and adolescents in communities? To fulfil a role in support of the most vulnerable, the TLYP program would require a deeper emphasis on rights and root causes of marginalisation in its skills building than it currently has. Interviews with current and former TLYP members showed discrepant degrees of understanding of the vulnerabilities and inequities caused by gender, poverty, disability or family background (for instance, illegitimacy). Interviewees who were closely involved with these issues cited other sources as well as TLYP for their information and inspiration.

For instance, though school dropout due to early pregnancy or early marriage was a recurring issue across the three cycles, none of the TLYP plenary recommendations reviewed identified necessary policy and paradigm shifts required to protect girls and boys from this trend. The work done on supporting teenage mothers to remain at school has been an exception to this, and an excellent example of supporting most vulnerable adolescents, but the agenda has been triggered by involvement with other organisations rather than TLYP.

4.5.5 To what extent, and how, has independence and protection of youth members been assured (do no harm safeguards)? It is surprising that safeguard mechanisms are not in place for protection of children from violence and sexual abuse when taking part in TLYP. TLYP members are aged 12 to 17, entitled to the special protection afforded through the CRC, and also a highly vulnerable age group for sexual abuse. TLYP process takes children from their homes, and therefore has an obligation to provide protection and supervision at all times. No stakeholders (adult or child) in the interview sample were clear on how this was being done. Though the evaluation also found no evidence that TLYP members had faced protection issues, a vital recommendation is for a system that protects children from harm. This requires clear assignment and documentation of roles and systems, including female as well as male supervisors when TLYP members are travelling.

Political independence is a principle of TLYP towards community-focused, non-partisan leadership, and well understood by current and former members. Candidates must acknowledge and address this criterion of independence in their application. Even so, political coercion is a challenge faced by some TLYP members, noted in the survey as well as in interviews. Four survey respondents (around 7% of total respondents) reported they had been approached to join political parties while under-age in the TLYP, and 12 (20% of respondents) said they had been approached since. When asked what they did about it, most did not rule out political engagement in the future but wanted to finish studies first. When discussed with interview participants, most did not see a problem with over-18 current or former TLYP members being approached. Some APFTL members were public on their political allegiance, and survey results plus interviews show that interest in politics as a career often increases as a result of TLYP membership.

Nobody reported the incident of political coercion at the time of it happening. There is no clear pathway to report such incidents in any case. Therefore, though the evidence available demonstrates personal resilience to political influence among TLYP members has been sufficient to date, the need for safeguarding against this child protection issue merits a formal protection and response system.

4.5.6 Are young people’s voices and opinions being heard and respected more than previously? To what extent is this linked to TLYP? As part of a broader movement for youth empowerment and social cohesion in Timor-Leste, TLYP’s three cycles have resulted in increased voice and action from individual, highly capable and motivated young adults; this was not only observed directly by the evaluator when working with TLYP and APFTL, but also reported consistently by partners who have worked with APFTL since 2015.

The TLYP annual plenary connects the voice and action of these young people to national policy process. It was not yet possible to identify the next step in the Theory of Change, of young people influencing national policy. When this occurs, it will be an exciting moment for all involved in TLYP, representing the likelihood of much broader reach and benefits for life skills, social and economic participation of young people across Timor-Leste than has been noted to date.

At local and municipal levels, the recognition and inclusion of youth as social and economic agents is a long-term goal. While government and NGO partners coordinate on this to a degree, interviews unearthed different viewpoints on how youth organisations worked together, and their effectiveness in doing so. It was generally agreed that National Youth Council, government youth centres and TLYP / APFTL had strong synergies of purpose; however, only TLYP had sustainable program funding and even this was not intended to be used for local, contextual activities.

A renewed approach to collaborate on effective youth participation is needed, especially considering the potential of the National Youth Policy to deliver vocational, peacebuilding and equity outcomes under a coordinated action plan. Stakeholders see the potential of establishing platforms for regular dialogue with MPs. TLYP on its own cannot instigate this level of planning, and so this observation is relevant to the youth sector more broadly

4.6 Sustainability

The likelihood that the program benefits will continue beyond the end of UNICEF support Questions:

6.1 To what extent did the benefits Summary of Findings continue after donor funding ceased? TLYP basis in law (Government Resolution No.23/2009) 6.2 What were the major factors which has generally assured sustainability of the TLYP as an influenced the achievement or non- institution under government, weathering changes of achievement of sustainability of the government leadership and the phase-out of UNICEF’s programme or project? dedicated project support over time. Sustainability of 6.3 What is the trend of state budget quality outcomes for direct participants is now evident, allocated for TLYP programme in the though with some dependencies on human resource past, and expected trend in the future? commitments long-term. Interviews found strong 6.4 Has the programme been support and internal advocacy from SSYS to resource successful in leveraging governments’ TLYP, in line with accountability to the 2009 government political will and financial resources for resolution, aiming for an annual budget of US$200,000. child and adolescent protection?

6.5 What is the significance of the Outcomes for child rights and protection, a UNICEF core Alumni Association to the mandate, are not fully guaranteed. Analysis of TLYP sustainability of Youth Parliament? policy platforms reveals an emphasis on adult youth 6.6 What is the best role for others policy. Members demonstrate less knowledge and (NGO partners, local institutions) to interest in child and adolescent issues than might be play in Youth Parliament and Alumni expected. Connecting this observation to curriculum activities? content, TLYP members also receive less exposure and training on children’s issues so, with the exception of adolescent sexual and reproductive health (ASHR), they do not usually discuss or make recommendations for children.

While government (SSYS, and TLYP secretariat) continues as the driver of TLYP process, increasingly APFTL can be seen as a driver of sustainable TLYP outcomes, expanding community outreach as a channel and network for planning and supporting initiatives that benefit youth. APFTL is a sustainable organisation and will grow each time a TLYP cycle ends; in fact, considering the reputation it has built, it may not grow as fast as is needed to meet expectations of its services.

4.6.1 To what extent did benefits continue after donor funding ceased? TLYP’s continued operation, along with positive feedback from current TLYP members, demonstrate that the structure is sustainable; benefits to members continue. Since UNICEF donor funding ceased in 2015, TLYP’s third cycle (2016-18) has been managed, funded and monitored by government, through the agreed mechanism of the TLYP Secretariat under SSYS. As of October 2018, all stakeholders interviewed expected that there would be a fourth cycle, that it would reflect original principles and inclusive design, and that it would be supported appropriately through government financial and human resources.

Most recently in 2018, some changes in government structure including reduced positions for the TLYP Secretariat have affected its smooth implementation. Preparation for 2019 TLYP member recruitment is somewhat behind schedule and there are gaps in capacity to provide mentoring or coordination compared to 12 months beforehand.

Several interview respondents raised this shortfall as an issue but were not sure what would be done about it. Some thought that APFTL would be expected to take up more implementing responsibilities, and others believed the positions would be restored under a different reporting structure. Though the solution has not yet been decided or announced, discussions throughout the evaluation process found good transparency and will to resolve human resource challenges. This indicates likelihood that the right solution will be identified in time for the significant intensity of the election phase in 2019.

4.6.2 What were the major factors which influenced sustainability of the program or project? UNICEF’s contribution to sustainability began from the planning phase, prior to 2009, when parliamentary and presidential endorsement was key to launching the initiative. Sustainability has also been supported by the Government of Timor-Leste’s youth focus over this time: for instance, acknowledgement of youth needs for nation-building in most policy documents and the Constitution; the specific National Youth Policy of 2016; the emphasis on nationwide geographic coverage of the National Youth Council (CNJTL) and TLYP membership.

Current members remain positive about their experiences and opportunities through TLYP, but also report with consistency that their mandate period contained fewer activities than previously. Some reputational risk is inherent in this, particularly as results are not measured or celebrated, and current in-school / media promotion of TLYP as an opportunity for children was barely visible in field trips. The upcoming election cycle will test whether TLYP is still recognised and demanded by demand high-calibre applicants, and it is advisable to start as early as possible on scaling up pro-active, youth-friendly messaging on TLYP advantages.

4.6.3 What is the trend of state budget allocated for TLYP program in the past, and expected trend in the future? The 2016 National Youth Policy calls on several different ministries to invest and support in youth. SSYS currently has sole responsibility for TLYP, though most of its funding is tied to infrastructure budget for sports facilities. Social accountability and civic strengthening for the ‘youth bulge’ does not have a specific budget line under any ministry, and so it is challenging to monitor cross-ministry allocation to child and youth participation and voice.

Resourcing volatility for SSYS as a whole is evident in recent past (see 4.2.1: Efficiency). Government stakeholders are working to secure a more definite annual budget in the general range of US$200,000, with good likelihood of success in the short to medium term. Longer term, some cost-sharing between ministries, for instance MoE or MSSI, particularly on monitoring children’s and youth contribution to the National Youth Policy, could enhance a sense of shared ownership and responsibility for TLYP. Monitoring is currently assigned to CNJTL but, as previously discussed, without resources or clear guidance to take up the responsibility.

4.6.4 Has the program been successful in leveraging governments’ political will and financial resources for child and adolescent protection? No connection between TLYP and improved child protection was identified in evaluation data, where the language and dialogue centres around youth needs. Child protection needs (from violence in homes, schools, communities, from sexual abuse, from hazardous work, from online abuse) have been overshadowed by more adult themes of employability, personal development and productivity. TLYP recommendations largely overlook child protection and gender, including gender-based violence, and do not address social welfare policies such as shelters and justice for violence survivors. Early pregnancy has been a core issue for young women’s advocacy work after TLYP, but is more commonly referred to as ‘early marriage’ in TLYP documentation, focusing on abstinence rather than gender-based vulnerabilities and protection needs.

Looking at the focus of study visits and other exposure activities in the first two cycles, the evaluator noted they did not connect to child protection or gender rights issues. TLYP members therefore are not receiving first-hand understanding of child protection gaps in Timor-Leste. Training on rights is provided but is not comprehensive: a day on human rights including child rights, and a day on gender equality. The information was among the least used to help others (see Section 4.3: Effectiveness). More deliberate connection to social welfare services, disability organisations and international movements for gender and child rights is likely to trigger increased interest and action from TLYP members on child protection issues, now and as young adults in the future.

4.6.5 What is the significance of the Potential for expansion of APFTL through Alumni Association to the sustainability associate membership Other young people are interested in taking part of Youth Parliament? in APFTL, even if they are not TLYP alumni. APFTL has been highly significant to sustainability. This quote from a boy in his final year at school Without it, the full multiplier effect of TLYP would be in Dili summarises the problem faced in doing less likely. The organisation’s professionalism and so: integrity have been noted by clients, resulting in a “I think it is not fair. I came close to getting sustainable annual program with a core into Youth Parliament but I failed, and I responsibility for TLYP training and advice. In its don’t know why. After that nothing third year, it is already recognised as the next step happened for me. I admire the APFTL and for TLYP members and interest in its continuation is I would really like to join them when I leave assured. school and train other young people like they do. But because I wasn’t in Youth APFTL formation, based on the self-identified desire Parliament I can never be in Alumni.” of graduating parliamentarians to continue to TLYP members confirmed this was a common connect and serve, can be considered an outcome desire among their friends, and that if training of the empowering elements of TLYP. Its inclusion and mentoring opportunities expanded, it would is a key lesson for refinement and replication of be simple to find young people to take up similar youth parliament models in other contexts. these roles. At the same time, it should be noted that the spontaneity of APFTL formation may have dependencies on personalities and capacities within the group at the time, and may not be as successful in other situations.

Interviews and direct reflection with APFTL brought up three considerations to maximise future social impact of the APFTL mandate:  Given its structure and size, APFTL cannot effectively reach every school in Timor-Leste. It needs a more definite strategy for why and how often it will target schools. This could include associate membership under a train-the-trainer approach (see boxed text)  APFTL currently bases its annual planning on grant availability. More consistent youth outreach could be possible under a proactive, multi-year strategy seeking donors for school or community work – potentially engaging government youth centres as partners.  APFTL does not have resources for internal capacity development, to learn new facilitation skills or to refresh existing knowledge. This will eventually limit member ability to provide quality up-to-date information and training; ongoing personal development for members is highly recommended.

4.6.6 What is the best role for others (NGO partners, local institutions) to play in Youth Parliament and Alumni activities? While sustainability of TLYP process was found to be high, long-term partners including UNICEF have a strategic interest to stay engaged and support ongoing evolution of the model. Project- based technical support respects the government’s ownership of the program and can fill gaps identified in this evaluation. Several ideas for doing so were tested for feasibility at the validation workshop. The following were agreed as relevant and practical; the process stopped short of assigning roles and action planning, and this will be a key next step.  Media and social media support: Two gaps were noted with regard to media and communications for TLYP: firstly, a gap in students’ and teachers’ knowledge of TLYP which indicates a need for more consistent messaging to promote TLYP purpose, process and value for candidates, their schools and peers; secondly, the survey found media and journalism skills were not as well remembered or used as other skills. Quality media engagement is crucial to effective social activism. A media project for TLYP and other youth representatives could resolve messaging gaps and allow participants to manage their own public relations and media directly.  Standardised training: Training materials have not been substantially revised since 2010 and are not packaged, branded or easily available for retraining purposes. The data from this evaluation on levels of retention and use of different skills can be used as a starting point for refreshing and updating a full TLYP curriculum.  Mentoring and co-hosting community events (especially supporting younger members): APFTL and other youth organisations could connect with greater formality to mentorship of current TLYP members under a community workplan or similar, to help achieve goals for community and peer influence. Interviews with current and former members showed a preference for conducting community work in groups, rather than as individuals. Partners could help TLYP members, especially those younger or less confident, to identify opportunities for groupwork in municipalities.  Specialist training and exposure to rights/environment issues: Interviews with APFTL members and school students noted gender and environment as areas where specialist organisations have been able to offer more to TLYP members and peers than the existing schedule allows. There may be more areas where individual interests can be expanded through government joint planning with technical partners: for instance, UNICEF on child rights and protection.  Monitoring and reporting results; learning: Interviews with TLYP members, government and APFTL saw a common thread emerge on the need for effective monitoring to demonstrate value of TLYP and APFTL, to identify influential practices, and to maximise learning and expanded impact. A project to incorporate monitoring practices into TLYP training, CNJTL partnerships and APFTL initiatives, supervised by SSYS TLYP Secretariat, could create sustainable systems for monitoring and reporting long-term. 67

6. Conclusions

6.1 Conclusions by OECD-DAC criteria

RELEVANCE The evaluation concludes that the model and objectives of TLYP remain highly relevant to the agenda for youth in Timor-Leste and should HIGHLY continue. Policy analysis confirms TLYP program objectives remain highly RELE- relevant to the Timor-Leste youth context, with alignment to national youth VANT policy, to UNICEF child, adolescent and youth priorities and the 2030 SDG agenda. Original principles of gender and geographical inclusion and youth- driven agenda remained as guidance for decisions taken.

EFFICIENCY The evaluation concludes good efficiency and performance in managing the process of TLYP is evident, but with opportunities for greater MAINLY innovation and quality programming under a more flexible budget. EFFIC- IENT Government records and stakeholder feedback confirm the program was efficiently administered, though activities did not always run to schedule. TLYP was found not to duplicate other youth programs, but could connect with them more effectively. Return on investment calculations show strong potential for large-scale indirect (outreach) influence with other young people.

EFFECTIVENESS The evaluation concludes effective practices throughout the TLYP cycle are contributing to achievement of process objectives, though objectives MAINLY for social change are unreached or currently unmeasurable. Targets for EFFEC- TIVE recruitment, training and parliamentary process have been achieved consistently over three cycles. TLYP members act as peer role models as envisaged. Stakeholders (government, APFTL) confirm the effectiveness of UNICEF’s support to TLYP. Records show that TLYP policy recommendations on adolescent and youth policy have been tabled and accepted by government ministries each year; however, ministry actions as a result of TLYP advocacy are not tracked or reported.,

IMPACT The evaluation concludes partial success is evident against impact intention, with conclusions limited by lack of tracking for impact. All

SOME sources of evidence suggest TLYP has greatly enhanced opportunities for IMPACT members, through training as well as positive and supportive interactions with Secretariat, Alumni and peers. The self-forming Alumni Association has been an unexpected and significant positive outcome of TLYP implementation.

Negative effects are not apparent, though safeguards against political coercion and child protection could be strengthened.

HUMAN RIGHTS The evaluation concludes TLYP strongly reflects most of its envisaged AND EQUITY human rights and inclusion principles, and has potential to guide human rights outcomes for children and youth in Timor-Leste. Examples were GOOD found of former TLYP members acting directly to reduce adolescent and youth RIGHTS vulnerabilities. Through the plenary and parliamentary policy submissions, BASIS young people’s voices and opinions are increasingly heard and respected at national level. Outcomes are less evident in local-level consultations. Analysis of membership demographics confirms realisation of TLYP’s original inclusion principles regarding gender, age, background and geographical disparity. However, it has struggled with the challenge of disability inclusion. UNICEF’s elements of child protection, gender and human rights in TLYP training remain core, but interviews and survey results show these are not priority skills from TLYP member perspectives.

SUSTAINABILITY The evaluation concludes that sustainability has already been tested and is assured; TLYP is an independent government-funded institution for HIGHLY children’s civic participation. The program continues three years after formal SUSTAIN- funding from UNICEF concluded. While State budget fluctuates annually, ABLE government stakeholders advocate consistently and successfully for the necessary budget to TLYP. The Alumni organisation APFTL is crucial to sustainability of outcomes for young people generally through ongoing self- funded outreach activities in schools and communities.

6.2 General conclusions

The evaluation concludes that TLYP has been effective in its process for engaging and empowering children of Timor-Leste through its local-to-national civic participation agenda. While TLYP is not the only mechanism for increasing leadership capacity and community responsibility in Timor-Leste, it is unique in connecting to parliamentary process and policy-making. Careful selection and support of members has resulted in retention over the mandate period and beyond, despite competing time demands on these young people. Benefits for young people taking part are clear in interviews, survey results and the evaluation team’s direct observation of capacities and interests. Interview respondents consistently stated the primary goal of TLYP was to ‘prepare young people to be future leaders’ and this goal is being met. Consultation with young people on policy is formalised through the annual plenaries. For youth to take up their

69 role as drivers of change, establishment of regular dialogue between TLYP members and MPs on children’s issues could also scale policy and impact for children in Timor-Leste in line with the National Youth Policy priorities.

To achieve other goals for increased civic responsibility and participation of all young people in Timor-Leste, it is vital to ensure the ‘multiplier effect’ is also working. Future support to TLYP should emphasise this multiplier objective, leveraging policies including the National Youth Policy and community development funds, expanding capacity and systems for effective results monitoring, and recognising individual and networked youth initiatives to which TLYP members have contributed. Observations throughout this report underscore the value of APFTL in this regard. Considering the geographic reach of the alumni network, which will expand once more as current members graduate in 2018-19, there is untapped potential for better coordination (with youth centres, CNJTL and other youth organisations) and quality outreach to improve abilities and opportunities for a much broader representation of Timor-Leste’s youth. While better planning of community youth work is not the sole responsibility of TLYP Secretariat, they can play a strong role; it lies within the remit of SSYS to deliver greater youth benefits under closer working and planning relationships between partners.

Involving other children in similar opportunities to learn and act would increase relevance of this member-only organisation for a broader range of young people. The nature of TLYP and others like it (see Annex 13) limits the number of children who can take part directly. Shortlisted but unsuccessful candidates, as well as young people participating in TLYP and Alumni training, have so far been an untapped resource. A second tier of active participation engaging these children at more local levels (municipal, sub-municipal) would enhance the reach of APFTL objectives as well as clarifying and popularising the purpose and benefits of the TLYP.

7. Lessons Learned

A key lesson from the methodology has been that the participatory approach involving children and youth was highly valued on all sides. This evaluation provided a great opportunity to apply principles from the UNICEF Strategic Plan 2018-2021 for working with adolescents, recognising the critical second decade in life. School and university students joined and led conversations on research planning and application, set questions in line with utilisation principles and their rights and interests as stakeholders, collaborated on site visits and other data collection, and contributed to analysis. TLYP members (all under 18) chaired the findings workshop together with SSYS, as primary representatives to endorse findings and recommendations. This level and success of beneficiary participation was a strength of UNICEF’s methodology, and children proved themselves to be indispensable contributors.

Other several lessons from this evaluation are likely to be of broader interest to those working within the field of adolescent development and empowerment, namely:

 Training is valued most highly when there are opportunities to use the skills directly and immediately. While skills such as public speaking and debate were put to immediate use, more theoretical knowledge such as gender and child rights, proposal writing and journalism was remembered, rated and used less. This has limited some results that are key to UNICEF’s global strategy, because indirect outcomes for children and adolescents, especially the most vulnerable, have not been prioritised by TLYP members. At the same time, some members connected to further training in line with their interests – for instance, more intensive gender rights training through other agencies – and developed personal strengths and commitments in these areas. Implementers of similar models should therefore consider partnerships and pathways beyond training to put skills into action.  Up to date, consistent information is an ongoing need in such programs, to ensure schools, friends and parents are supportive of children’s commitment and choices. Support and full understanding at home and in the community is crucial to positive outcomes for members. Survey results indicated this type of support was usually high, though slightly less on average among peers than family, and some children had experienced envy from their peers. It should not be presumed that the privilege and responsibility of participating is understood throughout a child’s social sphere, and implementers of similar models should monitor and meet community information needs.  More needs to be done to challenge existing attitudes and stereotypes if inclusion principles for the most marginalised are to be realised. As with many programs and initiatives, tightly held attitudes and stereotypes existed in children’s homes, schools and communities. As a result, members with a disability did not remain engaged, out-of-school children did not apply, pregnant teenagers or young mothers were not made welcome. If 71

no-barriers inclusion remains the goal then these gaps must be understood and addressed, with full participation of decision makers associated with the program. Realistically, such changes take time, and may require specific technical support from inclusion advocates to mainstream rights and inclusion in all activities and programs.  Child protection principles and child-safe organisational processes are specialist areas; it should never be presumed that they are in place. While no child protection incident was reported over the three cycles of TLYP, it is still necessary for any initiative involving children – especially when that involvement takes them away from their homes – to have stringent child protection protocols in place. This may not always be within the scope or capacity of implementing partners to recognise. A key role for UNICEF is to ensure all their associated programs are aware of and applying child protection mandates.

8. Recommendations Evaluation recommendations were reached through a process of consultation and refinement:  Firstly, analysing key findings and lessons to identify suggested ways forward. This included stakeholder commentary as well as the evaluator’s own research and experience of youth participation projects in other contexts.  Next, shortlisting and presenting recommendations to three audiences separately: APFTL, UNICEF and SSYS ministry representatives, to help with prioritising and grouping recommendations for final review  Finally, bringing stakeholders together for a validation workshop (see methodology section): as part of this, group work to refine wording, ensure Tetum language translations matched with intent in English, and eventually to agree and endorse a final set of recommendations.

When presenting recommendations at the workshop, the evaluator grouped them into four categories: TLYP process; improving role model and multiplier outcomes; measuring and demonstrating impact; enhancing personal development. This was useful for thematic discussions. Since that time, they have been reworked once more, to resemble a timebound action plan. It should be noted that most recommendations are not made to UNICEF; this was an evaluation commissioned by UNICEF on behalf of the government of Timor-Leste, and many recommendations apply equally or solely to other partners in TLYP.

Recommendation Priority: within six months Medium/long-term: six months to two years for..? UNICEF  Support SSYS in partnering with  Support design and dissemination of appropriate technical expertise to communications and IEC materials that promote review and revise the current TLYP understanding in communities (especially curriculum design for TLYP. schools, including students and teachers).  Formalise connections and learning between youth leadership programs (especially TLYP) and peacebuilding initiatives (government / NGO), encouraging standardised curriculum and coordinated implementation.  Support TLYP Secretariat to build and implement an achievable monitoring framework including TLYP within broader National Youth Policy goals. Consider Most Significant Change and other youth-friendly monitoring techniques, as well as quantitative reporting of community outreach by TLYP/APFTL.

 Provide technical support to APFTL on realistic expansion plans in the face of increasing demand; including connections to training refresher courses and ‘train the trainer’ capacity in members. Government of  Formalise the shift from two to  Develop an action plan for greater consultation Timor-Leste three years for TLYP mandate with disability organisations, including disability- (SSYS) cycle, aiming for consistent specific social analysis skills. government budget commitment  Draw clear Standard Operating Procedures to per year; ensure election costs can acknowledge changes made since 2009 based on be covered under this budget. formal learning and experience; standardise  Install complaint and safety annual activities and schedules over the three- mechanisms for TLYP members to year cycle. report problems they experience.  Partner with appropriate technical expertise to  Establish a formal platform for revise the current curriculum and make it available regular consultation with Members for all to use, including online, branded as Timor- of National Parliaments and TLYP Leste’s approved youth leadership training on child issues, including course. connecting to the SDG Roadmap  Work with gender and inclusion specialists to working group. increase overall curriculum emphasis on equality, addressing gender stereotypes in Timor-Leste; identifying missing or under-represented themes.  Using data from new monitoring framework, deliver a formal publication at the end of each TLYP cycle, presenting output and outcome data at community, municipality and national levels. Other partners,  Consider a network and program  Identify APFTL members who have a clear including APFTL for shortlisted / substitute passion for training, designate them as lead and other youth candidates, potentially trainers and connect them to refresher courses in NGOs implemented through municipal facilitation and training techniques, including alumni networks. increased usage of online and multimedia-based  Look at roles in training based on learning approaches. expertise, and seek specialist  Set up a mentoring system, using administrative providers to expand rights-based post / municipal youth networks (APFTL, modules (gender, disability, child government youth centres) to support TLYP protection, human rights) members on first-time community and chefe suco policy engagement.

9. References and further reading

EC/USAID 2013, Youth, Democracy and Peacebuilding in Timor-Leste: A joint baseline survey, available at: https://www.sfcg.org/wp-content/uploads/2014/09/SFCG_YEPS_DAME-Baseline-Report-_FINAL-publication.pdf

GCPYC 2015, Evaluation of Child and Youth Participation in Peacebuilding, Eastern DRC, Global Partnership for Children and Youth in Peacebuilding 2015, available at: https://www.sfcg.org/wp- content/uploads/2014/11/2015July_Eval-of-ChildYouth-Peacebuilding-DRC.pdf

GOTL 2014, Situation analysis of children in Timor-Leste, available at: http://www.statistics.gov.tl/wp- content/uploads/2014/09/Situation_analysis_of_children_in_Timor-Leste.pdf

GOTL 2016, National Youth Policy, available at: https://www.mof.gov.tl/wp-content/uploads/2016/07/06-National-Youth- Policy.pdf

IPU 2016, Youth Participation in National Parliaments, Inter-Parliamentary Union, Geneva, Switzerland 2016, available at: http://archive.ipu.org/pdf/publications/youthrep-e.pdf

IDEA 2015, Increasing Youth Participation Throughout the Electoral Cycle, Round Table November 2015, Pretoria, South Africa, International Institute for Democracy and Electoral Assistance (IDEA), available at: https://www.idea.int/sites/default/files/publications/increasing-youth-participation-throughout-the-electoral-cycle.pdf

Kirby, P. and Bryson, S. (2002), Measuring the Magic? Evaluating and researching young people’s participation in public decision-making, Carnegie Young People Initiative, London, available at: http://www.participatorymethods.org/resource/measuring-magic-evaluating-and-researching-young-peoples- participation-public-decision

Lansdown, G. 2011, A Framework for Monitoring and Evaluating Children’s Participation, available at: https://resourcecentre.savethechildren.net/library/framework-monitoring-and-evaluating-childrens-participation- preparatory-draft-piloting

PMNCH/Women Deliver 2017, Advocating for Change for Adolescents! A practical toolkit for young people to advocate for improved adolescent health and wellbeing, Partnership for Maternal, Newborn and Child Health/Women Deliver May 2017, available at: http://www.who.int/pmnch/knowledge/publications/advocacy_toolkit.pdf

Rogers, Patricia (2014). Theory of Change: Methodological Briefs - Impact Evaluation No. 2, Methodological Briefs no. 2

UNDP 2017, Timor-Leste’s Roadmap for the Implementation of the 2030 Agenda and the SDGs, UNDP/Government of Timor-Leste, available at: http://timor-leste.gov.tl/wp-content/uploads/2017/08/UNDP-Timor-Leste_SDP- Roadmap_doc_v2_English_220717.pdf

UNEG 2011, Integrating Human Rights and Gender Equality in Evaluation ‐ Towards UNEG Guidance, United Nations Evaluation Group, New York, available at: http://www.uneval.org/document/detail/980

World Vision 2015, Children’s Parliaments and Children’s Councils in World Vision Programs, available at: http://www.wvi.org/sites/default/files/Children’s%20Parliaments%20and%20Children’s%20Councils%20in%20WV%20 programmes.pdf

UN IWG 2016, Young People’s Participation in Peacebuilding: A Practice Note, Inter-Agency Network on Youth Development Working Group on Youth and Peacebuilding, January 2016, available at: http://www.un.org/en/peacebuilding/pbso/pdf/Practice%20Note%20Youth%20&%20Peacebuilding%20- %20January%202016.pdf

10. Annexes Annex 1: List of documents consulted 2007  November: Proposal to set up a Youth Parliament in Timor-Leste (UNICEF) 2008  July: Proposal to set up a Youth Parliament in Timor-Leste (Timor Aid) 2009  February: Proposal for the Establishment of the Timor-Leste Youth Parliament (SSYS/UNICEF)  May: Summary of progress, National Timor-Leste Youth Parliament (Timor Aid)  October: Training Plan for Youth Parliament (SSYS/UNICEF)  November: Jornal da Republika (Government Resolution) N. 023, Youth Parliament, passed November 2009 (Government of Timor-Leste) 2010  March: TLYP Strategy (powerpoint)  December: Press release, first sitting of Timor-Leste Youth Parliament (UNICEF) 2011  Report on Development of the Strategic Plan and Capacity Analysis of the Secretariat for Timor-Leste Youth Parliament (UNICEF/SSYS) 2012  December: Recommendations, final plenary of Parliament 2010-2012 2013  December: Recommendations, first plenary of Parliament 2013-2015 (SSYS) 2014  Training manual, 2nd period 2013-2015, Secretariat of State for Youth and Sports/Youth Parliament Secretariat – Modul Formasaun Ba Membrus PFN 2nd Periodo 2013-2015, Secratira de Estado Da Juventude e Do Desporto/Secretariado Parlamentu Foinsa’e Nian  August: Summary results, Youth Parliament Study Visit to Viqueque August 2014 (UNICEF/SSYS)  Recommendations, second plenary of Parliament 2013-2015 2015  January: Youth Parliament Election planning (meeting minutes) (UNICEF/SSYS/CNJTL)  December: Recommendations, third plenary of Parliament 2013-2015 (SSYS)  State SSYS Budget and TLYP Budget 2016  State SSYS Budget and TLYP Budget 2017  August: Terms of Reference, TLYP evaluation  State SSYS Budget and TLYP Budget  December: Recommendations, first plenary of Parliament 2016-18

Annex 2: Terms of Reference, TLYP Evaluation

UNICEF – Timor-Leste Terms of Reference (TOR) for individual Contract (External: for Circulation) (Note: This TOR is original before rescheduling due to political situation in Timor-Leste)

Requesting Section: Child Protection and Participation Nature of Consultancy: Evaluation of Timor-Leste Youth Parliament Programme Background and Purpose of Assignment: Timor-Leste is a country of young people, with nearly half of the population being children under 18 years of age25, the age group defined as a “child” under UN Convention of the Rights of the Child (CRC), which was ratified by the Government of Timor-Leste (GOTL) in 2003 immediately after its independence in 2002. The National Youth Policy (2016) of the GOTL defines “youth” as young people aged 15 to 24 years old with further disaggregation by “adolescents” (15-19 years old) and “young adults” (20-24 years old), and emphasizes that these periods are “important stage in youth development” and “transition from the life as a child into life as a youth and initial stage of adulthood” (p.3, National Youth Policy). With this representation, adolescence and youth deserved to have special focus in the National Youth Policy of Timor-Leste, with the view to motivate young people to exercise their rights and duties that are inherent as citizens and as a fundamental base for sustainable development.

Considering the pivotal role of youth and adolescents in national development, UNICEF supported the GOTL through the Secretary of State for Youth and Sports (SSYS) for the establishment of Timor-Leste Youth Parliament (TLYP) after the approval of the Government Resolution No.23/2009. The Timor-Leste Youth Parliament (TLYP) was inaugurated on 16th August 2010 with the aim to provide a forum for Timorese youth to promote civic participation of young people and serve as a unique school for future leaders and provide opportunity for building a dialogue between youth, community and leaders to promote involvement and meaningful participation of young people and their concern in nation building (Ref. Strategic Plan for the Timor-Leste Youth Parliament, 201126).

The TLYP consists of 132 national youth parliamentarians including 2 special representatives of disabled adolescents. Gender and geographic balance is maintained through community election process in which each community elects one male and one female youth parliamentarian to represent their community. During their two-year mandate, TLYP members receive a wide range of trainings covering topics such as: leadership, citizenship, environmental issues, public speaking and debate, basic planning and organizational management, problem analysis, and writing skills including drafting letter to government/ Members of Parliament for advocacy; human rights including CRC, Convention on Eradication of Discrimination against Women (CEDAW), among others. In addition to these trainings, the TLYP members attend relevant workshops on youth and adolescents. During the mandate, they also conduct a field visit to selected municipalities (administrative unit after nation in Timor-Leste) in order to collect information around youth issues affecting them, such as quality of education, health, early marriage, unemployment and basic infrastructure, security, and social cohesion. These findings are brought to the plenary sessions of the annual National Youth Parliament to discuss and submit key recommendations to the relevant line ministries of Government.

Since the establishment of TLYP Programme, a total of 392 adolescents have directly benefitted from the programme as youth parliamentarians including the current batch of 2016-2018. Whereas the mandate of each youth parliamentarian is only for 2 years, the alumni took their own initiative and established an Alumni Association in August 2015. As an independent body, the Alumni of Youth Parliament Association has been collaborating with various institutions including the GOTL and other development partners such as UNICEF on key youth related programmes areas including Life Skills Based Education, Child Online Protection, among others. In 2016, a Memorandum of Understanding (MoU) was established between the Secretary of State for Youth and Sports (SSYS)/GOTL and Association of Alumni of Youth Parliament to organize the inauguration and plenary session of Youth Parliament.

While several key activities were actively conducted since the approval of the Government Resolution No.23/2009 in 2009, there is a lack of documentation of TLYP to inform evidence-based policy actions and to share valuable experiences/lessons learned with other countries. SSYS

25 Government of Timor-Leste, Population and Housing Census 2015.

77 also expressed their needs to develop a standard training/orientation package for TLYP based on the lessons learned. In this regard, GOTL and UNICEF agreed to conduct a formative evaluation on the TLYP under UNICEF Country Programme Action Plan (CPAP) 2015-2019 signed between UNICEF and the GOTL (Ref. Five-years IMEP in CPAP 2015-2019). There is no Theory of Change (ToC) developed for TLYP while strategic plan (Annex 1) indicates overall outline of TLYP including vision and strategic objectives. This evaluation will provide an opportunity to revisit TLYP strategic plan, and update it according to current context of the country including National Youth Policy (2016) and 2030 Agenda on Sustainable Development Goals (SDGs).

Purposes and Objectives of the Assignments Overall Purpose The evaluation’s main purpose is to generate substantial knowledge and evidence on the TLYP in the context of Timor-Leste to inform national development policies and strategic planning processes as well as guide UNICEF CPAP 2015-2019 Outcome 3: Child Protection and Participation at the time of mid-term review in 2017. A formative evaluation with a utilization focused approach is suggested to ensure the usefulness and relevance of this evaluation exercise in Timor-Leste. Specific Objectives - To assess the effectiveness, efficiency, relevance, impact, sustainability, and equity of the Timor-Leste Youth Parliament (TLYP) programme. - To draw recommendations on how the TLYP Programme can be further enhanced towards contributing to achieving results for children in Timor-Leste - To review and provide recommendation for the contents of the standard training/orientation package for TLYP. - To document key findings and lessons learned of TLYP with local and international community - To provide strategic guidance to GOTL and UNICEF in determining its focus areas of support to adolescent and youth participation - To identify synergy areas with other initiatives related to youth and adolescents at national and sub-national levels (municipality). Primary audience The primary audience for this evaluation is the Government of Timor-Leste (GOTL) and UNICEF Timor-Leste. A small reference group consisting of UNICEF and GOTL will be established to guide the evaluation and ensure the participation of youth and adolescent (Annex 2). Being the first comprehensive evaluation for the TLYP Programme, other stakeholders working in the areas of adolescent participation will also benefit from the findings and results for youth related policy implementation and programming especially under current 2nd phase (2016-2020) of National Strategic Development Plan in which fostering a human resource is one of the key priorities.

The consultant should adhere to UNICEF’s Evaluation Policy; UNEGs Norms and Standards for Evaluation, and ethical guidelines for UN evaluations; UNICEF Publication guidelines27; UNICEF style book28; and to UNICEF Procedure for Ethical Standards in research, evaluation, data collection and analysis29.

Proposed Methodology and Evaluation Questions The evaluation will adapt formative evaluation approach with utilization focus. The evaluation uses mixed method approach covering both, quantitative and qualitative data analysis based on desk review, secondary analysis of existing data, designing and administering questionnaires for all current YP members together with YP Secretariat Office, Focus Group Discussion (FGD) and case studies in selected two municipalities. At national level, the consultant should meet with representatives of the TLYP members, Secretariat of Youth and Sports (SSYS), Secretariat of National Youth Parliament under SSYS, Representative of Alumni of Youth Parliament Association, National Youth Council (CNJTL) representative, NGOs (Alola Foundation, Red Cross, etc) and relevant development partners and UN agencies. At sub-national level, the consultant should meet with the local authorities, representatives of the Youth Parliament, stakeholders in the community (community leaders, parents, school teachers, Youth Center representatives, etc). This evaluation needs to follow the core principles of Human Rights Based Approach, and findings need to be analysed with gender and equity lens (sex, age, and disability). The consultant is expected to apply a participatory approach by encouraging adolescent’s engagement and adolescent’s empowerment through this evaluation exercise30. This evaluation is also expected to contribute to the national evaluation capacity building in collaboration with General Directorate of Statistics (GDS) through; knowledge sharing on evaluation training session/workshop on the use of data collections tools (interview, observation, Focus Group Discussion, etc); facilitate multi-stakeholder engagement, ethical review for an evaluation, among others31.

27 To be provided during phase 1. 28 To be provided during phase 1. 29 In 2015, the Director, Division of Data, Research and Policy at UNICEF approved the UNICEF Procedure for Ethical Standards in Research, Evaluation and Data Collection

and Analysis to guide UNICEF’s evidence generation activities and to support the integrity of UNICEF’s evidence base in order to ensure that UNICEF’s programmes, policy and advocacy activities are grounded in ethical principles and practices. All evaluation and research/studies in UNICEF involving human subjects or analysing sensitive secondary data are supposed to follow the aforementioned ethical standards procedure. 30 UNICEF Innocenti Research Center 2017, Adolescent Participation in Research: Innovation, rational and next steps. Available at https://www.unicef- irc.org/publications/879/ 31 UNICEF Evaluation Office (2010), Evidence for Children. Developing National Capacities for Country-led Evaluation Systems – A conceptual framework

Key evaluation questions are proposed as below based on the OECD-DAC evaluation criteria32 and Human Rights Based Approach (HRBA)33. The below evaluation criteria and questions need to be further refined by the Consultant during the inception phase together with definition of evaluation criteria, methodology on how to collect key information to answer each question.

Relevance: The extent to which the programme is suited to the priorities and policies of the target group, county, UNICEF, and global agenda such as 2030 agenda on Sustainable Development Goals (SDGs). - To what extent are the objectives of the programme still valid? - Are the activities and objectives of the programme consistent with the overall goal/vision? - Has the programme been aligned to governments and UNICEF’s priorities/policies/reform agendas? - Is the programme relevant to 2030 agenda on SDGs? If so, how? - To what extent has TLYP been inclusive in terms of gender, socio-economic background, geographical disparities (urban/rural), and disability of children? - How the TLYP programme in line with the Government Resolution No.23/2009? Efficiency: Efficiency measures the outputs -- qualitative and quantitative -- in relation to the inputs. - Were activities/programme cost-efficient in achieving the programme intended goal? - Were objectives achieved on time through the inputs? - Was the programme implemented in the most efficient way compared to alternatives? - Are there any synergy areas with other programme related to youth and adolescent participation? - What strategies/ approaches of UNICEF have been most efficient in influencing improvements in guaranteeing child and adolescents’ participation rights? Effectiveness: A measure of the extent to which the objectives have been achieved and the major factors influencing the achievement and non- achievement of objectives. - To what extent have the planned results been achieved (quantitative and qualitative)? - How effective was UNICEF’s support to the achievement of youth parliament program? - To what extent did the advocacy made by TLYP influence government decisions concerning youth and adolescent issues? - How did the youth parliament programme benefit other youth who are not TLYP members? - To what extent has the programme contributed to the capacity building of the adolescents in the country? - How successful was the programme for other children and adolescents in the community especially for those most vulnerable and marginalized? Sustainability: Sustainability is concerned with measuring whether the benefits of an activity are likely to continue without donor funding. Projects need to be financially sustainable. - To what extent did the benefits of a programme/activities continue after donor funding ceased? - What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project? - What is the trend of state budget allocated for TLYP programme in the past, and expected trend in the future? How sustainable is it? - Has the programme been successful in leveraging governments’ political will and financial resources for child and adolescents protection? Impact: The positive and negative changes produced by an intervention/programme, directly or indirectly, intended or unintended. This involves the main impact and effects resulting from the activity on the local social, economic, environmental and other development areas. The examination should be concerned with both intended and unintended results and must also include the positive and negative impact of external factors. - What has happened as a result of the TLYP programme? - What real difference has the activity made to the beneficiaries, and which activities contribute more impact? - How many people have been affected directly and indirectly? Scope of the Evaluation Programme Area and Specific Project Involved: Country Programme Action Plan 2015-2019: Child Protection and Participation Programme Outcome 3: Child Protection and Participation34. Geographic Scope: National as well as sub-national level ( 2 municipalities) 1. Work Assignments and work schedule: Deliverables/End Product (s) and Time Frame Estimated duration of assignment is a total of 35 working days. The consultant needs to elaborate a detailed time frame for each activity/deliverable in technical/financial proposals. (International travel time is not included into above working days). Activity Expected Outputs Deliverables Time Frame Phase I. Inception Phase (in-country) 1.1 Minutes of the meeting including 7 working days in overall timeframe of the evaluation 1. Introductory meeting with UNICEF Timor-Leste and stakeholders October 2017 exercise 2. Conduct desk review of relevant documents. (UNICEF will 1.2 Inception report, including the Note: 2 weeks’ period provide relevant information and documents with consultant elements mentioned in Activity 3, including UNICEF style book and UNICEF publication guideline is required between Phase 1. and manual of Youth Parliament training) 1.3 Final inception report including the activity 3 and 4 3. Submit an inception report including : context of the country, elements mentioned in Activity 3, history and background of TLYP programme, evaluation (Phase1) in order to Phase 1, based on the feedback framework, work plan, Theory of Change (ToC) on TLYP

32 The DAC Principles for the Evaluation of Development Assistance, OECD (1991), Glossary of Terms Used in Evaluation, in ‘Methods and Procedures in Aid Evaluation’, OECD (1986), and the Glossary of Evaluation and Results Based Management (RBM) Terms, OECD (2000). OECD- DAC evaluation criteria. Available at http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm 33 These criteria mean that the activity ensures all children and other beneficiaries receive opportunities, without discrimination, bias or favouritism 34 By 2019, all children and adolescents grow up in a protective environment free from violence, exploitation, abuse, neglect and unnecessary separation from their families and that, in particular, adolescents are empowered with knowledge to adopt positive practices and means to participate in local governance processes and mechanisms 79

Programme, methodology with proposed tools/interview guides, from evaluation reference group collect feedback from list of evaluation questions with source/methodology, limitation, with an additional appendix with the evaluation risk/means of mitigation, list of reference. (Format needs to be table which clearly shows aligned with UNICEF style book and UNICEF publication “comments received” “what degree reference group. guideline) have comments have been 4. Submit revised final inception report based on the feedback accepted by the consultant” and received from evaluation reference group (UNICEF and GOTL) “how comments addressed in revised report”.

Phase 2. Data collection, Drafting evaluation report and power point presentation (in-country; 2nd visit) 2.1. Draft evaluation report and power 15 working days in 1. Meeting and interviewing key stakeholders at national level point presentation, including all elements November 2017 2. Conduct field visits to interview at the sub-national level. mentioned under activity 3 Phase 2. 3. Draft evaluation report and power point presentation. Evaluation report needs to be well-structured and aligned with UNICEF Publication guideline/UNICEF style book, viz.; executive 2.2 Final evaluation report and power summary; context, purpose methodology; findings; conclusions; recommendations; lessons learned; and annexes including point presentation, including all elements TOR, list of interviewees and site visits; data collection mentioned under activity 3 Phase 2. instruments; evaluation matrix; list of reference; two Human Interest Stories with pictures ( detailed check list will be provided for reference), outline of standard training manual for Youth

Parliament. 4. Presentation of key findings to evaluation reference group for verification of preliminary findings. 5. Submit final draft of evaluation report and power point presentation including all elements mentioned under activity 3 Phase 2. Phase 3. Finalization and Development of Dissemination Materials 3.1 Final evaluation report and 13 working days in power point presentation, 1. Submit final evaluation report and power point presentation January 2018. including all elements deliverables based on the comments from UNICEF and other mentioned under activity 3 Note: stakeholders and counterparts Phase 2. 2. Develop a reader friendly summary booklet on the evaluation of 2 weeks’ period is 3.2 Reader friendly summary TLYP Programe intended to be shared with children and required between booklet for children and youth adolescent including NYP members activity 5 (Phase 2) including NYP members (not 3. Develop an evaluation brief of TLYP Programme evaluation with and 1 (Phase3) in exceeding more than 6 pages) visual information (table, figures, pictures, etc), key findings, order to collect 3.3 Summary brochure for recommendations, and lessons learned intended to be shared feedback from the advocacy with a wider audience including other line ministries of GOTL, evaluation reference

development partners, and other countries (UNICEF will provide group. the example such as “Joint Formative Evaluation of Child- Friendly School Policy Implementation in Cambodia 2016”) 1. 12. Qualifications or Specialized Knowledge / Experience Required Both technical and financial proposals are required together with CVs and sample deliverables to be considered for this consultancy. Essential:  Advanced university degree in Social Sciences or other related field  Professional experience to conduct an evaluation (sample evaluation report is essential to submit together with technical/financial proposals)  At least 8 years work experience, out of which at least 5 years practical experience in programme planning, management, monitoring and evaluation relevant to adolescents and youth participation  Good understanding of UNICEF’s global and regional programme strategies particularly in Child Protection and Participation, and Human Rights Based Approach.  Strong analytical and writing skills  Excellent written and oral English Asset  Familiarity with Timor-Leste or region  Working experiences in developing countries  Knowledge of Tetum/ Portuguese 13. Contract Supervisor: Chief of Planning, Monitoring and Evaluation, Social Policy, UNICEF Timor-Leste. (Note: Evaluation reference group will be established as a quality assurance mechanism; see Annex ) 15. Consultant’s Work Place: Upon the consultant’s arrival to Timor-Leste, UNICEF will provide an office space. UNICEF will support the consultant in establishing contact with necessary stakeholders and arrange meetings with all relevant stakeholders. Laptop or computer will not be provided.

Annex 3: Evaluator biodata and suitability

Chalk It Up: philosophy and function Chalk It Up is the international development branch of Project Chalk, a small consultancy offering learning, communications and project management services to not-for-profit and public sectors. Project Chalk sees itself as a support service with a core duty to assist community programmes towards quality outcomes. Based in Melbourne, Australia, Project Chalk calls on a number of specialisations to meet client requirements for programme and policy research, evaluation and audit, learning events, curriculum and training including e-learning, technical writing and editing.

Chalk It Up reflects the experience and values of lead consultant Katie Chalk. Katie worked across a variety of advisory and planning roles with World Vision International for more than a decade and has been freelancing for a number of different aid and development clients since 2014.

Katie brings extensive insight into solid and practical methodologies for conducting research, social surveys, capacity building, learning and application of learning for quality programming. Katie also offers significant expertise in creating user-friendly reports, databases and multimedia to enhance audience and opportunities for sharing lessons and practices in international development.

Previous experience relevant to the work required by UNICEF in this instance includes:

 Review of UNICEF/H&M Foundation Alternative Pre-Schools and Parenting Education project, 2017  Close the Gap 10 Year Evaluation, Oxfam/Australian Human Rights Commission, 2016  Review of Government of Norway Strengthening Justice and Welfare Systems for Children in Timor- Leste project, UNICEF, 2016  Factsheets and case studies on youth advocacy initiatives, Nepal, Philippines (2010), Democratic Republic of Congo (2014)

______

Annex 4: Timor-Leste Youth Parliament Strategic Plan

Annex 5: Online survey demographics

TLYP Started Completed Completed Added based Total Telephone MEMBERS all most on direct sample interview IN SURVEY consultation used validation 2010 Male 10 7 9 1 10 1 2010 Female 14 10 11 0 11 2 2010 TOTAL 24 17 20 1 21 2013 Male 12 9 10 0 10 2 2013 Female 6 1 5 0 5 1 2013 TOTAL 18 10 15 0 15 2016 Male 8 5 6 1 7 2016 Female 9 3 5 1 6 2016 TOTAL 17 8 11 2 13 TOTAL 59 35 46 3 49 6

Annex 6: List of people interviewed; site visit schedules In line with the ethical framework for conducting this evaluation, actual names of people taking part have been withheld from this report. Interview transcripts are stored with the consultant in confidence.

Direct consultation with TLYP members

Focus group Focus group Individual Consultation Total young discussion: discussion: semi- on draft people consulted Youth YP with other structured evaluation Duplicates Parliament young interview findings removed – some members people took part in more only than one way # interviews / 2 1 6 1 groups 2010 Male 3 1 4 2010 Female 2 1 3 2013 Male 1 1 2 2013 Female 1 2 3 2016 Male 2 2 1 3 6 2016 Female 1 1 3 4 TOTAL 22

Direct consultation with other stakeholders, Dili and field-based

Government UNICEF Implementing Teachers / Chefe Youth representatives stakeholders partners (UN / principals sucos centre team (SSYS) NGO) members 4 6 3 3 3 4

Trip schedule for field-based consultations

Location Dates Student focus Youth Centre Administrative post groups team interviews / chefe suco interviews Dili October 26-31 2017 2 0 0 Dili Sept 24-28 2018 2 1 0 Liquica October 1 2018 2 1 1 Ermera October 2 2018 2 1 2

Annex 7: TLYP member 2015 6. Rate the following training; how did you use it? survey tool Not Use- Great – Don’t Didn’t useful ful a high- remem- do it light ber it Life skills Based Process Education  Survey was presented as online Social analysis freeware (SurveyMonkey) interface. Human rights and  Alumni Association supported children rights dissemination of the link to the survey, Civic education via “WhatsApp” for former youth members. Basic leadership

 Results were discussed with alumni Gender rights and representatives participating in the domestic violence evaluation field research, to form initial Proposal writing conclusions for validation through semi-structured interviews and focus Public debate Public speaking groups. Environment Content: Basic journalism Anti-corruption and Link to Tetun version governance https://www.surveymonkey.com/r/LMT2LRH This survey is for current and former members of the Timor- Other (please specifiy) Leste Youth Parliament. It is your chance to have your say on what happens next. We want to know what you think about being part of Youth Parliament, how it has helped you personally and how it has helped other young people in I have I have I have your community. The survey will take about 20 minutes. I used it – used it trained Thank you for your help. haven’t comm- pers- others Name used it unity or onally in it peers Life skills Based . 2 Age Education Social analysis

3. School level Human rights and children rights Under Year 9 Civic education Year 9 Basic leadership Year 10 Gender rights and Year 11 domestic violence Other (please specifiy) Year 12

University 7. Rate the following activities Not Use- Great Don’t Didn’t Not at school useful ful – a hig- remem- do it If not at school, what was the final year you completed? hlight ber it

If not at school, are you working? Field research – 4. Location my own

(Drop down for municipality) municipality Adminstrative post Field research – other municipalities Field research –

international 5. Which Youth Parliament were you part of ? Swearing in ceremony 2010 Youth Parliament Plenary 2013 Other (please specify)

11. My confidence and knowledge Weak Naton Strong Very Not sure

strong Did you take part in the 2014 Municipal Consultations? Yes/No At the start If yes, did you think your views were represented in the final National Youth Policy? At the end Can you describe this? 8. Did you organise any of the following activities? 12. My interest in government process Yes No Weak Naton Strong Very Not sure strong Conducted training for other young people in my community At the start

At the end Conducted training for adults and community groups in my community . My interest in youth and community issues 13 Weak Naton Strong Very Not sure Shared the results of field trips or plenary strong sessions at school At the start Shared written materials on youth issues in my school or community At the end Put a proposal in for community work 14. My public speaking and debating skills Took up a role in community or NGO Weak Naton Strong Very Not sure work strong Started campaigning on youth issues (for instance, teen pregnancy, early At the start marriage, school dropouts) At the end 9. If you conducted training, shared results or written materials, approximately how many people did you reach? 15. My employability and life skills On average, per session Weak Naton Strong Very Not sure Overall, all sessions strong How did people around you support your participation in At the start Youth Parliament? During Didn’t Mostly Very At the end nomination really understanding understanding and understand and and 16. My behaviour, choices and positive attitude shortlisting or support supportive supportive Weak Naton Strong Very Not sure My parents strong and family At the start

At the end My school and teachers 17. Support from other people my age to be my best My friends Weak Naton Strong Very Not sure and peers strong At the start Didn’t Mostly Very During really understanding understanding At the end TLYP understand and and or support supportive supportive 18. Support from adults to be my best My parents Weak Naton Strong Very Not sure and family strong At the start My school and At the end teachers

My friends My interest in becoming a community leader and peers Weak Naton Strong Very Not sure strong What difference has it made? At the start

Thinking about yourself, what has changed for you as a At the end result of your time in Youth Parliament?

My interest in entering politics Stro Disagre Neither Agre Strongl Weak Naton Strong Very Not sure ngly e agree / e y agree strong disa disagree At the start gree d Youth Parliament At the end activities were

relevant to me Thinking about your community, and other people your age, Youth Parliament what has changed for them as a result of your time in Youth activities helped Parliament? me understand 19. Other young people’s confidence and knowledge disability disadvantage Weak Naton Strong Very Not sure strong Youth Parliament At the start activities helped me understand poverty At the end disadvantage

20. Other young people’s interest in government process Youth Parliament activities helped Weak Naton Strong Very Not sure me understand strong disadvantage for At the start girls and young women At the end Youth Parliament made me feel more supported in my 21. Other young people’s interest in youth and community issues own decisions and actions Weak Naton Strong Very Not sure Youth Parliament strong members are likely At the start to stay involved in community and At the end NGO work Youth 15. Other young people’s employability and life skills Parliament Weak Naton Strong Very Not sure contibutes to strong peace and social At the start cohesion (working At the end together for peace building) 16. Other young people’s behaviour, choices and positive in Timor-Leste attitude Youth Parliament Weak Naton Strong Very Not sure plenary strong recommendations At the start contribute to success of national

youth policy (they At the end bring change)

Youth Parliament 23. Adults’ and leaders’ attitudes to young people’s views is a worthwhile Weak Naton Strong Very Not sure government strong investment At the start Youth Parliament should continue long-term At the end 

27. What advice would you give new Youth Members? 24. Have you talked with local leaders (chefe suco, . What advice would you give the organisers of Youth municipal council) about the needs of young people in your 28 Parliament community? ? 29. What else do you want to tell us? Yes 30. We are looking for volunteers to take part in a further phone call interview to talk about their experiences in more No detail. Would you be interested in taking part? 25. If yes, what happened? Yes

26. Please rate your opinion of the following statements No about the Youth Parliament

Annex 8: Follow-up Telephone

Validation interview tool

Targets:

 Current and former TLYP members, who have already filled out the survey and

volunteered to give a further phone interview. 1. What made you want to be part of TLYP?  Roughly equal participation of F/M, and multiple municipalities represented 2. How supportive were your parents? How  Require eight interviews supportive was your school? Hello. My name is _____. I am calling on behalf of the Youth Parliament Evaluation Committee, as part 3. What have been the highlights for you? of the evaluation of current and former Youth

Parliament, commissioned by UNICEF. You gave us 4. What would you change if you could? your name as part of a survey of Youth Parliament members, and said you would be happy for a phone 5. Have you ever felt out of your depth, threatened interview to tell us a bit more about your experiences. or unsupported in your TLYP role? Does this sound familiar to you? Yes/no.

6. (If so) what did you do, who did you ask for help? If no, give a little more information about evaluation survey, for instance: 7. TLYP intended to include young people with  Link sent through Youth Alumni disability, and young people out of school, but it has  You might have answered in December/January, or when the link was not been successful. Do you think it would work to sent again in September try this again?  Survey asked about your training, activities and what you did in your own community as a result of TLYP. ADDITIONAL QUESTION IF RELEVANT, BASED

Is now a good time to conduct the follow up ON SURVEY ANSWERS interview? It should take around 15 minutes, unless You said you were currently looking for work. How you have a lot to tell us. is that going? How have your skills learned in TLYP If no, when is a good time for us to call back? helped with your employment opportunities?

There are seven questions on my list but feel free to Is there anything else you wanted to talk about talk about anything else that you think we should regarding your time in TLYP, positive or negative, know. I will be taking notes and then afterwards I’ll that we didn’t already cover in the survey or this translate them in summary to English so the lead interview? consultant can understand and include your main points. What you tell us is confidential and nobody THANK YOU FOR YOUR TIME apart from the consultant will see my notes. Is it OK to start?

Yes/No

Annex 9: TLYP participating schools analysis

School attendance was not a pre-requisite for TLYP members. However, considering the age group and the competitive criteria for entry, young people outside school did not usually apply to join. In theory, schools with TLYP membership should benefit from the role model and multiplier effect of discussions and training passed on from member involvement. The analysis below uses data from 2009/2010 records on the number of schools in each municipality35, and finds that:

 TLYP members in this cycle attended 39% of all schools in Timor-Leste;  25 schools had more than one TLYP member; one school had four and another had five  TLYP selection represents but does not favour private schools

The potential reach for TLYP advantages of leadership and civic responsibility through schools is therefore strong, underscoring the importance of support and mentoring to help TLYP members deliver training in their schools. It also raises a question on whether schools should be permitted to have more than one TLYP member; however, some of the duplication at schools has been caused by the transition from lower to upper secondary school, or the geographical proximity to other youth activities which support successful candidature (Scouts, youth centres)

Proportion public, private, technical

Municipality # admin posts / # TLYP # schools Total schools % schools members attended Aileu 4 posts / 8 members 5 13 38% Ainaru 4 posts / 8 members 7 17 41% Baucau 6 posts / 12 members 7 38 18% Bobonaro 6 posts / 12 members 9 13 69% Covalima 7 posts / 14 members 6 17 35% Dili 6 posts / 12 members 9 41 22% Ermera 5 posts / 10 members 8 17 47% Lautem 5 posts / 10 members 6 12 50% Liquica 3 posts / 6 members 3 10 30% Manatuto 6 posts / 12 members 9 13 69% Manufahi 4 posts /8 members 8 18 44% (maximum achieved) Oecusse 4 posts / 8 members 6 9 66% Viqueque 5 posts / 5 members 6 23 26%

35 https://www.epdc.org/country/timorleste/search?indicators=648-718-653- 655&year_from=2009&year_to=2010 – note, more recent data is not available through this source, but the consultant checked manually in three locations and did not find changes. 89

Annex 10: Ethics, privacy and informed consent

This methodology is dependent on the willing and informed participation of people, including children under 18, as key informants and participants in the Timor-Leste Youth Parliament. This participation is vital and in itself an ethical responsibility. At the same time, certain steps must be taken to ensure that individuals who provide personal information are protected from any risk associated with their participation, under principles of autonomy, benefit/’do no harm’ and equity of treatment across the sample of participation. In this respect, the evaluation will follow UNICEF guidelines for ethical standards including rights and gender considerations36. Principles of informed consent, confidentiality and voluntarism are to be applied, including written information to each participant or focus group leader with the option to withdraw at any stage. A technical reference group comprising UNICEF, government and university partners are to provide a quality assurance mechanism including ethical review, between the submission of this evaluation plan and the start of consultation with children and communities.

2.1 Provisions for ethical consultation of children, community and duty bearers Enquiry that focuses on human behaviour is always considered to contain an element of risk; however, the consultant proposes that this risk is negligible when mitigated by the following conditions of confidentiality, voluntarism and informed consent.

 Ethical considerations for consulting children (under 18)

Children (12 to 17) are the primary subject of outcome-focused interviews, and will be asked directly about their lives, the positive and negative implications of TLYP, and changes they have witnessed. Because of the themes considered by youth parliamentarians, there is likelihood that these conversations will include themes such as discrimination, sexual and reproductive health, violence against women and children, and other subjects which may be considered of disclosure risk to child participants. The risk is mitigated to some degree because parliamentarians will be talking about others. However, there remains a strong ethical duty in line with UN/UNICEF standards to provide options to child participants to withdraw or question their participation at any stage. Interviewers will be briefed in these ethical options.

Children interviewed as the subjects of human interest stories will be consulted, informed and respected in line with UNICEF’s protocols for child-centred communications, including name changes where a child is considered to be vulnerable. At all times a trusted adult (teacher, parent or other caregiver) will be present when children are telling their stories. For in-depth interview, for instance, human interest stories, informed consent will be obtained from parents or caregivers of young people under 18, before the interview commences.

Questions and themes for interviews are to be reviewed in advance by the Technical Reference Group including UNICEF, government and academic partners.

 Ethical considerations for consulting community members

Consultation of community members focuses solely on duty bearers and caregivers with a clear perspective on current or former Youth Parliamentarians. As the questions will focus on perspectives of change for young people, they are unlikely to venture into areas of personal risk or trespass. However, due to the methodology of Most Significant Change and the need to validate information quickly, consultation with community members may at times be unexpected to them. ‘Do no harm’ principles will be considered on a case-by-case basis, and informed consent using the form shown at Annex 2.2 will be sought. The names of participants remain confidential to the consultant.

 Ethical considerations for consulting duty bearers

People who hold responsible positions associated with the project, such as government staff, NGO partners and UNICEF staff, are purposively selected for their knowledge of the project. This reduces full confidentiality of participation. However, as with all other participants, comments will remain confidential to the consultant and will not be shared or used in a way that identifies the source. Key informants will be asked for signed consent on their participation including information on how to opt out later.

At all times, the consultant will be guided by UNICEF protocols for ethical research involving children37, and by decisions and input from the Technical Reference Group.

Use, storage and confidentiality of human data Transcripts of interviews and other ‘raw’ data are to be stored securely with the consultant for a period of at least two years. Transcripts that do not identify participants can be made available to UNICEF for accountability and cross-check on request. Data may not be used for any other purpose than that stated to the participant during data collection.

36 UNICEF Procedure for Ethical Standards in Research, Evaluation, Data Collection and Analysis, available at: https://www.unicef.org/supply/files/ATTACHMENT_IV-UNICEF_Procedure_for_Ethical_Standards.PDF ; UNEG Guidance on integrating Human Rights and Gender Equality in Evaluation, available at: http://www.uneval.org/document/detail/1616; UN- SWAP Evaluation Performance Indicators, available at: http://www.uneval.org/document/detail/1452. 37 https://www.unicef-irc.org/publications/pdf/eric-compendium-approved-digital-web.pdf ; https://data.unicef.org/wp-content/uploads/2015/12/EPDRCLitReview_193.pdf 90

2.2 Consent forms Thank you for agreeing to take part. The interview is about the activities and results of Timor-Leste Youth Parliament (TLYP) and will cover the period 2008 – 2017.

People taking part in interviews include:  Ministry staff and officials (SSYL/SSS, others where relevant)  NGO partners in training, planning and implementation  TLYP members, 2010-2017  Community representatives, leaders, parents and teachers

What you tell us will be used to improve TLYP in the future so that young people taking part can fulfil the TLYP goals of youth participation, better policy and enhanced opportunities for young people. Good ideas and positive experiences can be repeated, and any past mistakes or challenges avoided.

About the researcher Katie Chalk (Chalk It Up) is an independent consultant, contracted by UNICEF to conduct the TLYP evaluation. She is not associated with TLYP, the Government of Timor-Leste or with UNICEF in Timor-Leste. Katie will hand over the evaluation report to UNICEF for final release in March 2018.

Participation, privacy and confidentiality All participants in this review were selected for their knowledge of the Timor-Leste Youth Parliament. We need you to be honest, speak openly and raise issues or concerns. Your words remain confidential to the consultant. Katie will keep a record of your conversation for her future reference only, and will only use it for the purpose described above. UNICEF and partners will not read anything you say in a way that identifies you.

Your participation in this review is voluntary and you are entitled to withdraw at any time. We ask that you sign below to acknowledge your ‘informed consent’ to take part in this interview. However, if you change your mind at any time for any reason, you can contact Katie on the details below, and she will delete your information.

Should you have any queries or concerns about the review, you can also contact Katie on the details below. Katie Chalk / Chalk It Up Email: [email protected] Mob: +61425 801 137

Consent form - Tetum Obrigadu tanba ita bo’ot hakarak atu hola parte. Entrevista ida ne’e konabá atividade no rezultadu hosi Parlamentu Foinsa’e Timor-Leste (PFTL) no sei kobre períodu 2008 – 2017.

Sira ne’ebé hola parte iha entrevista ida ne’e mak hanesan:  Funsionáriu no ofisiál husi ministériu (SEJD, seluk tan wainhira relevante)  Parseiru ONG sira ba formasaun, planeamentu no implementasaun  Membru PFTL, 2010-2017  Reprezentante hosi komunidade, líder, inan-aman no profesór sira

Saida mak ita bo’ot hato’o sei uza hodi hadi’ak liután PFTL iha futuru atu nune’e foinsa’e sira ne’ebé mak hola parte bele realiza objetivu hosi PFTL ba partisipasaun foinsa’e-nia, polítika ne’ebé di’ak liu no loke oportunidade ba foinsa’e sira. Idea sira ne’ebé di’ak no esperiénsia positiva bele repete filafali, no bele evita buat hirak ne’ebé laloos ou sai hanesan dezafiu.

Konaba Peskizadór Katie Chalk (Chalk It Up) mak nu’udar konsultór independente ida, kontratadu husi UNICEF atu hala’o avaliasaun ba PFTL. Nia laiha afiliasaun ho PFTL, Governu Timor-Leste ou ho UNICEF Timor-Leste. Katie sei hato’o relatóriu ba UNICEF ne’ebé lansamentu final sei hala’o iha fulan Marsu 2018.

Partisipasaun, privasaun no konfiabilidade Partisipante hotuhotu iha revizaun ida ne’e nomeadu bazeia ba sira-nia koñesimentu konabá Parlamentu Foinsa’e Timor-Leste. Ami presija ita bo’ot nia onestidade, ko’alia ho laran kma’an no hato’o problema ou preokupasaun ruma. Konsultór sira sei rai metin ita bo’ot nia liafuan. Katie sei rai gravasaun husi ita bo’ot nia konversa hanesan ninia referénsia deit ba futuru, no sei utiliza informasaun hirak ne’e tuir buat ne’ebé temi tiha ona iha leten. UNICEF no parseiru sira sei la lee buat hirak ne’ebé ita bo’ot hatete ho meiu ne’ebé bele identifika ita bo’ot nia an.

Ita bo’ot nia partisipasaun iha revizaun ida ne’e sei halo ho voluntáriu no ita bo’ot bele dada an iha tempu ne’ebé deit tuir ita bo’ot nia hakarak. Ami husu atu ita bo’ot bele asina iha kraik atu rekuñese ita bo’ot nian ‘formuláriu konsentimentu” atu hola parte iha entrevista ida ne’e. Maske nune’e, karik ba oin ita bo’ot iha hanoin seluk iha tempu ne’ebé deit ho razaun saida deit, ita bo’ot bele kontaktu Katie ho detalle hanesan iha kraik ne’e, i nia sei la uza ita bo’ot nia informasaun .

Karik ita bo’ot iha dúvida ou preokupasaun ruma konaba revizaun ne’e, ita bo’ot bele kontaktu Katie ho detalle hanesan iha kraik.

Annex 11: TOR of Reference Group

Terms of Reference (ToR) for the Reference Group for Youth Parliament Programme Evaluation (Draft as of 5th June 2017)

Note: This TOR will be updated after new election schedule in May 2018 and following VII government establishment

June 2017

An evaluation reference group will be formed with members from SSYS, UNICEF and other key partners in the country. The reference group will be chaired by the evaluation manager, of SSYS, supported by UNICEF Timor-Leste Chief of Planning, Monitoring and Evaluation and Social Policy (PMESP), contributing to ensuring the ownership of the evaluation exercise.

Composition of the Reference Group: Evaluation Manager: National Director of Policy and Planning, SSYS Vice Evaluation Manager: Secretary: Youth & Adolescent Dev Officer UNICEF, Secretary Member: Commission for the Rights of the Child General Directorate of Statistics (GDS) Peace Center, University of Timor-Leste Child Protection Manager, UNICEF Chief of PMESP, UNICEF M&E Officer, UNICEF

The chief of PMESP has a responsibility to obtain technical inputs and clearance from UNICEF Regional Office and Head Quarters before finalization of the key documents related to the evaluation (ToR, the inception report, and final evaluation report) and share it with the evaluation manager.

Evaluation Manager has a responsibility to coordinate with key directors within the SSYS and other relevant ministries.

The broad task of the reference group will be to assist the evaluation manager in monitoring and supervising the evaluation.

Following are the specific tasks of the reference group in line with the ToR for the evaluation: 1. Review the inception report and provide timely feedback a. Review the inception report particularly for the methodological approach, ethical issues, evaluation design, the key evaluation questions, and data collection and analysis methods and tools, and work plan submitted by the evaluation team and provide inputs for refinement

b. Review of the refined inception report by the evaluation reference group to check if the inputs from the reference group has been incorporated or not.

2. Provide necessary support to the evaluation team for access to key informants / respondents at national, sub- national for data collection on the identified evaluation questions according to the inception report (in line with the approved work plan)

3. Verify the quality of the report and provide inputs for finalization

a. Review of the draft report by individual reference group member. b. Sharing of comments on the draft report and providing feedback on the findings and making recommendations through participation in the session on presentation of the draft report by the evaluation team.

4. Review the revised evaluation report and recommend for approval by SSYS and UNICEF a. Review of the revised evaluation report by the evaluation reference group, particularly, the evaluation manager to check if the inputs from the reference group has been incorporated or not. b. Identify key actions to address to the issues raised in the recommendations of the evaluation report, and develop management response plan. c. Once the evaluation manager and reference group members are satisfied with the revised evaluation report, then the evaluation manager will share the report with SSYS and UNICEF for approval

Note: Currently exploring the possibility to invite the EvalPartners Parliamentarian Forum (http://www.pfde.net/) as a reference group member.

Annex 12: Revised timeline The original timeline for this evaluation exercise was changed due to the political situation in Timor-Leste and the Government’s decision to hold an early election in May 2018. In order to ensure “national ownership” on evaluation exercise and “use of evaluation results”, stakeholders agreed to revise the schedule as below to restart the National Youth Parliament evaluation with new government.

Month Sep Oct Nov Week 1 2 3 4 1 2 3 4 1 2 3 National Youth Parliament Evaluation 3 Pre-phase: Presentation of final draft working inception note ( additional 3 working days; 1 days day workshop and 2 days revision) (WDs) Phase 2 Data collection municipality Most Significant Change workshops; human interest stories identified and appropriate permissions sought 8 WDs Analysis of data; draft findings presented to UNICEF internally for utilisation check 4 WDs Final consultations, Dili; results presentation to key partners (external) and to UNICEF senior leadership team (internal) 3 WDs Phase 3. Finalize report Human interest stories, draft report available. 7 WDs Feedback from reference group 10 WDs Finalise report, ppt, reader-friendly summary based on UNICEF feedback. 6 WDs

Compared to original TORs, major changes were made as below: 1. Extension of the assignment until 23rd November 2018 to accommodate above schedule change 2. Additional three working days and in-country mission to brief inception report to new government and accommodate changes if necessary before starting Phase 2 data collection.

Considering the criticality of the NYP evaluation as a key milestone for both GOTL and UNICEF Timor-Leste, this Annex hereby recommends the cost associated extension of the contract with Ms. Katherine Chalk.

Annex 13: Reflections on Youth Parliament models

The concept of Youth Parliament as a way to familiarise young people with the workings, duties and accountabilities of democratic governments is widely used. A 2016 study of youth participation in national parliaments surveyed 76 countries and found that half had a Youth Parliament, though models, membership and process varied (IPU 2016). The study noted three broad functions of Youth Parliament: deliberative (giving young people a voice in youth policy), awareness of parliamentary process, and political empowerment to promote youth-led advocacy and influence. All are highly relevant in the Timor-Leste context. A fourth, to promote positive interaction, social cohesion and community development within youth networks, is a common element of youth empowerment initiatives designed to contribute to peacebuilding outcomes (UN IWG 2016) and also very important to Timor-Leste.

Although Youth Parliaments are common, their processes and outcomes differ. There is no one global model. Searches of ‘grey’ literature available from governments and non- government organisations online do not provide much information, indicating that results of such initiatives are not consistently or well documented. However, some examples can be found of policy or social impact resulting from national youth parliament, in both wealthy and developing countries. These results, though isolated, demonstrate that Youth Parliament is more than a token institution given the right process and circumstances. For instance:

 In 2017, Malaysia’s National Parliament tabled four motions from the Malaysian Youth Parliament, including a motion for greater action on the region’s Rohingya refugee crisis38.  UK Youth Parliament presents formally to Parliament each year, based on constituency consultation (‘Make Your Mark’) on the most important issues affecting young people: for instance, minimum wages, social cohesion, protection39.  Members of the Democratic Republic of Congo’s Children’s Parliament (PARDE) have worked successfully to reduce political exploitation and recruitment of minors (GCPYC 2015).

A light review of general online information about Youth Parliaments forms the basis of the following matrix, which compares attributes of different parliaments with that of TLYP.

38 http://www.themalaymailonline.com/malaysia/article/youth-parliament-passes-motion-calling-on-govt-to-help-rohingya- refugees#uyaU8ZdFX0ClKeeJ.97 39 http://www.byc.org.uk/news/members-of-parliament-support-make-your-mark-ballot 94

TABLE 1: COMPARISON OF YOUTH PARLIAMENT MODELS

Where? When Who coordinates How does it work? Age Age of voters Membership Geographically Formal did it it? mirrors representative? representation start? electorates? in Parliament? India 1966 NGOs and Originally known as ‘mock parliament’, this model used a schools competition Second- N/a – school No No No sponsors/ patrons in its early years. The mandate for members has evolved to become more ary representational year-round, but without a formal link to government students institutions. Philippines 1996 National Youth Individuals self-nominate and self-fund, capped at 200 participants. The role 15-30 N/a – committee No No Yes Commission is consultative but once-off, every two years. Recommendations are raised to (Govt. affiliate) Parliament by relevant MPs Sri Lanka 2010 National Youth Members are usually leaders of existing youth clubs, shortlisted through 16-25 13-29 No Yes – youth clubs No Services Council election then interviewed by existing parliamentarians. Two year mandate, (Govt. affiliate) with annual sitting and ongoing consultation from government. UK 2001 Government Members represent the same electorates as formal Parliament; voted by 11-18 11-18 Yes Yes Yes peers (between 11 and 18). Annual sitting results in recommendations to parliamentary process; ongoing policy consultation directly with relevant government departments. Malaysia 2015 Government Members are elected (online/social media campaigning and voting) with a 18-30 15-40 Yes Yes Yes quota based on geographic youth population. There is a direct response mechanism with Parliament. Nepal 2010 NGO Members join through chapters, similar to youth groups, with delegates from 16-40 N/a – No Not yet Not sure each chapter connecting to national and international youth parliaments representation of chapters Australia 1985 NGO A taskforce is elected, but members self-nominate and undergo interview. 16-25 N/a No No Yes (Victoria) Meet annually in groups to identify policy before consensus negotiations, culminating in a formal submission to Victorian parliament. Europe 1988 NGO 40 ‘National Councils’ meet 3 times a year for policy discussions. Executives 14-28 (not sure yet) No Yes No are elected by members; anyone can join. Timor- 2010 Govt Members are nominated at suco-level and elected at sub-district by 12-17 12-17 Yes Yes Yes Leste other young people. They sit annually to generate policy recommendations, and take a lead in their local communities on empowerment and civil participation of children and youth.

Annex 14: Analysis of TLYP Plenary Recommendations General observations Policy recommendations from 2014, 2015 and 2017 were included in the analysis below. All were drafted formally on government letterhead and submitted by the Youth Parliament Secretariat to relevant ministers during Parliamentary process. In total 22 recommendations were passed in 2014, 21 in 2015 and 54 in 201740. The ability to frame recommendations around the five thematic visions of the National Youth Policy gave focus to Youth Parliament recommendations in 2017, with five different policy papers naming appropriate ministries and secretariats to take responsibility for actions.

Analysis of the scope of recommendations indicates a wide coverage of issues based on appropriate community consultation and contextual findings: for instance, concern over contaminated water sources leading to skin disease in 2017, a growing awareness of HIV and other sexually transmitted diseases in the same year, and the observations from frontier study visits in 2015. Recommendations on sexual and reproductive health decisions are consistently at the forefront each year, as are calls for appropriate accountability and budget on services for young people in health, education and vocational support.

At this stage a gap in the plenary process is formal response from ministries to these policy asks. The table below groups recommendations by issue, and shows which year and to what degree the issue was raised. National Policy 2014 2015 2017 Explanatory Key: - issue not raised Youth Policy recommendation Notes goal scope issue raised once

issue raised repeatedly

Cross- Addressing - - Language of the 2017 policy recommendations in particular cutting regional emphasises rural and remote community needs

disadvantage Early marriage / Social and health disadvantages of early pregnancy are well early pregnancy understood. TLYP has called for stronger adolescent family

/ SRH planning and SRH in schools and community clinics, plus moral leadership from parents and church. Human resource Better teaching standards and support to teachers are and recurring themes. TLYP also wants improved school facilities

infrastructure including WASH, and more accessible health services. Youth disability Disability needs are acknowledged in a general way; extent

of consultation with disability organisations and

representatives is unclear. Youth - Recommendations reflect TLYP focus on an active, productive

productivity and youth sector and call for competitions, training, music and

creativity sports to bring young people’s talents together. Civic Child rights - - Child rights awareness has not been a strong theme within participation TLYP consultation. Consultation Each year TLYP has called for more intuitive process in

consulting with youth representatives, especially at suco and municipal levels.

40 In 2017, three recommendations, though not duplications, were very similar in content; they have been merged for the purpose of this analysis, so 54 recommendations were used. - 96 -

Electoral - - Civic registration and voting rights were important in 2017, due participation to the timing just prior to the repeated national election.

Youth civic TLYP consistently notes gaps in young people’s capacity understanding of how policy works and how they can play a role in its influence. Education Corporal - - Corporal punishment in schools was raised within 2017 punishment recommendations on teacher quality; it was not mentioned in any year as a more general child protection issue. Reaching most - - TLYP in 2017 showed general awareness and support for the vulnerable youth needs of out-of-school and illiterate youth.

School retention - Though not a primary theme, MoE and community strategies

for helping students stay at school was among recommendations. Employment Labour laws, TLYP recommendations included tighter arrangements for and border laws immigrant workers, voiced most strongly after study visits to employability border areas in 2014 and 2015. Transport - TLYP made recommendations to improve roads and bridges

to support safe youth labour and value chain markets. Vocational TLYP noted each year that vocational training was lacking or opportunities for did not connect to appropriate local opportunities, and

youth highlighted agriculture, horticulture and fisheries. Youth business - Recommendations for entrepreneurship sought to match and

development coordination capacity of young people with business

management and market development. Healthy Drugs, tobacco, - TLYP recommended youth-focused public health campaigns

lifestyles alcohol to address uptake of detrimental habits. HIV / TB - TLYP in 2017 responded to increased overall concerns on HIV

prevalence. TB treatment for youth was also raised.

Environment - TLYP environmental focus has been on water management and infrastructure pollution.

Food and - TLYP raised food and nutrition issues generally rather than

nutrition connecting them to youth policy. Online safety - - Raised strongly in 2017, after safe internet usage was and protection introduced to the youth parliament training curriculum.

Water, - - TLYP in 2017 raised skin infection as an issue for young sanitation, people, especially in remote areas.

hygiene Violence and Gender - - Equality and protection components of gender are not crime strongly represented in TLYP recommendations. Police - TLYP recommendations aim to fill capacity and accountability

accountability, gaps in current police services for young people.

information, enforcement Strengthening - - In 2017, TLYP gave several recommendations to increase community community role (teachers, parents and church) in positive

support social influence. Strengthening - Reach and influence of youth organisations has been a minor

youth but recurring theme in TLYP recommendations. organisations

- 97 -

Annex 15: Survey results, detailed Personal skills and attributes, before/after TLYP

Before After Accept- Very Accept- Very Weak able Strong strong Weak able Strong strong (score (score (score (score (score (score (score (score Before After % = 1) = 2) =3) =4) = 1) = 2) =3) =4) (average) (average) increase My confidence and knowledge 18 15 3 3 0 2 19 17 1.77 3.39 91.88% Others' confidence and knowledge 16 18 2 1 1 8 17 12 1.68 3.05 82.17% My civic interest 24 9 4 1 0 7 21 10 1.53 3.08 101.72% Others' civic interest 21 11 5 0 1 14 18 4 1.57 2.68 70.69% My interest in youth issues 17 11 5 4 0 0 12 27 1.89 3.69 95.16% Others' interest in youth issues 17 15 4 2 0 8 23 7 1.76 2.97 68.66% My employability 23 8 2 2 0 14 14 9 1.51 2.86 89.19% Others' employability 16 15 5 0 0 16 14 3 1.69 2.61 53.80% My outlook and attitude 7 23 4 3 0 1 20 18 2.08 3.44 65.10% Others' outlook 7 11 2 0 0 4 14 2 1.75 2.90 65.71% My leadership among youth 10 19 5 3 0 4 19 16 2.03 3.31 63.18% Others' leadership 15 17 5 1 1 7 23 6 1.79 2.92 63.12%

Activities led by TLYP members

Yes No % yes- yes- yes- 2010 2013 2015 Training in community 46 2 96% 100% 92% 92% Shared study visit results 30 18 63% 73% 69% 38% Shared materials 32 16 67% 86% 62% 38% Made a proposal for community work 26 22 54% 59% 69% 31% Worked in community or NGO 30 18 63% 64% 69% 54% Campaigned on a youth issue 35 13 73% 91% 62% 54%

Support from family, school, peers, Secretariat (note, only 21 answers on this question, means disaggregation by TLYP cycle is not meaningful) Not Very Not Very support- Support- support- support- Support- support- Preparing ive ive ive ive ive ive / applying (score (score = (score = Average During (score (score = (score = Average for TLYP =1) 2) 3) (max. 3) TLYP =1) 2) 3) (max. 3) Family 1 6 14 2.62 Family 0 6 16 2.73 School 2 7 12 2.48 School 1 8 13 2.55 Peers 4 3 14 2.48 Peers 2 7 13 2.5 Secretariat 0 3 18 2.86 Secretariat 0 5 17 2.78

- 98 -

Rating of TLYP activities

Average Very Forgot/ TOTAL (max = Not useful Useful useful Highlight didn't do ANSWERS 3) Study visit - my own municipality 2 12 19 0 10 43 2.52 Study visit - another municipality 1 14 19 1 9 44 2.54 Study visit – international 4 3 9 0 15 31 2.31 Inauguration ceremony 2 15 18 2 5 42 2.49 Plenary sessions 0 15 31 0 0 46 2.67 Municipal consultations 1 6 6 2 3 18 2.47

Rating of TLYP training

Not Very Highlight useful Useful useful (score (score = (score (score remains as Forgot/ TOTAL average 1) = 2) =3) a 3) didn't do ANSWERS (max. 3) Life skills 3 14 26 0 5 48 2.53 Social analysis 1 20 24 1 2 48 2.52 Human rights / child rights 0 18 25 1 3 47 2.59 Civic education 0 18 29 0 0 47 2.62 Leadership 0 18 29 0 0 47 2.62 Gender equality 1 20 20 1 6 48 2.48 Proposal writing 4 9 7 2 7 29 2.23 Public debate 0 10 20 1 0 31 2.68 Public speaking 0 9 21 0 0 30 2.7 Environment 2 13 17 0 0 32 2.47 Journalism 1 11 7 1 8 28 2.35 Anti-corruption 1 14 12 0 2 29 2.41

Rating TLYP training, disaggregated by year:

2010 2013 2015 Life skills 2.44 2.46 2.75 Social analysis 2.48 2.62 2.50 Human rights / child rights 2.63 2.62 2.50 Civic education 2.64 2.46 2.75 Leadership 2.55 2.62 2.75 Gender equality 2.42 2.69 2.30 Proposal writing 2.23 2.50 2.14 Public debate 2.46 2.78 2.89 Public speaking 2.57 2.75 2.88 Environment 2.56 2.50 2.25 Journalism 2.22 2.20 2.67 Anti-corruption 2.43 2.33 2.43

- 99 -

Use of training

used in USED didn't used trained community used in USED WITH use myself others work advocacy SUMMARY MYSELF OTHERS Life skills 12% 80% 56% 52% 8% Life skills 80% 56% Social analysis 32% 68% 28% 32% 16% Social analysis 68% 32% Human rights / Human rights / child rights 28% 56% 36% 40% 16% child rights 56% 40% Civic education 12% 76% 64% 52% 12% Civic education 76% 64% Leadership 4% 76% 64% 52% 24% Leadership 76% 64% Gender Gender equality 28% 72% 40% 44% 20% equality 72% 44% Proposal Proposal writing 28% 64% 20% 32% 8% writing 64% 32% Public debate 8% 92% 48% 44% 28% Public debate 92% 48% Public Public speaking 0% 88% 56% 64% 36% speaking 88% 56% Environment 16% 64% 40% 44% 16% Environment 64% 44% Journalism 48% 44% 20% 24% 4% Journalism 44% 24% Anti-corruption 44% 44% 24% 16% 20% Anti-corruption 44% 24%

Analysis: Value of training and activities Respondents scored their training and activities on a scale of one to three: not useful, useful or very useful41. They were also asked how they had used the information they received in training. Results show that public speaking and leadership have been consistently valued and used, which was confirmed through discussions and focus groups with youth parliamentarians. Young people discussed their obligation to become more confident and to speak out so that their age group could be represented.

While all training received high scores overall, the use of ‘not useful’ as a rating was used most in life skills (three respondents, or around 7% of total) and in proposal writing (two respondents, around 5%). At the same time, proposal writing was the only course to receive more than one ‘highlight’, which indicates the different needs, interests and learning styles of TLYP members.

Gender and anti-corruption knowledge were least likely to be passed on or used by trainees. Every respondent had used their public speaking training, including strong usage in community work and advocacy.

The graphs below show areas of training, arranged by score from most useful to least useful, as well as proportion of respondents who used their training in different ways: for themselves (at work or school); to train others; to conduct community work; and to conduct advocacy. Using

41 A fourth rating, ‘a highlight’, was also offered but not used sufficiently to add weight to findings; for instance, two respondents rated Proposal Writing a highlight, but a further two rated it as ‘not useful’ and trainees have used proposal writing skills less than any other. - 100 - the skill directly was always the most likely application, and advocacy usually the least likely, which matches with other reflection and survey questions on the outcomes of training for youth parliamentarians.

Training curriculum is roughly comparable over the three cycles; however, trainers changed, with first cycle training often handled through outsourcing – for instance, UNICEF on Life Skills and Alola Foundation on Gender Equality – but all training conducted by the APFTL by the third cycle.

Enthusiasm for all training is strongest in respondents from the most recent cycle. Discussion with young people suggested a few reasons for this:  As trainers, Alumni from 2010 had hindsight in thinking about how to share information;  Consistency of trainers assisted with engagement and ‘hitting the ground running’ for the current parliamentary group;  The age group (12 – 17) makes it less likely that current TLYP members will critique.

However, current TLYP members were aware that fewer resources had been spent on them; for instance, no study visits and less direct support on proposing community work. This imbalance came up in all interviews with current members.

How training was used, all years

- 101 -

- 102 -

- 103 -

Annex 16: Case studies Eskola Sekundária Públika Nino Koni Santana Schools in Timor-Leste are not given a budget for extra-curricular activities, and are dependent on the proximity and capacity of local government and NGO programmes for learning outside the classroom. Without these services, most students would have no opportunity to learn about key issues affecting their futures: adolescent life skills and healthy choices, sexual and reproductive health, leadership and civic participation rights, environmental sustainability, gender equality, peacebuilding and the democratic process.

Eskola Sekundária Públika Nino Koni Santana, located centrally in Ermera municipality, is fairly typical in this regard. It has around 600 students, in the upper secondary range (Year 10 to 12). The principal does what he can to create extra- curricular opportunities for his students: for instance, nature walks, disaster risk reduction and horticulture projects. He also makes sure students from the school take part in Ministry of Education quizzes and public speaking competitions, which increase students’ Principal of Eskola Sekundária Públika Nino Koni Santana confidence, and in municipal sports which enhance fitness, leadership and social interaction. Beyond this, he depends on external providers for more formal training and knowledge building.

The school has used the services of APFTL trainers three times in the past. While he cannot select his own topics for training opportunities, the principal is always grateful to host APFTL and trusts them to be promoting appropriate and needed youth messages.

The Student Council at the school, made up equally of girls and boys, helps to organise and enrol students for external training. Most providers use a cascading approach to adolescent and youth courses, training one or two representatives per class on the understanding that they will share what they have learned with others. Selection to attend is seen as a reward for students who have shown good interest and aptitude, so some students actively and repeatedly participate in extracurricular opportunities while others receive information mainly second-hand.

- 104 -

The students saw some problems in effectiveness of peer-based cascading information. One student said: “It would be good if training took place at the school and involved everyone, regularly. At the moment, even if the whole school learned something last year, the new (Year 10) students this year have missed it.” Students involved with the Student Council confirmed and appreciated APFTL training opportunities Prompting each other’s memories, the Student Council were able to recall 13 training providers working with their school over the last two years, and covering 9 topics: three in sexual and reproductive health, two in gender including gender- based violence, two in youth peacebuilding, plus youth rights and law, disability inclusion, first aid, online safety and democracy/civic participation. The final two were APFTL courses, with online safety funded through UNICEF. Student Council members also remembered APFTL giving out information on democracy and election participation through church and other community gatherings, in the lead up to the 2018 repeat election.

Out of all extracurricular training available, the students were asked to choose just one which had been of most value to them. While every answer was different, different aspects of health were raised most often, followed by youth rights, law and democratic participation.

The group of Student Council members knew the basics of TLYP but had never thought about applying and were unsure of the benefits. Last year, a TLYP member was attending this school, but had since moved on; nobody could confirm that she had instigated any training or other activities in her time at the school. The principal was also hesitant on the details of TLYP and its relevance to his school: “We’ve known about it since Mr Ramos Horta was president, and I’m very supportive of it, because this nation needs quality leaders for the future. But we need more information on it, especially on how children become members, otherwise we don’t benefit from it here.”

- 105 -

Youth leadership in challenging times: Martinho’s story Martinho is from Tilomar, in Covalima municipality. He has grown up in a difficult time for Timor- Leste and remembers the pain and violence during Timor-Leste’s bid for independence. Even now, the legacy of that time affects opportunities for children and youth.

“Because of the history of Timor- Leste’s move to independence, it was normal that everyone aged 15 and above should already be working. Though we were still children, we really only applied the definition to children below 12. Economically it was hard for my family if I stayed at school. I never thought about university.” Martinho with his parents at home in Covalima; he is the first person in his family to attend university (photo supplied)

Martinho was part of the first cycle of Timor-Leste Youth Parliament (TLYP), which brought together 130 promising young people aged between 12 and 17 to represent the children and youth of their local area. All of Timor-Leste was covered, with one girl and one boy selected from every electorate nationwide. He heard about the opportunity from the leader of the local youth centre, who was also involved with Timor-Leste’s National Youth Council.

“I was quite active at the youth centre as a peer educator, and my church also had youth activities. If you are young and proactive in Timor-Leste you can find these opportunities, but not everyone is. My mentor at the National Youth Council was interested in the TLYP at first, but was too old to apply, so he suggested I put in an application.”

At first Martinho thought the initiative was NGO-based and local, as much of the National Youth Council work is based on local grants. When he found out it was a national, government-run organisation, he became even more interested. He thought maybe it was a way to visit Dili, something he’d never previously imagined being able to do.

“Two candidates from each suco were selected, so eight total for two positions. We presented in public. I knew that at least three of them were active in more organisations than I was. But I got through. I think some of the girls were probably stronger than me, but they could only choose one of each.”

- 106 -

Martinho describes the mandate of TLYP not only to represent children of Timor-Leste but also to prepare future leaders to take up responsible roles. As one of the oldest members in the first TLYP, Martinho noted some different levels of knowledge to start with, but found that this decreased as the training began. “Everyone went through the same process of selection, and when we started to learn, for instance, public speaking, we found similar maturity across age groups. We felt as though we were the best.”

Martinho’s personal journey has been strongly influenced by his time with TLYP. Not only achieving his goal to visit Dili, he also joined an international exposure visit to Indonesia. Learning about different national and international perspectives expanded his world view. He took part in the first ever parliamentary plenary, attended by national heroes Xanana Gusmao and Jose Ramos Horta.

“If I hadn’t joined TLYP, maybe I would never have even seen Dili. I would probably not have gone to university, or become who I am today. I was the first from my family to make it to university; they couldn’t really afford it but made efforts to provide small amounts, and I had the life skills and networks to pay my own way for the rest. They were very proud, and so was my school. They wanted me to be a voice for youth, to access and make the most of opportunities.”

Martinho has now finished university and is working for the government in Dili. He also continues to volunteer with the TLYP Alumni Association, mainly in mentoring current TLYP members but also in public engagement and awareness work on youth issues.

“I don’t know if my experience is representative of all TLYP members. Only some of us remain active. We all continue to face our own challenges, and only some of us overcome them. I will say though, that my story is representative of what is possible for young people if they are fortunate enough to be chosen for TLYP.”

- 107 -

Gender advocate begins her journey with TLYP At 22, Maria do Ceu Gusmao is a gender advocate driving a national campaign to reduce stigma against adolescent pregnancy: specifically, to create policy that mandates re-entry for young mothers. She is now working for UNICEF Timor-Leste to promote child rights. Maria believes that education is a right and that being pregnant or a young mother is no reason to drop out from school. No national policy exists either way, and so it is left to the discretion of schools to allow re-entry of teenage mothers: a rights challenge that Maria believes can be resolved by stricter policy.

Maria do Ceu, youth advocate for gender She attributes her motivation to three key influences in rights in Timor-Leste (photo supplied) her life: firstly, her membership in Timor-Leste Youth Parliament, then her increased gender rights exposure through local NGO Alola Foundation; lastly international exposure through regional youth CEDAW discussions. All have been pivotal, but began with her curiosity and determination as an outgoing 14 year old in rural Timor-Leste.

“I first heard about TLYP in 2010. It was starting for the first time. In Year 9, I was old enough to apply, just, and I was very keen to try out, but I’d recently moved to Liquica municipality and I didn’t know the teachers well enough to ask for their support. It was hard to think about convincing people I didn’t know yet that I could win a selection process. So, I waited and tried again in 2013.”

Maria had just turned 17 at the time and was in her final year at school. She describes the recruitment process:

“It was very competitive. Applications were open to anyone but you had to do many things – go to the village leader, introduce yourself and get the information, then submit the birth certificate and fill out a good application. Then, if they select your documents, you go to the suco elections to make a public speaking presentation on your vision and mission. I was certainly nervous presenting but I had friends who had been members, and they coached me, helped me understand the difference between concepts of dreams and visions. They also came to watch me, supported me and voted for me. Finally, once I got through the suco competition, we had the exam.”

- 108 -

Maria says her school was supportive and proud of her, though a little perplexed about her sudden interest in civic process. “I had been very focused on science and ecology beforehand, and one teacher asked me why I wanted to learn about politics suddenly.”

But Maria found TLYP was about much more than politics. “The life skills component was so important, helping us to know about life, respecting ourselves. Even now when I am faced with a decision I still use the same techniques: clarifying what I need versus what I want, and weighing the balance.”

TLYP was a busy time for Maria, especially in her final year at school and first year at university, and she didn’t take on any additional peer education duties. “I shared what I learned with family and friends, but it wasn’t until I finished and joined the Alumni that I started to help with youth training. I joined an Alumni colleague to conduct training on reproductive health in Covalima (away from home). Then we started to make applications to the suco for community funding. Working together on awareness work was easier and more rewarding. We do it less now because some of us are married and can’t give the time, but we reached at least 90 people whenever we did it.”

As a TLYP member, Maria was invited to join Alola’s youth gender training in 2013. The training also selected promising young women from National Youth Council and other recognised youth groups. Maria says that though TLYP gave her basic insight into rights issues, the intensive course offered by Alola took things much further.

“We learned a lot of inspiring things through that training: gender equality, human rights, children’s rights, advocacy techniques. We went on to meet with policy makers specific to women’s issues in Timor-Leste.”

Maria and her TLYP associates were then invited by UN Women to present the situation for adolescent girls in Timor-Leste at an international conference. They talked about reducing the stigma of early pregnancy in the context of Timor-Leste, given its prevalence, and how they could use CEDAW to establish a formal schools re-entry policy for girls after they give birth.

Maria keeps trying. “We have been a little successful but have not yet made the policy. This year has not been the right time to advocate for policy change due to the political uncertainty. But we plan to try again next year. In the meantime we continue our sexual health campaign – we try to explain that any young person is free to do what they want but with knowledge, respect for women, and equal responsibility for consequences.”

- 109 -

Annex 17: Evaluation Matrix (Evaluation and Evidence Framework)

TABLE 1: EVALUATION MATRIX (EVALUATION AND EVIDENCE FRAMEWORK) Questions Sources (document Sources (primary) NB: questions based on specific learning requests by UNICEF or other stakeholders during review) the inception phase consultation appear in italicised bold font 1. Relevance: The extent to which the programme is suited to the priorities and Government of Timor- Youth member survey policies of the target group, country, UNICEF, and global SDG agenda. Leste youth policy Semi-structured 1.1 To what extent are programme objectives still valid, and activities consistent with SDG literature interviews with: these? (including UNICEF current / former youth 1.2 How well is the programme aligned to UNICEF’s priorities/policies/ reform agendas and other agencies) members (including 2030 SDG agenda), and with government’s national youth policy including Analysis of TLYP NGO partners Govt. Resolution No.23/2009 (Youth Parliament)? member Government 1.3 How visible and relevant are original principles of democratic election and youth- demographics representatives driven agendas? Other community 1.4 To what extent do the role model and multiplier aspects envisaged through members (e.g. TLYP remain relevant? parents, teachers) 1.5 Are outcomes and measures identified in the 2012 TLYP strategy still valid? 2. Efficiency: measures the outputs -- qualitative and quantitative -- in relation to the Budget and As above inputs. outgoings, at least 2.1 Were activities cost-efficient in achieving the program’s intended goal? two years. 2.2 Were objectives achieved on time through the inputs? Partner and 2.3 How well does the programme connect with and complement other youth programs and government initiatives in Timor-Leste (and can this be done better?) summaries of other 2.4 Can a value-for-money proposition be made, that the results evident for young youth/ peacebuilding people at community level (indirect beneficiaries) justify investment in youth initiatives. parliamentarians (direct beneficiaries)? 3. Effectiveness: the extent to which the objectives have been achieved and the major UNICEF and partner As above factors influencing the achievement and non-achievement of objectives. programme 3.1 To what extent have the planned results been achieved (quantitative and qualitative)? documentation, TLYP 3.2 How effective was UNICEF’s support to TLYP goals and process? 3.3 To what extent did TLYP advocacy influence government decisions concerning youth and adolescent issues? 3.4 What components (training modules, orientation, exposure trips) are of most value to young people for a/ personal empowerment and b/ helping others? 4. Impact: The positive and negative changes produced by an intervention/ Monitoring records As above programme, directly or indirectly, intended or unintended. Media / social media 4.1 What has happened as a result of the TLYP programme, and which activities have review contributed most to this impact? Alumni Association 4.2 What unexpected, including negative, effects are evident for youth parliamentarians? documents and 4.3 What progress can be seen as a result of TLYP community engagement towards the agreements five strategic outcomes of the 2016 National Youth Policy? Peacebuilding 4.4 Is there a correlation between TLYP community engagement and social documentation, GIZ cohesion/peacebuilding outcomes? If so, how can this result be replicated and scaled?

- 110 -

5. Human Rights and Equity: The ways and extent to which the intervention contributes to ‘civil, cultural, economic, political and social rights inherent to all As above As above human beings, regardless of one’s nationality, place of residence, sex, sexual orientation, national or ethnic origin, colour, disability, religion, language etc.’42 5.1 To what extent has TLYP reflected its original human rights and equality principles: gender, socio-economic background, geographical disparities (urban/rural), child and youth participation, and participation of children and young people with disabilities? 5.2 What UNICEF strategies/ approaches have been most efficient in influencing improvements to child and adolescents’ participation rights? 5.3 How did the youth parliament programme benefit other youth who are not TLYP members, in terms of a/ local policies and behaviours and b/ personal capacity and resilience? 5.4 How successful was the programme in supporting the most vulnerable children and adolescents in communities? 5.5 To what extent, and how, has independence and protection of youth members been assured (do no harm safeguards)? 5.6 Are young people’s voices and opinions being heard and respected more than previously? To what extent is this linked to TLYP? 6. Sustainability: Whether the benefits of an activity are likely to continue without UNICEF budget As above donor funding. allocations, five years 6.1 To what extent did the benefits continue after donor funding ceased? Government budget 6.2 What were the major factors which influenced the achievement or non-achievement of allocations, at least sustainability of the programme or project? two years 6.3 What is the trend of state budget allocated for TLYP programme in the past, and Alumni Association expected trend in the future? documents and 6.4 Has the programme been successful in leveraging governments’ political will and agreements. financial resources for child and adolescent protection? 6.5 What is the significance of the Alumni Association to the sustainability of Youth Parliament? 6.6 What is the best role for others (NGO partners, local institutions) to play in Youth Parliament and Alumni activities?

42 UNEG 2011: p.11 - 111 -

Annex 18: Outcome-level framework: The Theory of Change .

Figure 21: Retrospective Theory of Change (Consultant, in collaboration with government, UNICEF, APFTL)

- 112 -

Annex 19: Output-level framework: the Results Matrix

TABLE 2: RESULTS MATRIX (based on 2012 TLYP Strategic Plan)

Level Results Statement Indicator measure/ statement Result Confiden Data source (2012 TLYP Strategic Plan) (2012 TLYP Strategic Plan) ce of finding Objective 1: Influence through advocacy and discussion Output TLYP has a strong, Yes/No Yes High Consensus from stakeholder transparent and independent interviews governance structure Output Youth Parliament raises the Youth Parliament are Yes High Stakeholder interviews; review voices of young people providing annual policy briefs of policy briefs formally to policy makers to appropriate government ministries. Output Alumni take responsible roles Proportion of former youth (estimated) Medium Self-reported in survey; in youth participation and members active in civil 33% estimated by APFTL policy, including at national society/NGO roles. secretariat level. Outcome Young people have Number of reforms (or None Low Stakeholder interviews; online contributed to reform across progress towards reforms) identified survey results (no examples key government portfolios. evident as a result of TLYP identified) or alumni actions. Outcome Young people feel connected Proportion of former youth (estimated) Medium Online survey agreement with to Government process. members who feel connected 89% statement “Plenary to Government process. recommendations contribute to youth policy”; stakeholder interviews with current and former TLYP members Objective 2: Empower through education, training and practical experience Output Activities are well Yes/no Mainly High Secretariat records; online coordinated and aligned. survey; stakeholder interviews Outcome Young people actively source Proportion of youth members 100% Self: Multiple questions in online employment, are skilled and reporting they/their peers are Medium survey; stakeholder interview employable. more skilled and employable. Peer: Low Outcome Local resolutions are Number of local resolutions None Low Stakeholder interviews; online achieved with assistance reported by youth members. identified survey results (no examples from skilled young people. identified) Outcome Young people are included Proportion of youth members None Medium Stakeholder interviews with more in decisions affecting reporting they/ their peers are identified current and former TLYP them. being included more in members community consultations. Objective 3: Increase engagement and participation for young people in decision making Output Youth Members are more Youth members self-report 100% High Online survey results; confident and equipped to increased confidence, stakeholder interviews; lead. knowledge, support and observation resource. Output Young people in all districts # / proportion districts with 100% High TLYP secretariat records are nominating to join TLYP. youth parliamentarians. Objective 4: Lead by example Output Youth members conduct # activities involving children Estimate Medium Online survey; APFTL records; child / youth participation and young people conducted 92% TLYP but not strongly backed up activities independently in by youth members. members through field-based communities. conducted consultations in schools, community communities. events Outcome Input from TLYP is sought # examples of TLYP Municipal High Municipal consultation records; out and valued. supporting political or civil consultation stakeholder interviews; other society process outside s 2015 examples evident especially in annual plenary. Dili.

- 113 -

Annex 20: Summary of youth providers and their services (as identified by school/youth centre interviewees):

 Red Cross (Cruz Vermelha de Timor-Leste): first aid training to schools; opportunities to participate in community events (clean-up days, youth summits).  Scouts (União Nacional dos Escuteiros de Timor-Leste): fitness, sports and community events; municipal and national meetings.  MOFFE, Alola, Marie Stopes: gender / SRH partners offering different levels of extra- curricular information and life skills.  CNC/Chega!: A global peacebuilding movement with strong youth engagement strategies in Timor-Leste.  ChildFund: in three out of six schools, with leadership and life skills training; however, respondents were unclear on the overall goal of the programme.  RYPEN: for two schools in Dili, a leadership programme supported by Australian Rotary International.

It is likely that other local and international NGOs are also operating similar programmes, though not in the sites visited.

- 114 -