The Evaluation of UNFPA Thailand’s 9th Country Programme (CP9)

By: Supattra Srivanichakorn, MD, MPH Kerry Richter, PhD Bang-on Theptein, PhD

For: The United Nations Population Fund (UNFPA)

March, 2011

TABLE OF CONTENTS

Page

ACRONYMS ...... iii Executive summary ...... vi Part 1: Introduction ...... 1 I. Background ...... 1 II. Evaluation Purpose and Methodology ...... 2 Part 2: Findings ...... 9 I. Reproductive Health (RH) Programme ...... 9 1.1. UNFPA’s achievements and the level of effectiveness, efficiency and relevance of strategies, key actions of RH programme ...... 10 1.2 UNFPA’s efforts to develop managerial capacity to improve the quality of RH services ...... 24 1.3 UNFPA RH programme linkages with national policies and priorities ...... 26 1.4 Factors contributing to the RH programme’s achievements and shortcomings ...... 27 1.5 Cross-cutting Issue of Gender and Rights in RH programme ...... 28 II. Population and Development (PD) Programme ...... 30 2.1. South to South Cooperation in Reproductive Health and Population Development 30 2.2 Monitoring and Strengthening Statistical Capacity (NSO) ...... 32 2.3. Evidence-based Responsiveness to the Emerging Challenges of Population Ageing in Thailand (OP) ...... 39 Part 3: Conclusions and Recommendations ...... 47 I. Key Findings & Recommendations for the RH Programme ...... 47 II. Recommendations for the Population and Development Programme ...... 50 III. Programme Management ...... 53 Annex ...... 56 Annex 1: Outcome and Output Indicators for UNFPA 9th Country Programme ...... 57 Annex 2: Budget, Expenditures and Implementation Rate of Projects under CP9 .. 64 Annex 3: List of Documents Reviewed ...... 66 Annex 4: List of Key Informants ...... 72 Annex 5: Methodologies to Answer the Evaluation Questions ...... 76 Annex 6: Evaluation Criteria and Related Questions ...... 89 Annex 7: Interview Guides ...... 90 Annex 8: Terms of Reference ...... 96 i | Page UNFPA Thailand’s CP9 Evaluation Report as of May 30, 2011

LIST of TABLES

Page

Table 1: Summary of South to South Project Activities 2007-2010…………………… 32

Table 2: Proportion of NSO Project Funds Budgeted and Spent by Major Strategy and Output, 2007-2010...... 35

LIST of FIGURES

Page

Figure 1: Overall Programme Evaluation Framework...... 4

Figure 2: Key Achievements of the NSO Project 2007-2011...... 37

Figure 3: Key achievements of the OP project...... 42

Figure 4: Budget and Expenditures by Year and Implementing Agency, OP project 2007-2010...... 45

Figure 5: Percent Expenditure by Type of Activity, OP Project 2007-2010...... 46

ii | Page UNFPA Thailand’s CP9 Evaluation Report as of May 30, 2011

ACRONYMS

AIHD ASEAN Institute for Health Development, Mahidol University ARH Adolescent Reproductive Health AWP Annual Work Plan BATS Bureau of AIDS TB and STIs CCM Country Coordinating Mechanism CCP Comprehensive Condom Program CM Chiang Mai CMU Chiang Mai University CP9 UNFPA Thailand’s 9th Country Programme CPAP Country Programme Action Plan CPS College of Population Studies, Chulalongkorn University DDC Department of Disease Control DHC District Health Office DOH Department of Health DOM Department of Medical Services EMOC Emergency of Obstetric Care EU European Union FAO Food and Agriculture Organization of the United Nations FC Female Condom FON/CMU Faculty of Nursing, Chiang Mai University FOPDEV Foundation of Older Persons’ Development FSW Female Sex Worker GFATM round RCC Global Fund on AIDS, TB and Malaria round Rolling Continuation Channel GIS Geographic Information System GO Government Organization HAI Help Age International HRD Human Resource Development ICPD International Conference on Population and Development IPC International Programs Centre IPSR Institute for Population and Social Research, Mahidol University LAO Local Administration Organization LP Lampang M&E Monitoring and Evaluation MCD Management Capacity Development MDGs Millennium Development Goals MFA Ministry of Foreign Affairs MHS Mae Hong Son MICS Middle Income Country Study iii | Page UNFPA Thailand’s CP9 Evaluation Report as of May 30, 2011

MMS Making Motherhood Safer MOI Ministry of Interior MOL Ministry of Labour MOPH Ministry of Public Health MOU Memorandum of Understanding MPM Men as Partner in Maternal Health MSDHS Ministry of Social Development and Human Security MVHV Maternal Village Health Volunteers NGO Non-Governmental Organization NHSO National Health Security Office NRT Narathiwat NSO National Statistical Office OP Older Persons OPTA Office of Population Technical Assistance Team OPW Older Persons Watch PATH The Program for Appropriate Technology in Health PBRI Praboromrajanok Institute for Health Workforce Development PD(S) Population and Development PHC Population and Housing Census PHO Provincial Health Office PLA Participatory Learning Action PLWA People Living With AIDS or Persons Living With AIDS PLHIV People Living With HIV/AIDS RBM Results-based Management RH Reproductive Health RTG Royal Thai Government SCCT Senior Citizen of Council of Thailand SK Songkhla S-S South to South Cooperation SSRH Strengthening Sexual and Reproductive Health STI Sexual Transmission Infection STRONG Strengthening SRH/HIV services for vulnerable groups in decentralization context SW Sex Worker SWING Service Workers In Group Foundation SWOT Strengths, Weaknesses, Opportunities, and Threats TAO Tambon Administration Organization TBAs Traditional Birth Attendants ThaiHealth (THPF) Thai Health Promotion Foundation TICA Thailand International Development Cooperation Agency TOT Training of Trainers UN United Nations UNAIDS United Nations Programme on HIV/AIDS UNDP United Nations Development Programme iv | Page UNFPA Thailand’s CP9 Evaluation Report as of May 30, 2011

UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund UNPAF United Nations Partnership Framework USAID United States Agency for International Development VHV Village Health Volunteer VHW Village Health Worker WCC Well Child Clinic WPPHC World Programme on Population and Housing Censuses YAP Youth Advisory Panel YSRH Youth Sexual and Reproductive Health

v | Page UNFPA Thailand’s CP9 Evaluation Report as of May 30, 2011

Executive summary

The ASEAN Institute for Health Development (AIHD) and the Institute for Population and Social Research (IPSR), Mahidol University was appointed to conduct an evaluation of UNFPA’s 9th Country Programme in Thailand (CP9). There are two main programme components with gender as a cross-cutting issue including : (1) Reproductive Health (RH) comprising of five projects i.e. Reproductive Rights (RR), Men as Partners in Maternal Health (MPM), Making Motherhood Safer (MMS), Youth Sexual Reproductive Health (YSRH), and Strengthening Sexually Reproductive Health/AIDs for Vulnerable Groups (STRONG); and (2) Population and Development, comprised of three projects i.e. South to South Cooperation (S-S), Monitoring and Strengthening Statistical Capacity, and Evidence- based Responsiveness to the Emerging Challenges of the Population Ageing in Thailand (OP).

Overall the CP9 programme was designed in line with Millennium Development Goals (MDG) 5&6 and International Conference on Population and Development (ICPD) guidelines, as well as with national priorities within the local context of Thailand. The programme was planned with key stakeholder involvement and based on evidence drawn from situation analyses, the 10th National Economic and Social Development, and other relevant exercises.

Reproductive Health:

RH programmes were built on evidence that the involvement of communities, leaders and volunteers is a successful strategic factor to improve accessibility to RH information and services. RH services and HIV/STIs prevention in implementing areas have been strengthened by trained health providers to increase their awareness and broaden their perspectives to work with various groups of clients with respect to their gender and cultural sensitivity. Multiple modalities of media and IEC materials were developed to improve communication channels for each specific vulnerable group.

As a result, there was an improvement in the availability of RH information and services in implementing areas for vulnerable groups. The MPM model was developed through the recognition that male involvement is an effective strategy for HIV prevention as well as for the improvement of the quality of maternal health services. Lessons and experiences learned from the MPM model implementation can be an effective policy advocacy tool to improve HIV prevention and maternal health services nationwide. The Youth Advisory Panel (YAP).has given youth greater involvement in reproductive health policy development and given peer educators, teachers and health care providers greater awareness of the specific needs of youth. However, the behaviour change outcome of condom use did not improve which due to the limited and inadequate coverage of information and services. Networking of GOs and NGOs working in RH was achieved through creating interesting websites using web linking. While website visitor numbers increased significantly, it is premature to assume that safe sex practices improved among youth who accessed the information and services vi | Page UNFPA Thailand’s CP9 Evaluation Report as of May 30, 2011

(i.e. counselling, testing and treatment). At this time, the evaluation team also notes the drawback that internet access is quite limited in the remote areas.

Programmes for sex workers included the establishment of STI clinics in various settings for various groups of sex workers, including outreach services and “drop-in centres”. It appears that these have increased accessibility for certain groups of sex workers. Also, Comprehensive Condom Programming was developed to improve the management of the condom programme; a committee was set up to oversee this system and a strategic plan for condom programming was drafted. But endorsement by the local government to function as one of the suppliers is still in the training process under the developed guidelines. Moreover, the HIV prevention programme for sex workers under UNFPA support is linked to the plans of other donors such as GFATM, which is providing support to improve services for sex workers and reposition the image of the condom so that it is not always associated with commercial sex. This resulted in the expansion of condom distribution to wider areas.

To increase utilization of reproductive health information in underserved areas, the UNFPA has supported programme implementation in two pilot areas. Traditional birth attendants and Maternal and Child Health volunteers have attained greater knowledge and skills on Emergency Obstetric Care. Referral services network was strengthened in order to increase access to RH services of the target population. Health staff and health managers at district and sub-district levels were assisted to improve their information management systems, while considering cultures, norms and influential factors of the mothers and their families. Mobile clinics in communities were introduced and a strategy to increase involvement of Muslim religious leaders in the unrest and cultural sensitive areas initiated to increase access by vulnerable groups to RH information and services. However, to ensure the sustainability of the programme there is a need to advocate for the investment of extra resources for remuneration of MCH community workers. Alternative resources may need to be identified to substitute these expenses when the project ends. Lessons learned in the implementing areas can be synthesized for policy formulation to improve access to RH services in culturally sensitive and underserved areas.

Gender and Reproductive Rights

The CP9 contained a separate umbrella project devoted to promoting Reproductive Rights and Policy Advocacy (RR). The project was primarily implemented by the Bureau of Reproductive Health, but some activities involving policy advocacy on both reproductive health and population development issues were implemented directly by UNFPA. The Thai government has promoted a number of policy strategies, legislation, and working guidelines and has also conducted training for health care providers for their attitude change. Most of the projects were extremely effective in raising awareness of health service providers with respect to gender and cultural sensitivity of various vulnerable groups. Although there was evidence that freedom of choice and adequate information for decision making was offered to clients, the program is still oriented to a “service provider” perspective rather than truly geared to identifying the needs of potential clients.

Another separate initiative under the RH programme focused on the rights of PLHIV to access counseling and RH services. This was primarily implemented at the central level in Bangkok under the auspices of the Department of Health (DOH), MOPH. The Guidelines for Family Planning for PLHIV were developed by the Bureau of Reproductive Health, DOH. vii | Page UNFPA Thailand’s CP9 Evaluation Report as of May 30, 2011

One weak point of this initiative was that the peer educators were trained only once; in addition, DOH did not conduct any follow-up research or obtain any feedback from the peer educators.

Population Development :

This project was implemented under bilateral collaboration between the Thailand International Development Cooperation Agency (TICA), the Ministry of Foreign Affairs and the UNFPA. The key achievements are: 1) enhanced capacity of participants from Asia and Pacific countries on RH and PD related topics especially on Migration Health, Community Based Participation in Older Persons management programme, HIV/AIDS prevention in Youth, and Maternal health Care Service; 2) organised ICPD15 forum to exchange and update situations, knowledge and innovations on ICPD related issues especially Maternal Health, YSRH and Programmes for Older Persons; 3) raised training management capacity of the potential Thai RH/PD Training Institutes; 4) Mapped Thailand RH and PD training resources; 5) developed web-based S-S mechanism for knowledge sharing among Thailand and other countries on RH and PD related issues; 5) based on needs survey and expectations towards Thailand S-S Cooperation, revised Thailand and UNFPA S-S Cooperation strategy to be more long-term and sustainable. New modality of institution to institution capacity development has been developed and piloted on Maternal Health Care in Laos and Bhutan and Sexuality Education with five selected countries (Vietnam, Philippines, Indonesia, Cambodia and Bhutan). Shortcomings of the S-S programme are: the follow-up visits and reports were not conducted after the trainings; therefore it is difficult to assess the effectiveness of the project. Another shortcoming is that the official S-S Cooperation procedure of UNFPA and TICA present major administrative and managerial challenges. These delay the processes for implementing the programme, especially regarding the application process, overseas participant recruitment, and the budget management system.

Monitoring and Strengthening Statistical Capacity of the National Statistical Office

The national focal partner of this project is NSO. The project created a strengthened partnership between UNFPA and NSO, and also between NSO and technical experts from academic institutions. The project contributed to the availability of high quality data and to its use for planning and policy. The 2010 Population and Housing Census was a major focus for project activities and expenditures. Advocacy activities for the Census had high visibility, and capacity development activities were seen as valuable. However, some short- term efforts at technical assistance and capacity development did not achieve their objectives, and few activities included local and provincial staff. Support, through technical assistance, of two other surveys; the National Survey of Older People and the Reproductive Health Survey, was implemented through small-scale consultancies and dissemination activities. Strengthening of technical assistance through longer-term partnerships with academic institutions would provide more consistent support. Analysis and utilization of the Census data is of critical importance to achieve project outcomes in the next country plan.

2.3. Evidence-based Responsiveness to the Emerging Challenges of the Population Ageing in Thailand (OP)

Two key achievements under this project include: 1) A multidimensional and multi-sectoral model to raise awareness and provide evidence for advocacy on policy issues affecting older viii | Page UNFPA Thailand’s CP9 Evaluation Report as of May 30, 2011 persons. Concrete achievements of the model programme include health services, a high level of participation by older people in the community-based activities, and expansion to two other provinces. 2) Policy advocacy that contributed to several major policy decisions at the central level which included: adoption of long-term care as a priority issue for national policy; revision of the national social pension scheme from a means-tested older persons allowance to basic universal social pension; approval of the amendment of the Social Security Act, Article 40 to increase benefits for informal sector workers; approval of contributions from the central and local governments to community-based saving funds; and revision of the Second National Plan for Older Persons Policy. Advocacy activities included efforts by UNFPA’s country office to increase and influence the policy dialogue on older person’s issues, collaboration with national and local media to increase coverage on ageing issues, and the creation of older persons’ networks that participated in policy advocacy.

Factors contributing to the RH and PD programme’s achievements and shortcomings:

A project planning process that uses a “bottom-up approach”, according to the perspective of health service providers and PD stakeholders, has ensured that project implementers have a strong professional and personal interest in participating in project activities. The quality and relevance of training at the field implementation level contributes to health care providers being able to obtain new information, as well as acquire specific technical skills to help them become more effective service providers and/or project managers. The ability of the UNFPA to invite NGOs with a wide range of practical experience (e.g. PATH, SWING, HelpAge) to work as partners and/or to introduce new strategies and methodologies into the project implementation design have enhanced the programmes.

There have also been several factors that have led to shortcomings in expected RH Programme achievements. These include project-based management rather than integration as programmes. A clear design for the supporting system after UNFPA withdrawal has not been overseen from the beginning. Limited interaction with several key policy makers and/or national programme officers resulted in lower awareness of CP9 project objectives, achievements and valuable lessons learned that could have relevance and possible further application on a larger scale in Thailand. Attention to identifying key counterparts was limited to the public sector and specifically to the MOPH, rather than broadening partnerships to other important players from different ministries or from the private and/or NGO sectors.

ix | Page UNFPA Thailand’s CP9 Evaluation Report as of May 30, 2011

Part 1: Introduction

I. Background

• Thailand is among the countries that have had a short fertility transition. Thailand reached a replacement level of fertility in the early 1990s and although high contraceptive use (of approximately 70-80%) contributes to low fertility in Thailand, many women still face unwanted pregnancies. Due to a rapid decline in fertility and increasing life spans, Thailand is rapidly heading toward an ageing society with 10.7% of its population over 60 years of age. In addition, the annual growth rate of 3.7% is among the highest in the region. In 2009 the welfare assistance programme for indigent persons over 60 years became an entitlement for everyone older than 60. The government has introduced several initiatives to strengthen income security at old age, life-long education, and other health and social services for the elderly. As Thailand has emerged as a middle-income country, urban growth has also proceeded rapidly. This growth is the result of both long-standing migration patterns and new settlement trends. About one- third of the population now lives in urban areas. At the same time, international migration, both into and out of the country, is an important economic and social force. • UNFPA’s cooperation in Thailand commenced in 1971 shortly after the launch of the government’s National Family Planning Programme. This cooperation has evolved to meet changing needs in maternal and child health; improving access to reproductive health services including HIV/AIDS prevention; information, education and behavioural change communication; data management including surveys and censuses; population and development; human rights and gender equity; South-to-South cooperation, and emerging population issues such as migration and ageing. Cumulative assistance through 2010 is approximately $47 million, with support to over 100 collaborative efforts. • UNFPA’s Executive Board approved the 9th Country Programme (CP9) for Thailand in October 2006. It was developed as an integral part of the UN Partnership Framework (UNPAF) and was later harmonized to contribute to the achievement of UNFPA’s Strategic Plan (2008-2013) and the Millennium Development Goals. • The main components of the CP9 include: (a) Population and Development Strategies (PDS), including South-South cooperation, strengthening statistical capacity, and addressing the needs of the ageing population; and (b) Reproductive Health (RH), including maternal health, development of management capacity, adolescent reproductive health (ARH), and HIV/AIDS prevention. Gender is not a separate component but integrated into both PDS and RH. The Country Programme Action Plan of the CP9 was signed by Thailand International Development Cooperation Agency and UNFPA in January 2007 followed by the Annual Work Plans (AWPs). Four outcomes and five outputs were initially developed within the above-mentioned focus areas during 2007-2009 and then 8 outputs were developed after 2009. (Details of the outcome, output and project indicators are given in Annex 1).

1 | Page UNFPA Thailand’s CP9 Evaluation Report

• CP9 introduces RH services tailored to the needs of target populations in the reproductive age groups, especially pregnant women and youth in Lampang and Mae Hong Son in the North and Songkhla and Narathiwat in the South. On emerging population issues, a collaborative project with representatives from line ministries, Chiang Mai University, and an NGO (Help Age International) for policy advocacy and model development to improve the quality of life of older persons was introduced in Chiang Mai, Lampang and Mae Hong Son in collaboration with a national policy dialogue and extension of social security coverage, welfare and comprehensive long-term care for older persons. On strengthening statistical capacity, CP9 supports development, data collection, and data analysis as well as utilization of results of The Population and Housing Census, the National Survey on Reproductive Health, and the National Survey on Older Persons.

• In alignment with these focus areas, from 2007-2009 the AWPs for five and later eight expected outputs were developed. The Project Steering Committees provided guidance to adapt or mainstream the projects and programmes. The annual review of CP9 was undertaken between UNFPA and implementing agencies at the end of each year for strategic review and adjustments to be made to improve the programme performance. A series of results-based management training workshops were provided to counterparts to strengthen their knowledge and skills in programme development and monitoring. Regular monitoring of progress has been conducted jointly by UNFPA and counterparts. A mid-term review of CP9 was not undertaken but rather a series of reviews on measurements of expected results were conducted to strengthen measurement and consistency of the indicators used. • During the implementation of CP9, the UN Country Team (UNCT) in Thailand conducted a review to suggest how to reposition the role of UNCT in Thailand in the context of a middle-income country. The results of this two-phase strategic review (Middle Income Country Study-MIC I&II) have given a firm ground for development of the next UNPAF (2012-2016), which will be a framework for development of the UNFPA’s 10th Country Programme (CP10).

II. Evaluation Purpose and Methodology

The evaluation has two primary purposes according to the Terms of Reference (TOR) (Annex 8): (1) To assess achievements of the 9th Country Programme’s (CP9) planned outputs and outcomes and the factors that facilitated or hampered achievements. (2) To provide key inputs for the development of the next United Nations Partnership Framework (UNPAF) and CP10, which will be submitted to UNFPA’s Executive Board for consideration in 2011.

Specific Objectives As UNFPA is a results-based organization, the overall objectives of the evaluation are: (a) To assess and apply five evaluative criteria, specifically the relevance, effectiveness, efficiency, sustainability, and impact of CP9.

2 | Page UNFPA Thailand’s CP9 Evaluation Report

¾ Relevance The evaluation team (ET) assessed whether and how contributions of outputs to outcomes of CP9 are aligned with the beneficiaries’ requirements, Thailand’s national goals, UNPAF outcomes, the Millennium Development Goals (MDG and MDG+), and the Programme of Action of the International Conference on Population and Development (ICPD). ¾ Effectiveness The ET assessed the achievements and the relative effectiveness of each intervention including factors affecting effectiveness. It also assessed the effectiveness of management to deliver the interventions, including whether, implementation arrangements (such as partnerships) were adequate for delivering the interventions. ¾ Efficiency The ET assessed the efficiency of the programme by comparing resources used with the outputs of the programme. These were assessed in terms of efficient utilization of inputs, cost effectiveness, duplication of other programme efforts, and strengths and weaknesses of overall programme management (including monitoring and evaluation, resource mobilization and utilization, and financial management). ¾ Sustainability The ET assessed the extent to which the programme/project results are likely to continue/remain after termination of the UNFPA’s assistance. The ET examined the evolution of roles of both UNFPA and implementing partners over the time period of the evaluation for evidence of durable capacity development and increasing partner ownership and direction of activities. ¾ Impact of CP9 The ET assessed the impact of CP9 on Thailand’s national goals and UNPAF outcomes by focusing on strategic partnerships, integration into counterpart programmes, paradigm change and the capacity improvement of key partners. (b) To identify and explore elements for development of the next UNFPA Country Programme including strategies for repositioning the UNCT in Thailand in the context of a middle-income country.

Scope of CP9 Evaluation

The ET has assessed the following: 1. UNFPA’s contribution by programmatic areas. Review of documents and field data collection has been conducted to assess the achievements of CP9 against its expected results and its contribution to Thailand’s development through partnership development. Due to the limited time available for this evaluation, the team has focused on the outcome and impact of the overall Reproductive Health (RH) and Population and Development (PD) programmes rather than analyze each project in detail. The focus is on strategic activities and approaches that contribute to achievements. Moreover, the project activities during 2011 AWP were not evaluated since this evaluation was conducted during January 2011. 2. UNFPA’s positioning and strategies. UNFPA’s positioning and strategies have been analyzed from the perspective of the organization’s mandate, development needs and the country’s priorities. This has entailed systematic analysis of UNFPA’s position within the development and policy space in the country, as well as strategies used by UNFPA to maximize its contribution. The focus is on relevance,

3 | Page UNFPA Thailand’s CP9 Evaluation Report

responsiveness, UNFPA’s effectiveness and its comparative advantage, particularly in promoting up-stream policy development.

Methodology

The Evaluation Framework used was presented in figure1.

Figure 1: Overall Programme Evaluation Framework

MDGs and ICPD, National priorities Relevance relevance, effectiveness sustainability efficiency impact UNFPA’s Strategic Plan sustainability impact

The Country Programme Action Plan of the CP9 Integration Participatory planning & co-ordinate management Evidence base

five outputs for RH relevance, Guide & three outputs for PDS sustainability Support impact Mechanism and Management Guide & RH Support PDS

Gender is mainstreamed in both components Projects Projects Effectiveness Effectiveness efficiency efficiency

Outcomes, Outputs, Key partners, key Strategies, policy/system change

Five key evaluation questions to be answered:

Q.1: Based on the UNPAF, UNFPA’s Strategic Plan, the Millennium Development Goals, and the Programme of Action of the International Conference on Population and Development (ICPD), to what extent has CP9 contributed to the achievement of national priorities, including outcomes and impacts? and Q 5 How effectively and efficiently has UNFPA implemented CP9 and ensured its relevance to Thailand’s needs and priorities? Q 2: What are the sustainable results of UNFPA’s contributions to Thailand’s National Reproductive Health Strategic Plan and its other population programmes and policies? Q 3: To what extent has UNFPA’s programme management mechanism and its application contributed to programme achievements (programme design, management, and implementation) Q 4: As Thailand becomes a middle-income country, what can and should be UNFPA’s strategic position and contributions to CP10? Sub-questions and analysis issues for answer the questions were developed as shown in Annex 5 and 6.

4 | Page UNFPA Thailand’s CP9 Evaluation Report

Evaluation Methods

The evaluation is based on qualitative analysis of data collected from semi-structured individual and group interviews and through document review. Data Collection Methods / Tools - Desk review and briefings: The ET has analyzed documents related to UNFPA’s programmes and projects under CP9 including the core documents suggested by the Thailand Country Office and the evaluation committee (as listed in the TOR). The ET also held briefing sessions with UNFPA staff to deepen their understanding of the agency’s work, strategic development, and policies or strategies that may not be fully documented. The list of documents reviewed is in Annex 3. - Topic-list semi-structured interviews – Topic guides were developed prior to interviews to ensure systematic coverage of questions and issues by team members working individually. The topics were developed around the evaluation questions, but grouped and targeted according to the organization or individual being interviewed. The semi - structured interview method allows interviewers to explore unforeseen avenues of enquiry as issues arise. Interview guides are provided in Annex 7. - Key informant interviews (listed in Annex 4) 1) Key counterparts: the responsible persons of the implementing organizations and/or the managers of the projects were interviewed at the national level for every program and at the provincial level in two provinces. The key issues investigated were: • planning process (situation analysis, evidence used, stakeholder participation, timing, budget sharing) • implementation (correspondence to the plan, obstacles and lessons learned) • management process and perspectives on UNFPA’s management process (financial, line of administration, communication, flexibility, time and other resources used, strengths, weaknesses, limitations). • support from the CP9 (technical, financial, collaboration, partnerships) • Achievements (benchmarks with the targets, quality, coverage, learning process, and resources used; both singly and relative to other work). • Perceptions of RH/PD situation and trends and opinions on the future role of UNFPA in the Thai context. 2) Key implementers / providers were interviewed at the district and sub-district level in two provinces. Key issues for the interviews include the following: • perspectives on the process of the projects: relevancy, effectiveness, efficiency obstacles, lessons learned and comparison to similar programmes (if applicable) • participation in the project planning and design process • support / assistance received from the CP9 programme • perceptions of the management of the projects relative to the ordinary system or working with other agencies 3) Experts and key stakeholders were also interviewed who are not CP9 counterparts but have a good understanding of issues and challenges facing Thailand and the UN agencies. These include those from government agencies, NGOs and the academic sector as suggested by the UNFPA country officers.

5 | Page UNFPA Thailand’s CP9 Evaluation Report

Participatory and consultative meeting A one-day meeting was held on 25th January 2011 with program counterparts, key implementers, other experts and key stakeholders and other representatives of related stakeholders:. The morning session consisted of a group meeting among stakeholders from Mae Hong Son and Narathiwat Provinces which the ET did not have a chance to visit. This allowed the ET to obtain additional qualitative in-depth data on their specific context and lessons learned. Moreover, other stakeholders who could not be interviewed before also were invited to the meeting to give more information, since the time for data collection was quite short. During the second half of the meeting, the major counterparts and stakeholders were invited to give feedback on the presentation of summary findings. The purpose was to verify the data and to solicit opinions on the challenges and opportunities facing Thailand, including the on-going and potential future role of the UNFPA.

Study Areas: Study areas were selected according to the location of the CP9 programmes in Bangkok and the targeted provinces. Two provinces, Lampang and Songkhla, were explored in depth to provide a more comprehensive evaluation. The criteria used for selecting these in-depth provinces were 1) provinces that contained more programmes being implemented; and 2) provinces that were feasible to access in the short time available. Other areas were explored for context-specific issues through document review, telephone interview and group interview during the stakeholders’ meeting.

Quality control: Peer Review team: While individual team members focused on specific projects, the team used a common framework and triangulated data by sharing among members of the whole team regularly. Research Validity: Through the qualitative research methods outlined above, detailed data were gathered through open-ended questions that provided direct quotations. All data records are maintained in the form of detailed notes and/or electronic recordings. The data have been triangulated among different key partners in the central and field operations and also against the content of related documents. Moreover the data have been triangulated and clarified by different sources of information such as document review, staff debriefing, peer debriefing and follow-up with key stakeholders in the field.

Ethical Considerations: The evaluation team, to the best of its ability, has adhered to the UN Ethical Guidelines for Evaluation, especially with respect to independence, impartiality, avoidance of any real or perceived conflicts of interest, and respect for respondents’ right to provide information in confidence. Although the report contains information that some respondents may consider to be sensitive, findings are presented in a way that ensures that none of this information can be traced to its source and that relevant individuals are protected from reprisals.

A note on conflict of interest: Two members of the evaluation team are from Institute of Population and Social Research (IPSR), which was also a subcontractor on several of the CP9 activities. For two of study these activities (the Census Quality Assurance project and some of the South-to-South activities), this is not an issue because neither of the team members on this evaluation worked on these projects. For UNFPA’s “Impact of Demographic Changes in Thailand” project, one of the evaluation team members (Kerry Richter) was a co- author. However, the output from this activity is not yet published and so the activity was evaluated in terms of its relevance. Finally, AIHD was a sub-contractor from TICA to

6 | Page UNFPA Thailand’s CP9 Evaluation Report

convene the workshop on “Make Motherhood Safer” on December 2010, but was not the main implementing partner.

Limitations of the study: This evaluation was limited by the short time period available to review numerous documents and to complete the field interviews. For this reason, the team identified several core issues to be explored rather than completing an exhaustive review. The interviews explored only issues that were supplemental to the documents and that could be used to verify some data. Some beneficiaries also could not be interviewed because of the limited time of field visit. Given the limited budget, the field study was limited to two sites. For this reason the data collected might not well represent the entire scope of the project. The team attempted to fill in the gaps by also exploring the context or related conditions. Also, many of the key persons from the other sites came to Bangkok for the stakeholders’ meeting. The CP9 programmes were located only in the Southern and Northern regions of the country. Therefore, the social and gender differentials and other data related to culture might also be limited.

The Evaluation Team: The evaluation team was divided into three teams so that the work could be done on time: 1) one team for projects related to RH headed by the RH expert 2) one team for projects with PD headed by the PD expert; and 3) one team for projects with systems strengthening and co-operation components headed by the ET leader. The teams were composed of content experts and research assistants. The data collection design, framework and main questions were developed together. Then, the team reviewed documents and collected data according to the allocated tasks. Discussion and brainstorming after data collection was done periodically to link the data and summary findings, including the conclusion. The team leader was responsible for the overall evaluation and management of the evaluation team and accountable for the completion of a report that UNFPA deems to be satisfactory and responsive to the agency’s needs. The experience and qualifications of the team evaluators and the team structure are shown below in Annex 9.

Time Period: The evaluation was conducted during December 2010 to 15 March 2010

7 | Page UNFPA Thailand’s CP9 Evaluation Report

Evaluation Plan Schedule:

Activities Time Period Dec. Jan. Feb. Mar 2010 2011 2011 2011 Writing the inception report 1-15 Desk/ document related project review More data collection: Interviews, site visits Meetings of the evaluation team x x x x Stakeholders meeting 25 Presentation of preliminary evaluation results to 31 the Evaluation Management Committee Submission of the 1st draft CP9 evaluation report 15th Submission of the final CP9 evaluation report 15th

8 | Page UNFPA Thailand’s CP9 Evaluation Report

Part 2: Findings

I. Reproductive Health (RH) Programme Outcome 1: Increasing utilization of reproductive health information and services by vulnerable groups and in underserved areas, and Outcome 2: Improving HIV prevention through safer sexual practices among vulnerable groups.

The 9th Country Program (CP9) of Thailand for RH programme aims to reduce inequality by improving accessibility and utilization of reproductive health information and services including HIV prevention among vulnerable populations namely youth and sex worker; and among general population in underserved areas namely Mae Hong Son in northern Thailand and Narathiwat in southern. Moreover, UNFPA developed a pilot model to support male involvement in improving maternal health including prevention of HIV. This latter program was implemented in four provinces of Mae Hong Son, Lampang, Songkla and Narathiwat. For youth and sex workers in particular, UNFPA introduced reproductive health services responsive to adolescents and a Comprehensive Condom Program responsive to demands among sex workers.

UNFPA has been working with two key counterparts namely Bureau of Reproductive Health and Bureau of AIDS, TB and Sexually Transmitted Infections (BATS). Three projects; RR, MPM/MMS and YSRH with 3 outputs were developed with Bureau of Reproductive Health. One other project, STRONG, was developed with BATS. The last one on MCD was developed under co-operation with Praboromrajanok Institute, MOPH. The output and the indicators of each project were developed (see details in Annex1).

These project objectives were aligned with the UNFPA outcome of increased access and utilization of quality social services and protection, effectively responding to people’s rights based on quality data, evidence-based planning and increased access to and utilization of comprehensive HIV prevention and treatment, care and support services.

The main components of reproductive health in the interests of CP9 were harmonized to the objectives of UNFPA’s Strategic Plan (2008-2013) and the Millennium Development Goals including maternal health, development of management capacity, adolescent reproductive health, and prevention of STIs.

9 | Page UNFPA Thailand’s CP9 Evaluation Report

1.1. UNFPA’s achievements and the level of effectiveness, efficiency and relevance of strategies, key actions of RH programme

1.1.1. Youth:

Achievements: The Youth Sexual Reproductive Health Project (YSRH) focused on improving access to “youth friendly RH services”. In the past, adolescents and young adults had access to RH health services, in both the public and private sector, but usually did not make optimal use of these services for a variety of cultural reasons. The YSRH aimed to create a supportive environment for youth to promote healthy behavior through the provision of life skill training, in order to better understand the importance of and to properly utilize sexual reproductive health information and services. The project also tried to increase access to information through better use of the media [e.g. websites and interactive DVDs]. A key project strategy was to help empower youth to engage in policy discussions regarding reproductive health policies and legislation through the establishment of a Youth Advisory Panel (YAP). The project was implemented in 2 districts of Lampang (Muang and Thoen). In order to achieve these objectives the project used two key methodologies which included establishing Youth Friendly Reproductive Health Clinics in 14 secondary and vocational schools as well as training of peer educators (teachers and students). This project design also included a separate component, implemented by PATH [Love Care Station], which developed an information and counseling service for youth through interactive websites and a “telephone call-in center service”.

It is somewhat difficult to assess whether the present intervention has in fact increased utilization of RH services, including HIV prevention, among youth, as the project was implemented only in a select number of schools in Lampang. Adolescents and young adults who had already graduated, or who were not attending school, were not included in the project design. From a survey conducted in 2009, in the target schools, it was observed that 24.8% of the respondents indicated that they had made use of the user friendly RH information and services. The survey also indicated that that there was a slight decline from 2008 to 2009 in condom use among those respondents who indicated that they were sexually active [i.e. from 47.5% to 42.6%]. However these figures are still significantly higher than that reported for the country as a whole.

With respect to the Love Care Station initiative there were a total of 52,103 website hits from May 2009 - October 2009. The website reportedly referred many of their contacts for various RH services and or counseling. This included HIV/AIDS/STIs counseling/services (30.1%), family planning (20%), cervical cancer screening (14.4%), counseling on unwanted pregnancy (7.7%), and shelters for youth with an unwanted pregnancy (3.7%). With respect to the “telephone call-in station” there only 279 telephone calls were made within this same time period. This included counseling on unwanted pregnancy (12.5%) and HIV/AIDS/STIs counseling/services (23.3%).

The evaluation team did not have an opportunity to directly interview or meet with youth in Lampang nor were they able to contact any of website/telephone call-in station users in Bangkok. Thus it is difficult to assess the views and opinions of the target groups with respect to project achievements aimed at increasing access to quality RH information and services. Project implementers had an overall impression that these initiatives promoted a more “user friendly environment” for youth to access information and specific RH services. In Thoen district health officials, responsible for the implementation of the project, felt that this UNFPA sponsored pilot project not only increased their own knowledge, through

10 | Page UNFPA Thailand’s CP9 Evaluation Report additional training, but more important it expanded their perspectives on the special RH needs of adolescents and young adults. As a result the project increased their ability to implement a wider range of related activities especially with respect to establishing linkages between the health sector and local youth networks. The Lampang Provincial Health Office served as the main implementer of the Youth Program in Muang district. They had previously initiated several youth oriented activities, but the present initiative helped to strengthen the local youth network and well as stimulate greater interest at the Provincial Governor’s Office to participate in a larger number of activities that affect youth.

PATH officials also felts that their user friendly website/telephone call-in station initiative has been extremely successful in generating interest among adolescents and young adults to seek information about RH issues as well as to learn where they can go to obtain specific counseling and RH services. The website format has, over the past couple of years, become more “attractive” and “interactive” and this has encouraged youth to make greater use of this new source of information and services.

Relevance: Epidemiological studies have shown that unprotected sex and unplanned pregnancies among young people have been increasing in Thailand. Approximately two- thirds of new HIV infections in females are found in women 15-24 years of age. Although the Contraceptive Prevalence Rate in Thailand is very high, information concerning contraception and access to FP methods is generally culturally restricted to married women. The YSRH Project was designed to address the gaps in the present national health care delivery and educational systems which are not “youth user friendly” or which do not contain relevant content for a “youth culture” in their course curricula. The project was designed to identify new strategies to engage young people to be aware of important reproductive health issues as well as to enable them to access specific information and services to meet their needs.

National standards for Youth Friendly Health Services were developed by the Bureau of Reproductive Health, Ministry of Public Health which served as a guide for establishing youth sexual and reproductive health service guidelines. An interactive educational video [iDVD] was also produced to help teach pre-teens learning about sex and sexuality. The content of the DVDs was based on input made by adolescents and young adults. Another important strategy incorporated into the project design was to train students, at selected schools participating in project implementation, to become peer educators. The peer educator model was designed to provide information in a more “user friendly” was so as to encourage other students to ask questions or seek advice from “friends who know” rather than going to an adult who they do not know and with whom they may not feel comfortable discussing embarrassing or personal topics. The content of the training of trainers curriculum [for peer educators] focused on topics identified by young people, such as, sexual and reproductive rights, safe sex, healthy relationships, condom use, and accessing appropriate contraceptive methods.

Effectiveness: The YSRH project contained 4 output indicators to indicate whether project interventions and inputs were effective.

Indicator #1: Percentage of young people confident to refuse unwanted/unprepared sex. The evaluation team did not have an opportunity to interview any of the students participating in the YSRH project in Lampang, and thus had to rely on data provided by project staff. Project staff conducted two surveys before and after project implementation. The baseline survey, conducted in 2008 indicated that 67.3% of the respondents felt

11 | Page UNFPA Thailand’s CP9 Evaluation Report confident that they could refuse unwanted/unprepared sex, while 20.9% were not sure, and 11.8% felt that they did not have the skills/ability to “say no”. The post invention survey, conducted in 2009, indicated that there was very little difference in the ability of young people to negotiate “safer sex practices” as a result of project inputs. That is 67.5% of respondents were confident that they could refuse sex, while 24.4% were not sure, and 8.1% did not feel that they could “say no”. One of the problems with comparing the results of the baseline and post intervention survey is that the sample size almost doubled in size between the first and second survey exercises. Another reason for the small difference in “safer sex negotiating skills” may be due to the fact that the respondents included in the second survey may not be the same individuals who were interviewed during the first or baseline survey. A third reason may also be that during the first survey respondents were not as candid as during they were during the second survey.

Project implementers did not conduct any baseline or post-intervention surveys with respect to the Love Care Station Website and Telephone Call-In Counseling Service initiative undertaken in Bangkok project.

Indicator #2: Percentage of sexually active young people using condoms during their last act of sexual intercourse: As indicated above the YSRH Project in Lampang conducted a baseline and post-intervention survey. The baseline survey indicated that 47.5% of the respondents had used a condom at the time of their last act of sexual intercourse, while this figure dropped to 42.6% in the post-intervention survey. One of the possible reasons for the relatively low percentage of condom use may be due to the fact that the “peer educators” did not provide condoms to interested clients, and thus the target population still may have difficulty in obtaining condoms for safe sex practices although they understand the need to do so.

Indicator #3: Percentage of young people that have used youth friendly service delivery points for information or services. The YSRH Project in Lampang was only able to obtain this type of information during the post-intervention survey, which indicated that 24.8% of the respondents had used the youth friendly service delivery points for information or services. This figure is not necessarily high, but then again many adolescents may not be sexually active and perhaps not in need of such services.

In Bangkok the Love Care Station was able to collect information on the number of people [i.e. “website hits”] using their service, the number of people calling the online counseling center, and the number of people referred to other service delivery points by the call-in center. The number of “website hits” from May-October 2009 was 52,103 (while during the previous 6 month period there were only 3,500 website hits). During this same time frame there were 279 using the call-in center online counseling service, while the number of individuals referred for RH services by the website was 4,674. In general the actual number of “youth friendly Love Care Station users” far exceeded expected targets set by project designers.

Indicator #4: Sexually active youth with unmet needs for contraceptives. The YSRH Project in Lampang and Bangkok did not collect information regarding the unmet needs of sexually active youth for contraceptive services. The baseline and post-intervention survey, in Lampang, only investigated whether or not condoms were used during the “last act of sexual intercourse”. It did not ask whether oral contraceptives or some other FP method was used instead of condoms, or in additional to condoms.

12 | Page UNFPA Thailand’s CP9 Evaluation Report

Efficiency: A total of approximately $ 350,000 was allocated for the implementation of the YSRH Project (2008-2010). The largest portion of these funds [~50%] was used to develop the Solution Exchange Website software package. One of the reasons why this input was so expensive is that the Solution Exchange Website had to be linked to a larger existing website and accordingly a large percentage of the funds were used to hire computer software experts in India. The process has not been implemented as smoothly or as efficiently as originally expected. At the present time it is still quite expensive to operate the website, and if the present system is continued it will probably cost a considerably amount of money, each year, to up-grade or modify the existing website software package. As such it may be useful to explore the possibility of using local Thai expertise to establish a comparable website in the future.

The second largest expenditure, under the YSRH Project, was devoted to the creation of Youth Friendly Services in Lampang. Approximately 26% of the total project budget was spend on capacity building activities for peer educators (175) as well as for training public health officials. The project did not specify the grade level of the peer educators, whether they continued to work as peer educators until they graduated secondary school, or whether the newly trained peer educators dropped-out of the project shortly after being trained. Up until now PATH has implemented 3 annual basic training courses for peer educators during the project life-time. Due to requests from students, and existing peer educators, the training workshops have been expanded from 2 to 3 days to cover additional topics. Project implementers have expressed the view that the training and deployment of “peer educators” has been a very cost effective and efficient intervention, which has already been expanded to another 15 provinces in the northern region [as part of the Global Fund].

The third largest expenditure item was the establishment of a Youth Advisory Panel (YAP) in Lampang. Approximately 12% of the total budget was devoted to this initiative. The Lampang Provincial Governor’s Office has used input from YAP to help with the strategic planning of programs and activities especially designed for adolescents and young adults. Project implementers hope that existing YAP members will continue to serve as mentors for new YAP members trained every year.

Sustainability: The youth-friendly health service has already been integrated into the National health care system. Moreover, the standard has been set up so it is feasible to scale up to other provinces. It is clear from the interviews that the awareness and attitude of health care providers in implementing areas was improved so that they were eager to provide services friendly to youth by trainings as well as they gain experiences and got the guidelines to improve the service quality. These are evidence to support the sustainability of the services. Nevertheless, a refresher course or re-training for new staff is also needed, so the services can be continued.

1.1.2. Men as partners in maternal health (MPM):

This project focused on increasing male involvement in general RH activities. Thai culture generally does not create a supporting environment for males to be directly involved in a number of basic RH issues. The intervention areas included the following 8 districts in 4 provinces: Lampang (Muang and Thoen), Mae Hong Son (Mae Sariang), Songkhla (Sadao and Rattaphum), and Narathiwat (Kolok, Bajaw, and Wang). The main objective of the project was to encourage husbands to accompany their wives during routine ANC examinations to better understand the needs of pregnant women and expectant mothers as

13 | Page UNFPA Thailand’s CP9 Evaluation Report well as to undergo HIV/AIDS screening for dual protection against maternal-to-child transmission of HIV/AIDS.

Achievements: This initiative was integrated into the MOPH’s routine ANC services and parenting school program implemented through the provincial and district health service delivery network. The district health office served as the key organizers for encouraging males to attend ANC clinics by using the existing VHV (Village Health Volunteer) network to provide important information to villagers as well as to motivate husbands to become more fully involved throughout their wives pregnancy. One of the problems in assessing the achievements of this initiative is the overall physical setting of project activities. Husbands, for example, were asked to accompany their wives to visit district and provincial hospitals, but for many men this request was not feasible as they could not take time off from their respective jobs to do so. The project did not make, for example, any provisions for ANC examinations, parent counseling, or HIV testing services to take place at local health centers or after official government working hours. The percentage of women whose husbands accompanied them to at least 3 ANC examinations varied between the districts and provinces. Overall approximately 27-40% of husbands accompanied their wives to ANC examinations. The only exception was Muang district of Lampang where the percentage was only 10%. This may have resulted from the fact that the provincial hospital did not fully support the activities of the district health office and the local VHV network. Another reason for this low figure may be due to the fact that Muang district is the site of a regional hospital which probably requires that patients/clients spend a longer period of time waiting for specific health services than at district hospitals.

The percentage of husbands and wives testing for HIV/AIDS, during scheduled ANC examinations, was very high. It was almost 100% at all service delivery points. Similarly the percentage of pregnant women who were informed about dual protection HIV testing by health providers during their ANC examinations was also nearly 100% at all service delivery points.

Relevance: A major focus of the project was to up-grade the capacity of RH service providers to be able to provide couple counseling services. To do so the project tried to enhance the understanding as well as increase the sensitivity of hospital providers to understand the needs and mindset of males who generally do not attend ANC examinations with their wives. This project was designed to be an extension and link to on-going initiatives aimed at preventing HIV transmission from mother to child. The original aim was to extend the duration of zero sero-conversion for as long as possible. This was part of the Thai national strategy to prevent HIV transmission among couples as well as to prevent possible further HIV transmission from mother to child. The “Stay Negative Project” was implemented as part of UNFPA CP8, and the present MPM Project was geared to scale-up efforts to promote HIV testing and dual protection by encouraging males to be actively involved in their spouses’ health and welfare at various times during pregnancy. The concept of encouraging active male involvement is quite new to the Thai cultural setting, but this approach can potentially have a dramatic impact on increasing access to and the quality of RH services for women, as well as encouraging males to become more responsible parents.

Effectiveness: The MPM Project, implemented in 8 districts in 4 provinces [Lampang, Narathiwat, Songkhla, and Mae Hong Son], contained 3 output indicators to indicate whether project interventions and inputs were effective.

14 | Page UNFPA Thailand’s CP9 Evaluation Report

Indicator #1: Percentage of women with male involvement during pregnancy. The percentage of women whose husbands accompanied them to at least 3 ANC examinations varied between the districts and provinces. Overall approximately 27-40% of husbands accompanied their wives to ANC examinations. The only exception was Muang district of Lampang where the percentage was only 10%. This may have resulted from the fact that the provincial hospital did not fully support the activities of the district health office and the local VHV network. Another reason for this low figure may be due to the fact that Muang district is the site of a regional hospital which probably requires that patients/clients spend a longer period of time waiting for specific health services than at district hospitals.

Indicator #2: Percentage partners tested for HIV/AIDs during pregnancy: The percentage of husbands and wives testing for HIV/AIDS, during scheduled ANC examinations, was very high. It was almost 100% at all service delivery points.

Indicator #3: Percentage of women who are informed about dual protection by health providers during ANC examinations: The percentage of pregnant women who were informed about dual protection HIV testing by health providers during their ANC examinations was also nearly 100% at all service delivery points.

Efficiency: The evaluation team was not able to directly interview husbands and wives attending the ANC examination. They were able to obtain, however, the impressions of the service providers. One of their key observations was that once husbands were exposed to this new experience, of attending ANC clinics with their wives and receiving parent counseling, they became very much interested in the welfare of their wives and the health of the fetus. They tended to make sure that their wives followed the recommended examination schedules and suggestions made by hospital staff. The health service providers claimed that the project helped them to better understand and appreciate the important role that males can have on improving ANC care in general and possibly playing a more active role future child rearing activities at home.

Sustainability: The team of gynaecologist and nurses in the implementing areas explained that educating pregnant women with their husband for dual protection and getting their male partner involved can be added into the Parenting School that has already become routine. Thailand’s Bureau of Reproductive Health has already adopted the MPM project to be integrated into the Royal Patronage Sai Yai Rak Project. So it shows that the male involvement in maternal health can be sustained in the implementing areas due to fact that the supporting system has already established and the capacity of health care can be built up. But it is not guaranteed that implementing MPM in other areas will be sustained because the health care providers’ capacity and attitudes need to be built up to provide the good quality services. Moreover, getting the men to get HIV tests faces more problems due to the men’s fear. More supplemental action is needed.

1.1.3. Sex Workers (STRONG project)

Although the Strengthening Sexually Reproductive Health (SRH) /AIDS for Vulnerable Groups Project (STRONG) had 2 expected outcomes, the project implementation design focused on HIV/AIDS prevention through safer sexual practices. The project used the 4 main strategies: This included (a) drop-in centers, (b) outreach services, (c) peer education, and (d) strengthening local STIs clinics.

The “drop in center” strategy used in Pattaya was implemented by a national NGO (SWING). The overall purpose of the “drop in centers” was to promote valuable life skills of sex

15 | Page UNFPA Thailand’s CP9 Evaluation Report workers, one of which was to have greater access to information concerning reproductive health as well as to take appropriate measures to prevent HIV/AIDS and STIs.

The SWING “drop in center” had been providing educational and social services to sex workers and later added HIV prevention through safer sexual practices into their repertoire of service. The “drop-in center” opened to any visitor interested in stopping by to talk about a wide variety of subjects, including HIV/AIDS and STIs prevention. The clinic also operated an “outreach component” whereby SWING volunteers went out into the streets of Pattaya to encourage sex workers to visit the center. The “drop in center” also linked its counseling services to further follow-up STIs services at Bangramung Hospital in Chonburi. The drop in center was a source of free condoms.

In Songkhla and Lampang SWING and the provincial health office established an outreach service network aimed at preventing as well as treating HIV/AIDS & STIs. One of SWING’s main roles was to help “map” the different entertainment venues and establishments where sexual services are available, as well as help health workers understand how to approach the owners of these establishments and/or how to effectively communicate with the sex workers. The situation in Lampang was considerably different and much easier to work in than in Songkhla. In Lampang sex workers usually operated out of karaoke bars or other “legal venues”, while in Songkhla the sex service venues were often “illegal” and many of the sex workers were illegal migrants. This made it more difficult for public health officials to operate “outreach services” to provide relevant information, counseling, and services. The services included VCT, pap smears, family planning counseling, distribution of condoms and other contraceptive methods, and HIV and VDRL testing. In Songkhla a similar outreach service system was implemented, but due to the nature of the target population, certain problems arose. One of main obstacles was that perhaps as many as 70% of the sex workers were migrants who were not fluent in Thai. Another related problem was that due to their “migrant status” the sex workers tended to move frequently from place to place with less chance for health workers to make follow-up contacts.

Another related strategy implemented in Lampang and Songkhla was the establishment of a “peer educator” network to provide information and counseling as well as to distribute condoms to sex workers who worked in a specific geographic locations identified through the “mapping exercise” mentioned above. SWING was responsible for identifying and training suitable “peer educators” but jointly worked with provincial and district health officers and other relevant parties.

Achievements: The achievements of the various strategies aimed at increasing access to HIV prevention and services were as follows:

“Drop-In Center” (Pattaya): The drop-in center initiate was implemented for only one year in the CP9 life-time. One of the problems in assessing achievements is that the project design did not contain many important key indicators to demonstrate success. One of its main indicators was that at 500 sex workers should visit the drop-in center. The total number of drop-in visitors was 596. The evaluation team was not able to interview any of the sex workers who visited the “drop-in center” to learn whether or not they learned how to better protect themselves against HIV/AIDS and STIs and/or subsequently took appropriate measures to reduce their changes of being infected with HIV/AIDS or STIs or whether they sought appropriate treatment for their condition. The data, from SWING project reports, also did not indicate the reasons why the sex workers “dropped in at the center”. Was it to obtain information, counseling or follow-up/referral services for HIV/AIDS and STIs or was it to

16 | Page UNFPA Thailand’s CP9 Evaluation Report simply learn a foreign language [e.g. English] to more effectively communicate with potential clients. SWING staff felt that the “drop in center” has been instrumental in improving the capacity of female sex workers to deal with their daily life situations. The “drop in center” has empowered sex workers to know how to use condoms to prevent HIV/AIDS and STIs, as well as how to better access referral counseling and services at nearby hospitals if necessary.

“Outreach service” and “Peer Education” (Lampang and Songkhla): The “outreach service” and “peer education” strategies were coordinated and implemented with provincial and district health officials in Lampang and Songkhla. The project established a number of key indicators to measure achievements. With respect to the number and percentage of sex establishments visited by outreach service staff/peer educators, the situation in Lampang was substantially different than that of Songkhla. In Songkhla the project intervention area contained between 174, 216, and 231 sexual service establishments from 2008-2010. Project staff were able to respectively visit 47.7%, 52.3%, and 70.6% of these sites. In Lampang the number of sexual service establishment were quite small [i.e. from 29-36 sites], and as such it was easier for outreach clinic staff and peer educators to make regular visits [i.e. from 81.6% to 94.6%]. The number of sex workers in the project intervention area in Songkhla was approximately 15 times larger than that of Lampang [i.e. from 126-172 compared to 1,443-2,272 during the project life-time]. Several output indicators specifically dealt with increasing access information as well as to specific RH and/or HIV/AIDS/STIs prevention services. An important indicator was the percentage of sex workers receiving an HIV test during the past 12 month period. In Songkhla, probably due to the fact that many sex workers were migrants who moved from place to place frequently, this number varied from 58.2%, 68.2%, and 50.1% from 2008-2010. In Lampang, with a smaller number of sex workers living in a more stable setting, the figures were respectively 89.2%, 61.6%, 83.1% and 93.8% from 2007-2010. With respect to the percentage of sex workers obtaining a RH service in the past 12 months the figures for Songkhla were 78.0%, 71.1%, and 72.5% from 2008-2010, while in Lampang the figures were 52.3%, 47.0%, and 69.8% [for pap smears], and 29.1%, 52.4%, and 68.3% [for FP services] from 2008-2010. In Lampang the percentage of sex workers consistently using condoms with general clients was extremely high [i.e. 98.9%, 98.7%, 96.4%, and 98.3% from 2007-2010], while the percentage of sex workers consistently using condoms with lovers/partners was 42.2%, 54.2%, 79.4% and 72.0% from 2007-2010. This data was not available from Songkhla. In general the figures in Lampang for consistent condom use, by sex workers, with respect to clients, other sex partners, and husbands/co-habiting partners was higher than for the nation as a whole.

Relevance: The UNFPA identified sex workers in Thailand as a vulnerable population, particularly with respect to sexual and reproductive health and rights. The UNFPA CP9 has in many ways offered Thailand an important opportunity to formulate strategic plans to improve the overall health situation for sex workers. The focus of the UNFPA’s initiative is to improve access to information, counseling, and RH services to prevent the transmission of HIV/AIDS. The program aims to strengthen the capacity of national and sub-national program managers as well as to develop and introduce a comprehensive condom distribution strategy for the prevention of HIV/AIDS and STIs among sex workers and their clients.

The project was designed within the overall context where special interventions for sex workers have steadily declined over the past decade. One the main reasons for this development is that the physical settings for sex work have changed from establishments

17 | Page UNFPA Thailand’s CP9 Evaluation Report such as brothels to other venues such as karaoke bars and restaurants. Another reason for the decline in access to HIV prevention services is due to the fact that STIs clinics, which were formally part of the provincial health office, have been transferred to hospitals as part of health reform measures in 2003. Many sex workers are not registered in the provinces where they work and thus not entitled to universal health coverage. The National AIDS Programme is also no longer in a position to fully fund the procurement and distribution of condoms so alternative solutions need to be found. Another rationale for designing the present project was to serve as a catalyst of continued action and support for the consortium of GOs, NGOs, and CBOs, working with sex workers that helped develop strategies for the 2007-2011 National AIDS Plan. The STRONG Project is designed to redirect interest as well as to identify new and appropriate interventions that specifically target sex workers.

The UNFPA supported another related strategy in Lampang that aimed to strengthen the existing STIs Clinic at the provincial health office. In the past, most provincial health offices, in Thailand operated large STIs clinics, but this function has gradually been shifted to provincial and district hospitals. The UNFPA Country Program 9 provided additional support to the Lampang Provincial Health Office to revitalize its STIs Clinic by procuring additional materials and supplies.

Effectiveness : The STRONG Project contained 3 output indicators to indicate whether project interventions and inputs were effective.

Indicator #1: Percentage of sex workers using condoms during last sexual intercourse with non-regular partners: This data is only available from Lampang where the percentage of sex workers using condoms during their last act sexual intercourse with non- regular clients were respectively 98.9%, 98.7%, 96/4%, 98.3% from 2007-2010.

Indicator #2: Percentage of sex workers who received RH and OB/GYN services within the last 12 months: The data from Songkhla is somewhat different from that collected in Lampang. In Songkhla it merely indicated the percentage of sex workers that received RH services in the past 12 months. For the years 2008-2010 the figures were respectively 78.0%, 71.1%, and 72.5%. In Lampang the data was collected according to specific RH service. Thus for pap smears the figures were 52.3%, 47.0%, and 69.8% while for FP services the figures were 29.1%, 52.4%, and 68.3% from 2008-2010.

Indicator #3: Percentage of sex workers attending government STIs clinics expressing satisfaction with counseling and other services: In Songkhla the percentage of sex workers attending government STIs clinics who expressed satisfaction with counseling or other services was respectively 54.3%, 62.2%, and 64.2% from 2008-2010, while in Lampang the figures were 90.0 and 92.6% in 2009 and 2010.

SWING did not collect this type of data for its “Drop-In Center” in Pattaya. It only collected information about the quantitative number of people who visited the center over a one year period [i.e. 596 against an estimated target of 500]

Efficiency: A total of approximately $690,000 was allocated to the STRONG Project which represented about 26% of the entire RH Programme budget. The largest portion of the budget was used for the Sex Workers’ Sexual Reproductive Health (SSRH) initiative, which included funds for baseline assessments of sex workers’ health needs, enhancing capacity and service for STIs clinics [e.g. procuring laboratory equipment and supplies], and for the training of health workers and sex worker “peer educators. Most of these inputs have long

18 | Page UNFPA Thailand’s CP9 Evaluation Report term implications for improving the capacity of health workers to address RH and HIV/AIDS and STIs needs of sex workers. The investments made in the procurement of laboratory equipment and supplies, as well as the training of health workers and peer educators can be used over a long period of time, even beyond the UNFPA CP9 life-time. At the present time, however, there are not sufficient financial resources to guarantee the procurement of on- going consumable lab test supplies.

With respect to the investment in training sex worker “peer educators” (i.e. approximately 236 trained in Songkhla and Lampang) outreach STIs clinic staff feel that these individuals have been very effective in helping to distribute condoms, provide advice, and refer fellow sex workers to outreach STIs clinics or nearby hospitals.

The second largest project expenditure was spent on the CCP activities, which included developing training curricula and training of trainers’ activities for local government officials. One of the objectives of this initiative was to encourage local government offices [i.e. the Tambol Administration Office] to take a leadership role in supporting future condom procurement as well as supporting a logistical re-supply network by using local financial resources. The project envisioned that the director of the TAO would be trained to serve as a leader and supporter for HIV prevention activities at the local level. At this point in time training activities have not permeated down to the implementation level and during the first two “pilot training sessions” most of the trainees did not include the director of the TAO. As such it presently difficult to assess whether this initiative will attain its expected outcomes.

The evaluation team was not able to directly interview any of the sex workers, peer educators, or other individuals involved in these initiatives (e.g. owners of establishments that offered sex services]. SWING staff, however, felt that these two strategies were very important in helping to educate sex workers about the dangers of HIV/AIDS and STIs, as well as encouraging them to obtain general RH services. SWING staff, however, acknowledged that they encountered many more constraints working in Songkhla than in Lampang. As indicated above one of the major problems was the overall environment of the sex worker/sex service delivery industry. In Songkhla the sex workers, for the most part, were illegal migrants, many of whom could not effectively communicate in Thai. In Songkhla sex worker services were available in a variety of different venues and settings and it was more difficult to gain access to this dispersed target population than was the case in Lampang. The fact that many of the sex workers in Songkhla may have been part of trafficking rings probably made it more difficult for peer educators or outreach clinic staff to closely work with the target population. SWING and public health officials, however, generally felt that these two strategies have nevertheless helped achieve the basic goal of preventing HIV/AIDS through safer sex practices. The number and percentage, as indicated above, of sex workers accessing RH services, going for routine HIV testing at least once a year, and consistently using condoms with clients [as well as with their regular spouses/partners] has increased.

1.1.4. Mothers in Underserved Areas

The Making Motherhood Safer Project (MMS) focused on improving access to high quality RH information and services for women living in underserved geographic areas. The project implementation area included Mae La Noi, Sob Moei, and Mae Sariang districts of Mae Hong Son, and Wang and Bajaw districts of Narathiwat. Mae Hong Son is a province in northern Thailand whose population is primarily made up of hill tribes, or ethnic minorities, many of whom still live in remote areas that have poor access to routine health services.

19 | Page UNFPA Thailand’s CP9 Evaluation Report

The population of Narathiwat is mostly Muslim, whose language and customs are considerably different from central Thai norms. With respect to maternal and child health practices, both target areas have less knowledge about as well as access to family planning and other RH services. The interventions of this strategy have been designed and developed according to the culturally sensitive approach. Therefore, interventions for both sites are definitely different. In Mae Hong Son, the project aim to build capacity of Maternal and Child Village Health Workers to be the change agent. In Narathiwat, the project worked closely with religious leaders, who are natural leaders as key change agents.

Achievement : In Mae Hong Son the project conducted a number of activities to improve both the quality and accessibility of comprehensive FP & RH services for women living in remote communities. A key component was to map and review the availability of Traditional Birth Attendants (TBAs) and RH-FP service delivery points for the target population. The project recruited a villager who is literate and speaks Thai and Karen fluently to be a Maternal and Child Village Health Worker. The MVHVs were trained to provide health education and selected RH services to women and mothers in their respective communities. An important component of the project was to increase accessibility to routine ANC, PNC, FP, EPI, and other basic MCH & RH services through mobile health clinics organized by nurses stationed at sub-district health centers. The main objective of the project, which was envisioned by the director of the Mae Hong Son provincial health office, was to ensure that there was a skilled birth attendant at every birth. Existing village level TBAs received refresher obstetrics care training at the district hospital. TBAs and MVHVs were responsible for encouraging expectant mothers to delivery at the nearest district hospital, by accompanying the mother to these sites before the delivery. The TBAs were supplied with an emergency obstetrics delivery kit, in the event that the delivery took place while traveling to the hospital. This hospital based delivery initiative also made use of a mobile telephone service to inform hospital personnel that an expectant mother was preparing to leave the village in order to delivery at the hospital. The project utilized key village leaders, such as village headmen and local tambol administration organization (TAO) officials, to motivate villagers to use government health services. These leaders also helped facilitate or organize local transportation services to take expectant mothers to the district hospital for their deliveries.

In Narathiwat the project aim to increase accessibility to comprehensive RH services by training religious leaders to take an active advocacy role to promote women and mothers to accept appropriate RH health services. Follow to the suggestion of Health Promotion Center Region 12 (HPCR-12), Yala, the project did not focus on working with Traditional Birth Attendants (TBAs). However, this strategy was further enhanced by expanding mobile health clinics to rural areas to provide routine ANC, PNC, FP, EPI and other basic PHC services and work closely with the communities. The project focused efforts on capacity building for women leaders as well as on raising local awareness about the importance of FP and RH services through bilingual radio campaigns and producing appropriate IEC materials.

The project apparently did not establish several key indicators to determine the extent to which project strategies have increased utilization of RH services by vulnerable groups in underserved areas. One indicator used to assess utilization of RH services was the percentage of deliveries attended by health personnel. In Mae Hong Son these figures were 56.6%, 53.3%, and 60.5% from 2008-2010, while in Narathiwat they were 94.0%, 95.1%, and 99.0%. From these figures it is difficult to determine whether or not there has been any

20 | Page UNFPA Thailand’s CP9 Evaluation Report improvement due to project interventions, as the percentage of deliveries taking place in hospitals throughout Thailand is over 95%. A better indicator may have been, especially for Mae Hong Son where the project intervention was situated in remote mountainous communities, to determine the percentage of “emergency/complicated deliveries” that took place in hospitals. Although the project promoted the concept of hospital-based deliveries, there are many constraints in remote areas that may have prevent a sizable portion of local women from seeking care at the nearest hospital. This includes (a) the availability of local transportation, (b) local weather conditions, (c) road conditions [especially in the rainy season], and (d) the general perceptions of local villagers with respect whether or not it is truly necessary to deliver at a distant health facility if the expectant mother is not experiencing any problems or adverse signs and symptoms and a local experienced TBA can assist in the delivery.

Another important health indicator that perhaps should have been included in the project design or monitoring and evaluation components was the percentage of married women/couples presently using a FP method. Although FP acceptance has greatly increased in hill tribe communities over the past 1-2 decades, CPR in many Muslim communities in southern Thailand is still generally low. It would have been interesting to observe whether advocacy activities with religious leaders and/or bilingual media campaigns had any effect on changing attitudes and behavior with respect to using an appropriate FP service to space births or limit further fertility.

The evaluation team had an opportunity to interview new mothers as well as TBAs in two village clusters in Mae Hong Son. With respect to mothers/pregnant women the respondents indicated that they are happy to go and delivery at the district hospital if this is at all possible. Otherwise they are confident that the local TBA can assist them at the time of their home delivery. They also indicated that the mobile health clinics make it easier for them to obtain many important FP & RH services as they do not have to spend time traveling to the sub-district health centers or need to spend money on transportation or other related costs. The women also have expressed satisfaction with their local MVHV who they can visit whenever they have questions or need advice.

The TBAs also appeared to be satisfied with the project as the interventions serve as a safety back-stopping mechanism in case they encounter potentially serious complications or if they feel that a delivery should be performed in a hospital setting. The TBAs are happy to be involved in the project as one of their main motivations in originally becoming TBAs was to help village women during the time of their delivery.

Sub-district health workers also appeared satisfied with the achievements of the project as it has improved their ability to address serious RH health issues. They have always, for example, operated mobile health clinics, but the present efforts are implemented in a more systematic fashion with input from community leaders, TBAs, and MVHVs. The use of mobile telephones to communicate with villagers as well as with district level supervisors has also greatly enhanced their ability to address emergencies while simultaneously informing the district hospital before the patient actually arrives. The mobile telephone service has also improved communication between sub-district health centers and district hospitals and district health offices, especially with respect to obtaining advice and recommendations to deal with specific problems and issues at the village level or in response to questions raised by villagers, TBAs, and MVHVs.

21 | Page UNFPA Thailand’s CP9 Evaluation Report

The evaluation team was not able to visit Narathiwat and accordingly could not obtain opinions from some of the key stakeholders, such as pregnant women, new mothers, couples of reproductive age and religious leaders as to whether project strategies have influenced accessibility to or the utilizations of comprehensive FP & RH services. Discussions with hospital staff indicate that they routinely conduct mobile health clinics, and that the present project intervention has facilitated this effort. Unfortunately hospital staff did set specific health indicators to indicate whether there has been an increased utilization of RH services, in rural communities, as a result of the increased frequency of mobile clinics. The district hospitals have collected a great deal of MCH & FP related data but it does not necessarily indicate whether these services were provided to any specific geographic target population or whether they are part of routine services provided by hospital staff at the hospital or through the overall mobile clinic initiative.

Relevance : The MMS Project is aligned with the MOPH’s First National Reproductive Health Strategic Development Plan and Policy [2010-2014] that states that every birth must be wanted, safe and of quality. The project was designed to ensure that women living in remote geographical areas, or who are part of ethnic minority groups that have different norms than that followed by central Thais, can have deliveries performed by trained health personnel and if possible in a hospital setting. The project interventions were planned in accordance with local needs assessments that included participation from local health care providers as well as local community members.

Effectiveness : The MMS Project contained 3 output indicators to indicate whether project interventions and inputs were effective.

Indicator #1: Percentage of deliveries attended by health personnel: There was a considerably difference in the percentage of deliveries attended by health personnel in the project intervention areas in Mae Hong Son and Narathiwat although both target populations were comprised of ethnic minorities [i.e. hill tribes and Muslims] with different cultural norms than central Thais. For Mae Hong Son the figures were respectively 56.6%, 53.3%, and 60.5% from 2008-2010. In Narathiwat the figures were 94%, 95.1%, and 99% over the same 3 year time frame. The relatively low percentage of deliveries performed in a hospital setting in Mae Hong Son compared to Narathiwat or Thailand as a whole probably reflects the fact the project intervention was located in remote mountain communities, where transportation may not be readily available during the rainy season.

Indicator #2: Percentage of persons surveyed who know three primary warning/danger signs of obstetrics complications: The data for this question came from a baseline survey conducted in 2009 for pregnant women living in UNFPA supported project intervention areas. In Mae Hong Son the figure was 70% while that of Narathiwat was 85%.

Indicator #3: Percentage of clients expressing satisfaction with counseling and other services: The data for this question also came from the same survey conducted in 2009, with the figures respectively being 75% and 80% in Mae Hong Son and Narathiwat.

Efficiency : A total of approximately $370,000 (i.e. around 12% of the RH Program) was allocated to the MMS Project. The largest expense, in Mae Hong Son, was spent on providing monthly stipends to the MVHVs [i.e. approximately $120/month/person]. In determining whether this is an efficient use of project expenditures, project staff expressed the opinion that in the initial stages of this type of project intervention it is necessary to invest financial resources to encourage local villagers to assume greater responsibilities aimed at

22 | Page UNFPA Thailand’s CP9 Evaluation Report increasing accessibility to safe delivery services. The main issue is whether or not it is necessary to pay village volunteers a regular monthly salary once the new MVHV system is established and operational. Payments made to MVHVs can potentially cause internal conflicts within the village as TBAs and other village leaders who cooperate in project activities do not receive any form of financial remuneration. Another related issue is whether payments made to village volunteers is sustainable if this initiative is eventually replicated on a larger scale throughout Mae Hong Son or to other provinces. The next largest budget item was spent on training activities for TBAs [on emergency obstetrics care], village leaders, and health workers stationed at the sub-district health centers and district hospitals. Project implementers felt that these inputs were very cost efficient and effective and can be used for long term sustainability of the project as trainees are able to use their new knowledge and skills on an continual on-going basis.

In Narathiwat the largest budget line item was devoted to strengthening mobile outreach clinic services in rural communities. The largest portion of this budget component [i.e. approximately 50%] was spent on per diem payments to mobile clinic staff as well as for related transportation and administration costs. Although the mobile clinic strategy has greatly increased accessibility to RH services, project implementers are concerned with whether or not this modality is sustainable without external financial resources.

In Mae Hong Son the project conducted a number of activities to improve both the quality and accessibility of comprehensive FP & RH services for women living in remote communities. A key component was to map and review the availability of Traditional Birth Attendants (TBAs) and RH-FP service delivery points for the target population. The MVHVs were trained to provide health education and selected RH services to women and mothers in their respective communities. An important component of the project was to increase accessibility to routine ANC, PNC, FP, EPI, and other basic MCH & RH services through mobile health clinics organized by nurses stationed at sub-district health centers. 1.2 Provision of sufficient UNFPA support to implementing counterparts

A number of factors have affected the expected project achievements within the UNFPA’s influence and control. The most important problem has been the delayed disbursement of project funds, especially at the beginning of each calendar year. Projects generally do not receive any funds for the first three months and this can interfere with the implementation of scheduled activities. These project funds include salaries of MVHVs, the procurement and distribution of HIV/STIs testing supplies, and funds for training and outreach clinic activities. The delay in implementing training activities adversely impacts upon the ability of health workers and peer educators to perform as expected.

Project implementers have commented that the UNFPA has provided a great deal of technical support but this generally comes at intermittent times throughout the project life- time rather throughout the entire project planning and implementation cycle. Another problem is that much of the technical assistance has been devoted to project design as well as conducting various needs assessments, but there has been little support to teach program managers and project implementers how to effectively use or interpret the data for future project planning, implementation, and evaluation purposes. At times UNFPA projects conduct baseline, or “one-time”, surveys but do not fund follow-up survey exercises to help assess the impact of project inputs or interventions. Another related problem is the overall use of project financial resources. For some project the first two years of project implementation are devoted to project planning and assessment exercises, often leaving

23 | Page UNFPA Thailand’s CP9 Evaluation Report insufficient time to fully implement project interventions, strategies, and activities. As such it is often difficult to assess the actual or potential impact of these initiatives as there has not been sufficient time for implementation purposes.

1.2 UNFPA’s efforts to develop managerial capacity to improve the quality of RH services 1.2.1 Improvement of RH Information System The different projects in the UNFPA CP9 have conducted needs assessments as well as designed specific project output indicators, but in reality they have not attempted to systematically improve the existing RH information systems nor have they greatly enhanced the capacity of project planners and implementers to use routine information collected by health workers, MVHVs, nor peer educators. The projects tend to be activity oriented with little attention made to qualitatively analyzing the impact of project inputs and interventions with respect to final project outcomes. Some key indicators, identified under specific projects, do not appear to be linked to project outcomes, which makes it difficult to assess project achievements.

1.2.2 Up-grading Managerial Capacity to Improve Quality of RH Service

The UNFPA CP9 contained two major interventions to deal with up-grading the managerial capacity of program managers, project implementers, and service providers. This included the Management Capacity Development Project (MCD) and Results-based Management (RBM) Training.

• The Management Capacity Development Project (MCD):

The main objective of this initiative was to enhance institutional capacity at national and sub- national levels for the planning, implementing, managing, and monitoring of the RH programmes. This MCD Project was not included in the original RH Programme design, and only came into being during the second year of project implementation in response to specific needs of project implementers. The UNFPA tried to identify potential organizations to undertake managerial training for its counterparts as part of the RH Program and as such invited PBRI to do so. The Praboromrajanok Institute for Health Force Development (PBRI), of the MOPH, is the main unit responsible for health resource manpower development for health personnel in Thailand. One of the shortcomings of the MCD Project is that it was implemented only in Lampang and Mae Hong Son and not in all provinces participating in the UNFPA CP9. Another drawback is that PBRI staff have designed project inputs directly with project implementers at the provincial and/or district with little input and/or administrative support from the national or sub-national level. Key strategies were aimed to increase data- based planning and management skills at all levels, promoting effective and undertaking corrective actions, enhancing the ability of policy makers, programme managers and service providers to encourage client participation in plans and programs and in implementing culturally sensitive activities in specific target areas.

In reality the PBRI has worked with project implementers to conduct training needs assessments as well as several follow-up training activities in Lampang and Mae Hong Son. In Lampang the MCD Project has been implemented with little input from the provincial

24 | Page UNFPA Thailand’s CP9 Evaluation Report health office. In Thoen district interventions focused on personal awareness (i.e. psychology) training for project implementers and in developing appropriate management information systems. At the present time health staff have learned how to design a data collection manual but have not yet learned how to use or analyze data for planning, implementation, and monitoring purposes. In Muang district the training focused on teaching project implementers how to network with community leaders and CBOs to increase community involvement in local health activities. The district health office and sub-district health center trainees have learned how to design appropriate training modalities but they have not actually implemented any of these activities on their own.

In Mae Hong Son province PBRI staff have utilized an on-site training approach to train health workers and health officials from local health centers, district health offices, and the provincial health office how to collect, use, and interpret information obtained at the village level. The training emphasized how to approach cultural sensitive issues with local community leaders and members. In Mae Hong Son the PBRI has been able to conduct several joint training sessions for project implementers in 3 districts. The training is conducted in a sequential manner. Trainees have commented that at first it was difficult to follow some of the concepts introduced or to understand how to interpret data that has been collected from communities. At the present time, however, they are more confident in their ability to do so as well as see the value of this type of training in improving their ability to provide relevant health services to the target population.

One of the most important achievements, or lessons learned, is that PBRI training personnel have also learned how to design specific training programmes that are tailor-made to address the needs of the trainees. PBRI personnel have learned to appreciate that it is crucial to constantly receive input and feedback from the trainees in order to ensure that the training package is relevant. PBRI staff feels that the lessons learned in Lampang and Mae Hong Son should be shared with colleagues from other training institutes as well as with other trainees in provinces that face similar challenges and problems.

The main shortcoming of the MCD Project is that it has focused on improving capacity only at the operational level and not at the planning or national level at the MOPH [e.g. RH planners at the Department of Health].

The MCD Project did not include any of the output indicators included in the RH Programme. A total of approximately $ 205,000 was allocated to the MCD Project [about 7% of the RH Programme budget]. Project funds were primarily used to implement training activities as well as to cover related administrative costs Trainees were extremely satisfied with their experiences during the training sessions and felt that they personally have increased their capacity to deal with local health issues. The trainees in Thoen have stated, for example, that they have learned how to better communicate with others, as well as to listen to what other individuals have to say both in their work setting as well as in their homes and communities.

• Results-based Management (RBM):

This initiative commenced during the second year of the UNFPA CP9 after a new staff member joined the UNFPA Country Office. The main objective of this initiative was to improve the management capacity of key project implementers by learning how to use evidence based data for more effective planning, implementation, and monitoring purposes. This initiative was fully implemented by a UNFPA Country Office staff member who visited

25 | Page UNFPA Thailand’s CP9 Evaluation Report each province in the UNFPA CP9 intervention area to conduct training for all project implementers. One of the shortcomings of these exercises is that although project implementers have learned how to better organize and arrange data for planning purposes the data is not necessarily being properly utilized for monitoring and/or evaluating project achievements.

• Process of Monitoring and Evaluation:

The UNFPA Office organizes and conducts a monitoring and evaluation workshop twice a year, one time at the provincial level and another time in Bangkok. UNFPA Office staff also undertake on-site field visits to monitor specific RH project activities. Overall monitoring and evaluation activities generally focus at the project activity level rather than at the programme level. Annual project progress reports serve as one of the key monitoring and evaluation tools, being supplemented by quarterly financial reports. One of the shortcomings of this process is the overall quality of the annual reports. They tend to focus on summarizing activities rather than on identifying results, problems, constraints, and/or the need to identify or implement new strategies. Another shortcoming is that project implementers generally do not receive adequate feedback from these reports which could help them broaden their perspectives or up-grade their professional capabilities. The annual RH Program meeting takes place over a 1-2 day period. This time frame is insufficient to share experiences and learn from others. The meeting primarily serves as an opportunity to summarize the various activities that have been implemented during the past year.

One of the shortcomings of the overall monitoring and evaluation process is that there appears to be very little documented feedback from the different target populations who are the intended beneficiaries of each of the projects in the RH Programme [e.g. youth, sex workers, couples, women living in remote or underserved areas] to help assess whether project inputs have made a qualitative improvement in their lives and/or their ability to have greater access to RH services.

A total of approximately $360,000 was allocated for M & E activities in the 5 Projects under the RH Programme. This represented around 10% of the RH Programme budget.

1.3 UNFPA RH programme linkages with national policies and priorities

1.3.1. Increasing access for vulnerable populations

The UNFPA RH Programme has been developed to be in line and to support a number of national health priorities. This includes the following:

• The National AIDS Plan and Policies (2007-2011) in which the Thai Government hopes to dramatically expand its HIV prevention efforts, focusing on young people, women, sex workers and their clients. The UNFPA has been asked to serve as a lead agency for HIV prevention among young people and sex workers.

• The National Health Plan which focuses attention on promoting equity and quality of health services for everybody including vulnerable groups.

• The Royal Patronage Maternal and Child Health Programme which promotes maternal safety and close bonding between mothers and newborn infants.

26 | Page UNFPA Thailand’s CP9 Evaluation Report

• The Thai Constitution which contains numerous clauses that outline the rights of Thai citizens that include universal access to health care and social services.

1.3.2. Improving quality of services along with gender and cultural sensitivity

The UNFPA CP9 is also aligned to support several initiatives outlined in the Tenth National Health Development Plan. This includes:

• Improving the overall quality of health services. The plan, however, does not make any specific reference to gender or cultural sensitive issues.

• Improving access to health information for youth.

1.4 Factors contributing to the RH programme’s achievements and shortcomings Several important factors have contributed to RH Programme achievements. They are as follows:

• The project planning process has followed, according to the perspectives of health service providers, a “bottom-up approach”. This has ensured that project implementers have a strong professional and personal interest in participating in project activities

• The quality and relevance of training at the field implementation level. Project implementers have generally been able to obtain new information as well as acquire specific technical skills to help them become more effective service providers and/or project managers. As such they are very enthusiastic when it comes to participating in project activities.

• The ability of the UNFPA to identify and be able to invite NGOs with a wide range of practical experience [e.g. PATH] to serve as technical advisers and/or to introduce new strategies and methodologies into the project implementation design.

There have also been several factors that have led to shortcomings in expected RH Programme achievements. These include the following:

• Although programme managers and project implementers talk about results-based management, in fact the RH programme is guided by a project-based management perspective that focuses attention and resources on the implementation of project activities.

• There are few linkages between the different projects even if they take place within the same province. As such financial and technical resources are not optimally utilized. The service providers, for example, in the MMS, MPM, YSRH projects are usually the same individuals or belong to the same health units, but project activities tended to be implemented in a vertical rather than integrated manner.

• The UNFPA has focused attention on identifying key public sector counterparts that are part of the MOPH rather than other important players from different ministries or from the private and/or NGO sectors. The YSRH Project, for example, should have perhaps included involvement from the Ministry of Education (MOE) or Ministry of

27 | Page UNFPA Thailand’s CP9 Evaluation Report

Social Development and Human Security (MSDHS) rather than simply relying on contacts at the MOPH. The SSRH initiative could have perhaps been better coordinated with the National Health Security Office (NHSO) as the latter is the key funder for basic health services which also includes support to the National HIV Prevention and Reproductive Health Programme.

• The UNFPA has limited interaction with several key policy makers and/or national programme officers and as such the latter are not necessarily aware of CP9 project objectives, achievements, or some of the valuable lessons learned that could have relevance and possible further application on a larger scale in Thailand.

1.5 Cross-cutting Issue of Gender and Rights in RH programme The UNFPA CP9 contained a separate umbrella project devoted to promoting Reproductive Rights and Policy Advocacy (RR). The project was primarily implemented by the Bureau of Reproductive Health, although some activities were directly implemented by the UNFPA, especially with respect to policy advocacy on reproductive and population development issues.

The project acted as an umbrella for the other separate RH projects including monitoring and evaluation. It was, however, somewhat difficult to assess project achievements as clear output indicators were not specifically established for the RR Project.

The Thai government has promoted a number of policy strategies, legislature laws, and working guidelines as well as conducted specific training courses to help health care providers change existing attitudes and practices. Some of the issues reviewed by the evaluation team included:

• Freedom and adequate information for making decision on choices of family planning / reproductive health issues – availability, accessibility, acceptability of services Although each project has tried to promote freedom and adequate information for target populations to have the capacity to make informed choices with respect to appropriate family planning and reproductive health issues the process is still “service provider oriented” rather than truly geared to identifying the needs of potential clients.

• Services which are hand-tailored/designed according to the needs/culture of each of the sub-populations The MMS, MPM, YSRH, and SSRH projects were extremely affective in raising the awareness of health service providers to better understand gender and cultural sensitive issues that affect vulnerable or underserved groups in the project intervention areas.

The projects developed a wide range of IEC materials that assisted health service providers provide counseling and services to the key target groups. The MPM Project, for example, dramatically increased male involvement and made husbands aware of women’s health issues, which also enhanced bonding between husbands and wives during the latter’s pregnancy. According to the responses of health service providers in Narathiwat many Muslim women claimed that they never thought it would be possible to involve their husbands in their pregnancies, and were thus pleasantly surprised to learn that their husbands could be motivated take an interest in their health. Health service providers often assigned husbands, whose wives were

28 | Page UNFPA Thailand’s CP9 Evaluation Report

at risk to anemia during pregnancy, the responsibility of monitoring their wives’ folic acid medication intake. As a result, one pregnant woman expressed the opinion that she was thankful for “her anemic condition” as this allowed her to feel her husband’s love.

One pilot activity, implemented under the STRONG Project, was tailor-made to address the needs of female sex workers. This was the introduction of “female condoms” aimed at empowering vulnerable women to be better able to protect themselves to prevent HIV/AIDS and STIs. By using female condoms the sex workers did not need to negotiate with male clients to use condom.

• RH Service for People Living With HIV/AIDS (PLHIV) A separate initiative under the RH Programme focused on providing counseling and other RH services to PLHIV. The initiative was implemented at the central level in Bangkok under auspices the Department of Health, MOPH, as well as at 60 hospitals in 12 provinces. The main objective was to raise consciousness and awareness of the rights and RH needs of PLHIV. In collaboration with local non-profit organizations, namely AIDSNET and the Raks Thai Foundation, as well as with community groups such as the Network of Northern People Living with HIV/AIDS and the Network of HIV Positive Women, the Bureau of Reproductive Health developed a curriculum entitled “Provision of Reproductive Health Services to People Living With HIV/AIDS: a Curriculum for Peer Educators”. The Bureau also published a protocol and operations manual of RH services for PLHIV.

One of the key achievements of this initiative is that an important step has been taken for Thai society in declaring that PLHIV have the right to access RH services. Reproductive health services have accordingly been especially designed and made available to respond to the needs of HIV positive people.

The initiative trained both peer educators (i.e. PLHIV) and hospital staff by using newly designed culturally sensitive curricula. One of the main tasks of the peer educators was to educate fellow PLHIV about how to access appropriate RH services. One of the shortcomings of this strategy was that peer educators were trained only once, and this is probably insufficient to address the various needs and concerns of PLHIV who are often stigmatized or marginalized in Thai society and who may hesitant to seek health services in the public sector. Another weakness in the project design was that the Bureau of Reproductive Health did not make any provisions for any follow-up support for peer educators through on-going monitoring and evaluation of their activities.

29 | Page UNFPA Thailand’s CP9 Evaluation Report

II. Population and Development (PD) Programme

2.1. South to South Cooperation in Reproductive Health and Population Development

This project was implemented with Thailand International Development Cooperation Agency (TICA), Ministry of Foreign Affairs, which was designed to achieve the following outputs and outcomes:

Outcome: Improved knowledge-management and networking mechanisms for inter- country communication and action on population, reproductive health, HIV/AIDS, gender, migration and population ageing issues in the region.

Output: Enhanced capacity and improved mechanisms for S-S in sharing experiences, information and professional and technical know-how in population in related areas.

This project is in line with the Country Programme Action Plan (CPAP) for 2007 to 2011 TICA and UNFPA agreed to promote Thailand’s contribution on South to South development as well as to Thailand to take a leading role in sub-regional and regional cooperation initiatives. Thailand has a great wealth of experience to share with other countries regarding poverty reduction policies and programmes as well as to demonstrate how to increase accessibility to reproductive health service for vulnerable groups.

Achievements: A summary of South to South Project activities are presented in Table 1. With respect to international training, a total of 8 training sessions/workshops were conducted. One workshop was cancelled in 2010. Late submission of nomination requests for trainees, from recipient countries as well as lack of relevant documents, caused the cancellation. It is difficult to assess whether or not the trainees are eventually able to apply their new knowledge and/or skills as the project design did not make provisions for further trainer-trainee follow-up. The training topics varied from year to year. The two key Thai institutes employed to conduct training, from 2007-2009, were the Chiangmai University Faculty of Nursing and the College of Population Study at Chulalongkorn University.

An important component of this project was to identify and up-grade the capacity of appropriate Thai academic institutions that could subsequently conduct training activities for South to South member countries invited to Thailand. This activity was conducted in 2007 and 2009.

In 2010 two new initiatives were undertaken by the project. This included knowledge management workshop and the development of a sharing of experiences network between Thai and other country institutions. A website was developed in 2010 to facilitate the sharing of information for interested parties.

It is difficult to assess why certain activities were discontinued as these developments were not documented in annual project reports.

The Project M&E strategy includes (a) monthly meetings between TICA and UNFPA to follow-up with planned activities and (b) the submission of evaluation report by training institutions after the completion of training courses. The evaluation team was not able to review TICA’s annual reports during the evaluation exercise, thus it is difficult to assess the level of follow-up provided by this agency.

30 | Page UNFPA Thailand’s CP9 Evaluation Report

Table 1: Summary of South to South Project Activities 2007-2010

2007 2008 2009 2010

ƒ Field assessment • Training on • Workshop to • Consultative of universities in Enhancing Enhance and Workshop on the southern and Community Strengthen Training Making northeastern Capacity in Management of Motherhood region Managing the Potential Thailand RH Safer (MMS) ƒ Training on Emerging and PD Institutes. Moving Toward Challenges of • Annual Meeting- Health Ageing: Population • Workshop on Program The Promotion of Ageing Population Ageing Review the Elderly’s Well- being Through • Workshop on Community Monitoring Migrant Participation Health.

• Training on • Sharing Experiences ƒ Monitoring and Migration and Forum to Evaluation of Health Related Commemorate Adolescent Issues the15th Anniversary of Reproductive ICPD Conference Health • Needs Assessment and Recommendations Survey Among Participating Country Members

Relevance : This programme is aligned to the global partnership objectives under UNPAF as well as with Thai foreign policy priorities. The Royal Thai Government, through TICA, has expressed interest in providing collaboration with other countries in the region and as well as would like to be part of the UN agency network. The RTG has already set aside its own budget allocations to support collaboration for capacity development among developing countries.

Effectiveness : The Project was able to conduct international training and workshops according to request made by participating countries through the joint collaborative efforts of TICA and the UNFPA.

The project offered Thai academic institutions an opportunity to improve their training management capacity. The participants, both trainers and trainees, have generally expressed a high level of satisfaction with the quality of training and/or the new knowledge gained during project activities. A total of 122 participants enrolled in training courses and workshops.

South to South Cooperation Strategy was revised and Institution to Institutional Capacity Development model was developed to ensure long-term and sustainability of the

31 | Page UNFPA Thailand’s CP9 Evaluation Report collaboration and network between Thai and other institutions in participating countries through. Maternal Health Care, Sexual Education, Data Management and Older Persons are identified as the key areas for improvement from the member countries.Instituton to institutional capacity model have been started and implementing on Maternal Health Care and Sexuality Education with Laos PDR, Bhutan, Vietnam, Indonesia, the Philippines, and Cambodia.

Website was developed and implementing as the South to South cooperation sharing mechanism.

Efficiency : This is the only Project in the UNFPA CP9 where the Thai government has allocated a supplementary budget for project implementation purposes. The project contribution from the UNFPA of approximately $235,000 represents about 3-7% of the annual UNFPA CP9 budget.

Contribution 2007 2008 2009

RTG 71,000 83,100 87,687

UNFPA 71,000 85,100 79,731

In discussing whether the South to South Project is efficient, it should be noted that both TICA and the UNFPA follow very rigid administrative and financial guidelines so that it is a very lengthy process to approve and fund requests. This has led to delays in implementing project activities and accordingly the project has only spent only about 50% of the allocated budget.

Sustainability : TICA has been supporting South to South collaborative efforts with many UN and other international agencies for many years. TICA has the financial capacity to continue this type of support to improve RH and PD programme networking and communication between Thailand and participating countries in the region. The UNFPA can assist this process by helping TICA and Thai institutions identify effective transferring of knowledge/experience modalities to other countries.

Impact : It is still too early to determine the impact of transferring Thai expertise and experiences to other institutions or countries in the region. The level of formal networking between institutions and/or experts is still relatively low, and it will probably take at least several years to observe more concrete results.

2.2 Monitoring and Strengthening Statistical Capacity (NSO) Output 4: Enhanced capacity at national and subnational levels to collect and analyze data and to conduct policy research on development issues, including reproductive health, HIV/AIDS, gender, migration and population ageing.

Achievement : Three main NSO efforts were the focus of these activities. These were the 2010 Population and Housing Census, the 2009 Reproductive Health (RH) Survey and the National Survey of Older People (OP) (conducted in 2007 and planned to be conducted

32 | Page UNFPA Thailand’s CP9 Evaluation Report again in 2011). In addition, UNFPA is producing a monograph on the impact of demographic change in Thailand that will be released in 2011. This output utilizes NSO surveys and the previous Census with the objective of packaging the information in a useful form for Thailand’s next strategic planning cycle.

Budget and expenditures by strategy and output are seen in Table 2. The annual work plans for the NSO project allocate about half of the total budget to Census activities, mainly in the realm of advocacy and capacity development. The RH and OP surveys received relatively minor budgetary allocations (17% total). Other outputs received 18% of the budget; this is largely allocated to the monograph (13%). Administrative costs, which included field visits, technical backstopping and the salary of a programme assistant who worked on all of the Population and Development activities, were allocated 16% of the budget. By strategy, advocacy accounted for about one-quarter (24%) of the budget, capacity development 25% and data utilization 35%

In terms of actual expenditures for the four-year period, advocacy activities took 39% and data utilization activities were reduced to 18%. By output, 64% of the expenditures were on the Census and only 7% was spent on the OP & RH surveys. Many of the discrepancies between the annual work plan and activities took place in 2007, before a formal agreement with NSO was in place and before priorities were set (see more on planning below). Others are due to changes in priorities; in particular there was an urgent need to contribute to advocacy efforts for the census in 2010. Efforts of some key personnel were shifted from survey to census work, and thus some planned survey activities were reduced in scope.

Table 2: Proportion of NSO Project Funds Budgeted and Spent by Major Strategy and Output, 2007-2010 Budget Expenditures Strategy Advocacy 24.2 38.8 Capacity development 25.2 24.5 Data utilization 35.0 18.1 Administrative 15.6 18.6 Total 100.0 100.0 Output Census 49.3 63.8 RH survey 9.1 4.2 OP survey 8.0 2.6 Other 18.0 11.7 Administrative 15.6 17.8 Total 100.0 100.0 Total budget and expenditure $440,664 $405,640 Implementation rate 92.1%

While the NSO project output calls for strengthening research on “reproductive health, HIV/AIDS, gender, migration and population ageing”, not all of these topics were addressed in the current period. Reproductive health and population ageing were the major focus of efforts. Priorities shifted in alignment with information required for policy advocacy as well as the availability of the evidence. The RH Survey included questions on antenatal HIV screening, HIV knowledge and sexual behaviour among adolescents. Regarding gender, the RH survey included a sample of both male and female adolescents aged 15-24 to examine

33 | Page UNFPA Thailand’s CP9 Evaluation Report adolescent sexual behaviour and contraceptive youth (for both married and single adolescents. There were no questions on men’s role in reproductive health however and older men were not interviewed. Though there were no specific activities focused on migration, umbrella funding was used to support the dissemination of a report on migration’s impact on the elderly.1

Results-based management efforts were weak for the NSO project. Annual reports were perfunctory, only documenting budgetary expenses and activities, without any analysis of lessons learned or concrete achievements. The indicators for the project (Table A1.2 in Annex 1 ) were for the most part not measurable, either because the wording of the indicator is not specific (“Extent of NSO's participation in reviewing methodology and analysis of data”) and/or because no targets were set (“Advocacy materials produced/ distributed”). There are plans to use a Likert scale questionnaire with key informants from NSO in 2011 to help monitor the project.

It is important to note that the NSO project was planned with the full participation of NSO. Indeed, the NSO conducted a SWOT2 analysis in 2007 to identify gaps that were not being funded by the RTG or other sources. NSO specifically identified areas where they requested UNFPA involvement. Thus the UNFPA marshalled resources to provide technical expertise and funding in response to NSO’s own needs assessment. NSO staff who were interviewed appreciated this participatory design as it met their needs directly; they also found the UNFPA to be flexible in planning as priorities changed.

The key achievements of the NSO project are summarized in Figure 2.

Relevance : With the increased emphasis on evidence-based planning in the public sector and elsewhere, and with results-based management becoming the norm for providing transparency and accountability in funding, the collection of high-quality data on a national and subnational level is a clear priority. Moreover, there is a critical gap in skills needed for utilizing data for policy development and planning. This is particularly true at the local level, as decentralization shifts decision making to provinces and localities.

The MIC study phase II called for the UN to make “fundamental changes in approach and behaviour” if the UN is to have a true partnership with Thailand.3 In particular, the study pointed out that a true partnership would mean that the UN would give authority to the RTG for planning and identifying needs.

UNFPA was proactive in forming a true partnership with NSO in 2008. This was accomplished through planning for the project based on NSO’s own needs assessment. Most of the activities were designed based on specific requests from NSO. On that basis, the relevance of the themes pursued by the project is clear. The largest percent of UNFPA funding went to Census efforts, both for capacity development and advocacy. This is a clear priority for this period, particularly in a time when rapid changes in technology and in the responsiveness of Thai citizens to the Census effort compel that methodologies be modified. The importance of the National Survey of Older People, in providing a greater understanding

1 Knodel, J., Kespichayawattana, J., Wiwatwanich, S. and Saengtienchai, C. 2007. Migration and Intergenerational Solidarity: Evidence from Rural Thailand. Bangkok: UNFPA. 2 Strengths, weaknesses, opportunities and threats. 3 Porter, I.C. and Suddhi-Dhamakit, S. 2010. UN Operations in a middle-income country: formulation of a strategy for enhanced UN coherence and effectiveness in Thailand – Phase II. p.7.

34 | Page UNFPA Thailand’s CP9 Evaluation Report of older people’s situation and making this information available to policymakers, meshes well with national priorities in a rapidly ageing society. The Reproductive Health survey provides critical information for understanding issues such as adolescent sexual behaviour and contraceptive use, and for understanding gaps in reproductive health that may be utilized for planning at the local level.

While the activities planned for the NSO project are clearly relevant to national priorities, some needs were not addressed during the CP9 period. The most critical of these is the need for capacity development at the local level for utilizing data. Two training workshops on improving Census methodologies (held in 2007 and 2008) included local NSO staff while the other capacity development activities focused on central and higher level staff, with a few provincial staff from larger provinces included.

35 | Page UNFPA Thailand’s CP9 Evaluation Report

Figure 2: Key Achievements of the NSO Project 2007-2011

2007 2008 2009 2010 Advocacy Advocacy Advocacy Advocacy

ƒ Consultative meeting on use ƒ ƒ Workshops held for ƒ Advocacy activities and of vital registration for the parliamentarians, line meetings for the Census Census ministries and the private sector ƒ Consultative meetings on Census ƒ Poster contest for Census ƒ IEC materials for Census

Capacity Development Capacity Development Capacity Development Capacity Development

ƒ Training on Census ƒ SWOT analysis by NSO to ƒ Technical support on RH ƒ Census Quality Assurance methodologies sharing set priorities and gain survey Project international experiences ownership of the project ƒ Study visit to Malaysia ƒ 2nd Training on Census methodologies sharing international experiences ƒ Study visit to India

Data Utilization Data Utilization Data Utilization Data Utilization

ƒ Review of the Survey of ƒ Technical review for ƒ Re-analyze and re-package ƒ Dissemination for RH survey Population Change MDG/ICPD monitoring OP survey ƒ Monograph on Impact of ƒ Re-analyze and re-package Demographic Change the OP survey

36 | Page UNFPA Thailand’s CP9 Evaluation Report

Effectiveness : Following is an assessment of the effectiveness of UNFPA’s work in filling the gaps in technical support for the major outputs with involvement during this period.

• For the major output, the 2010 Census of Population and Housing, it is not yet possible to make an assessment of the effectiveness of the NSO project at the output level—that is, the quality of the data or the extent that it is used for policy and planning. Also, it would not be appropriate to hold UNFPA responsible for the overall quality of the Census, as its role is to fill needed gaps using technical support in a partnership framework.

• Data from the National Survey of Older People has been used by the National Commission on the Elderly for their annual report, in papers for international conferences and in international journals.4 Plans are underway to collect the next round of the survey in 2011 with technical assistance from the College of Population Studies, Chulalongkorn University. This survey makes a valuable contribution to knowledge on older people in Thailand.

• As discussed below, dissemination efforts for the Reproductive Health Survey were delayed by problems with the estimation of key indicators and analysis issues. A preliminary report has been released and the full report will be released shortly.

• Although not planned at the outset of CP9, the monograph on the Impact of Demographic Change in Thailand has achieved a high profile, as a high-level dissemination symposium was recently held with the Prime Minister giving the keynote address. The monograph received a very favourable response and discussants felt that it provided extremely useful information for understanding population dynamics and for planning.

More general observations of the effectiveness of the NSO project activities are given below, described in terms of whether they achieved the planned results and thus made a contribution to the output.

• The activities were effective in building up networks for technical assistance and advocacy. Stakeholders cited the relationships that they made through the activities as continuing and valuable. The Assistant Representative at UNFPA who is the focal point of the NSO activities was described as a valued and well-respected colleague who was influential in advocating for the collection of high quality data and for using it effectively. This positive relationship has contributed to the project’s effectiveness. Moreover, the technical assistance needs were identified by NSO’s own analysis, allowing NSO to take ownership over the priorities of the project.

• Many of the activities were one-time events, such as workshops, consultative meetings or technical reviews. Such activities are effective if they lead to make a lasting

4 National Commission on the Elderly. 2009. Situation of the Thai Elderly 2008.; Sakunphanit, T. & Suwanrada, W. 2010. Thailand Universal health coverage Scheme (UCS) 500 Baht Universal Pension Scheme. Presented at the Workshop on Sharing Innovative Experiences on “the Social Protection Floor”, Turin, Italy, 8-9 July 2010; Lloyd-Sherlock, P. 2006. Identifying vulnerable older people: insights from Thailand. Ageing and Society 26: 81-103.

37 | Page UNFPA Thailand’s CP9 Evaluation Report

contribution to skills or result in the development of a continuing strategy. Based on these criteria, the effectiveness of the project was mixed. Some activities (the technical assistance on vital registration data, the study visit to India, and the technical review of the Survey of Population Change) appear to have no long-lasting results. Others, such as the advocacy workshops, were universally seen by stakeholders to have reaped benefits in terms of visibility and networking. Stakeholders said that the opportunity for NSO staff to get to know high-level decision makers through these activities had given the office greater visibility, and that informal networking between the office and parliamentarians continued. In terms of capacity development, the partnership with CPS on the OP Survey, the study visit to Malaysia and the international consultant visits were seen to be particularly valuable in the perception of the participants, as they were able to see new methodologies with their own eyes and attained confidence that they could be put in place in Thailand.

• The timing of some of the activities limited their effectiveness. Some stakeholders felt that the efforts to combine vital registration data and the Census were ineffective and came too late for them to be integrated into planning for the 2010 enumeration. The technical assistance on the RH survey did not begin until after the data was collected, and several flaws in the design prevented the ability to calculate some key indicators (such as unmet need and the total fertility rate). While this was due to the selected consultants being unable to provide assistance at the appropriate time, the results reduced the usability of the data.

• Some stakeholders cited the lack of resources for advocacy work for the Census as limiting its effectiveness for raising awareness to the general public (for example, funding was not available to hold a workshop for the media). However, the responsibility of UNFPA in doing public relations for the Census must necessarily be limited, as such responsibility lies with the RTG.

Efficiency : The largest expenditures for the project from 2007-2010 were the advocacy activities for the Census in 2009 ($96K). This accounts for nearly 25% of the budget for the four years of activities. Activities included three high-level advocacy meetings, a poster contest for schoolchildren, IEC materials and several consultative meetings. Because the advocacy workshops were designed to attract stakeholders at the highest levels of government, as well as the private sector, it is understandable that the cost of this activity was high; discussions with the stakeholders involved indicate that the activities were planned as economically as possible. The workshops did attract senior level government officials and industry leaders.

The other large expenses for the project were the monograph on demographic impact ($43K), the Census consultative meeting in 2007 which included consultants from other countries ($34K) and support for a programme assistant for the whole Population and Development programme (from $14K to $34K in 2008-2010). Other activities, such as support to the RH and OP Surveys, were mainly on a small scale—less than $10K. The larger question about efficiency is not whether there was wasteful spending on any one activity, but whether the activities themselves were effective. In the case of the RH and OP surveys, the technical assistance was given in direct response from NSO’s request, and the two surveys investigate critical issues for Thailand. In other cases, although the amounts involved were small, some of the short-term consultancies did not have a sustained impact on capacity development.

38 | Page UNFPA Thailand’s CP9 Evaluation Report

Sustainability : When asked whether the activities they had been involved with would have a sustainable impact, stakeholders had a generally favourable view. With regard to the Census, even though some of the activities did not result in a change or improvement in methodologies for 2010, stakeholders felt that the foundation is set for improvements in the future. This would include further refinement of the multi-modal methods, possible exploration of the use of vital registration data, and greater quality control. The Quality Assurance project, which sent expert demographers into the field to observe Census training and procedures, has a set of recommendations for improving the quality of the Census in future rounds. However, the final report has not yet been received by UNFPA. Key informants that were interviewed felt that the advocacy efforts have improved support for the Census and other data collection efforts. This foundation includes the informal networks for technical assistance that have been created, enhancing informal information sharing and ad hoc requests for technical support. Informal international networks have been created through the capacity development workshops and study visits as well, and stakeholders felt that Thailand can now give technical assistance as well as receive it.

Impact : The strengthened partnership between UNFPA and NSO, and between NSO and technical experts from academic institutions, contributes to the availability of high quality data and its use for planning and policy. For sustained impact, the foundation built by the NSO activities needs to be continued, and to be sharpened in some areas. Lessons learned from the Quality Assurance of the Census and the visibility gained from the Census advocacy activities should be cultivated and energized so that these efforts are institutionalized.

2.3. Evidence-based Responsiveness to the Emerging Challenges of Population Ageing in Thailand (OP)

Output 7: Improved access of older persons to economic security, health and social services.

Achievements : Details of the project activities are found in Figure 3. There are two major achievements of the OP project:

1) A model to meet the multidimensional needs of older persons was established in Lampang and Chiangmai with multi-sectoral participation from NGOs, line ministries and academic institutions. The model is also in the process of being established in Mae Hong Son. The project activities are implemented by various organizations through two main partners, namely the Faculty of Nursing, Chiangmai University (FoN/CMU) for the health component and HelpAge International (HAI) for the economic and social component. The model is evidence-based, using research and development (R&D) and participatory action research (PAR) for planning and monitoring. Key components of the model include the development of institutional- and community-based long-term care and hospice care. This was accomplished through training, experience sharing, meetings and study visits of relevant groups on health care and services and issues related to health of older persons. At the same time, the project enhances economic and social security for older persons by establishing community saving groups and occupational groups. It also promotes older persons’ empowerment through the establishment of Watch Groups, which provide information on older persons’ rights and assist them in accessing their entitlements and

39 | Page UNFPA Thailand’s CP9 Evaluation Report available services, and advocacy groups which engage with local authorities and submit funding proposals for social welfare activities.

The purpose of establishing the model was to raise awareness and provide evidence for advocacy on policy issues affecting older persons. Thus its objective is to investigate older persons’ needs using participatory methods and demonstrate how they can be met at the community and provincial levels. As a model project, its outputs go beyond increasing access to services at the local level but include showing the way for developing policies at the national level that will ultimately increase access for all Thai citizens.

2) By using the evidence generated by the model programme, the OP project contributed to several major policy decisions at the central level. These included: adoption of long-term care as a priority issue for national policy; revision of the national social pension scheme from a means-tested older persons allowance to basic universal social pension; approval of the amendment of the Social Security Act, Article 40 to increase benefits for informal sector workers; approval of contributions from the central and local governments to community- based saving funds; and revision of the Second National Plan for Older Persons Policy. Advocacy activities included efforts by UNFPA’s country office to increase and influence the policy dialogue on older person’s issues, collaboration with national and local media to increase coverage on ageing issues, and the creation of older persons’ networks that participated in policy advocacy. National conferences and forums have been held to raise awareness of the society on the significance and necessity of promoting older persons’ quality of life through long-term care.

40 | Page UNFPA Thailand’s CP9 Evaluation Report

Figure 3: Key achievements of the OP project

Sub‐National Level National Level

1. R&D in

Health Care Improved access 4. Policy Advocacy 2. Enhancing of older persons Social & Economic & Awareness Security Raising to economic security, health & 3. Capacity Building & Empowerment of older persons

1. Health Care and Services 3. Capacity Building and Empowerment Models for health care and services were established in implementing sites: Capacity building, knowledge sharing and Lampang employment of older persons were developed through: • Integrated institutional LTC and community-based care in Muang and Hang • Trainings, meetings, workshops, Chat districts conferences and study visits at national Chiangmai and international levels • Nursing care in OP residential home • Older Persons Watch setting in Muang district • Multi-purpose centres • Community-based LTC and Hospice care in Muang district 4. Policy Advocacy and Awareness • Community-based LTC in the lowland and Raising highland areas of Mae Taeng district Mae Hong Son Policy advocacy and policy dialogues and awareness raising were conducted through: • Comprehensive LTC integrated with OP Multi-purpose centre in Muang district (on- • National conferences and forums going) • Community-based care for ethnic OPs in • IEC materials the highland area of Mae La Noi district • Media (radio broadcasting, posters, short (on-going) film and VDO clip contests) 2. Economic and Social Security • National Day of Older Persons Economic and social security was enhanced • International Day of Older Persons through:

• Community saving groups

41 | Page UNFPA Thailand’s CP9 Evaluation Report

Relevance : The need for preparing Thai society to respond to the future challenges of the rapidly ageing population has been emphasized in the Second National Plan for Older Persons (2002-2021) and the Act on Older Persons (2003). The Tenth National Economic and Social Development Plan (2007-2011) also addresses the transition to ageing society and its impacts. The project is also in line with the three pillars of Madrid International Plan of Action on Ageing (MIPAA 2002), namely older persons and development, advancing health and well-being into old age, and ensuring enabling and supportive environments. The relevance of the issues is also evidenced by the achievements of the programme in advocating for Thailand’s policy development on older persons’ issues mentioned in the previous section.

Effectiveness : As outlined above, successful results from project sites have been used for raising awareness and for evidence-based policy advocacy at the sub-national and national levels. The effectiveness of the project can be attributed to several factors:

• Significant coordination efforts by UNFPA to foster collaboration among concerned GOs, NGOs, civil society and stakeholders at the national and sub-national levels. Regular and in-depth monitoring by UNFPA of the ongoing activities with guidance and valuable suggestions provided.

• As an international organization, UNFPA has more voice and credibility than other local organizations in policy advocacy and raising awareness in Thailand.

• Multi-sectoral approach used to address the emerging challenges of rapid population ageing, involving multiple organizations and requiring collaboration from all relevant parties.

• Evidence-based design, using the findings from the 2007 situation analysis and continuing participatory research investigating the emerging challenges of older persons in project sites. All stakeholders, GOs and NGOs such as older persons’ groups, community- based organizations, TAOs, public health workers and concerned government officials at community and national levels are involved in every step of the research and planning process. This evidence has been used as key inputs into policy and project planning processes and development of the interventions according to local context and settings. Feasibility studies on saving for old age and job opportunities for older persons were conducted to ensure the possibility of the proposed activities.

• Gender issues have been addressed through building in women’s participation and including the gender issue in training sessions and IEC materials. Monitoring tools include participation of older people disaggregated by gender. Women’s participation in community-based activities averages around 60%. Women leaders are identified and highlighted in the programme.

However, stakeholders raised concerns about some aspects of the project. Though the involvement of multiple sectors with a broad range of technical expertise is a vital part of the project, the complexity of the project has its challenges. Some of the partners lack formally stated roles and responsibilities, which in some cases has caused confusion and delay. With limited resources, it is at times difficult to include all partners efficiently or effectively. Additionally, some stakeholders pointed to gaps in in-depth collaboration at community as

42 | Page UNFPA Thailand’s CP9 Evaluation Report well as national levels, both within and between project components. Communication gaps between implementing partners sometimes led to difficulties for people in the sites.

With regard to gender, there is some indication that older women are less able to participate in the awareness raising and advocacy activities that are held outside of their own communities; only 40% of participants were women on average in 2010. Many factors contribute to this, including mobility, health, education and cultural issues around women’s leadership. The issue is being addressed through highlighting the women leaders who do exist, giving them a stronger voice with the goal of modelling women’s full participation.

Results-based management efforts of the OP project were weak. The output indicators for the project as specified in the Annual Work Plans (AWP) (Table A1.3 in Annex 1) are not measurable. For example, “increased meaningful participation of older people” is not defined. Recently a more specific set of indicators for each component was developed with the help of consultants. However, no targets have been set for these indicators. Since the purpose of project indicators is to clearly define how the project results are measured, and to use as a tool for RBM from the outset, it is unfortunate that the indicators have not been available for use until now.

Efficiency : The OP project had the biggest budget of the CP9 portfolio ($964,910 from 2007-2010 including $155,949 from United Nations Trust Fund for Human Security (UNTFHS)). As seen in Figure 4, the budget increased as full implementation was reached. The corresponding budget for FoN/CMU and HAI increased proportionally, though the amount budgeted for UNFPA’s central activities declined from 2009 to 2010. Expenditures by UNFPA were lower than budgeted in 2009 and 2010. Overall the project spent 91.3% of the budget in the four-year period.

In Figure 5 the proportion of expenditures by type of activity is shown. One-third of expenditures were on advocacy activities and awareness raising, with about one-quarter on research and development on health care and 10% on social welfare activities. The remainder of the budget, about one-third, was spent on management and M&E (14%) and on operations (20%). While this proportion of expenditure is not unreasonable given the nature of the project, it can be said that significant coordination efforts were required to foster collaboration among concerned GOs, NGOs, civil society and stakeholders at national and sub-national levels. These efforts were contributed by UNFPA and the two implementing agencies. As a model programme, it was important to have the involvement of a wide number of stakeholders both from GOs and NGOs, and the coordination expenses were seen as necessary for start-up and for advocacy. Alternative designs that would have been less expensive, such as involving fewer stakeholders, would likely have been less effective for advocacy efforts.

43 | Page UNFPA Thailand’s CP9 Evaluation Report

Figure 4: Budget and Expenditures by Year and Implementing Agency, OP project 2007-2010

Budget 400,000

350,000

300,000

250,000 UNFPA HelpAge 200,000 MoPH 150,000 MSDHS FoN, CMU 100,000

50,000

‐ 2007 2008 2009 2010 Expenditures 400,000

350,000

300,000

250,000 UNFPA HelpAge 200,000 MoPH 150,000 MSDHS FoN, CMU 100,000

50,000

‐ 2007 2008 2009 2010

44 | Page UNFPA Thailand’s CP9 Evaluation Report

Figure 5: Percent Expenditure by Type of Activity, OP Project 2007-2010

R&D on health Enhancing care and economic and services social security 23% 10%

Management and monitoring and evaluation (M&E) 14%

Policy advacacy, awareness raising and Operations and capacity miscellaneous building 20% 33%

Sustainability : The major achievement of the OP project is its contribution to several major policy changes at the central level that address the needs of the ageing population. Thus the project has led to long-term benefits for its target population. Because the model programme was set up to provide evidence for advocacy, scaling up the model or exxtending it to other sites was not the main objective. Nevertheless, the success of the model at the community level would argue for replicating at least some components in new locations. The demonstrated results could be promoted at the local level in other provinces, perhaps in a less comprehensive framework. UNFPA could play an effective role in continued advocacy to extend the model, with government counterparts taking the main responsibility.

The achievements of the project to-date have created a constituency to support its continuation. Stakeholders express their willingness to continue the activities. Their enthusiasm and eagerness to participate in the project are clearly observed. The older persons’ groups established by the programme continue to actively engage with their respective local authorities.

However, some internal management issues affect the prospects for sustainability of the model. Because lines of responsibility were not clearly established at higher levels, local administrative officers were not always willing to take part. While the project activities should be complementary with the existing system and resources, some “push” might be required to influence local GOs to engage in the project as policy. The previous success at the central level may be leveraged to establish the systems needed to establish the model.

In some ways, the success of the project has resulted from key individuals rather than clear systemic mechanisms. For this reason, any changes in the focal point or relevant partner may affect the working process significantly. For sustainability, attention should be paid to systemizing the structure of the model, rather than have it depend on individual or organizational relationships.

45 | P a ge UNFPA Thailand’s CP9 Evaluation Report

Impact : Evidence of the long-term impact of the project is seen in the success of the advocacy efforts in developing and revising government policies, outlined above. Concrete social and economic benefits for older persons will result from these new policies. Awareness raising at the local and national level was accomplished with a firm foundation in evidence that demonstrates OP needs and the potential for community-based participatory programmes to meet them. Besides the impact on policy, the project has clearly created greater access to services for the model programme’s beneficiaries. Multipurpose and rehabilitation centres have been established in project sites, and health care policies and strategies have also been clearly stated in the community plans.

46 | Page UNFPA Thailand’s CP9 Evaluation Report

Part 3: Conclusions and Recommendations

I. Key Findings & Recommendations for the RH Programme

1. Sexual Reproductive Health Services Including HIV Prevention for Young People

1.1 At the present time government policies acknowledge that RH services need to be improved and provided for youth. This issue was included as an important component in the RH Strategic Plan #1 implemented by the MOPH and allied partners. A key finding of the CP9 evaluation exercise is that the overall process of identifying appropriate strategies and means to increase access to RH and HIV/STI prevention information and services for youth is not complete. As such the UNFPA should continue to play an important technical support role to increase the capacity of government policy makers to get fully involved in and support programme initiatives. At the present time programme managers and health services providers continue to focus attention on implementing activities rather than on evaluating whether or not outputs contribute to improving accessibility or the quality of services, or whether the interventions truly address the needs of youth.

1.2 The pilot projects, implemented under the UNFPA RH Programme, have supported more choices and opened new channels for young people to access services. These include establishing interactive websites, utilizing school-based peer educators, and setting up a YAP network. The results of these new strategies should be better disseminated to appropriate policy makers in different ministries in order to make more optimal use of the valuable lessons learned. These lessons should be incorporated into the experiences of other related agencies, such as the Global Fund, different departments/organizations under the auspices of the MOPH, and other ministries, all of which have expressed interest in developing more expansive user friendly policies and programmes for youth. A “knowledge management approach” can be an effective strategy to inform others about good practices and lessons learned as well as help to establish appropriate guidelines for specific situations and settings. This process can be further used as the basis for making useful recommendations for the development of UNFPA CP10.

1.3 The YSRH Project component employed a “peer-educator” model that was primarily school-based. This model has demonstrated that it has the potential to dramatically increase access to RH information and services for youth. One serious drawback, however, is that it does not address the needs of youth who have either graduated from or are not enrolled in school. This obviously includes a considerable proportion of the target population who are also in need of information and RH services including the prevention of HIV/STIs. The project design has also not included the needs of migrants in Thailand the majority of who are adolescents and young adults. Many of these individuals work in factories or other non-school settings. As such it would be beneficial for UNFPA and its partners to work more closely with other government agencies, as well as NGOs, to design appropriate “user friendly” strategies that are more inclusive for youth living in Thailand, and which also can address a more diverse range of RH and social needs than that found in a school-based population.

47 | Page UNFPA Thailand’s CP9 Evaluation Report

1.4 Several government agencies, such as the Ministry of Education, Ministry of Culture, Ministry of Social Development and Human Security have general policies and programmes that deal with youth, but they do not have specific guidelines dealing with RH and/or HIV/STI prevention interventions or services. The UNFPA should accordingly consider working as a key facilitator to promote greater harmony in integrating policies related to youth. In general all relevant agencies need to review their present policies, rules, and regulations on youth so that they are more relevant to real life situations and the expressed needs of this vulnerable target group. The UNFPA can assist in this process by helping to identify lessons learned from other countries that can possibly be adopted and modified so that they can be successfully applied to the present situation in Thailand.

2. Increasing accessibility for sex workers (SW) and their clients with respect to RH and HIV prevention information and services:

A key general finding of the evaluation exercise is that the UNFPA should better collaborate with Implementing Partners (IPs) in the following areas of programme implementation to improve access to services:

2.1. Evidence-based policy improvement on SSRH: Since the MOPH does not have specific policies and/or regular budget support exclusively devoted to sex workers it is difficult at times to access adequate financial resources to address the reproductive health needs of this vulnerable group. During the CP9 life-time the UNFPA was able to allocate additional resources that improved access to quality health services. This included outreach visits, HIV/AIDS & STI screening, as well as care and treatment for sex workers. In order to increase sustainability of project inputs and achievements the UNFPA and partner agencies should disseminate key lessons learned and clearly indicate the financial requirements for further programme implementation. This may include recommendations to modify the existing health service delivery structure so that RH services are culturally designed for sex workers and not merely included as part of general health services. This may also require policy changes from the MOPH, NHSO, and Bureau of the Budget, as well as from service delivery outlets managed under the Universal health coverage (UC) system.

2.2. Quality and continuity of the service: A key finding from the evaluation exercise is that NGOs, such as SWING, have the experience and cultural sensitivity skills to closely work with sex workers to help increase accessibility to information and services. Government agencies, whether they are the MOPH, other ministries, or local government authorities, should consider including new partners when working with vulnerable groups such as sex workers. This may require modifications in the government budgetary allocations process so that non-government partners can access financial resources needed for programme implementation. The UNFPA should try to act as a catalyst to help identify new modalities to modify existing government budget regulation so that NGOs can more easily work with government agencies in a flexible manner.

2.3 Expanding coverage to additional categories of female and male sex workers: At the present time the UNFPA RH Programme has focused attention on female sex workers who operate out of clearly identified sex service establishments. It does not provide coverage to, or address the needs of, other categories of female sex workers who may work on a freelance basis or within other settings. The UNFPA and MOPH should work together to identify appropriate strategies so as to increase RH and HIV/STI prevention services to this large vulnerable group, which may include migrants as well as to young male and female teenagers.

48 | Page UNFPA Thailand’s CP9 Evaluation Report

2.4 Condom Distribution: A major objective of the Comprehensive Condom Programme (CCP) under the UNFPA RH Programme was to encourage local administration organizations [i.e. LAO] to take greater interest in and responsibility for the procurement and distribution of condom commodities within their jurisdiction. A key strategy has been to train senior LAO officials, but the project is still in the early “training of trainer” stage and has not actually begun to train the target population. A major problem is that the nationwide training process is taking a considerable amount of time, without being able to observe whether this modality is appropriate or successful. The UNFPA should consider implementing part of this strategy on a smaller “pilot project” to determine whether the lesson learned justify further modifications in project design and/or replication to a larger geographic area. The CCP Committee should also include senior LAO members to assist in relevant project strategy and implementation design.

3. Policy advocacy to include Reproductive Health (RH) for the vulnerable populations included in the National Reproductive Health Strategic Plan (2010-2014): The present National Reproductive Health Strategic Plan indicates that RH services should be available to everybody as part of Universal health coverage. The plan indicates the need to increase access to RH information and services for youth, but it does not mention other vulnerable groups such as sex workers, migrants, and/or men and women living in underserved areas. The UNFPA Country Office should take a leading role to encourage its key counterparts, especially the Department of Health, MOPH, to establish new guidelines and/or policies that address the RH needs of all vulnerable groups to ensure that they have greater access to information and specific RH services.

4. Encourage male involvement in order to improve the overall quality of RH services for the general population

Key findings from the MPM suggest that increasing male involvement dramatically improves the overall quality of RH health services for women. Dual testing of couples helps reduce the chance of HIV/AIDS transmission between husbands to wives as well as from mothers to children. Male involvement also increases the quality of antenatal services as husbands tend to encourage their wives to more regularly attend ANC clinics as well as to follow recommendations made by health service providers during pregnancy. The UNFPA should help the MOPH apply the lessons learned from this intervention to develop proper guidelines for expanding this strategy to other areas, but it should do so in a sequential and systematic manner to ensure that it is relevant to the local cultural context.

5. Reproductive Health Services for Cultural Sensitive and Underserved Areas

Several projects in the UNFPA RH Programme have demonstrated their effectiveness in increasing access to information and RH services for vulnerable groups in underserved areas. The lessons learned from these initiatives have broad implications for similar populations located in the northern and southern sections of Thailand. The UNFPA should consider organizing a policy forum with the NHSO, MOPH, and perhaps the Ministry of Interior (MOI) to discuss ways to develop specific policies, strategies, and guidelines designed to meet the special RH needs of these vulnerable groups by employing culturally sensitive methodologies. The forum should outline specific measures to undertake as well as identify potential sources of funding that can be utilized to expand and sustain the present efforts.

49 | Page UNFPA Thailand’s CP9 Evaluation Report

6. Increase organizational capacity to properly use a Results-based Management (RBM) approach to more effectively plan, implement, monitor, and evaluate the RH Programme

There is a great diversity in Thailand with respect to social, economic, and cultural issues and situations. As such it is vital that decision makers and programme managers at the national and sub-national (provincial) level understand the context of these issues and their implications while designing and planning implementation strategies. The projects in the UNFPA CP9 have employed a RBM approach, but as indicated above this process has not been implemented in a comprehensive or systematic manner. Although all projects generally commenced with needs assessment activities, the findings were not fully utilized in the different stages of project implementation. For the most part data has been used for planning rather than monitoring and evaluation purposes. Another important finding is that RBM training opportunities have primarily been offered to project implementers and have not always included central level policy makers or provincial level programme managers. Therefore there are a number of gaps in the RBM process, which can indirectly result in lack of ownership by key stakeholders. In addition project implementers have used their new RBM skills to document project activities rather than looking at overall programme objectives and expected outcomes. As such there has been very little analysis of the factors or constraints that have influenced project outcomes and achievements. The UNFPA should, during the next cycle its Country Programme, try to encourage its partners to use the entire RBM approach process by inviting all key stakeholders to participate in the various stages of programme development and implementation.

II. Recommendations for the Population and Development Programme

1. South to South Cooperation in Reproductive Health and Population Development

1.1. Most of the activities implemented under the South to South initiative have taken the form of workshops and short-term training courses organized and conducted by leading academic institutions in Thailand. It is presently difficult to assess whether Thailand’s South to South Cooperation efforts have improved the capacity of individual trainees or workshop participants since there was no systematic follow up mechanism to observe to what extent new knowledge and skills, acquired in Thailand, are being utilized. The UNFPA and TICA should plan and budget resources for follow-up on-site visit activities so that Thai experts can continue to closely work with their counterparts on a longer term basis.

1.2. Many of the requests for Thai technical expertise, under the South to South Cooperation initiative, are geared to the particular short-term needs of a small number of individuals [trainees/participants] and do not include long-term planning or programme goals aimed at building up the capacity of in-country level institutions. The UNFPA and TICA should closely work with their counterparts in other countries to identify comprehensive programme development or institutional capacity building needs with which Thailand, or specific Thai institutions and/or experts can provide assistance.

1.3. One of the strategies of the South to South Cooperation Project is to strengthen regional RH/PD networks, but up until now this process has really not been implemented as originally envisioned. This finding is somewhat related to the one mentioned above with respect to lack of follow-up and communication between Thai experts/institutions and

50 | Page UNFPA Thailand’s CP9 Evaluation Report counterparts/individuals who have received participated in South to South Cooperation Project activities. TICA and the UNFPA should try to identify a focal point to serve in this capacity.

1.4. At the present time it is not clear whether international participants visiting Thailand come from GOs, NGOs, or CBOs. This has important implications for making optimal use of the experiences and lessons learned in Thailand as many successful strategies and interventions were not implemented by the Thai public sector, or at least initially they were designed and introduced by CBOs and NGOs [especially with respect to innovative strategies dealing with vulnerable groups]. As such TICA and the UNFPA need to ensure that appropriate training or “experience sharing” opportunities are opened for members of civil society as well as for members of the public sector.

1.5. As Thailand becomes a MIC it needs to consider efficient and effective alternative modalities to promote its capacity to participate in South to South Cooperative initiatives in order to deal with new country requests based on new emerging issues and/or global changes. This may require a review of how Thailand should provide foreign assistance in general to its neighbours and friends. The UNFPA and TICA should conduct knowledge based management activities to learn how the public and private sectors, in other countries around the world, provide international support or cooperation. Identifying and implementing new collaboration modalities will require the involvement of senior policy and decision makers at the Ministry of Foreign Affairs, the director-general level of relevant ministries, and the UNFPA country representative.

1.6. The UNFPA budgetary cycle is somewhat rigid. Project activities are currently planned for and approved on an annual basis, rather than on a more comprehensive multi-year basis. The latter would enable participating Thai institutions adequate time to make appropriate preparations and arrangements for the provision of technical assistance as requested.

2. Monitoring and Strengthening Statistical Capacity of National Statistics Office

A foundation has been built through previous work on advocacy and capacity development for the promotion and utilization of the 2010 Population and Housing Census. It is important to continue these activities as the census data becomes available and to provide technical assistance on developing methodologies to utilize lessons learned from the Census Quality Assurance Project.

As new techniques continually emerge, it is critical to continue to provide technical assistance to strengthen national statistical capacity and data utilization. However such assistance would be more valuable through long-term partnerships, as some efforts may fail due to a mismatch in coordination process. More long-term and joint planning would help avoid this in the future and allow for greater informal assistance.

The greater attention to evidence-based planning and Thailand’s decentralization has led to a critical need for capacity development on the use of statistics at the local and provincial levels. Attention and resources in this area would be of great relevance and benefit to the country.

51 | Page UNFPA Thailand’s CP9 Evaluation Report

Recommendations:

2.1 Support workshops at the regional level for the NSO to discuss lessons learned from the 2010 Census and to plan for the improvements of the 2020 survey. Long-range planning will help to incorporate with new technological solutions as well as to exchange knowledge with other countries. Workshops should include provincial NSO staff from all provinces.

2.2 Promote and facilitate use of the 2010 Census data, through capacity development efforts on data analysis for NSO and through sponsorship of analysis by academics and others. Promote gender disaggregated analysis where appropriate and also to address the need for data disaggregated by locality, age and urban status.

2.3 Sponsor and facilitate long-term institutional linkages between NSO and academic institutions that can provide technical support in both substantive and statistical areas. These linkages will improve data quality, promote collaboration and increase data utilization.

2.4 Continue the positive partnership with NSO with regular needs assessments and participatory planning. Adhere to the RBM strategy by developing realistic and measurable indicators and targets and by requiring annual reports from NSO on their progress.

3. Evidence Based Responsiveness to the Emerging Challenges of the Population Ageing in Thailand

The Older People (OP) project gathered evidence about the complex needs of Older People and how they may be addressed through a comprehensive, multi-sectoral and participatory programme. It used this evidence to advocate for policy changes benefitting older people’s social welfare and access to health at the central level.

Because the main purpose of setting up the model programme in Lampang and Chiangmai was to provide evidence for advocacy, the replication of the programme to other sites was not a key objective. Moreover, the model programme in its current form is not sustainable, due to the high level of coordination involved and its cost. Nevertheless, the successes of many of the programme’s elements argue for continuing it in some form. Community members are strongly behind the programme and actively involved in its implementation and the participatory nature of the planning and monitoring have made it well-grounded in meeting older people’s needs.

Recommendations:

3.1 UNFPA’s advocacy at the central level to improve policies affecting older people has been extremely successful and should be continued. UNFPA’s voice and credibility for raising awareness is critically important for this issue.

3.2 UNFPA should conduct a participatory assessment to streamline the programme; more cost effective and high-impact components should be maintained, while the level of coordination and the number of stakeholders reduced. Those GOs whose roles and responsibilities are not well defined could be phased out if their participation is not needed for advocacy reasons. Community-based elements that are sustainable with a low level of support and coordination should be retained. If UNFPA plans to continue to do Research

52 | Page UNFPA Thailand’s CP9 Evaluation Report and Development (R&D) for advocacy purposes, this streamlining process will help to make these efforts more cost effective.

III. Programme Management

1. Programme planning and implementation

1.1 Improve the planning process for programme based management:

The UNFPA RH Programme planning and management process is geared to focus on specific projects, or project activities, and accordingly does not facilitate the necessary linkages and integration of new strategies and interventions to determine whether they contribute to the achievement of strategic goals or outcomes. This process is even evident in provinces simultaneously implementing several RH projects. This perspective is also apparently shared by UNFPA staff responsible for different RH projects who rarely discuss potential linkages between the different project interventions in order to better achieve specific programme outcomes. The UNFPA and its counterparts should accordingly make more efforts during the project design and planning stages to specifically identify intermediate output indicators that make appropriate linkages between the various strategies and interventions of each project. These intermediate indicators can serve as useful on- going monitoring and evaluation tools to help assess whether final project outcomes are realistic and feasible.

The UNFPA and its counterparts should also develop a strategic 5 year road map or action plan that contains clear identifiable milestones for each key strategy and/or intervention. This process will help programme managers develop appropriate, realistic, and consistent annual work-plans.

1.2 Improve UNFPA and counterpart capacity to ensure for programme success and sustainability

At the present time the UNFPA Programme has not established clear models to help identify the various factors that need to be taken into consideration in order to ensure the success and sustainability of project interventions. The UNFPA and its partners should clearly identify, during the early stages of programme planning and design, relevant and realistic risks and assumptions that can positively or adversely affect eventual programme success. They should also clearly identify how new strategies and project interventions will continue once UNFPA financial and technical support is eventually withdrawn. Sustainability assessments should identify important stakeholders that need to participate during the project life-time and beyond.

1.3 Increase active participation and involvement of senior policy makers and programme managers throughout the entire programme planning, implementation, and evaluation process

At the present time key stakeholders, including senior policy makers and programme managers, are not involved in all of the major stages of programme design and implementation. This has potentially serious implications for project success as well as for future replication and/or sustainability of new strategies and interventions. The individual project activities within the RH programme are primarily implemented by service providers with little day to day involvement of more senior programme managers. The UNFPA should

53 | Page UNFPA Thailand’s CP9 Evaluation Report explore new modalities to encourage senior decision makers to be more involved in the entire programme process. These individuals are instrumental in helping to identify realistic risks and assumptions during the project planning stages. They can also help identify potential or actual resources for project use, as well as make commitments to sustain successful strategies and interventions in the future.

2. Improve programme monitoring and evaluation

Although each of the UNFPA supported projects conducted Results-based Management (RBM) training sessions project implementers have focused efforts on monitoring activities rather than factors that lead to success of strategies or interventions. They do not focus attention on results or the quality of achievements, but rather tend to document the completion of activities and the expenditure of corresponding project funds. The UNFPA should ensure that its programmes and projects have specific output and process indicators that can be measured within a specified time period. These indicators monitor changes in the target groups as well as system and process changes. The UNFPA should consider modifying its current project/programme monitoring mechanisms which rely on regularly submitted reports and annual meetings. This process does not provide adequate opportunities for “sharing and listening” to and by stakeholders and/or project implementers. The UNFPA should try to facilitate interactive feedback sharing and listening sessions in all provinces where project interventions take place. On-site visit study tours to observe actual project interventions would greatly assist this process.

3. Improving the management external technical support

At the present time much of the technical support provided by the UNFPA has focused on conducting situation/needs assessments, capacity building activities, and evaluating project inputs. This has generally been arranged on an ad hoc basis rather than as part of a long- term technical assistance plan aimed at strengthening institutional or individual professional capacities. The UNFPA should work with IPs to develop a long-term institutional development plan to facilitate more relevant and effective external consultations. The long term institutional development plan should also be geared to help transfer technical capabilities to selected academic institutions and NGOs that could possibly serve in the future as technical consultants/advisors for the UNFPA in Thailand and/or in neighbouring countries.

4. Modify UNFPA roles and priorities so that it focuses more attention and resources on policy and systems reorientation in Thailand

As an emerging MIC Thailand has established an effective infrastructure capable of providing basic health and social services for its population. Within the past decade a number of structural systems reforms have taken place that has changed the way government line agencies operate. The Universal Health Coverage programme and decentralization of power to local administration organizations have resulted in some major improvements in the provision or health services. At the same time some of these changes have not kept pace with other developments so that gaps remain in the provision of important health and social services for vulnerable groups. In the past the UNFPA has focused on “micro-issues” at the field level that directly deal with the needs of specific target populations. In the future the UNFPA should consider modifying its role so that it focuses more attention and resources at the “macro national policy and programme level”. This

54 | Page UNFPA Thailand’s CP9 Evaluation Report includes improving the central level’s capacity to anticipate and plan, as well as to effectively monitor and evaluate, new strategies and programmes that are evidence based. UNFPA pilot projects should be designed to help policy and decision makers understand the needs of specific groups that are not generally covered by existing policies and problems. This is especially relevant for developing practical and relevant policies for vulnerable groups such as migrants, sex workers, youth, and a rapidly ageing population. As local administration organizations take over some of the functions previously delegated and implemented by government line agencies there will be a tremendous need to strengthen the technical and administrative capacity of these units.

5. Expand the role of the UNFPA in providing appropriate technical support on emerging health and social issues

As Thailand becomes a MIC it will undoubtedly face many new and complex social, economic, demographic, and health related issues that it has never previously encountered. Thailand will need to learn how other countries have successfully addressed many of these same issues and to do so it will accordingly need to expand its current “base of external expertise and experience” to include other geographic areas outside of the Asia and Pacific Region. The UNFPA should accordingly help identify appropriate institutions and experts who can assist Thailand in this process. The UNFPA should also support overseas study tours to developed countries to observe and learn how key stakeholders have identified and addressed many of the problems and issues that have already started to emerge in Thailand or which will probably soon appear upon the horizon. .

6. Improve linkages with other international and domestic funding and development agencies to improve overall programme efficiency

Although one of the UNFPA’s great strengths lies in its technical expertise and long experience working with RH and PD programmes it nevertheless has a limited budget. As such the UNFPA should try to more effectively link its technical resources with other international funding agencies that have similar programme goals and objectives, such as the Global Fund for AIDS, TB, and Malaria (GFATM). The UNFPA has should make greater efforts to share valuable lessons learned from models it helped develop as part of the MPM, YSRH, and SSRH expand support for vulnerable groups such as sex workers and youth. Improving linkages between technical and funding sources should also be pursued with national Thai agencies and institutions, such as the Thai Health Promotion Foundation, the National Health Security Office, and academic institutions and local CBOs.

55 | Page UNFPA Thailand’s CP9 Evaluation Report

Annex

ANNEX 1 Outcome and Output Indicators for UNFPA 9th Country Programme

ANNEX 2 Budget, Expenditures and Implementation Rate of Projects under CP9

ANNEX 3 List of Documents Reviewed

ANNEX 4 List of Key Informants

ANNEX 5 Methodologies to Answer the Evaluation Questions

ANNEX 6 Evaluation Criteria and Related questions

ANNEX 7 Interview Guides

ANNEX 8 Terms of Reference

ANNEX 9 Evaluation Team

56 | Page UNFPA Thailand’s CP9 Evaluation Report

Annex 1: Outcome and Output Indicators for UNFPA 9th Country Programme

Table A1.1: UNFPA 9th Country Programme Outcome and Output Indicators and Project Output Indicators

National priorities: addressing disparities of opportunities and improving the quality of social services and protection as well as self-empowerment of the most vulnerable

UNPAF outcome: by 2011, increased access to, and utilization of, quality social services and protection, especially for vulnerable groups and in underserved areas, resulting in reduced disparities

Reproductive Health (RH) Country programme Country programme outputs, indicators, Projects output and indicators outcomes, indicators, baselines and targets Outcome1: Output 1: RR Output 1: Increased utilization of Improved access to high-quality, gender-sensitive promote reproductive rights and SRH demand reproductive health reproductive health information, counselling and for vulnerable population information and services by services, including HIV prevention for vulnerable vulnerable groups and in groups and underserved areas Output indicators: underserved areas Output indicators: 1. Number of government hospitals providing Outcome indicators: •% of service delivery points in targeted areas RH services for PLHIV In selected areas: offering information, counselling and services for 2. Number of STI clinics offering RH services • Need met for modern family planning, HIV prevention, safe motherhood for sex workers contraceptive methods and newborn health •% of service delivery points MCD Output 8: (age- and group-specific) providing youth-friendly Enhance institutional capacity at national and • Proportion of deliveries by reproductive health services sub-national level for planning, implementing, skilled birth attendants •% of underserved populations (migrants, poor, managing and monitoring of RH program • Percentage of in- and out- rural, young people, commercial sex workers) Expected results for year 4: At the end of of school youth using having access to information and services for 2011: Lesson learned on MCD models and information family planning, HIV prevention, safe motherhood alternatives implemented in selected RH and services on and Project areas by: reproductive newborn health Capitalized of MCD experiences and lesson health and HIV •% of clients satisfied with quality of reproductive learned health services Scaled up to policy level

57 | Page UNFPA Thailand’s CP9 Evaluation Report

Country programme Country programme outputs, indicators, Projects output and indicators outcomes, indicators, baselines and targets Output 2: Enhanced institutional capacity at national and MMS / MPM Output 2: subnational levels for planning, implementing, Improve access to quality maternal health managing and monitoring reproductive health information and services Programmes Output indicators: MMS Output indicators: •% of planning and programme management staff 1.% of deliveries attended by health personals at all levels who have all recent, relevant 2.% of persons surveyed who knows three- reproductive health data available for use primary warning/danger signs of obstetrics •% of managers and service providers with complications appropriate knowledge of and attitudes toward 3.% of clients expressing satisfaction with gender-sensitive and culturally sensitive counselling and other services reproductive health information and services • Involvement of primary stakeholders (particularly MPM Output: migrants, rural women in southernmost provinces 1.% of women with male involvement during and young people) in planning culturally pregnancy appropriate reproductive health services 2.% of partners tested for HIV during •% of service delivery points receiving regular pregnancy reproductive health monitoring visits and feedback 3.% of women who are informed about dual protection by health providers during ANC Outcome 2: Output 3: STRONG Output 3: Improved HIV prevention Improved access to information, counselling and improve access to information, counselling and through services for HIV prevention among young people, services for HIV prevention among sex safer sexual practices commercial sex workers, clients of commercial sex workers, client of sex workers in underserved among workers and migrants in underserved areas areas vulnerable groups Output indicators: area: National Lampang Songkhla Bangkok Outcome indicators: • Percentage of service delivery points providing Pattaya In selected areas: youth-friendly information, counselling and services Output indicators • Percentage of young for HIV prevention 1.% of sex workers using condoms during last people • Percentage of service delivery points providing sexual intercourse with non-regular partners. demonstrating behavioural high-quality voluntary counselling and testing for 2.% of sex workers who received RH and change towards safer sex HIV infection OB/GYN services within the last 12-months

58 | Page UNFPA Thailand’s CP9 Evaluation Report

Country programme Country programme outputs, indicators, Projects output and indicators outcomes, indicators, baselines and targets practices • Percentage of young people, sex workers, clients 3.% of sex workers attending government STI • Percentage of sex workers of sex workers, and migrants (workers and clinics expressing satisfaction within reporting consistent use dependents) having access to target group-specific counselling and other services of condoms with clients and information, counselling and services for HIV YSRH Output 4: non-client partners prevention Improve access to RH information, including •% Condom use at last high- • Percentage of young women who are confident HIV prevention for young people risk sex they have the skills to get their partners to use National Lampang Bangkok condoms Output indicators 1.% of young people confident to refuse unwanted/unprepared sex 2.% of sexually active young people using condoms during last sexual intercourse 3.% of young people who have used youth- friendly services delivery points for information or services 4. Sexually active youth with unmet needs of contraceptives

Sources: DP/FPA/CPD/THA/9, Country programme for Thailand, 11 October 2006 Annual work plan of each program Term of reference Evaluation of UNFPA Thailand’s 9th Country program (2007-2011) Annex1

59 | Page UNFPA Thailand’s CP9 Evaluation Report

Table A1.2: NSO M&E Framework, 2007-2011

UNFPA Strategic Programme (PD) PD Outcome: Data on population dynamics, gender equality, young people, sexual and reproductive health and HIV/AIDS available, analyzed and used at national and sub-national levels to develop and monitor policies and programme implementation

2008-2011 SP Outcome Indicators: 1. Proportion of countries that have completed their 2010 round of population and housing censuses as planned 2. Proportion of countries that have conducted a national household/thematic survey that includes ICPD-related issues 3. Proportion of national development plans that include time-bound indicators and targets from national/sub-national databases CPAP/CPD Outcome: Increased utilization of disaggregated data for policy and programme formulation at national and sub-national levels for addressing maternal and newborn health, adolescent reproductive health, HIV/AIDS, gender, migration and population ageing

Outcome indicators: 1) Key policies and programmes are evidence-based ; 2) Policymakers support formulation and implementation of elderly friendly policies and programmes (qualitative data) CPAP/CPD Outputs: Enhanced capacity at national and sub-national levels to collect and analyze data and to conduct policy research on issues relating to reproductive health, HIV/AIDS, gender, migration and population ageing

CPAP/CPD Indicators: 1. HIV/AIDS and gender issues are integrated into reproductive health and population and development policies ; 2) Effective policies and programmes to deal with population ageing formulated and implemented; 3) Analytical reports on gender-related topics available and disseminated (qualitative data)

60 | Page UNFPA Thailand’s CP9 Evaluation Report

AWP 2007 2008 2009 2010 2011 1. Trained NSO core staff 1. Existence of materials 1. Trained NSO core staff 1. Trained NSO core staff effectively manage and and guidance to help effectively manage and effectively manage and undertake census activities strengthened quality of key undertake census and undertake census and surveys and population and survey activities survey activities housing census 2. Availability of reliable 2. Utilization of knowledge 2. Extent of NSO's 2. Extent of NSO's data of key RH and to improve quality of key participation in reviewing participation in reviewing population indicators for surveys and population and methodology and analysis data collection planning and housing census of data methodology and analysis implementation of data 3. Analytical reports on RH 3. Availability of reliable 3. Extent of NSO's related topics available and data for key MDG participation in promoting disseminated indicators utilization of census and survey data 4. Effective policies and 4. Improved access to 4. Analytical reports on programmes to deal with existing data for policy population and RH, and population ageing planning at national and reproductive health related formulated and provincial levels topics available and implemented disseminated 5. Advocacy materials 5. Strengthened capacity of produced/distributed NSO core staff on public campaigns for population and housing census

Note: All of above indicators require a combined qualitative and quantitative data. The quantitative data for selected indicators will be collected from key informants from NSO comparing before and after NSO's participation in UNFPA's supported project. A brief Likert scale questionnaire will be given in 2011.

61 | Page UNFPA Thailand’s CP9 Evaluation Report

Table A1.3: OP M&E Framework, 2007-2011

UNFPA Strategic Programme (PD) PD Outcome: Data on population dynamics, gender equality, young people, sexual and reproductive health and HIV/AIDS available, analyzed and used at national and sub-national levels to develop and monitor policies and programme implementation

2008-2011 SP Outcome Indicators: 1. Proportion of countries that have completed their 2010 round of population and housing censuses as planned 2. Proportion of countries that have conducted a national household/thematic survey that includes ICPD-related issues 3. Proportion of national development plans that include time-bound indicators and targets from national/sub-national databases CPAP/CPD Outcome: Increased utilization of disaggregated data for policy and programme formulation at national and sub-national levels for addressing maternal and newborn health, adolescent reproductive health, HIV/AIDS, gender, migration and population ageing

Outcome indicators: 1) Key policies and programmes are evidence-based ; 2) Policymakers support formulation and implementation of elderly friendly policies and programmes (qualitative data) CPAP/CPD Outputs: Enhanced capacity at national and sub-national levels to collect and analyze data and to conduct policy research on issues relating to reproductive health, HIV/AIDS, gender, migration and population ageing CPAP/CPD Indicators: 1. HIV/AIDS and gender issues are integrated into reproductive health and population and development policies ; 2) Effective policies and programmes to deal with population ageing formulated and implemented ; 3) Analytical reports on gender-related topics available and disseminated (qualitative data)

AWP 2007 2008 2009 2010 2011 1. Reports on situation and 1. Policies and strategies 1. Policies and strategies 1. Policies and strategies policy analyses and addressing specific healthcare addressing health and well addressing health and well recommendations for the and services, social and being of older persons at being of older persons at interventions in addressing economic security of older national, provincial and national, provincial and population ageing with a focus persons at national and sub- community levels developed community levels developed on poverty, livelihood and national levels developed income security in the upper northern a) Lampang: Rural poor and b) Mae Hong Son: Underserved groups/ethnic minorities in the highland

62 | Page UNFPA Thailand’s CP9 Evaluation Report

AWP 2007 2008 2009 2010 2011 2. Policies and strategies 2. Comprehensive 2. Comprehensive 2. Comprehensive addressing well-being of older programmes in addressing programmes in addressing programmes in addressing persons at national, provincial well being of older persons well being of older persons in well being of older persons in and community levels related to health, social and the area of health, social and the area of health, social and developed economic security economic security appropriate economic security appropriate appropriately for the different for local context and settings for local context and settings groups of Ops taking into developed and in place developed and in place account the local context 3. Increased meaningful 3. Increased meaningful 3. Increased meaningful 3. Increased meaningful participation of older persons participation of older persons participation of older persons participation of older persons in needs assessment, policy in needs assessment, policy in needs assessment, policy in needs assessment, policy planning and programme planning, programme planning, programme planning, programme development in addressing development and monitoring development and monitoring development and monitoring population ageing the implementation of OP the implementation by GOs in the implementation by GOs in policies/programmes/actions addressing population ageing addressing population ageing by public sector (GOs) 4. Increased support and 4. Increased support and 4. Increased support and involvement of key involvement of key involvement of key stakeholders in assessment, stakeholders in assessment, stakeholders in assessment, planning and implementing planning and implementing planning and implementing the programmes on older the programmes on older the programmes on older persons persons persons

5. Enhanced networking in 5. Enhances networking in 5. Enhances networking in working towards better working towards better working towards better livelihood and income security livelihood, social and income livelihood, social and income of older persons at national, security of older persons at security of older persons at provincial and community national, provincial and national, provincial and levels community levels community levels

6. Improved public awareness 6. Improved public awareness 6. Improved public awareness on population ageing for on population ageing for on population ageing for enabling supportive enabling supportive enabling supportive environment and positive environment and positive environment and positive image of ageing image of ageing image of ageing

63 | Page UNFPA Thailand’s CP9 Evaluation Report

Annex 2: Budget, Expenditures and Implementation Rate of Projects under CP9

Table A2.1: Overall Figures by Project in 2007

2007 Project Partners Budget Expense Imp. Rate RH Plus (THA9R201) DOH, LP, MHS, NRT 508,681.00 440,904.90 86.68

HIV (THA9R208) BATS 119,852.00 104,983.06 87.59

OP (THA9P203) MSDHS, FoN/CMU 131,013.00 127,059.67 96.98

NSO (THA9P101) NSO 76,000.00 43,308.28 56.98

64 | Page UNFPA Thailand’s CP9 Evaluation Report

Table A2.2: Overall Budget, Expenditures and Implementation Rate 2008-2010 by Project

2008 2009 2010 Project Budget Expense Imp. Prop. Budget Expense Imp. rate Prop. Budget Expense Imp. rate Prop. Rate THA9R11A 272,100.57 258,427.22 94.97 24.18 186,129.42 162,779.56 87.46 12.82 186,213.00 187,020.79 100.43 13.09 RR* THA9R19A 50,149.00 38,395.58 76.56 3.59 81,399.97 73,317.59 90.07 5.77 85,680.00 87,391.09 102.00 6.12 MCD** THA9R22A 250,627.81 248,625.87 99.20 23.27 238,991.27 232,832.00 97.42 18.33 206,648.00 212,850.79 103.00 14.90 MMS/MPM* THA9R43A STRONG/ 177,738.00 152,811.87 85.98 14.30 202,484 193,972.98 95.80 15.27 194,090.00 179,987.58 92.73 12.60 SSSRH*** THA9R54A 127,939.60 119,434.45 93.35 11.18 165,099.00 146,336.98 88.64 11.52 193,464.66 182,083.30 94.12 12.75 YSRH* THA9P15A SOUTH- 69,600.00 39,246.99 56.39 3.67 129,671.54 94,886.52 73.17 7.47 80,656.16 42,557.54 52.76 2.98 SOUTH THA9P36A 74,000.00 60,632.14 81.94 5.67 127,912.42 124,471.66 97.31 9.80 171,822.00 174,217.74 101.39 12.20 NSO THA9P47A 177,687.00 150,971.22 84.96 14.13 281,383.38 241,446.52 85.81 19.01 374,827.00 362,152.48 96.62 25.36 OP TOTAL 1,199,841.98 1,068,545.34 89.06 100 1,449,578.27 1,308,902.83 85.02 100.00 1,493,400.82 1,428,261.31 95.64 100 * In 2007 was included in RH Plus Project (THA9R201) ** In 2008 The MCD was established under THA9R22B *** THA9R43A was separated from HIV (THA9R208), RH Plus Project (THA9R201) in 2007 **** In 2008 THA 9R201 is separated into 3 AWPs under DOH THA9R11A, THA9R22A, THA9R54A

65 | Page UNFPA Thailand’s CP9 Evaluation Report

Annex 3: List of Documents Reviewed

1. UNCT Joint Documents 1.1 United Nations Partnership Framework (UNPAF) 1.2 MIC Study Reports (Phase I and II) 1.3 ICPD

2. National Plans, Policies 2.1 The 11th National Social and Economic Development Plan (On-going draft) 2.2 The National Reproductive Health Strategic Plan 2.3 The 2nd National Plan of Older Persons 2.4 The National Statistical Act

3. UNFPA Project/Programme Documents • UNFPA CP9 Evaluation: Guideline for the inception report • UNFPA CP10 Development: Process and Agenda • Need Assessment and Gap Analysis for UNFPA’s Ninth Country Program (CP9) 2006-2011 • 2008-2011 Strategy Plan: Accelerating Progress and National ownership of the ICPD Programme of Action • Country Programme Action Plan (CPAP) 2007-2011 • UNFPA 2007-2001 United Nations Partnership Framework Thailand • Thailand Common Country Assessment • UN Operations in a middle-Income /country Formulation of a strategy for enhanced UN coherence and effectiveness in Thailand

3.1 Reproductive Health

1) Strengthen Access to and Utilization of Reproductive Health Information, Counselling and Service (The RH Plus Project) 2) Need Assessment, Gap and Situation Analysis and the Recommendation of the Strengthen access and utilization of reproductive health information, counselling and service (the RH Plus Project) SSRH, YSRH,MMS,MPM Narathiwat Province 3) Need Assessment, Gap and Situation Analysis and the Recommendation of the Strengthen access and utilization of reproductive health information, counselling and service (the RH Plus Project) SSRH, MMS, MPM Songkhla Province 4) Strengthen Access to and Utilization of Culturally and gender-sensitive information, counselling and service of MH 5) Strengthen Access to and Utilization of Reproductive Health Information, Counselling and Service (The RH Plus Project) 6) Annual Report 2007 - 2010 7) Financial Report 2007 – 2010

66 | Page UNFPA Thailand’s CP9 Evaluation Report

1. RR (Promote reproductive rights and SRH demands for vulnerable populations) and MMS/MPM (Strengthen access to and utilization of culturally and gender-sensitive information, counselling and services of maternal health) 1) Maternal and Child health situation analysis focusing on birth spacing in Waeng and Bajoh Districts, Narathiwat province 2) Monitoring the Situation of Children and Women 3) Multiple Indicator Cluster Survey of Children and Women in Thailand 2006 4) Situational analysis and needs assessment report of health service utilization in Lampang province 5) Situational analysis and needs assessment study to increase access to maternal and child health and to make motherhood safe, formulate strategy to promote male partnership in maternal health in Mae Hong Son province 6) Study of accessing to information, counselling and services of reproductive health in Mae Hong Son province 7) Baseline report on MMS in Narathiwat and Mae Hong Son 8) Report on Knowledge Management on MMS project implementation in Narathiwat and Mae Hong Son 9) Report on Knowledge Management on MPM project implementation in Narathiwat, Songkla, Lampang and Mae Hong Son

2. MCD - Management Capacity Development Project 1) THA9R19A AWP (2008-2010) 2) Report on evaluation of accessibility to the needs of Maternal and Child Health at Lam Pang (Thai version) 3) Report on evaluation of accessibility to the needs of Maternal and Child Health at Mae Hong Son (Thai version) 4) PBRI’s Document; namely a. Lam pang lesson learn document b. Mae Hong Son lesson learn document c. Self learning (power point presentation) d. Self learning Manual e. How to conduct self learning by Computer (power point presentation) f. Development of Curriculum Model (power point presentation)

3. STRONG/SSRH - Strengthening SRH/HIV services for vulnerable groups in decentralization context 1) 2007 Survey of Sexual and Reproductive Health of Sex Workers in Thailand 2) Agreement between DDC, MOPH and UNFPA for the implementation of the UNFPA - funded Annual Work Plan (AWPs) 2007- 2011 Country Programme 3) Annual Meeting: Meeting documents of HIV/AIDS Programme (2007-2009) 4) CCP: Meeting documents, working group appointment orders and other CCP related documents 5) FC Promotion Posters and Brochures 6) FC: Meeting documents and other FC programme related documents 7) Final Report: Assessment of capacity development and skills building needs of public and civil society implementing partners in the local response to AIDS (Thai and English) 8) Final report: Enhanced GFATM Grant Implementation in Thailand through local partner capacity building and improved multi-sectoral local response programming for AIDS

67 | Page UNFPA Thailand’s CP9 Evaluation Report

9) Health Migrants for Security of Thailand 10) HIV – Migrant SW project 11) Proceedings: Consultative Workshop on Developing a National Evaluation Agenda for HIV/AIDS in Thailand, Cha-Am 2010 12) Relevant Statistics of SW in Lampang 13) Relevant docs: RH of Migrants in Thailand 14) Relevant meeting Documents 15) Report: Best Practice for HIV/AIDS Response in Thailand 2008-2009 16) Report: Size Estimation and Approach to Sex Workers in Diverse Settings, 2007 (Thai and English) 17) Report Card: HIV Prevention For Girls and Young Women -THAILAND (Thai and English) 18) RH/HIV/SW project in Songkhla 19) Situation on STI/HIV Prevention in Sex Work Settings in Thailand ( Thai and English) 20) Some relevant documents: HIV Prevention for FSW in Pattaya 21) Stigma and Discrimination against PLHIV: meeting documents and relevant documents/reports 22) Strengthen Access to and Utilization of Reproductive Health Information, Counselling and Service (The RH Plus Project) 23) The 2nd National Migrants in Thailand 24) UNFPA Thailand - THA 9R43A Annual Work Plan 2009 25) UNFPA Thailand - THA 9R43A Annual Work Plan 2010 26) UNFPA Thailand- Annual Work Plan 2008 Output 3 27) Young Sex Workers in Thailand: Report prepared for UNFPA-APRO (2009) 28) Guideline to provide reproductive health for PLHIV 29) Guideline to develop HIV/AIDS policy at local level (DDC, DLA, UNFPA, UNDP, UNAIDS, UNICEF) 30) Project report “Strengthening HIV/AIDS and STIs prevention project in Lampang 2010” 31) Sexual transmitted infections survey among vocational school and university students in Muang district, Chiang Mai 32) Coordinating mechanism in Ministry of Public Health in implementing HIV prevention among international migrants 33) Survey of sexual establishments and service workers in Samnugkham sub- district, Sadoa disctrict, Songkhla in 2008 34) Study of factors associated with perceptions of HIV/AIDS prevention among female service workers in Sadoa district, Songkhla province 35) Secret (unrevealed): colour of the secret (pocket book) 36) Field manual “technique to conduct female service worker survey in establishment 37) People living with HIV/AIDS and Family Planning 38) Partnership and better environment for better health volume 3: “Reproductive health and family planning for HIV infected people” 39) Study on “ improving access to information, counselling and service for HIV prevention among female service workers in Songkhla 2008 40) Feasibility study of human rights on AIDS campaign and development of stigma and discrimination indicators

68 | Page UNFPA Thailand’s CP9 Evaluation Report

41) Situation and opinion study of condom management among health service providers and local administrative organizations 42) Consolidated report of national seminar: Sharing experiences and strengthening networking of GO and NGOs working STI/HIV prevention among sex work 43) Summary report “Voices and choices of women living with HIV/AIDS 44) Sexual health of female service workers: knowledge to practice 45) Training manual of “Capacity building of local administrative organizations of condom management in promoting condom use in STI and HIV prevention 46) Training manual for peer educators at holistic care centre in promoting RH services among PLHIV

4. YSRH - Strengthen access to and utilization of SRH/HIV information, counselling and services for young people 1) Adolescent Sexual and Reproductive Health (ASRH) Policy and Programmes in Thailand 2) Annual Work Plan for Lampang Province (2007-2010) 3) Consultation with adolescent in vulnerable situation on access to adolescent reproductive health information and service 4) Love Care Station 5) Love@1st_Click DVD 6) Review IEC Materials on SRH and young people/ adjusted to develop DVD for pre-teen (Script Animation) 7) Consolidated report of Solution Exchange in 2008 about knowledge and views of reproductive health among young people 8) Standard Youth Friendly Service 9) Summary Report: Situation of contraception towards young Thai people 10) UNFPA: Love care station, Youth Friendly Sexual Health Service 2007 , 2008 , 2009 11) Y-PEER Compact disc (CD) 12) YAP Annual Report 2008-2009 13) Situational analysis of management information system of reproductive health among young people aged 10-24 years old 14) Standards of youth friendly health service 15) Study of family planning and unintended pregnancy among young people

Population Development

5. South to South - South to South Cooperation in Reproductive Health and Population 1) THA9P36A – AWP (2007-2010) 2) S-S Report year 2008-2010 (4 files) 3) South to South project Profile 2010 4) Full Version: S-S Survey Report Feedbacks from UNFPA COs and line ministries, and government organizations 5) Document of Conceptual Framework: Sustainable Capacity Development Programme under South-South Cooperation 6) PROJECT, ANNUAL PROGRESS REPORT, 2008 AWP, South to South Cooperation in Reproductive Health and Population Development

69 | Page UNFPA Thailand’s CP9 Evaluation Report

7) PROJECT, ANNUAL PROGRESS REPORT, 2009 AWP, South to South Cooperation in Reproductive Health and Population Development 8) PROJECT, ANNUAL PROGRESS REPORT, 2010 AWP, South to South Cooperation in Reproductive Health and Population Development 9) Proceeding on Work shop report by Chiang Mai University, year 2007-2009 a. Moving towards Healthy Ageing: The Promotion of the Elderly’s Well-being and Community Participation (During 17 - 28 September 2007) b. Enhancing Community Capacity in Managing the Emerging Challenges of Population Ageing (During 10 – 21 November 2008) c. Enhancing Community Capacity in Managing the Emerging Challenges of Population Ageing (During 9 – 20 September 2008) 10) Proceeding on Work shop report by Mahidol University, year 2007-2009 a. Monitoring and Evaluation of Adolescent Reproductive Health (During 8 – 19 October 2007) by Institute of Population and Social Research b. Consultative workshop on Making Motherhood Safer (MMS) by ASEAN Institute (During 6 – 11 December 2010) 11) Proceeding on Work shop report by Chulalongkorn University, year 2007-2009 a. Migration and Health Related Issues (During 20 - 31 October 2008) b. Monitoring and Evaluation of Migration and Health-Related Programs (During 26 October - 6 November 2009)

6. NSO - Monitoring and Strengthening Statistical Capacity 1) Summary Report of the Preparation for the Population and Housing Census 2010 for the National Statistical Officers: Quality assurance of the Population and Housing Census. 10 - 14 November 2008, Meeting Room, National Statistical Office, Bangkok. 2) Summary Report of the Preparation for the Population and Housing Census 2010 for the National Statistical Officers: Reaching Out in the Cities. 29 October - 2 November 2007, Prince Palace Hotel, Bangkok. 3) Summary Report of the Quality Assurance Observation of the 2010 Thailand Population and Housing Census 4) National Statistical Office. 2010. Key Findings from the Reproductive Health Survey. 5) National Statistical Office. 2011 (forthcoming). Reproductive Health Survey 2009 Report (draft). 6) Archavanitkul, K., Punpuing, S., Prohmmo, A., and Bryant, J. 2009. Lessons learned from pilot census in areas where there are a concentration of migrants in Thailand. 7) Annual Work Plan 2007-2010 8) Annual Report 2007-2010 9) Financial Report 2007-2010

7. OP - Evidence-based Responsiveness to the Emerging Challenges of the Population Ageing in Thailand 1) Older Persons in Thailand: Situation and Policy 2) Population Ageing and the Well-Being of Older Persons in Thailand: Past Trends, Current Situation and Future Challenges 3) Thailand’s Implementing Strategy on Older Persons under the Madrid International Plan of Action on Ageing (MIPAA) 2010 4) Summary of the Brainstorming Meeting on the Needs of Older Persons for Health Services

70 | Page UNFPA Thailand’s CP9 Evaluation Report

5) Report on the Revision of the Second National Plan for Older Persons Policy (2002-2021) 6) The Second National Plan for Older Persons Policy (2002-2021), First Revision 2009 7) The Development of Long-term Care for Older Persons from Institution to Community in Lampang 8) Monitoring Report on the Preparation for Ageing Society in Thailand 9) Summary Report of Project Feasibility, Preparation for Occupational and Income Generating Activities in Chiang Mai and Pilot Site - Hang Chat District, Lampang. 31 May 2010, Chiang Mai Labour Office. 10) Implementing Strategy for Older Persons in Lampang for Fiscal Year 2011 11) Strategic Plan for Occupational and Income Generating Activities for 2011, Chiang Mai, Lampang and Mae Hong Son Labour Offices 12) Annual Work Plan 2007-2010 13) Annual Report 2007-2010 14) Financial Report 2007-2010

71 | Page UNFPA Thailand’s CP9 Evaluation Report

Annex 4: List of Key Informants

Project Key Informants Experts Mr.Promboon Phanichapuk , Rak Thai Found. Dr.NunTha AoumKul, Consultant of DOH Dr.Pechsri Sirinirund , NAMc Dr. Viroj Tangcharoensathien, IHPP RR Dr. Taweesap Siraprapasiri, UNFPA Bureau of Reproductive Health , DOH Dr.Kitthiphong Saejeng, Director Mr.Suthon Panyadilok Mrs.Renu Chunin Advisor, DOH Ms.Pornsinee Amornwichet Women's Health Advocacy Foundation Ms.Natthaya Boonpakdee MPM/MMS Ms.Sukanya Thongthumrong, UNFPA Bureau of Reproductive Health • Dr.Kitthiphong Saejeng, Director • Mr.Suthon Panyadilok • Mrs.Renu Chunin Narathiwat • Mrs.Kemmawan Leangwatthaphong, PHO • Mrs.Nuleeha Wasalama, PHO • Ms.Niwara Sengsridang , PHO Bajao • Ms.Wayao Wayeng, PCU Laojude Songkhla • Ms.Sunun Tririrat , PHO • Ms.Boonyarat Suwanphansa, PHO • Ms.Sopen Phopongsa, PHO Rattaphum • Ms.Prapapan Sangkomin, PHO Rattaphum • Ms.Sutthida Maijon , PHO Rattaphum • PHO Sadao Team Lampang • Dr.Chalermchai Wuttipitthamongkol,PHO • Ms.Sirikul Kusuwan, PHO • Dr.Pratuang Keamphongsaputthi Lampang Hospital • Ms.Sopich Kantha , Lampang Hospital • Ms.Wanpen Leelaporn , Lampang Hospital • Dr.Surachai Phonglorpisit , Thoen Hospital • Ms.Siriporn BanKum , Thoen Hospital • Ms.Thanissa Suwanlert, PHO Thoen Meahongson • Ms.Pempit Laoreandee ,PHO • Mrs.Supanee Julathana,PHO • MCH health workers , mothers, VHV and community leaders in Ban Pong nam ron and Ban Nong ngwon , Mae La Noi district MCD Ms. Duangkamol Ponchamni , UNFPA PBRI, Praboromarajchanok Institute for Health • Dr. Tipapron Sukosit Deputy Director, PBRI • Dr. Jureerat Kijsomporn, PhD. Workforce Development Mea Hong Son • Dr.Suwat Kithidilokkul ,PHO Mea Hong Son

72 | Page UNFPA Thailand’s CP9 Evaluation Report

Project Key Informants • Ms. Penpit Loareindee ,PHO • Ms. Supane Jurathna,PHO • Mr. Anan Noisuwon,PHO • Mr. Boonsom Pongpripoom,PHO Lampang • Dr.Sirichai Patthanutaporn ,PHO • Mr.Chuchat Kawila PHO Thoen • Mr.Sornsin Mungkornkeaw PHO Thoen • Mr. Chumpol ,PHO Mung • Mr.Chuchat Kawila PHO Thoen • Mr.Sornsin Mungkornkeaw PHO Thoen STORNG Dr. Taweesap Siraprapasiri, UNFPA BATS • Dr.Patchara Siriwongrungson ,Direct of BATS • Mrs.Vipada Maharatthanaviroj • Mrs.Thumneab Sungwanprakaysang SWING • Ms.Surang JunYam Songkhla • Ms.Tassanee Phongpaiboon , PHO • Mrs.Sudee Jaruphan , PHO • Ms.Manee Patphong ,PHO • PHO Sadao team Lampang • Ms.Penkae Doungkamsawas, PHO • Mr.Chuchat Kawila PHO Thoen • Mr.Sornsin Mungkornkeaw PHO Thoen • Mrs.Yaowares Kawilkrea Thoen Hospital • Mrs. Jureemas Saingam, Thoen Hospital • Ms.Panisa Kunsai , Thoen Hospital Meahongson • Dr.Suwat Kithidilokkul (Chief Medical for Provincial Public Health Office during 2007-2010) YSRH Mrs. Srisuman Sartsara, UNFPA Ms Paradee Chansamorn, Bureau of Reproductive Health Dr.Watchara Phumpradit ,PATH Dr. Nunta Aoumkul,DOH Lampang • Ms.Tassanee Srijun, PHO • Mrs.Kulrat Chaiprom ,PHO • YSRH Team PHO.Mung and Lampang Hospital • Mr.Chuchat Kawila PHO Thoen • Mr.Sornsin Mungkornkeaw PHO Thoen • Mrs.Yaowares Kawilkrea Thoen Hospital • Mrs. Jureemas Saingam, Thoen Hospital • Ms.Panisa Kunsai , Thoen Hospital South Mr.Najib Assifi ,UNFPA Representative and Deputy Regional Director South Ms. Duangkamol Ponchamni , UNFPA Assist.Prof. Achara Entz , Collage of Population Study, Chulalongkorn University Ms Saranya Chaisang, Faculty of Nursing, Chiang Mai University International Organisations Partnership Branch (Multilateral), TICA • Mr. Banchong Amornchewin, Director

73 | Page UNFPA Thailand’s CP9 Evaluation Report

Project Key Informants • Miss Suchata Thaibunthao , Deputy Director General of TICA) • Mrs. Sasitorn Wongweerachotkit, Development Cooperation Officer NSO Dr. Wassana Im-em, UNFPA Mr. Prasert Sripanaratanakul, Information Centre Administrative Development, MoL Ms. Jirawan Boonperm, MICT NSO • Ms. Rajana Netsaengtip • Ms. Pattama Amornsirisomboon • Ms. Jirawan Matoum • Ms. Aphaporn Amornthum • Mr. Pairoj Boonlue, NSO Chiang Mai College of Population Studies, Chulalongkorn University • Dr. Vipan Prachuabmoh • Dr. Napaporn Chayovan Institute for Population and Social Research, Mahidol University • Dr. Pramote Prasartkul • Dr. Sureeporn Punpuing, • Dr. Kritaya Archavanitkul OP Ms. Viennarat Chuangwiwat, UNFPA Ms. Siriwan Aruntippaitune, Bureau of Empowerment for Older Persons, MSDHS Dr. Manoon Vathisoonthorn, DOH, MOPH Ms. Orawan Kuha, DoM, MOPH Mr. Suchart Premsurity,Social Security Office, MoL Ms. Suwattana Sripirom, Consultant Chiang Mai Dr. Linchong Pothiban ,FoN, CMU Ms. Usa Khiewrord, HAI Mr. Chakrit Taburi, HAI Mr. Sawang Kaewkantha, FOPDEV Mr. Janevit Wisojsongkram, FOPDEV Mr. Panat Limpisathien, Prime mover and SCCT Mae Taeng District Dr. Pongsiri Prathnadi, SCCT Chiang Mai Ms. Chanya Busayawong, Consultant Lampang Dr. Sirichai Pattharanutaporn, PHO Ms. Pitsamai Kongsuriyasak, PHO Mr. Somsak Wongkhamtan, MoL Dr. Orarn Yingseree, Hang Chat Hospital Ms. Ampai Aunsri, TAO Hang Chat Mae Tan Participants in focus group discussion at Hang Chat Hospital

74 | Page UNFPA Thailand’s CP9 Evaluation Report

Participants of Consultative Meeting

On The Evaluation of UNFPA Thailand’s 9th Country Programme (CP9, 2007 -2011) On January 25, 2011 08:00 AM – 4:30 PM Rajadamnoen Meeting room, Royal Princess Larnluang Hotel, Bangkok, Thailand

No. Name Workplace 1. Mrs.Khemmawan Luangwatthnaphpong PHO Narathiwat 2. Mrs.Nuleeha Wasalamae PHO Narathiwat 3. Ms.Niwara Sengsridang PHO Bajao Narathiwat 4. Ms.Wayao Wayeng PCU Laojude , Wang 5. Dr. Suwat Kithidilokkul PHO Samutsongkram 6. Ms. Penpis Laoreamdee PHO Mae Hong Son 7. Ms. Supranee Julathana PHO Mae Hong Son 8. Dr. Sirichai Pattharanutaporn PHO Lampang 9. Ms. Tassanee Phongpaiboon PHO Songkhla 10. Ms. Orawan Kuha Department of Medical Services (DoM) 11. Mr. Chakrit Taburi HelpAge International (HAI) 12. Mr. Janevit Wisojsongkram Foundation for Older Persons’ Development (FOPDEV) 13. Ms. Romyawadee Saraketrin TICA 14. Ms. Saaitron TICA 15. Ms. Pattha Into TICA 16. Ms. Daungkamol Withapitukwong TICA 17. Dr. Nuntha Aoumkul Consultant of DOH 18. Mr. Suthon Panyadilok Bureau of Reproductive Health, DOH 19. Mrs. Renu Chunin Bureau of Reproductive Health, DOH 20. Mrs. Vipada Maharattanawiroj BATS 21. Ms. Surang Junyam SWING 22. Mr. Bunjong Amornchewin TICA 23. Ms. Usa Khiewrord HelpAge International (HAI) 24. Ms. Jirawan Matoum NSO 25. Ms. Aphaporn Amornthum NSO

75 | Page UNFPA Thailand’s CP9 Evaluation Report

Annex 5: Methodologies to Answer the Evaluation Questions

General Questions:

Questions1: Based on the UNPAF, UNFPA’s Strategic Plan, the Millennium Development Goals, and the Programme of Action of the International Conference on Population and Development (ICPD), to what extent has CP9 contributed to the achievement of national priorities, including outcomes and impacts?

issues & detailed Analysis / consideration Source of Data / methods questions

To what extent are the - participation/ involvement - review evidence used and RH and PD components of the national authorities in participation of stakeholders of the CP9 aligned with the planning and monitoring in the planning process national priorities? process To what extent have - Review the national health expected results been - Consider goals and output / plan, UNFPA’s strategic achieved and what outcome indicators of CP9 plan,ICPD, MDG, CPAP, empirical evidence is that aligned with national AWP, progress report and available to support ? priorities annual report of each The result or impacts that happen according to the project. -Compare the achievement plan and beyond the plan of each program and the -Review the assessment explore the other projects that may contribute the trend with the expected report related to RH and PD same goal and if there is, output/outcome indicators comparing CP9 -Interview responsible approach with the others - results / effects that were person in UNFPA, not identified in the plan counterparts and managers at the provincial and district/ - explore conditions and sub-district level in some factors contributing to the piloting area including some results. experts in these fields.

76 | Page UNFPA Thailand’s CP9 Evaluation Report

Question 2: What are the sustainable results of UNFPA’s contributions to Thailand’s National Reproductive Health Strategic Plan and its other population programmes and policies?

issues & detailed Analysis / consideration Source of Data / methods questions

Perceptions, understanding What gained and how it was -Interview key counterparts and experiences of key gained. and related stakeholders at counterparts at all levels: different levels what they gained in-term of Whether it will be sustained? plan, approach, strategies, - Review progress report, Supporting system for the annual report and the related methodologies, knowledge or change: plan, budget some skills related to RH, PD plan of the RH/PD linked and management. Comparing with the other authorities programs support by other agencies.

Lesson learned , obstacle

The key system changes Compare the result, plan, after implementing the CP9- policies of RH/PD of related link activities authorities according to time series, piloting areas and the Change of policies and plan others of related authorities including the resource allocation and supporting system

Facilitating and constraining Factors contributing under factors ( design, different conditions management, leadership etc.) influencing to the result quality and sustainability

77 | Page UNFPA Thailand’s CP9 Evaluation Report

Question 3: To what extent has UNFPA’s programme management mechanism and its application contributed to programme achievements? (Program design, management, and implementation)

issues & detailed Analysis / consideration Source of Data / methods questions

The management pattern What can be done according Interview the key person of and mechanism of to the guidelines and plan? UNFPA implementing CP9 program (program planning, resource Obstacles, limitation, lesson Review the annual report and allocation, line of learned evaluation documents administration, HRD, financial management, selecting counterparts, M&E )

Perception and The weak, strength, Interview key responsible understanding of key limitation points on the person of the counterpart at counterparts towards perspective of counterparts each level UNFPA’ program management and mechanism involved in the CP9.

Have relevant strategic List of stakeholders in the Review documents related to stakeholders been planning and implementation planning process and result. included in the plan. development and Interview key counterpart at implementation of CP9? Coverage of stakeholders different level How and how involved and the result effectively? To what extent have the Existence of Interview key counterpart at RH/PDS programmes activities/strategies/ target different level. effectively integrated and group of RH/PD reflected gender equality, programmes /gender, rights Review progress report and cultural sensitivity, and issues in the operational annual report including some human rights? plan of implementing unit evaluations report (if have) including the way they did integratively.

Existence of these related issues in the operational plan or implementation process

To what extent has there The system set for adjust the Interview key program been flexibility in strategic plan and annual managers and key addressing emerging work plan of CP9. counterpart at all level issues experienced during CP9 and how well The evidence used during Comparing the AWP with the did the UNFPA respond monitoring and annual work

78 | Page UNFPA Thailand’s CP9 Evaluation Report

issues & detailed Analysis / consideration Source of Data / methods questions

to the challenges it faced plan formulations. strategic plan and the targets during implementation of CP9? Experiences of some change during implementation

Obstacles faced.

What were the main Existence of the main -Review the plan documents assumptions and risks assumptions and risks in the and progress report/annual identified that have planning process and the report affected expected actions response results? Were these Interview key stakeholders assumptions appropriate? To what extent and with Understanding and Interview key counterparts at what consequences has experiences of key partners all level. UNFPA applied results- on results-based based management management (usefulness, during its development limitation, obstacle) and implementation of CP9?

To what extent were Evidence of HR capacity Review AWP, annual report, UNFPA’s and its considered in the planning meeting report partners’ capacities and implementation process. (financial, technical and Interview key manager of the management) considered Component of human counterparts in the development and resource development in the management of CP9? plan: content, frequency,

Were UNFPA’s Plan of M&E Review progress. annual monitoring mechanisms report, evaluation report and appropriate and the Frequency, regularity and meeting reports results of monitoring actions after monitoring appropriately used to including its effect Interview key counterpart of make necessary each mechanism adjustments? Obstacle and lesson learned

Question 4: As Thailand becomes a middle-income country, what can and should be UNFPA’s strategic position and contributions to CP10?

79 | Page UNFPA Thailand’s CP9 Evaluation Report

issues & detailed Analysis / consideration Source of Data /methods questions

• Thailand potential, strength • Mapping Thailand’s needs • Review the report of UN and weak points related to with the strength of operations in a Middle RH and PD and gender UNFPA. income country phase I and • The strength and potential • The gap of output II value added of UNFPA /outcome with the • Review key changes of the • Perception and opinions of expected goal in CP9 and strategic plans of UNFPA key stakeholders in its trend in the past. Thailand in the past to Thailand and other related present. international agencies • Key stakeholders meeting working with Thailand on and interview positioning of UNFPA Thailand

Question 5: How effectively and efficiently has UNFPA implemented CP9 and ensured its relevance to Thailand’s needs and priorities?

issues & detailed Analysis Source of Data / methods questions

Relevancy Evaluate the issues of project Review the assessment purpose that aligned with report of RH and PD in Thailand need and priorities Thailand, the Thai National health plan and AWPs

Effectiveness and efficiency Compare outputs and Review document resources used to the objectives of the projects in Interview key partners at CP9 and also comparing the all level similar programs support by other agencies

Is UNFPA’s budget for CP9 sufficient to ensure achievement of the expected results? Were financial and human resources used efficiently and cost effectively? What factors have facilitated or constrained achievement of the expected results? Which of these factors are within UNFPA’s influence or control? What could have been done to make UNFPA’s efforts more effective and efficient?

Key Question about Population and Development Strategies

80 | Page UNFPA Thailand’s CP9 Evaluation Report

Q_PD1: What are the UNFPA’s overall results in terms of its expected outcomes?

1.1 To what extent have UNFPA’s efforts contributed to the development of statistical capacity in strengthening the quality, collection, analysis, and reporting of data on population? issues & detailed questions Analysis Source of Data / methods

¾ Were expected outcomes met, ¾ Compare outputs and ¾ Compare annual in the view of technical experts resources used to the workplans with progress and through technical review objectives of the reports of outputs? projects in CP9 ¾ Technical review of NSO ¾ To what extent did UNFPA ¾ Technical review outputs inputs contribute to the quality ¾ Triangulation of ¾ Review of capacity and efficiency of census interview data with building plans, curricula efforts? technical review etc. ¾ Did capacity building efforts ¾ Interviews with technical address key gaps in the ability experts to collect high quality and ¾ Interviews with NSO accurate data? stakeholders ¾ Were capacity building efforts effective and sufficient? ¾ the utilization of the National Survey on Reproductive health and the National survey on Older persons

1.2 What are UNFPA’s achievements in enhancing capacity development and improving South-South cooperation in sharing technical know-how in population development and RH? issues & detailed questions Analysis Source of Data / methods

Issues: Content analysis will be Review AWP, Annual Building capacity of national conducted of the reports report, evaluation report institute and organization to Interview key conduct international education stakeholders( Thailand and training International knowledge management under Development project Cooperation Agency training which involve other (TICA), Ministry of countries Foreign Affairs, MOPH) international network established Obstacle and lesson learned

81 | Page UNFPA Thailand’s CP9 Evaluation Report

1.3 What are UNFPA’s unique contributions in addressing Thailand’s population ageing, particularly in improving access of older persons to economic security, health and social services, and empowerment of older persons? issues & detailed questions Analysis Source of Data / methods Have UNFPA’s advocacy efforts -compare achievements -document review resulted in benefits for the ageing with expected outcomes including project reports population? and other outputs -views of key stakeholders at the program and policy -key informant level interviews with program To what extent did participatory staff, policymakers and approach succeed in advancing other stakeholders policy agenda? Were key stakeholders engaged and effective?

To what extent was the networking approach used to improve programmes and extend coverage? Was the approach effective and did it improve efficiency?

Is there evidence that public awareness was raised towards the issues affecting older people?

Were health care and service -compare achievements -document review systems appropriately analyzed with expected outcomes including project reports and addressed? Are there concrete and other outputs achievements in defining and -views of key stakeholders providing policy recommendaions? at the program and policy -key informant level interviews with program staff, policymakers and other stakeholders

Did participatory action research lead to concrete achievements in improving social and economic security?

Did the networking and participatory approach lead to greater empowerment among participants vis a vis setting the national agenda?

82 | Page UNFPA Thailand’s CP9 Evaluation Report

Q_PD2: How well is the UNFPA’s PDS programme linked with national policies and priorities?

issues & detailed questions Analysis Source of Data / methods

To what extent are the PD - participation/ involvement - review evidence used components of the CP9 aligned of the national authorities in and participation of with national priorities? the planning and stakeholders in the monitoring process planning process

- Consider goals and output - Review the national / outcome indicators of health plan, UNFPA’s CP9 that aligned with strategic plan, MDG, national priorities CPAP, AWP, progress report and annual report -Compare the achievement of each project. of each program and the trend with the expected -Review the assessment output/outcome indicators report related to RH and PD - results / effects that were not identified in the plan -Interview responsible person in UNFPA, - explore conditions and counterparts and factors contributing to the managers at the results. provincial and district/ sub-district level in some piloting area

83 | Page UNFPA Thailand’s CP9 Evaluation Report

Q_PD3: What factors contribute to the PDS programme’s achievements and shortcomings; what constraints or difficulties within UNFPA’s influence or control affect further achievements?

issues & detailed questions Analysis Source of Data / methods

-Did program design provide a -Identify constraints and/or -interviews with key feasible and appropriate road map difficulties affecting stakeholders including to achieve outcomes with the time achievements UNFPA staff, programme and budgetary resources staff and other allocated? -Situation analysis of how stakeholders difficulties arose and -To what extent did unforeseen attempted solutions -document review difficulties affect achievements?

-When problems arose, were solutions sought in a creative and effective way? Was support given to work through difficulties or use a different approach?

Key Questions about the Reproductive Health Programme (RH)

Q_ RH1: What are the UNFPA’s overall results in terms of its expected outcomes?

1.1 What are UNFPA’s achievements in improving access to quality RH services including HIV prevention in underserved areas and in vulnerable populations, including pregnant mothers and couples, young people, sex workers, and people living with HIV?

issues & detailed questions Analysis Source of Data / methods

Achievement of improving of Review of relevant good quality of RH services (look documents; i.e. annual at accessibility to the services) reports, project reports Time trend of (THA9R11A, THA9R22A, • Do utilization of RH THA9R43A and THA9R54A) services by vulnerable accessibility, and or groups and in benchmark the Key informant interviews underserved areas utilizations and increase? satisfaction (if available) Counterparts • Do HIV prevention of RH services with other (DOH/PPHO/Implementers) through safer sexual provinces without and clients practices among interventions vulnerable group increase? Perception and opinion of • Do RH services sensitive users In-depth interviews (2-5 to gender issue in users) providing good quality services?

84 | Page UNFPA Thailand’s CP9 Evaluation Report

1.2. What is the level of effectiveness, efficiency, and relevance of strategies, key actions, and UNFPA management in the RH programme?

issues & detailed questions Analysis Source of Data / methods

How program design, management • MIC II, project and implementation fit to context of reports, national needs and evolving • Rapid technical assessment of circumstances (Relevancy)? selected projects sites (Lampang and • To what extent are the RH • Health care system Songkhla) components align with national milestones • Operations priority • RH strategy changed or evaluation mission • RH strategies responded to the adjusted with these and field needs perceived by the central involving circumstance observation and state (National) • UNFPA interventions • interviews key support essential sector persons from DOH policy and institutional and DDC of MOH, reforms? implementer counterparts To what extent have the expected results been achieved (Effectiveness & Efficiency)?

• Level of achievement • Achievements • Annual reports comparing to output • In-depth interviews • Financial and HR used effectively? indicators • Achievement justified its cost (unit/cost) compared with others methods • Opinion of implementing organizations (GOs and NGOs) UNFPA provide sufficient support to the implementing counterparts

• Factors have facilitated or constrained achievement of • Timeliness and • Review project projects within UNFPA influence adequacy of reports and control counterparts funding • Rapid technical • Procurement and assessment of implementation issues selected projects • Technical assistance sites (Lampang and from UNFPA Songkhla) • Interviews government organizations

85 | Page UNFPA Thailand’s CP9 Evaluation Report

1.3 Have UNFPA’s efforts to develop managerial capacity improved the quality of RH services?

issues & detailed questions Analysis Source of Data / methods

Improvement of RH information • Improvement of Statistics report system reporting system Review project reports

Sufficient and relevant trainings of • gained management Interview key partners Human resource capability of personnel at each level in the pilot Group meeting of the Key change of process of areas trainees management training • Need assessment • Satisfaction of trainees Review report and Capacity for data utilization for • change of RH service feedback planning and adjust implementations providing in different context

Process of Monitoring and • change process of M&E Evaluation (evidence use, interpretation, participation of stakeholders) Q_RH2: How well is the UNFPA’s RH programme linked with national policies and priorities?

issues & detailed questions Analysis Source of Data / methods

How well UNFPA’s programme MIC 1 and MIC2 linked with national policies? Alignment of the project National plan, policy - Increase access for the objectives with national and priority vulnerable population? policies and priorities - Improve quality with gender and cultural sensitive? Output, outcome - limitations / positive effect achievement of the project Interviews GOs at under what conditions that support the national national level (DOC priorities under different and DOH of MOPH) conditions and time trend Group meeting of providers and users

86 | Page UNFPA Thailand’s CP9 Evaluation Report

Q_RH 3: What factors contribute to the RH programme’s achievements and shortcomings; what constraints or difficulties within UNFPA’s influence or control affect further achievements?

issues & detailed questions Analysis Source of Data / methods

• Factors contribute/ shortcomings • comparison the UNFPA • Periodic report/ ( update Information, technical programs with others minutes of support, design, attitude and skill • output according to time meetings of operators, budget support) trend • Interviews • constraints / difficulties ( • perception / opinions of implementing management rigidity, time stakeholders partners at selected management, budget regulation provincial levels etc)

87 | Page UNFPA Thailand’s CP9 Evaluation Report

For Gender and Rights Issues

The team had been explored the following issues

• Freedom and adequate information for making decision on choices of family planning / reproductive issues – availability, accessibility, acceptability of services • Services which tailored, design according to the need / culture of each sub- populations • Explore the policy, strategies, and implementation that aware / concern rights and genders or even adjust to respond these. • Perceptions and attitude of personnel at all level on these issues

88 | Page UNFPA Thailand’s CP9 Evaluation Report

Annex 6: Evaluation Criteria and Related Questions

Addressing the evaluation objectives will require the evaluator to provide answers to the following questions:

1. Effectiveness ƒ To what extent have the project’s objectives been reached? o To what extent was the project implemented as envisaged by the project document? o If not, why not? o Were the project activities adequate to realize the objectives? ƒ What has the project achieved? Where it failed to meet the outputs identified in the project document, why was this? o Have any significant developments taken place since the project started, if so, explain how they affected the project goal and activities and evaluate the impact on the project?

2. Relevance ƒ Were the objectives of the project in line with defined needs and priorities? o Should another project strategy have been preferred rather than the one implemented to better reflect those needs and priorities? Why? o Were risks appropriately identified by the projects? How appropriate are/were the strategies developed to deal with identified risks?

3. Sustainability ƒ To what extent has the project established processes and systems that are likely to support the continued implementation of the project? o Are the involved parties willing and able to continue the project activities on their own (where applicable)? ƒ Are the project outcomes likely to be sustainable? If not, why not? Which remedial actions would have been good to take?

4. Project design and performance assessment/Efficiency ƒ Was the project design appropriate? If not, why not? o Was the project, including its finances, human resources, monitoring, and oversight and support managed efficiently? ƒ What was the role played by the implementing agency(ies) and, where applicable, the executing agency in leveraging resources, internal or external, and expanding partnerships with other actors to support and expand this project? o Assess the appropriateness of current formal and informal communication channels between national stakeholders, implementing and executing agencies and UNDEF staff, including recommendations for improvement

5. Impact ƒ To what extent has/have the realization of the project objective(s) had an impact on the specific problem the project aimed to address and on the targeted beneficiaries? o To what extent the project has caused and is likely to cause changes and effects, positive and negative, foreseen and unforeseen, on society? ƒ Is the project likely to have a catalytic effect? How? Why? Please provide examples ƒ Have the needs of project beneficiaries been met by the project? If not, why not?

89 | Page UNFPA Thailand’s CP9 Evaluation Report

Annex 7: Interview Guides

NSO Interview Guide:

1. General description of activities and role of respondent:

UNFPA had three objectives in its NSO activities: technical assistance in data collection, advocacy, and increased data utilization. We’d like to get your perspectives on their strategy, their achievements, management issues and other constraints, and sustainability.

First we’d like to know about your role in the different NSO activities, including:

Census:

¾ Workshops at national/local level ¾ Quality assurance project (IPSR) ¾ Field study visits ¾ TPDInfo—Delhi; online database for TAO Surveys:

¾ Older people survey, including data collection, dissemination, utilization ¾ RH survey, including data collection, dissemination, utilization ¾ Income distribution survey, including measurement workshop ¾ Youth and child survey (dissemination)

Work on other outputs:

¾ National statistical master plan (workshop only?) ¾ Gender statistics report

Workshops/trainings:

¾ Panel data analysis ¾ Workshop on mapping MDG indicators ¾ Technical workshop for provincial govt. agencies ¾ Seminars to review the national statistical master plan

Advocacy:

¾ Improved access to data for policy planning at local and provincial level ¾ Public awareness on the census ¾ Policy space for gender statistics report

90 | Page UNFPA Thailand’s CP9 Evaluation Report

2. Planning/strategy 2.1. What is the history of the partnership of UNFPA with NSO? 2.2. Were you involved with UNFPA’s planning process for these activities? To what extent did NSO participate in the planning process? 2.3. To what extent were UNFPA’s and its partners’ capacities (financial, technical and management) considered in the development of CP9? 2.4. Have relevant strategic stakeholders been included in the development and implementation of CP9? How and how effectively? 2.5. Do you feel that UNFPA’s strategies and objectives align with NSO priorities and Thailand’s national priorities in general? 2.6. Do the NSO and OP activities help fill gaps in the national strategy (providing activities not duplicated by other partners)?

3. Achievements/value added of the UNFPA support 3.1. What do you see as the key achievements of the UNFPA activities? (Can go through activities mentioned in question 1) 3.2. What empirical evidence is available to support these achievements? 3.3. Did capacity building efforts address key gaps in the ability to collect high quality and accurate data? 3.4. Census: To what extent did UNFPA inputs contribute to the quality and efficiency of census efforts? How is the information shared at the workshops and the study tours being used by NSO? How was it used to improve the quality of the Census? 3.5. RH Survey: Was technical assistance provided by CPS-Chula to the RH survey effective? What were the challenges? What could UNFPA have done better? 3.6. Other outputs: achievements/value added 3.7. Did the NSO activities create a network for technical assistance and information sharing? 3.8. Are there unexpected results, either exceeding project targets or supporting project activities? (explore the other projects that may contribute the same goal and if there is, comparing CP9 approach with the others) 3.9. What were the main assumptions and risks identified that have affected expected results? Were these assumptions appropriate? 3.10. To what extent have the RH/PDS programmes effectively integrated and reflected gender equality, cultural sensitivity, and human rights?

91 | Page UNFPA Thailand’s CP9 Evaluation Report

4. Utilization 4.1. Are the outputs from the NSO activities (National Survey on Reproductive health and the National survey on Older persons) used by policy makers and planners? Why or why not? Is the UNFPA support contributing to greater utilization of data, such as the surveys, statistical master plan, etc.? 4.2. Is data being used at the local level for planning? Did UNFPA contribute to these efforts? 4.3. Is there concrete evidence of greater utilization? Who could we talk to about this?

5. Management 5.1. How has UNFPA management assisted in efficient implementation for the NSO and OP activities? What are the main challenges in implementing these activities that could be alleviated through UNFPA management mechanisms? 5.2. Are the reporting requirements of UNFPA reasonable or burdensome? 5.3. Have any previous inefficiencies and implementation problems been rectified? Have there been improvements to management based on lessons learned during CP9? 5.4. To what extent has there been flexibility in addressing emerging issues experienced during CP9 and how well did the UNFPA respond to the challenges it faced during implementation of CP9? 5.5. To what extent and with what consequences has UNFPA applied results-based management during its development and implementation of CP9? 5.6. To what extent were UNFPA’s and its partners’ capacities (financial, technical and management) considered in the management of CP9? 5.7. Were UNFPA’s monitoring mechanisms appropriate and the results of monitoring appropriately used to make necessary adjustments? 5.8. Is UNFPA’s budget for CP9 sufficient to ensure achievement of the expected results? 5.9. Were financial and human resources used efficiently and cost effectively? 5.10. What factors have facilitated or constrained achievement of the expected results? Which of these factors are within UNFPA’s influence or control?

6. Advocacy 6.1. What have been the results of advocacy activities during CP9? (key issues: utilization of data for planning, strategies)

7. UNFPA’s strategic position 7.1. In your opinion how can UNFPA best contribute to Thailand’s needs and interests in population development?

92 | Page UNFPA Thailand’s CP9 Evaluation Report

Interview Guide: IPSR Quality Assurance of the Census:

1. General description of activities and role of respondent:

UNFPA had three objectives in its NSO activities: technical assistance in data collection, advocacy, and increased data utilization. IPSR was involved in the Quality Assurance Project for the Census and also some advocacy workshops. We’d like to get your perspectives on their strategy, their achievements, management issues and other constraints, and sustainability of your work. We’d also like to get your opinion on UNFPA’s strategic position for CP10.

2. Planning/strategy 2.1 How did your work on Quality Assurance come about? Was IPSR involved in planning and design? How early in the planning process was IPSR involved? (Is there any written description of the project?) 2.2 Have relevant strategic stakeholders been included in the development and implementation of the project? How and how effectively? 2.3 same for advocacy workshops—get details on number of workshops, planning strategy etc., but may do this separately with Aj. Krit

3. Achievements/value added of the UNFPA support 3.1 Did the activities go as planned? Did they achieve the planned results and meet the deadlines? If not, what were the constraining factors? 3.2 What do you see as the key achievements of the UNFPA activities? Besides the Quality Assurance project, do you see other ways that UNFPA contributed to the quality of the Census? 3.3 What empirical evidence is available to support these achievements? 3.4 Did you see evidence of a participatory approach in the UNFPA efforts? 3.5 (If there were capacity building efforts) Did capacity building efforts address key gaps in the ability to collect high quality and accurate data? 3.6 Were results used to improve the current Census? How were the lessons learned incorporated into current methodologies? 3.7 How will the findings of the study be disseminated? Are there plans to turn these findings into capacity building efforts, technical assistance, or other ways to assure that they are built into the next census? 3.8 For advocacy workshops, did they help to create a network for technical assistance and information sharing? 3.9 Are there unexpected results, either exceeding project targets or supporting project activities? What were the main assumptions and risks identified that have affected expected results? Were these assumptions appropriate? 3.10 To what extent have the RH/PDS programmes effectively integrated and reflected gender equality, cultural sensitivity, and human rights?

4. Utilization/Sustainability/Advocacy 4.1 Are the outputs from the advocacy workshops used by policy makers and planners? Why or why not? Is the UNFPA support contributing to greater utilization of data, such as the surveys, statistical master plan, etc.?

93 | Page UNFPA Thailand’s CP9 Evaluation Report

4.2 Is data being used at the local level for planning? Did UNFPA contribute to these efforts? 4.3 Is there concrete evidence of greater utilization? Who could we talk to about this?

5. Management 5.1 How has UNFPA management assisted in efficient implementation for the activities? What are the main challenges in implementing these activities that could be alleviated through UNFPA management mechanisms? 5.2 Was the budget sufficient? Were human resources used efficiently and cost effectively? What could have been done to make it more efficient/cost effective? 5.3 Are the reporting requirements of UNFPA reasonable or burdensome? 5.4 To what extent has there been flexibility in addressing emerging issues experienced during CP9 and how well did the UNFPA respond to the challenges it faced during implementation of CP9? 5.5 To what extent and with what consequences has UNFPA applied results-based management during its development and implementation of CP9? 5.6 To what extent were UNFPA’s and its partners’ capacities (financial, technical and management) considered in the management of CP9? 5.7 Were UNFPA’s monitoring mechanisms appropriate and the results of monitoring appropriately used to make necessary adjustments? 5.8 Is UNFPA’s budget for CP9 sufficient to ensure achievement of the expected results? 5.9 Were financial and human resources used efficiently and cost effectively? 5.10 What factors have facilitated or constrained achievement of the expected results? Which of these factors are within UNFPA’s influence or control?

6. UNFPA’s strategic position 6.1 What are the strengths of UNFPA? In your opinion how can UNFPA best contribute to Thailand’s needs and interests in population development? What is their value added?

94 | Page UNFPA Thailand’s CP9 Evaluation Report

Interview Guidelines for OP project

1. Project planning a. Please describe the history of your collaboration with UNFPA. Does it continue from the CP8? b. Have you been participated in the project from the very beginning? To what extent have you been involved? Were the outcomes/output of the project considered at the planning process? c. Have the beneficiaries/stakeholders been involved/participated in the project planning process? To what extent? d. Were empirical evidence/data on older persons used in the planning? Please give details. e. Do you consider your project align to national priorities or local policies?

2. Project activities a. Please describe project activities that you are responsible for. b. Do you consider them similar/different from those of other organizations? c. Is there any monitoring and evaluation process conducting along with the project implementation? If yes, has the result been used to improve the project implementation?

3. Output/Outcome a. Have all project activities been implemented according to the plan, to what extent? If not, what are the main problems/obstacles? b. What do you consider as the most success/lesson learned from the project? Please give details. c. Do you think beneficiaries/stakeholders of the project benefit in term of capacity building? Please give examples. d. To what extent does UNFPA play a role in the project? What are the advantages/disadvantages of UNFPA involvement? e. Is there any long-term plan for the project to be sustained? Do new personnel have been trained? f. Have outputs/outcomes of the project been used/expanded to the community, provincial or national levels?

4. Project management a. Do you consider UNFPA project management flexible or rigid comparing to other organizations’ management? b. Do you consider UNFPA management efficiency, helpful and supportive for your project activities? c. Are UNFPA assistances in terms of financial, personnel, technical, etc. sufficiently provided? d. Which type of assistance provided by UNFPA suits your needs the most? e. Do you still need UNFPA assistance in the future? If so, in what particular area? f. What is your opinion, what should be the role/direction of UNFPA in the future? g. Do you have any suggestions/comments for UNFPA, on the project or in general?

95 | Page UNFPA Thailand’s CP9 Evaluation Report

Annex 8: Terms of Reference

“Evaluation of UNFPA Thailand’s 9th Country Programme (2007-2011)”

Background

Thailand is among the countries that have had a short period of fertility transition. It reached a replacement level of fertility in the early 1990s. Although high contraceptive use (of approximately 70-80%) contributes to low fertility in Thailand, many women face unwanted pregnancies. Due to a rapid decline in fertility and increasing life spans, Thailand is rapidly heading toward an ageing society with 10.7% of its population over 60 years of age. In addition, the annual growth rate of 3.7% is among the highest in the region. In 2009 the welfare assistance programme for indigent persons over 60 years became an entitlement for everyone older than 60. The government has introduced several initiatives to strengthen income security at old age, life-long education, and other health and social services for the elderly. As Thailand has emerged as a middle-income country, urban growth has also proceeded rapidly. This growth is the result of both long- standing migration patterns and new settlement trends. About one-third of the population now lives in urban areas. At the same time, international migration, both into and out of the country, is an important economic and social force.

UNFPA’s cooperation in Thailand commenced in 1971 shortly after the launch of the government’s National Family Planning Programme. This cooperation has evolved to meet changing needs in maternal and child health; improving access to reproductive health services including HIV/AIDS prevention; information, education and behavioral change communication; data management including surveys and censuses; population and development; human rights and gender equity; South-to-South cooperation, and emerging population issues such as migration and ageing. Cumulative assistance through 2010 is approximately $47million, with support to over 100 collaborative efforts.

UNFPA’s Executive Board approved the 9th Country Programme (CP9) for Thailand in October 2006. It was developed as an integral part of the UN Partnership Framework (UNPAF) and was later harmonized to contribute to the achievement of UNFPA’s Strategic Plan (2008-2013) and the Millennium Development Goals. The main components of the CP9 include: (a) Population and Development Strategies (PDS), including older persons, development of statistical capacity, and South-South cooperation; and (b) Reproductive Health (RH), including maternal health, development of management capacity, adolescent reproductive health (ARH), and HIV/AIDS prevention. Gender is not a separate component but integrated into both PDS and RH.

The Country Programme Action Plan of the CP9 was signed by Thailand International Development Cooperation Agency and UNFPA in January 2007 followed by the Annual Work Plans (AWPs). Four outcomes and five outputs were initially developed within the above-mentioned focus areas. The implementation of the 2007 activities according to the AWPs was slightly delayed and developed to full capacity in the following years. From introduction of the UNFPA Strategic Plan (2008-2013) in 2008, the original five outputs were reformulated and expanded into five outputs for RH and three outputs for PDS now reflecting the main focus areas of CP9

96 | Page UNFPA Thailand’s CP9 Evaluation Report

CP9 contributes to the national priority to reduce disparities by improving accessibility and utilization of RH information and services including HIV among vulnerable population in underserved areas. CP9 introduces RH services tailored to the needs of target populations in reproductive age especially pregnant women and youth in Lampang and Mae Hong Son in the North and Songkhla and Narathiwat in the South. A pilot model was developed to support male involvement in improving maternal health including prevention of HIV. A Comprehensive Condom Programme was introduced, and access to RH and STI/HIV prevention among sex workers has been enhanced. Key national policies were developed including enhancement of male involvement for HIV prevention during pregnancy, development of the National Standard Service Protocol to increase access of youth-friendly RH services, and a strategic review to enhance reproductive rights of people living with HIV. Moreover the development of health providers’ management capacity as well as a system of technical support for human resource development in the implementing sites were enhanced to improve access to RH services.

On emerging population issues, a collaborative project with representatives from line ministries, a university, and an NGO for policy advocacy and model development to improve the quality of life of older persons was introduced in Chiang Mai and Lampang in collaboration with a national policy dialogue and changes in social welfare and health coverage for older persons. On statistics development, CP9 supports development, data collection, and data analysis as well as utilization of results of the Population and Housing Census, the National Survey on Reproductive Health, and the National Survey on Older Persons. South-South collaboration involves capacity development and knowledge sharing of Thailand’s experiences in RH and PD with participating countries in the region including a review on progress of Programme of Action of International Conference on Population and Development, the 15th Anniversary.

In alignment with these focused areas, from 2007-2009 the AWPs for five and later eight expected outputs were developed. The Project Steering Committees provided guidance to adapt or mainstream the projects and programmes. The annual review of CP9 was undertaken between UNFPA and implementing agencies at the end of each year for strategic review and adjustments to be made to improve the programme performance. A series of results-based management training workshops were provided to counterparts to strengthen their knowledge and skills in programme development and monitoring. Regular monitoring of progress has been conducted jointly by UNFPA and counterparts. A mid-term review of CP9 was not undertaken but rather a series of reviews on measurements of expected results were conducted to strengthen measurement and consistency of the indicators used. Annex 1 provides details about CP9’s intervention (outcomes, outputs, projects, key partners, implementing sites).

During the implementation of CP9, the UN Country Team (UNCT) in Thailand conducted a review to suggest how to reposition the role of UNCT in Thailand in the context of a middle-income country. The results of this two-phase strategic review (Middle Income Country Study-MIC I&II) have given a firm ground for development of the next UNPAF (2012-2016), which will be a framework for development of the UNFPA’s 10th Country Programme (CP10).

97 | Page UNFPA Thailand’s CP9 Evaluation Report

Evaluation Purpose

The evaluation has two primary purposes: (1) to assess whether CP9 has produced demonstrable results in achieving its planned outputs and outcomes and (2) to provide key inputs for the development of the next UNPAF and CP10, which will be submitted to UNFPA’s Executive Board for consideration in 2011.

Thus the evaluation of CP9 should assess UNFPA’s comparative advantage as a development partner in the areas of RH, PDS, and gender. The evaluation will also review and explore UNFPA’s potential contributions to “up-stream” policy development relevant to UNFPA’s mandates including evidence-based advocacy and revision and development of policies, laws, and budgets.

The evaluation will be shared with government agencies as well as with other UN agencies, key counterparts and stakeholders, civil society, and potential development partners for CP10 to extract key lesson learned and recommendations for development of the next country programme.

Objectives UNFPA is a results-based organization so the overall objective of the evaluation will be to (a) assess and apply five evaluative criteria, specifically the relevance, effectiveness, efficiency, sustainability, and impact of CP9 and (b) to inform the development of CP10. Accordingly, the evaluation will assess:

• Contribution of outputs to outcomes (national goals, from which UNPAF outcomes are aligned) • Effectiveness of the intervention in achieving programme outputs – this could be expanded to assess relative effectiveness of the different interventions, the factors affecting effectiveness of intervention, etc. • Effectiveness of management to deliver the interventions – e.g. was the management, implementation arrangements (e.g paertnerhsip) adequate for delivering the interventions. • Assess sustainable - whether the achieved results are or will be sustainable. • Identify elements for the next UNFPA Country Programme

Scope of CP9 Evaluation

A team of Consultants will work together to evaluate the programme as a whole. The Consulting team will assess the following:

3. UNFPA’s contribution by programmatic areas. Review of documents and field data collection will be made on the achievements of UNFPA CP9 against its expected results contributing to development results of Thailand through its partnership development with key strategic partners. The analysis will focus around relevance, effectiveness, efficiency, and sustainability. Although the focus areas are on reproductive health including HIV/AIDS and population development strategies, as gender is mainstreamed and cross-cutting in both components, the evaluation will also assess the rights-based approaches of the interventions to assess how gender and human rights were introduced in the programmes.

4. UNFPA’s positioning and strategies. UNFPA’s positioning and strategies are analyzed both from the perspective of the organization’s mandate and the development needs and country’s priorities. It would entail systematic analysis of UNFPA’s position within the

98 | Page UNFPA Thailand’s CP9 Evaluation Report

development and policy space in the country, as well as strategies used by UNFPA to maximize its contribution through adopting relevant strategies and approaches. The focus will be on relevance and responsiveness & UNFPA’s effectiveness and comparative advantage, particularly in promoting up-stream policy development.

Evaluation Questions

The following questions will be assessed in the evaluation:

General Questions: 1. Based on the UNPAF, UNFPA’s Strategic Plan, the Millennium Development Goals, and the Programme of Action of the International Conference on Population and Development (ICPD), to what extent has CP9 contributed to the achievement of national priorities, including outcomes and impacts?

2. What are the sustainable results of UNFPA’s contributions to Thailand’s National Reproductive Health Strategic Plan and its other population programmes and policies?

3. To what extent has UNFPA’s programme management mechanism and its application contributed to programme achievements?

4. As Thailand becomes a middle-income country, what can and should be UNFPA’s strategic position and contributions to CP10?

5. How effectively and efficiently has UNFPA implemented CP9 and ensured its relevance to Thailand’s needs and priorities?

Key Questions about Population and Development Strategies

1. What are the UNFPA’s overall results in terms of its expected outcomes?

6.2 To what extent have UNFPA’s efforts contributed to the development of statistical capacity in strengthening the quality, collection, analysis, and reporting of data on population?

6.3 What are UNFPA’s achievements in enhancing capacity development and improving South-South cooperation in sharing technical know-how in population development and RH?

6.4 What are UNFPA’s unique contributions in addressing Thailand’s ageing population, particularly in improving access of older persons to economic security, health and social services?

2. How well is the UNFPA’s PDS programme linked with national policies and priorities?

3. What factors contribute to the PDS programme’s achievements and shortcomings; what constraints or difficulties within UNFPA’s influence or control affect further achievements?

99 | Page UNFPA Thailand’s CP9 Evaluation Report

Key Questions about the RH Programme

1. What are the UNFPA’s overall results in terms of its expected outcomes?

1.1 What are UNFPA’s achievements in improving access to quality RH services including HIV prevention in underserved areas and in vulnerable populations, including pregnant mothers and couples, young people, sex workers, and people living with HIV? 1.2 What is the level of effectiveness, efficiency, and relevance of strategies, key actions, and UNFPA management in the RH programme? 1.3 Have UNFPA’s efforts to develop managerial capacity improved the quality of RH services?

2. How well is the UNFPA’s RH programme linked with national policies and priorities?

3. What factors contribute to the RH programme’s achievements and shortcomings; what constraints or difficulties within UNFPA’s influence or control affect further achievements?

Evaluation Team

The evaluation team will include a team leader supported by one or more local consultants with expertise in RH and PDS. The team leader will be (a) responsible for the overall evaluation and management of the evaluation team and (b) accountable for the completion of a report that UNFPA deems to be satisfactory and responsive to the agency’s needs. Annex 3 identifies the requisite experience and qualifications of the evaluators.

UNFPA will appoint an evaluation manager who will represent the organization during the evaluation and who will:

• Be UNFPA’s primary point of contact with the evaluation team; • Provide guidance or clarification to the evaluation team; • Facilitate the team’s access to background documents; • Arrange for the services and other assistance that UNFPA will provide to the team, including logistical support and arranging meetings and field visits; • Coordinate UNFPA's internal review processes; and, • Provide written approval of all deliverables.

Methodology

The evaluation team will design an appropriate methodology in collaboration with UNFPA. This design must reflect the Norms and Standards and the Ethical Code of Conduct established by the United Nations Evaluation Group (UNEG)5 as well the UNFPA’s Evaluation Guidelines.6 UNFPA will provide its guidelines to the evaluation team. The following activities will be undertaken:

• Desk review and briefings: The evaluation team will analyze the documents listed in Annex 4 related to UNFPA’s programmes and projects about CP9. The evaluation team may also request briefing sessions with UNFPA staff to deepen their understanding of the

5 UNEG/FN/Standards (2005) 6 UNFPA Evaluation Guidelines, June 2010

100 | Page UNFPA Thailand’s CP9 Evaluation Report

agency’s work, strategic development, and policies or strategies that may not be fully documented. • Data collection through interviews, surveys (analysis of existing data, not primary data collection), or focus group discussions with key stakeholders in Bangkok and at project/field activity sites selected in consultation with UNFPA. The data-collection strategy should reflect a participatory approach involving a broad range of stakeholders, including government representatives, policy makers in the field of RH and PD, UN agencies, partner agencies, implementing partners, and community members including youth and other vulnerable groups. The evaluation team should also use a variety of methods including triangulation to ensure that the data collected are valid. The evaluation team is required to use an appropriate tool (e.g., an evaluation matrix to present findings from multiple sources) to show that the findings have been validated.

Expected Outputs

The evaluation team will produce the following outputs, all of which must be in English:

1. An inception report. Within 15 days of award of contract, the evaluation team shall submit an electronic copy of a draft inception report to UNFPA’s evaluation manager. The inception report provides an opportunity for UNFPA and the evaluation team to ensure that their interpretations of the TOR are mutually consistent. The manager will review and approve the report, which will serve as an agreement between UNFPA and the evaluation team about how the evaluation will be conducted. This inception report shall:

• Explain the evaluation team’s understanding of what is being evaluated and why; • Describe the team’s strategy for ensuring the evaluation’s utility and applicability to the needs of UNFPA and those of key stakeholders; • Describe the evaluation team’s plans to engage and involve these stakeholders in the design (e.g., questions, objectives, methods, data-collection instruments), data collection, data analysis, and development of recommendations; • Explain how the evaluation questions will be addressed with respect to all evaluative criteria indicated above by way of proposed methods, evaluation designs, sampling plans, proposed sources of data, and data-collection procedures;

Note: The evaluation team is encouraged to suggest refinements to the TOR and to propose creative or cost- or time-saving approaches to the evaluation and explain their anticipated value.

• For each of the five evaluative criteria identified above, describe the measurable performance indicators or standards of performance that will be used to assess progress towards the attainment of results, including outcomes; • Discuss (a) the limitations of the proposed methods and approaches, including sampling, with respect to the ability of the evaluation team to attribute results observed to UNFPA’s efforts especially when there is no consideration of a valid counterfactual and (b) what will be done to minimize the possible biases and effects of these limitations; • Explain the team’s procedures for ensuring quality control for all deliverables; • Explain the team’s procedures to ensure informed consent among all people to be interviewed or surveyed and confidentiality and privacy during and after discussion of sensitive issues with beneficiaries or members of the public; • Explain how the evaluation will reflect attention to and mainstreaming of gender concerns and identify the member of the evaluation team who will be responsible for doing so; • Indicate familiarity with and agreement to adhere to (a) the requirements of the Standards for Evaluation in the UN System, especially standards 4.1 through 4.18

101 | Page UNFPA Thailand’s CP9 Evaluation Report

and (b) UNFPA’s Evaluation Quality Standards, which will be provided to the evaluation team; and, • Provide a proposed schedule of tasks, activities (including site visits), and deliverables consistent with this TOR. No site visits shall occur before UNFPA’s approval of the inception report.

The evaluation team may be asked to make an oral presentation of the inception report to UNFPA and the Evaluation Management Committee. UNFPA’s evaluation manager will provide written comments on the inception report to the team within 7 days of the report’s submission or completion of the oral presentation, whichever comes later. UNFPA reserves the right to modify the TOR in response to the inception report.

Note: Items marked with an asterisk should also be discussed in the evaluation report.

4. A draft evaluation report. The evaluation team shall submit an electronic copy of a draft evaluation report to UNFPA’s evaluation manager no later than January 15, 2011. The draft report should be thoroughly copy edited to ensure that comments from UNFPA and other stakeholders on content, presentation, language, and structure can be reduced to a minimum.

After UNFPA’s and stakeholders’ review of the draft report, the evaluation manager will provide written comments to the evaluation team. Based on these comments, the team shall correct all factual errors and inaccuracies and make changes related to the report’s structure, consistency, analytical rigor, validity of evidence, and requirements in the TOR. The team will not be required to make changes to conclusions and recommendations unless they are regarded as qualitative improvements. After making the necessary changes, the evaluation team will submit a revised draft evaluation report, which may lead to further comments from UNFPA. After the second round of review and, if necessary, further revision to the draft evaluation report, the evaluation team can then submit the final report pending UNFPA’s approval.

5. A final evaluation report. UNFPA will provide the evaluation team with a recommended outline for the final report. The report must contain a self-contained executive summary that provides a clear, concise presentation of the evaluation’s main conclusions and key recommendations and reviews salient issues identified in the evaluation. At a minimum, the final report shall contain the following annexes: • List of persons interviewed (if confidentiality permits) and sites visited; • Data-collection instruments (copies of surveys, questionnaires, etc.); • A bibliography or list of references; and, • The TOR for the evaluation.

All materials produced or acquired during the evaluation shall remain the property of UNFPA unless explicitly relinquished in writing. UNFPA will retain the exclusive right to publish or disseminate in all languages reports arising from such materials. The rights and duties specified in this paragraph shall continue, notwithstanding the termination of the contract for the evaluation.

102 | Page UNFPA Thailand’s CP9 Evaluation Report

Timeline

The evaluation report should be completed no later than January 31, 2011 to ensure that sufficient time is available to accommodate the main recommendations, build on good practices, and address identified obstacles and challenges.

Date Activity Within 15 days of award of • Submission of inception report contract Within 7 days after • UNFPA Evaluation Manager receiving the inception report October-November 2010 • Desk review of documents • Data collection No later than 15 Dec., • Presentation of preliminary evaluation results to the 2010 Evaluation Management Committee

15 January, 2011 • Submission of the 1st draft CP9 evaluation report

31 January, 2011 • Submission of the final CP9 evaluation report

February, 2011 • Dissemination of evaluation results to key stakeholders as advise by the Evaluation Management Committee

Payment of Consulting Fees

The expected number of working days and the consulting fee together with the fieldwork cost will be proposed to UNFPA by the evaluation team. Payment of the evaluation team will be in three installments, as follows:

ƒ 1st payment (30%) – Upon UNFPA’s approval of the inception report; ƒ 2nd payment (30%) – Upon submission of the first draft; and ƒ 3rd payment (40%) – Upon UNFPA’s acceptance of the final evaluation report.

103 | Page UNFPA Thailand’s CP9 Evaluation Report

Outcomes, Outputs, Intervention

Project Code THA9R11A THA9R19A THA9R22A THA9R43A THA9R54A THA9P15A THA9P36A THA9P47A

AWP Titles Promote Management Strengthen Strengthening Strengthen South to South MDG Evidence‐based reproductive Capacity access to and SRH/HIV access to and Cooperation in Monitoring and Responsiveness rights and SRH Development utilization of services for utilization of Reproductive Strengthening to the Emerging demands for Project culturally and vulnerable SRH/HIV Health and Statistical Challenges of vulnerable gender‐sensitive groups in information, Population Capacity the Population populations information, decentralization counselling and Development Ageing in counselling and context services for Thailand services of young people maternal health Known as RR MCD MMS/MPM STRONG YSRH South‐South NSO OP Counterparts Reproductive PBRI, MOPH Reproductive Bureau of AIDS, Reproductive Thailand National HelpAge Health Division, Health Division, TB, STIs Health Division, International Statistical International MOPH MOPH MOPH Development Office, MICT and Faculty of Cooperation Nursing, CMU Agency (TICA), and others MOF UNFPA R1 R1 R2 R4 R5 P1 P3 P4 Strategic Promote SRH Promote SRH Access to Demand and Access of young Population Population Emerging Planning Rights and Rights and Maternal Health Utility of HIV/STI to SRH and Dynamics and Gender and SRH Population Outcomes Demand Demand Service services gender Inter‐linkages Data for Issues in Development Development Country 1 2 2 3 4 Programme Increased utilization of RH information and services by Improved HIV Improved HIV Improved Increased utilization of Outcomes vulnerable groups and in underserved areas prevention prevention knowledge disaggregated data for policy and through safer through safer management and programme formulation at national networking sexual practices sexual practices mechanisms for and sub‐national levels for among among intercountry addressing maternal and child vulnerable vulnerable communication health and newborn health, ARH, groups groups and action on HIV/AIDS, gender, migration and population, RH, population ageing HIV/AIDS, gender, migration and population ageing issues in the region

104 | Page UNFPA Thailand’s CP9 Evaluation Report

Project Code THA9R11A THA9R19A THA9R22A THA9R43A THA9R54A THA9P15A THA9P36A THA9P47A

Country Programme 1 8 2 3 4 5 6 7 Outputs Promote Enhanced Improved access Improved access Improved access Enhanced capacity Enhanced capacity Improved access reproductive institutional to quality to information, to RH information, and improved at national level to of older persons rights and SRH capacity at maternal health counselling and including HIV mechanism for strengthen quality to economic demands for national and sub‐ information and services for HIV prevention for South‐South of survey and security, health vulnerable national levels for services prevention among young people cooperation in census data and social services populations planning, sex workers, sharing technical implementing, clients of sex know‐how in managing, and workers in population related monitoring of RH underserved areas areas programmes Strategies 1. Reproductive Management 1. Making 1. Sexual and 1. Youth Sexual 1. Capacity dev. Capacity Dev. and 1. Model Rights of Capacity Motherhood Safer reproductive and RH 2. International advocacy for development PLWA Development 2. Male health of sex 3. Solution training census, OP and RH 2. Capacity dev. 2. Policy involvement in workers Exchange 3. Networking surveys 3. Policy advocacy Advocacy maternal health 2. Comprehensive 4. Youth Advisory 4. M&E 3. M&E condom Panel/Y‐Peer 4. UNFPA core programme 5. Love Station 3. Sex Work (PATH) Project (BATS & SWING) Implementing sites National National National National National Region National National Lampang Lampang Lampang Lampang Lampang Mae Hong Son Mae Hong Son Songkhla Bangkok Chiang Mai Narathiwat Bangkok Mae Hong Son Songkhla Pattaya

Budget for 574,329* 189,379 682,613 * 616,746 432,095 * 234,592 432,301 861,509 programme

Total programme budget $4,522,096 (2007‐10) Total budget for $1,841,519 programme support (2007‐10)

GRAND TOTAL $6,363,615 in US$ Note: In 2007 - THA9R11A, THA9R22A, THA9R43A were combined under THA9R201 and the total budget was $498,532.

105 | Page UNFPA Thailand’s CP9 Evaluation Report

Evaluation Questions

Programme design, management, and implementation a. To what extent are the RH and PD components of the CP9 aligned with national priorities? b. What were the main assumptions and risks identified that have affected expected results? Were these assumptions appropriate? c. To what extent and with what consequences has UNFPA applied results-oriented management during its development and implementation of CP9? d. To what extent were UNFPA’s and its partners’ capacities (financial, technical and management) considered in the development and management of CP9? e. Have relevant strategic stakeholders been included in the development and implementation of CP9? How and how effectively? f. To what extent has there been flexibility in addressing emerging issues experienced during CP9 and how well did the UNFPA respond to the challenges it faced during implementation of CP9? g. Were UNFPA’s monitoring mechanisms appropriate and the results of monitoring appropriately used to make necessary adjustments? h. To what extent have the RH/PDS programmes effectively integrated and reflected gender equality, cultural sensitivity, and human rights?

UNFPA’s effectiveness and efficiency a. To what extent have the expected results been achieved and what empirical evidence is available to support the findings? To what extent can the achievements identified be attributed to UNFPA’s efforts? b. To what extent are the identified outcomes the result of the UNFPA’s interventions rather than external factors or other possible explanations? c. Is UNFPA’s budget for CP9 sufficient to ensure achievement of the expected results? d. Were financial and human resources used efficiently and cost effectively? e. What factors have facilitated or constrained achievement of the expected results? Which of these factors are within UNFPA’s influence or control? f. What are the opinions of stakeholders (policy makers, planners, data users and data collectors/processors) about the results? g. Do the results of CP9 justify its cost? h. What could have been done to make UNFPA’s efforts more effective and efficient?

Sustainability and programme replication a. Are the achieved results sustainable? If so, what evidence supports this conclusion? b. Did the design and implementation of CP9 include strategies to promote sustainability? c. Have UNFPA’s national counterparts contributed sufficient resources to promote sustainability? d. In what way have lessons learned, good practices, and effective innovative interventions been used to increase the quality of access and services? e. What are the facilitating and constraining factors within the UNFPA’s control or influence that affect sustainability and replication? Is there evidence indicating that some pilot models will be replicated or expanded after termination of UNFPA funding?

106 | Page UNFPA Thailand’s CP9 Evaluation Report

Annex 9: Evaluation Team

Researcher and Evaluation Expertise Institute Team

Overall program management

Dr. Supattra Srivanichakorn Primary health care and MD, MPH, Director, AIHD, health systems Mahidol University management, reproductive health/ ageing/ evaluation of HIV/AIDS projects

Dr. Aroonsri Mongkolchati Population and social PhD, Researcher, AIHD, research, public health Mahidol University

Miss Orawan Qwansri Nurse / psychology / MA, Project Coordinator, international co-ordination AIHD, Mahidol University

Miss Parinda Tasee Sociology/ evaluation of MA, Researcher, AIHD HIV/AIDS projects institute, Mahidol University

RH program

Dr. Bang-on Theptein Reproductive health/ PhD, Researcher, AIHD, behavioral health/ Mahidol University evaluation of HIV/AIDS

Assistant Professor Somsak Reproductive health / MPH, Lecturer, AIHD, Wongsawass statistics Mahidol University

Dr. Kanokwon Tharawan Women’s health advocacy / PhD,Medical Anthropologist, evaluation of HIV/AIDS Faculty of Social Science, projects Mahidol University

PD program

Dr. Kerry Richter Sociology/demography, PhD, Foreign Expert, IPSR, monitoring and evaluation Mahidol University

Dr. Sawarai Boonyamanond Demography development/ PhD, Lecturer, IPSR, Mahidol University economics

107 | Page UNFPA Thailand’s CP9 Evaluation Report