Evaluation of UNFPA/Cambodia's 3 Country Programme, 2006–2010

Evaluation of UNFPA/Cambodia’s 3rd Country Programme, 2006–2010

Dr. Richard Tobin, Team Leader Ms. Uk Toptosseda, Gender Specialist Dr. Em Sovannarith, Reproductive Health Specialist Mr. Hong Sokheang, Population and Development Specialist

December 2011 Phnom Penh, Cambodia

Table of Contents

Acronyms and Abbreviations ...... 3 Executive Summary ...... 4 Programme Overview ...... 8 Programme Management and Implementation ...... 10 Need, Purpose, and Scope ...... 10 Purpose of the evaluation ...... 11 The scope of the evaluation ...... 12 Methodology ...... 12 Data sources and data-collection methods ...... 13 Limitations of the evaluation ...... 13 CP3‟s results framework ...... 14 Mitigating the limitations ...... 17 Ethical Considerations ...... 19 Adherence to International Best Practices ...... 19 Findings...... 20 Relevance ...... 20 Effectiveness ...... 24 Efficiency ...... 37 Impact ...... 46 Impacts on gender and human rights ...... 50 Conclusions ...... 51 Key Recommendations ...... 53 Substantive recommendations ...... 54 Procedural recommendations ...... 55 Annex 1: Terms of Reference ...... 57 Annex 2: Methodological Framework for the Evaluation ...... 66 Annex 3: Documents Reviewed for the Evaluation ...... 68 Annex 4: List of People Interviewed ...... 71 Annex 5: Interview Guide ...... 78 Annex 6: Statement on Confidentiality and Informed Consent ...... 80 Annex 7: DOS Evaluation Quality Criteria ...... 81

Acronyms and Abbreviations

ADMC Associate Degree Midwifery Curriculum AusAID Australian Agency for International Development AWP Annual Work Plan AWPMT Annual Work Plan Monitoring Tool CCWC Commune Committee for Women and Children CDC Council for the Development of Cambodia CHEMS Cambodia Health Education and Media Service COAR Country office annual report CPA Complementary Package of Activities CPAP Country Programme Action Plan CPE Country programme evaluation CP3 3rd Country Programme of UNFPA/Cambodia CP4 4th Country Programme of UNFPA/Cambodia DFID Department for International Development DHS Demographic and health survey DOLA Department of Local Administration DOS Division for Oversight Services (UNFPA) EMC Evaluation management committee EmONC Emergency obstetric and newborn care FACE Funding Authorization and Certificate of Expenditure GMAG Gender Mainstreaming Action Group HC Health center ICHAD Interdepartmental Committee on HIV/AIDS and Drugs IP Implementing partner IPCD International Conference on Population and Development JPIG Joint Partnership Arrangement Development Partners Interface Group KYA Khmer Youth Association MDGs Millennium Development Goals MoEYS Ministry of Education, Youth, and Sports MoH Ministry of Health MoWA Ministry of Women‟s Affairs MPA Minimum Package of Activities NCPD National Committee for Population and Development NGO Nongovernmental organization NSDP National Strategic Development Plan OD Operational district OFA Operating fund account PD Population and development PHD Provincial Health Department RBM Results-based management RGC Royal Government of Cambodia RH Reproductive health UNDAF United Nations Development Assistance Framework UNDP United Nations Development Programme UNICEF United Nations Children‟s Fund UNFPA United Nations Population Fund

Executive Summary

he United Nations Population Fund (UNFPA) completed the implementation of T its 3rd country programme in Cambodia in December 2010. The five-year programme (2006–2010) focused its assistance in three thematic areas: (a) gender; (b) population and development; and (c) reproductive health and in 14 of Cambodia‟s 24 provinces. The programme was implemented through several nongovernmental organizations and six government agencies, including the Ministries of Health; Interior; Planning; Women‟s Affairs; Education, Youth and Sports; and the National Committee for Population and Development. In accordance with the Fund‟s corporate requirements, country programmes are typically evaluated in their penultimate year. For Cambodia‟s 3rd country programme, this would have been 2009. Given the paucity of data and the timing of two major data-collection efforts, namely a census of population in 2008 and a demographic and health survey in 2010, the country office postponed the evaluation until these efforts had been completed and their results available. The delay was intended to allow the evaluation to produce meaningful findings that would be useful for future programmes, including the current 4th country programme, which extends from 2011 through 2015. The evaluation is an external exercise and is aimed at generating an independent assessment of successes, challenges, and lessons learned in an effort to contribute to the improved performance of UNFPA‟s 4th country programme. The intended audience for the evaluation includes UNFPA/Cambodia, the agency‟s regional office in Bangkok, and the implementing partners with whom the agency collaborates in Cambodia. The evaluation considered UNFPA‟s performance in achieving development results and examined the country programme‟s relevance, effectiveness, efficiency, and impact, all of which are standard assessment criteria for development-related programmes, including those of the UN Population Fund. To complete the evaluation, the country office hired an independent, four-person team composed of an internationally recruited team leader and three locally hired and experienced experts – one for each of the three programmatic areas. In planning and executing the evaluation, the evaluation team adhered to three guiding principles. First, the team emphasized adherence to international best practices in evaluation, including the UN Evaluation Group‟s Standards for Evaluation in the UN System, its Norms for Evaluation in the UN System, and its Code of Conduct for Evaluations in the UN System. In particular, the evaluation team con- formed to best practices regarding informed consent and confidentiality. To the extent possible, all informants were informed of the evaluation‟s purpose, the confidentiality of their discussions with the evaluation team, and team‟s interest in their suggestions and recommendations about how the UNFPA can increase its effectiveness and efficiency in Cambodia. Second, the evaluation team emphasized stakeholders‟ meaningful participation and engagement in the evaluation. The team coordinated its work with the Fund‟s country office and its evaluation management committee, which included as many as 24 stakeholders, to build consensus around the evaluation‟s purposes, scope, and processes. The team shared its inception report with this committee and subsequently met with its members at the beginning and end of the evaluation process and individually with many members of the committee. All

4 members of the committee were also asked to review and provide comments on the draft evaluation report, and about half of the members did so. Third, the evaluation emphasized triangulation of data sources in an effort to validate the information included in the evaluation. The triangulation process involved a review of government reports, internal reports from the country office, three demographic and health surveys, and interviews with more than 200 people in Phnom Penh and in seven provinces. Limitations of time and resources prevented the team from visiting all of the provinces in which the agency focused its resources. Despite the evaluation team‟s best efforts, the methods it used were not without limitations. Attribution of results to the agency‟s efforts was problematic because others donor address the same issues as does the UNFPA in Cambodia. Furthermore, the UNFPA is not an implementing agency and typically does not come into direct contact with its intended beneficiaries and most of the individuals and government units that implement activities designed to advance the agency‟s agenda. As a consequence, some or even many of the agency‟s beneficiaries have no or only limited familiarity with the agency‟s work or support for their efforts. Finally, there were many changes in UNFPA‟s country staff during the programme period, with the consequent loss of institutional memory that was not readily available to the evaluation team. With respect to relevance, the 3rd country programme was highly relevant to a wide range of government policies and strategies, and for this the country office should be commended. Whether one considers the National Strategic Development Plan or one of several strategic sectoral plans, such as the one for health, UNFPA‟s three thematic areas of focus align closely with them. As the evaluation team was informed, the country programme was one of few such agency programmes in the Asia-Pacific region that was fully aligned with national programmes and priorities and in line with ongoing sectoral reform processes. Cambodia‟s country programme relied on and promoted national ownership and worked effectively through existing structures and processes in accordance with the Paris Declaration on Aid Effective- ness. No less important, the close alignment between the government‟s priorities and UNFPA‟s country programme means that it was also highly relevant to the needs of Cambo- dians, especially those in rural and previously underserved areas. UNFPA‟s effectiveness in achieving its intended results was mixed. On the one hand, Cambodia and its development partners, including UNFPA, can point to many successes and accomplishments in each of the three areas in which the agency provides support. One such example is UNFPA support for the recruitment, training, and deployment of secondary midwives, which were in short supply when the country programme began. To support the government‟s objective, UNFPA provided support to ensure that all of the country‟s health centers had at least one trained midwife by 2010. This target was exceeded; by the end of 2010, more than 50 percent of the health centers had more than one trained midwife. On the other hand, there were also several instances in which UNFPA did not achieve the results it had anticipated. Several reasons exist for this situation. Several of the country programme‟s indicators and their targets were not objectively verifiable or did not have objective standards that permitted a reliable judgment about whether the targets had been achieved. In other instances the best efforts of the agency and its implementing partners were

5 insufficient, so targets were not achieved. There were also several instances in which the UNFPA‟s targets were overly modest, but there were also some targets that were beyond UNFPA‟s ability to achieve, regardless of its efforts or diligence. The evaluation provides several examples of the administrative efficiency of UNFPA‟s country office. Its review and approval of implementing partners‟ annual work plans and their requests for advance funding and reimbursement of expenses incurred is exemplary. In contrast, the country office was less efficient in managing the financial resources transferred to these partners. They typically requested and UNFPA/Cambodia approved more in advance funding than these partners needed. In turn, in several years some implementing partners requested and were given more money than they were able to absorb or spend. UNFPA also faced challenges in ensuring understanding and proper enforcement of the agency‟s procedures for the reimbursement of indirect expenses for all of its implementing partners, including government agencies. Assessing UNFPA‟s legacy or longer-term impact is challenging. UNFPA supports several government ministries and much of the agency‟s success and accomplishments depend on how well these ministries perform their functions. Several factors affect how well these functions are performed, including high staff turnover, limited individual and organizational capacity, persistently low salaries that lead to low motivation and high levels of absenteeism, and long-standing cultural norms that are unsympathetic to gender equality. In the face of these challenges, UNFPA‟s limited resources provide only limited leverage and opportunities for impact that is attributable to the agency. Notwithstanding this constraint, there are some data that permit an assessment of impact through the use of counterfactual. A counterfactual approach asks what would have happened in the absence of UNFPA‟s intervention. Using data from the 2005 and 2010 demographic and health surveys and a random sample of women who had or had not been exposed to a UNFPA-sponsored media campaign intended to increase awareness about reproductive health, the evaluation team was able to compare the results of UNFPA‟s interventions for people that were or were not exposed to its interventions. On the whole, these comparisons suggest that UNFPA‟s interventions had little to no impact. Most of the comparisons reflect only negligible differences between those who had and those who had not been exposed to these interventions. The evaluation concludes with several priority recommendations and a series of procedural recommendations, all of which are intended to improve the effectiveness, efficiency, and impact of the current, 4th country programme. The priority recommendations include these: 1. UNFPA should identify its areas of particular competency and then limit its attention and resources to those few areas in which the country office can demonstrate a comparative advantage over all or most other development partners. This recommendation is especially germane in an environment in which (a) donors are increasingly requiring UN agencies to demonstrate improvements in people‟s lives and (b) the resources that many donors can provide to UNFPA are increasingly constrained. 2. UNFPA should develop a strategy for its efforts to develop capacity. The strategy should be demand driven, be based on a rigorous and comprehensive assessment of gaps and needs, and specifically identify the measurable goals and objectives of UNFPA‟s efforts. The strategy should be explicit in defining what should be achieved in terms of the

6 skills and abilities among the presumed beneficiaries of UNFPA‟s capacity-building efforts. The strategy should also include a means to measure and objectively evaluate the results, including the quality, of any capacity-building initiatives that UNFPA supports either directly or indirectly through its implementing partners. 3. UNFPA/Cambodia should increase its attention to the evaluation of the initiatives its supports. UNFPA should consider evaluations of its initiatives for out-of-school youth, the quality of midwives, efforts to improve facilities for emergency obstetric and neonatal care, and the consequences and benefits of including mention of gender, population, and reproductive rights in national and subnational plans and policies. Given the magnitude of the resources devoted to capacity building, an evaluation of capacity building is essential and should be accorded a high priority. 4. The country office should ensure that activities included in annual work plans of the UNFPA‟s implementing partners contribute to desired results. The agency should not fund activities that do not contribute or logically lead to these results, and the country office should ensure that its implementing partners understand this requirement. A related review of the current country programme could usefully consider the extent to which the indicators and their targets are specific, measurable, attributable, relevant, and time bound (i.e., SMART).

Programme Overview

he United Nations Population Fund opened its country office in Cambodia in T 1994 at the request of the Royal Government of Cambodia (RGC) following the UN-sponsored national elections in 1993. UNFPA has steadily increased its technical and financial assistance to Cambodia since then through successive programmes of assistance. UNFPA/Cambodia recently completed the implementation of its 3rd country programme (CP3) 2006–2010, the goals of which were to support the RGC in its pursuit of the Millennium Development Goals (MDGs) and the targets of the Programme of Action of the International Conference on Population and Development (ICPD) through the implementation of Cambodia‟s National Strategic Development Plan (NSDP) and the United Nations Development Assistance Framework (UNDAF) 2006–2010. CP3 focused its assistance in three areas: (a) gender; (b) population and development (PD); and (c) reproductive health (RH). As the Country Programme Action Plan (CPAP) for 2006–2010 indicated, CP3 included six expected outcomes and eight related outputs, as shown in Table 1. Of the US$27 million of proposed assistance under CP3, $18 million was intended for RH, $6 million for PD, and $2 million for gender. The remainder, $1 million, was allocated to UNFPA‟s programme coordination and assistance. The data in Table 2 show both the projected and actual expenditures for each of the three thematic areas. Several findings related to expenditures merit mention. First, less was spent on RH than had been projected, due largely to the fact that the country office obtained far fewer “other resources” (i.e., funds from donors provided directly to the country office) than had been projected. In addition, according to the country office, some of these other resources were used to support PD activities because of issues related to the capacity of the Minis- try of Health to use the funds. Second, and for the opposite reason, more was spent on PD-related activities than had been projected. Much of this additional money was used to support Cambodia‟s 2008 national census of population and the 2010 demographic and health survey (DHS). Geographically, CP3‟s activities were focused in 71 districts in 14 of Cambodia‟s 24 provinces (see figure 1). Cambodia‟s districts, which are administrative subdivisions of provinces, contain 1,621 communes. In turn, each commune includes about eight to nine villages. Within the CP3‟s 71 districts, UNFPA‟s activities encompassed 446 communes and slightly over 3,000 villages in the 14 provinces.1 As the CPAP noted, the locations reflected need as well as poor results in relation to RH, gender, and attention to population issues.2 These locations also did not have large-scale external assistance in the three thematic areas.

1 The provinces included Banteay Mean Chey, Battambang, Kampong Cham, Kampong Chhnang, Kampong Thom, Koh Kong, Kratie, Mondul Kiri, Oddar Mean Chey, Pailin, Preah Vihear, Ratanakiri, Siem Reap, and Stung Treng. 2 The country office noted that the priority areas were selected “based on need such as high levels of poverty, population growth, migration, gender inequality, and unmet need for family planning.” See UNFPA and RGC, Mid Term Review Report, October 2008. The country office subsequently added three additional provinces in 2009 when it received support from the Australian Agency for International Development (AusAid). These provinces, Kampot, Kandal, and Preah Sihanouk, did not meet the same criteria that had been used to select the original 14 priority provinces. Among Cambodia‟s 24 provinces, Kampot and Kandal had the least need for further attention to RH, at least according to the selection criteria that UNFPA/Cambodia used. According to the

Table 1: CP3‟s Intended Outcomes and Outputs Area Intended Outcomes Intended Outputs PD National and sectoral policies, decentralized Improved national and decentralized capacity to inte- plans and strategies take into account popula- grate and implement population, poverty, and devel- tion, poverty, and development linkages opment issues within national, sectoral, and decentralized plans The collection and utilization of age and sex- Strengthened national and local capacity in collect- disaggregated population and poverty data at ing, analyzing, interpreting, disseminating, and utiliz- national and decentralized levels are improved ing disaggregated population and poverty data for decentralized planning, monitoring, and policy making RH A policy environment that promotes RH and Strengthened national capacity to develop, imple- reproductive rights ment, and evaluate gender-sensitive RH and HIV policies, strategies, and protocols Increased access to and utilization of high- Strengthened capacity of relevant government insti- quality RH services tutions and NGOs to provide high-quality RH services, including those focusing on HIV/AIDS and sexually transmitted infections, in priority areas Increased access to high-quality, comprehensive, client-oriented, and gender-sensitive RH information and services (including those for the rural poor and vulnerable groups in priority areas) Increased awareness and empowerment of the Increased awareness of women, men, and youth population, particularly women and youth, about RH, reproductive rights and available services regarding their reproductive rights, including in priority areas RH services Gender Institutional mechanisms and sociocultural Strengthened capacity of priority ministries, selected practices promote and protect the rights of commune councils, and the media to promote the women and girls to advance gender equity empowerment of women and youth Increased awareness and empowerment of women and youth in the priority areas to claim their rights to gender equity

Table 2: Projected and actual expenditures, 2006–2010, for CP3 (in $000) Projected expend- Reproductive Population and Gender Programme Coor- Total itures Health Development dination Regular resources $11,000 $4,000 $2,000 $1,000 $18,000 Other resources $7,000 $2,000 0 0 9,000 Total $18,000 $6,000 $2,000 $1,000 $27,000 % of total CP3‟s projected expendi- 66.7% 22.2% 7.4% 3.7% 100.0% tures Actual expenditures Regular resources $10,506 $4,656 $2,015 $761 $17,938 Other resources $4,701 $2,583 0 0 $7,284 Total $15,207 $7,239 $2,015 $761 $25,222 % of total CP3‟s actual expendi- 60.3% 28.7% 8.0% 3.0% 100.0% tures Note: All currencies throughout this report are in U.S. dollars.

country office, the selection of the additional provinces was related to alignment of funding from AusAid with the geographic areas in which the GAVI Alliance (formerly the Global Alliance for Vaccines and Immunisation) was working.

Figure 1: Map of Cambodia

Source: Canby Publications

Programme Management and Implementation

he Council for the Development of Cambodia (CDC) is responsible for coordi- T nating all official development assistance including the assistance that the UNFPA provides. Implementing partners (IP) under CP3 included the Ministry of Planning; Ministry of Health (MoH); Ministry of Women‟s Affairs (MoWA); Ministry of Education, Youth, and Sports (MoEYS); the Ministry of Interior‟s Department of Local Administration (DOLA); the National Committee for Population and Development (NCPD); and several nongovernmental organizations (NGOs) including CARE, the Khmer Youth Association (KYA), Cambodia Health Education Media Service (CHEMS), and the Reproductive Health Association of Cambodia. Annual work plans (AWPs) were the primary tool for operational- izing UNFPA‟s collaboration with each of these IPs.

Need, Purpose, and Scope n accordance with UNFPA‟s corporate requirements, country programmes are typi- I cally evaluated in their penultimate year. For CP3, this would have been 2009.

Given the paucity of data and the timing of two major data-collection efforts, namely the 2008 census and the 2010 DHS, the country office proposed the postponement of the evaluation until these efforts were complete and their results available. The delay was intended to allow the evaluation to produce meaningful findings that will be useful for future programmes, including the current 4th Country Programme (CP4). With approval of UNFPA‟s regional office in Bangkok, the country programme evaluation (CPE) began in October 2011. The CPE is an external exercise and is aimed at generating an independent assessment of successes, challenges, and lessons learned.

Purpose of the evaluation The CPE focuses on the outputs and outcomes achieved through the implementation of CP3. The evaluation assesses the programme‟s relevance to national priorities and those of the UNDAF. In addition, the evaluation assesses UNFPA‟s and its IPs‟ achievements from 2006 through 2010 against intended results and the efficiency with which these results were achieved. As requested in the terms of reference (TOR) (Annex 1) for the evaluation, the evaluation also briefly assesses the extent to which CP4, as implemented, provides the best possible modalities for reaching the intended objectives, on the basis of results to date. The CPE provides recommendations for the five-year action plan of the CP4 and subsequent annual work plans and offer suggestions about the design of future UNFPA operations in Cam- bodia. The specific objectives of the evaluation are to assess CP3‟s:  relevance by examining the programme‟s external coherence and relevance with the RGC‟s NSDP 2006–2010, the UNDAF, and other key planning documents as well as the CP3‟s relevance to the needs of the target population, the relevant MDGs, and the ICPD‟s Programme of Action;  effectiveness in terms of achievement of intended results;  efficiency in terms of whether the outputs achieved were reasonable for the resources spent; and,  impact, including the overall effects (intended and unintended, short term and long term, positive and negative).

In addition to the four criteria just noted, the CPE also assesses: (a) UNFPA‟s leadership and management in terms of human resources, financial management, and systems; (b) the extent to which UNFPA‟s programs have integrated gender and human rights as cross- cutting themes and promoted gender equity and gender sensitivity; and, (c) the extent to which the UNFPA country office and its country programme responded to pertinent sector reform initiatives and the international aid effectiveness agenda.3 For the latter issue, the Par- is Declaration on Aid Effectiveness provides the performance benchmarks against which UNFPA‟s responsiveness was assessed.

3 Evaluations for UNFPA often include attention to sustainability, but the agency‟s Evaluation Guidelines indicate that such attention is not obligatory. In the present instance the country office decided to omit attention to sustainability to accommodate the resources and time available to complete the evaluation.

The scope of the evaluation

The evaluation covers the CP3‟s timeframe from 2006 through 2010 and seeks to determine what has worked, what has not worked, and the reasons why. In line with UNFPA‟s Evaluation Policy the evaluation focuses on “performance in achieving development results,” which includes outputs, outcomes, and impacts. This definition of results necessarily dimin- ishes attention to the number and variety of activities performed or completed. Completion of activities is not a direct indication of their effectiveness. The United Nations Development Programme‟s guidance on this issue is pertinent here: “Planning, monitoring and evaluation processes should be geared toward ensuring that results are achieved – not toward ensuring that all activities and outputs get produced as planned.” Attention to results also has a corollary within UNFPA. The agency has committed to mainstream results-based management (RBM) into all of its activities. At a minimum, this concept requires that all of the agency‟s processes, products, and programmes contribute to the achievement of desired results. RBM has several steps, all of which serve as an organiz- ing framework for the evaluation:

1. Identify clear and measurable changes and results to be achieved; 2. Select indicators that can be used to measure progress in achieving the results; 3. Set explicit targets for each indicator used to judge performance; 4. Develop performance monitoring systems to collect data on actual results; and, 5. Review, analyze, and report actual results vis-à-vis the targets.

Methodology

he evaluation was conducted by an independent evaluation team consisting of an T international team leader and three Cambodian technical experts – one for each of the three thematic programme areas. In planning and implementing the evaluation, the evaluation team adhered to a guiding principle: stakeholders‟ meaningful participation and engagement. The evaluation team coordinated its work with UNFPA‟s country office and its evaluation management committee (EMC), which included as many as 24 stakeholders, to build consensus around the evaluation‟s purposes, scope, and processes.4 The EMC reviewed the TOR and the evaluation team‟s inception report (as did UNFPA‟s regional office in Bangkok). The evaluation team also made an oral presentation to this committee during the first week of the evaluation. The presentation addressed the inception report, the field sites to be visited and the rationale for each, and the team‟s desire to meet with and solicit the opinions and recommendations of as many of the key stakeholder s as possible. The team also made a presentation to the EMC after completion of the draft evaluation report. Members of the EMC were invited to review and provide comments on the draft report and many did so.

4 The EMC included representatives from the Ministries of Health, Interior, Planning, and Women‟s Affairs, the National Committee on Population and Development, four UN agencies (including UNFPA), and several nongovernmental organizations.

Data sources and data-collection methods

The methodological framework for the evaluation (Annex 2) details the data sources and data-collection methods for each of the evaluative criteria. First, the team reviewed a wide range of reports and documents (Annex 3). Among these reports were the 2005 and 2010 DHS. Both of these surveys were invaluable because several of the indicators used to assess progress were derived from the 2005 DHS. In addition, the evaluation team also had access to the results of the 2008 national census. Second, the evaluation team conducted interviews in Phnom Penh and in seven of the 14 priority provinces. The field visits occurred in areas of high concentration of UNFPA field support, partnership, and coordination. Meetings in Phnom Penh and in seven provinces (i.e., Banteay Mean Chey, Battambang, Kampong Cham, Kampong Chhnang, Kratie, Pailin, and Siem Reap) allowed the team to meet with more than 200 people, including many officials at all levels of government as well as representatives of NGOs and the donor community (An- nex 4). The team developed a semistructured interview guide (Annex 5) to ensure attention to the central question: what is the value added of UNFPA‟s presence in Cambodia? In particular, the interview guide addressed three subquestions: (a) Is the right thing being done? (b) Is it being done well? (c) Are there better ways of doing it? The first question addressed the rationale and relevance of what UNFPA‟s programme in Cambodia does. The second question examined the effectiveness of the results achieved and assessed efficiency, with a view toward optimizing the use of resources, promoting sustainability, and leading to long-term impacts. The third question identified alternatives and suggests recommendations for UNFPA‟s response and action.

Limitations of the evaluation

The evaluation has several limitations:

1. Constraints associated with interpretation during interviews. The team leader joined the Cambodian team members for many of the interviews conducted during the field visits. These interviews were conducted primarily in Khmer, which is not a limitation but it was for the team leader who does not speak Khmer. 2. The evaluation drew on the expert opinions of key informants involved in the programme‟s development, implementation, and monitoring. Their opinions may be, at least partly, subjective, to the extent that they believed that favorable comments might facilitate UNFPA‟s continued financial support. Moreover, these informants might not have distin- guished between the completed CP3 and the present, ongoing CP4. As a consequence, outputs achieved in 2011 as a result of activities undertaken as part of CP4 may have been attributed incorrectly to CP3. 3. Linking results to UNFPA‟s efforts was also problematic because others donors, including other UN agencies, address some of the same issues as does the Population Fund. As an illustration, UNFPA, the United Nations Children‟s Fund (UNICEF) and the UN Development Programme (UNDP) have supported Commune Committees for Women and Children. UNFPA and UNICEF supported efforts to integrate attention to HIV in life skills

13 education in schools. In addition, UNFPA‟s support to the MoH‟s Second Health Sector Sup- port Programme has been in the form of pooled funding since early 2009 as well as discrete funding through CP3. Accordingly, the challenge of attribution to the UNFPA is of concern when multiple donors contribute to the same objective.5 4. Limitations of time and distance precluded visits to all of the priority provinces, several of which are “remote” and not easily accessible from Phnom Penh. Likewise, UNFPA has a large and diverse group of stakeholders at both the national and provincial levels. Given the time constraints of the evaluation it was not feasible to interview a representative sample of them in all the priority provinces. In consultation between the evaluation team, the EMC, and UNFPA, stakeholders were selected purposively for interviews, based on their level of engagement with the programme. 5. Many changes in UNFPA‟s country staff occurred during the programme period, with the consequent loss of considerable institutional memory. Several of the key programme staff, including the country representative and deputy representative, are relatively new to the programme and did not have an in-depth knowledge of CP3‟s development or early implementation. This turnover within UNFPA/Cambodia represented a major challenge as did the turnover among the government officials interviewed. Many of the people interviewed during the field visits were new to their positions, and several had little or only recent familiarity with CP3 or UNFPA.

CP3’s results framework

As UNFPA‟s Evaluation Policy explains, “A prerequisite for evaluation is a coherent results framework.” Such frameworks include statements of expected results, the logical sequence of the expected results, and indicate how activities lead to the results. In turn, results frameworks also identify relevant performance indicators, baselines, and targets. How well did CP3 meet these expectations, and with what consequences for the evaluation? The evaluation team has concerns about CP3‟s results framework and the weak (and occasionally absent) relationship between activities and outputs and between outputs and outcomes. In too many instances the former were not relevant to the latter; the sequence of expected results was often flawed. As UNFPA‟s Results Based Management Policy makes clear, the agency is accountable for demonstrating that its outputs make a contribution to the achievement of expected outcomes.6 Similarly, the agency and its staff should be able to explain why an activity contributes to desired outputs and outcomes. The CPAP asserted that the collection and utilization of age- and sex-disaggregated data will contribute to the promotion and protection of human rights. That is a questionable assumption. In another instance one of the intended outcomes for the CP3‟s focus on gender

5 UNFPA/Cambodia has a contrary opinion, claiming that its participation in the pooled funding allows the agency to maintain a “high degree of attribution for specific activities.” See UNFPA/Cambodia, 2004 Country Office Annual Report. The country office also contributed to pooled funding for the First Health Sector Support Programme. 6 UNFPA‟s Results Based Management Policy defines outputs as “changes in skills or abilities, or the availability of new products or services, produced by an intervention or activity” and outcomes as “institutional and behavioral changes in development conditions that occur between the completion of outputs and the achievement of goals.”

14 was an increase in the number of women in Cambodia‟s senate and national assembly as well as the number of women who are ministers, governors, and commune councilors. As the staff of the country office and other UN agencies acknowledged, however, none of the CP3‟s activities or outputs would logically have led to an increase in the number of women in Cambo- dia‟s parliament, even if all the activities had been completed and all the outputs achieved. In still another instance, the country office eliminated and did not replace an outcome target on youth policy. Despite this action, six “orphan” outputs remained. There was no outcome to which the outputs were supposed to contribute. To achieve an increase in the percentage of currently married couples where both partners approved of family planning, one AWP included only the following activities, none of which are arguably or directly related to the desired increase:

 Organize training for NGOs on adolescent and sexual reproductive health;  Conduct an orientation workshop on finance;  Organize the bidding of a contract;  Print manuals for district teacher trainers; and,  Purchase computers.

In other instances AWPs listed activities but no indicators, baselines, or targets.7 In one AWP, the description for the activity was to “build capacity and attend regional meetings” with no corresponding output, target, or objective. Several of the programme‟s targets confounded outputs and outcomes. Examples of outcomes identified as outputs include the proportion of births in facilities for emergency obstetric and newborn care (EmONC) and the number of health centers that have trained midwives. The CPAP identified a series of outcome and output targets in each of the three thematic areas, but several of them were neither objectively verifiable nor have objective standards that would permit a reliable judgment about whether the targets had been achieved. As an illustration, several of the population-related indicators refer to the incorporation of population, RH, and gender issues into provincial and commune-level plans. The meaning of “incorporation” is unclear. For gender, several of the targets refer to government plans that are “gender responsive” but do not indicate what that concept means. UNFPA‟s midterm review of CP3 in late 2008 acknowledged the lack of standard criteria for assessing gender responsiveness as a concern and noted that such criteria should be developed. Despite this acknowl- edgement, it does not appear that such development occurred. Four hundred commune committees for women and children were supposed to be “functional” by the end of 2010, but the meaning of the term is unclear. UNFPA‟s midterm review similarly identified the absence of criteria for this concept as a concern. UNFPA noted the need for discussion of possible criteria, but the results of any discussion that may have occurred are unknown.

7 This finding also applies to CP4, where similar problems with several AWPs were observed. In addition, the CPAP for CP4 includes several of the indicators that the present evaluation has identified as problematic (e.g., committees that are supposed to be “functional” and communal plans that are supposed to be “gender sensitive” and “incorporate” RH, youth, and population issues).

An RH-related output was the provision of “high-quality RH services.” This target is common throughout UNFPA. The term is vague and subject to multiple interpretations. Moreover, the concept has none of the characteristics of good indicators, and DOS notes that the absence of such indicators does not meet UNFPA‟s expectations.8 The country office was asked but was unable to indicate the meaning of the terms the team considered to be unclear and how these indicators should be measured. As discussed below, this situation required the evaluation team to devise its own means of measurement for several indicators. Some indicators had baseline data but no way to verify achievement of the targets. This problem occurred because the country office had selected several indicators and their baselines from the 2005 DHS. In several cases the 2010 DHS did not include the same questions that had been asked in 2005. The absence of comparable data across the two time periods was especially unfortunate with respect to several indicators on trafficking and gender- based violence. In a few occasions the country office selected baseline data from the 2000 DHS, with the consequence that these baselines were not valid at the start of CP3 in 2006. The 2000 DHS reported that 55.2 percent of women in Cambodia were aware of trafficking of women. The CPAP identified a target of 70 percent to be achieved by the end of 2010. This was a modest target; the 2005 DHS reported that 66 percent of Cambodia‟s women were aware of trafficking in that year.9 Other indicators had targets but no baselines until several years after the programme started. The Country Programme Document for Cambodia pledged that a “strategic baseline study” would be undertaken in 2006. The study was not completed. The CPAP established national targets for several RH-related indicators rather than targets solely for the priority areas in which UNFPA focused its efforts. The consequence is that events and activities outside of the programme‟s control or influence affected outcomes and compounded efforts to assess the programme‟s effectiveness within the priority provinces. The DHS provides provincial-level data, so UNFPA could have chosen provincial-level targets that encompassed its priority areas. Over the five years of CP3 the country office also made multiple changes in its output indicators. Several of the indicators and their targets identified in the CPAP were dropped while others were amended, occasionally with less ambitious targets than initially established in the CPAP. New indicators were added as a result of UNFPA‟s midterm review of CP3 in 2008; other indicators were added in 2010, the programme‟s last year. Other indicators were dropped because of the lack of relevant data. One output indicator was both added and then subsequently dropped.10 The indicators for RH changed the most. Of the 13 original RH output indicators in the CPAP, all were either dropped or amended. None were left unchanged, but 13 were added. In total, the number of output indicators in CPAP increased to 60 in 2010 from 28 in 2006. Inflation in the number of indicators is indicative of fragmentation.

8 See DOS, Evaluation Methodology for DOS Oversight Assessments of Country Programmes. 9 The opposite situation also occurred. The 2000 DHS reported that 52.4 percent of women had heard of any laws to protect women‟s rights. The CPAP target was set at 70 percent. The 2005 DHS found that only 41.6 percent of women had heard of any laws to protect women‟s rights. 10 As one person from the country office explained, “adding and dropping [the indicator] occurred in the process of review that would be the nature of development.”

Furthermore, the indicators and targets included in some AWPs did not always coin- cide with the indicators and targets that the country office provided to the evaluation team or that had been agreed upon following a midterm review of the programme. Table 3 summariz- es the changes that occurred in the output indicators between 2006 and 2010 as well as the total number of output indicators in each of the CP3‟s programmatic areas at the end of 2010. Each of the concerns just noted affected the ability of the evaluation team to make credible judgments about CP3‟s successes and accomplishments as well as its relative lack of success in other instances. As one member of the EMC noted, given the multiple constraints and limitations that exist, it is difficult to demonstrate that CP3 has been or will be develop- mentally effective. The evaluation team both shares and appreciates this concern.

Table 3: CP3‟s output indicators, 2006–2010 RH RH Communi- PD Gender Total cation # of indicators in CPAP (2006) 13 0 4 12 29 Status of indicators, 2006–2010 Unchanged 0 0 1 9 10 Amended 7 0 3 3 13 Added 13 20 1 3 37 Dropped 6 0 0 0 6 # of indicators at end of CP3 (2010) 20 20 5 15 60 Source: CPAP, AWPs for 2010, and information provided by country office. Note: Indicators with multiple targets are considered to be separate indicators. As an illustration, one of the indicators for RH communication was the percentage of women of reproductive age who know that some traditional practices are harmful for mother and child. There were targets for six harmful practices, so these are considered to be six indicators rather than a single one. Similarly, one of the desired outputs for RH focused on the percentage of the population living under the poverty line protected by a health equity fund (a) at the national level and (b) within the UNFPA‟s priority areas. The data in the table treat these as separate indicators, with one at the national level and a second for the priority areas.

Mitigating the limitations

The evaluation team tried to mitigate the limitations and to minimize possible biases through triangulation of methods, data, and investigators. As the Evaluation Office of the Global Environment Facility has noted, “In the scarcity and/or absence of a reliable set of quantitative data, triangulation can be a useful substitute for obtaining reasonably solid and reliable evaluation results.”11 Triangulation was achieved through three major evaluation approaches: perceptions, validation and documentation. Perceptions were elicited though interviews with internal and external stakeholders and key informants. Validation was achieved through stakeholder meetings, such as debriefing meetings with UNFPA staff and with members of the EMC; through direct observation during field visits; and through specific studies such as case studies, bene- ficiary assessments, impact studies, etc. Documentation included programme-related docu-

11 Global Environment Facility, Evaluation Office. 2010. Methodological Note on Triangulation Analysis in Country Portfolio Evaluations.

17 mentation, relevant policies, strategies, and action plans, national statistics, midterm analyses, external reviews, and other external documents. Credible evaluations routinely rely on the use of a counterfactual, which is the situation that would have existed if the intervention being evaluated had not occurred. Although perfect counterfactuals do not exist, evaluators often try to create valid counterfactuals through the use of experimental evaluation designs that compare groups of people or geographic areas that receive an intervention (the treatment group) with areas that did not receive the intervention (the control group). In ideal circumstances, the geographic areas are randomly selected and then randomly assigned to one of the groups. If the two groups are sufficiently large, they are likely to be similar before the intervention on both observed and unobservable characteristics. Since both groups are similar before the intervention and the only difference over time between the two groups is the intervention, differences between the two groups at the end of the intervention can be attributed to the intervention. If there are no or only negligible differences between the two groups after the intervention, then one can reasonably conclude that the intervention had little or no impact. This discussion of counterfactuals and impact is germane here for two reasons. On the one hand, in the absence of a valid counterfactual, attribution of observed changes to UNFPA‟s efforts is problematic. Although a change in the expected direction may be observed, such as a reduction in gender-based violence, that change cannot be linked persua- sively to a specific intervention in the absence of a counterfactual. The absence of a counterfactual means that rival or plausible alternative explanations for the change have not been eliminated. For the present evaluation, no valid counterfactuals are available for gender or population and development. On the other hand, and in contrast, using data from the 2005 and 2010 DHS it is possible to construct reasonably valid counterfactuals for several indicators related to reproductive health. For the evaluation of CP3, one counterfactual approach compares key indicators in CP3‟s priority provinces with those provinces that were unsupported by the programme. Accordingly, and as discussed below, the evaluation team was able to compare (a) relevant DHS data from 2005 with that from 2010 and (b) data from CP3‟s priority provinces (the intervention areas) with nonpriority provinces. Making these comparisons faced at least three challenges. First, and unfortunately, the 2005 and 2010 DHS did not include identical ) questions for several of the CP3‟s indicators (e.g., the 2005 DHS collected information on the percentage of women who had heard of any laws protecting women‟s rights; the 2010 DHS did not). CP3 had used the 2005 data on this item as a baseline for one of its indicators; the expectation was that the 2010 DHS would provide the endline data and a means to assess UNFPA‟s progress in promoting gender equality. Second, for purposes of reporting the data at the provincial level, the 2010 DHS combined data from one priority province (Koh Kong) with data from one nonpriority province (Preah Sihanouk). Third, CP3‟s activities were limited to selected operational districts within

18 the 14 priority provinces.12 In other words, some districts within these provinces were not with the “treatment” group of districts. Regardless of these challenges, the opportunity to construct a reasonably strong counterfactual is likely unique and unparalleled among evaluations for the UNFPA. The counterfactual approach allows identification of what works and what does not work rather than having to rely on speculation and anecdote. Using a counterfactual approach eliminates the need for speculation about whether an intervention provides the explanation for the outcome observed.

Ethical Considerations

NFPA‟s Division for Oversight Services (DOS) has provided guidance related U to ethical considerations for evaluators. This guidance notes that:  Minimum expectations for ethical considerations should include documentation of consent procedures where beneficiaries or members of the public are surveyed; and,  Brief descriptions of confidentiality provisions should be provided where personal information is used in the evaluation or the evaluation report.

The evaluation team adhered to international best practices and conducted its work in full compliance with UNFPA‟s Evaluation Guidelines and the UN Evaluation Group‟s Code of Conduct for Evaluation in the UN System.13 The evaluation team attempted: (a) to ensure that respondents understood the evaluation‟s purpose, objectives, and the intended use of findings; (b) to be sensitive to cultural norms and gender roles during interactions with all respondents; and, (c) to respect their rights and welfare by ensuring informed consent and rights to confidentiality before interviews. To ensure respondents‟ informed consent and their awareness of the scope and limits of confidentiality, interviewees were told about their informed consent, anonymity, and confidentiality to ensure that sensitive information could not be traced to its source (without the respondent‟s approval). A written statement (Annex 6) that had been translated into Khmer was typically provided to respondents before any substantive discussion occurred.

Adherence to International Best Practices

he evaluation was conducted in full compliance with the UN Evaluation Group‟s T Standards for Evaluation in the UN System and its Norms for Evaluation in the UN System. The evaluation team similarly attempted to adhere to the requirements of UNFPA‟s Evaluation Quality Standards.

12 An operational district, which is distinct from the government‟s administrative districts, is the most peripher- al subunit within the health system closest to the population. Cambodia presently has 77 operational districts. 13 All team members agreed, in writing, to abide by the Code of Conduct.

DOS assesses evaluations completed for the agency using seven quality assessment criteria, addressing issues such as the structure and clarity of reporting, design and methodology, and findings and analysis. This report attempts to ensure that the evaluation meets the expectations associated with these criteria, which provide another benchmark against which the evaluation report can be judged. The DOS quality assessment criteria, which DOS requires to be appended to all evaluations, can be found in Annex 7.

Findings Relevance

CP3 was highly relevant to a wide range of government policies and strategies, and for this the country office should be commended. Whether one considers the National Strate- gic Development Plan or one of several strategic sectoral plans, such as the one for health, UNFPA‟s three thematic areas of focus align closely with them. One example of this close alignment can be found in The Rectangular Strategy for Growth, Employment, Equity and Efficiency in Cambodia, which was completed in 2004. The report that accompanied the strategy presented the country‟s development partners with a summary of the RGC‟s priorities and its needs for external development assistance. The overall strategy had four “growth rectangles,” one of which was Capacity Build- ing and Human Resource Development, including: (a) enhanced quality of education; (b) improvement of health services; (c) fostering gender equity; and (d) implementation of population policy. Within each of these areas the strategy identified a series of actions and objectives. Several examples illustrate the alignment between these objectives and those of UNFPA‟s country programme.

 To improve the delivery of health services, the government declared that priority would be given to the implementation of a minimum package of activities (MPA) and a complementary package of activities (CPA) for health.14 CP3 provided funding and included targets for both MPA and CPA.  The Health Strategic Plan for 2008–2015 identifies three key priorities, the first of which is to reduce maternal, new born, and child morbidity and mortality and increase reproductive health through increases in access and coverage of health services. The plan recognizes that the competency of primary midwives is inadequate. A major thrust in CP3 was the training of midwives, improvement in their training curriculum, and support to the regional training centers where prospective midwives are trained.  To foster gender equity, the RGC emphasized its commitment to the mainstreaming of gender in national and local programmes and policies and the development of a national gender mainstreaming strategy and a related plan of action. CP3‟s gender- related initiatives were intended to address each of these priorities.  The RGC‟s population policies called for (a) the support of “all couples and families to be free and accountable for the decision on their desired number of children and

14 The MPA provides a basic package of preventive and curative services at the primary care level. The CPA provides a complementary package of services for inpatient and outpatient care at the hospital level.

birth control, and ensure their access to information, education, service delivery, and other means to fulfill their above decisions”; (b) a reduction in high rates of fertility and an increase in the use of birth spacing services; (c) a reduction in maternal mortality; and, (d) a consideration of population factors in all economic and social policies, plans and programs at all levels. CP3 included activities germane to each of these areas.

The National Strategic Development Plan for 2006–2010 similarly identified all of these issues as priorities for Cambodia‟s development. Here again are two other relevant examples of priorities included in the NSDP:

Gender equity issues permeate all activities, and are being addressed in all sectors, particularly in agriculture, health, education. The National Council for Women would continue to address the myriad aspects of gender mainstreaming in all aspects of governance and society.

The National Population Policy designed to improve the quality of life of all Cambo- dians will be implemented to provide information and knowledge to all couples and enable them to make the choice about the size and spacing of their progeny and provide them with needed services and supplies to attain their goals.

As one person informed the evaluation team, CP3 was one of the few UNFPA country programmes in the Asia-Pacific region that was fully aligned with national programmes and priorities and in line with ongoing sectoral reform processes. As this person explained, the CP3 relied on and promoted national ownership and leadership and worked through existing structures and processes in accordance with the Paris Declaration on Aid Effectiveness. The evaluation team fully agrees with this assessment. Moreover, the close alignment between CP3 and the RGC‟s priorities means that CP3 (and CP4) have been highly relevant to the needs of Cambodians, especially those in rural and previously underserved areas. This high and commendable level of relevance also leads the evaluation team to conclude that the design of both country programmes have responded well to and reflected Cambodia‟s current and future challenges in the three thematic areas in which UNFPA works. As several key stakeholders remarked, UNFPA‟s approaches are comprehensive, complementary, and well coordinated with the government‟s priorities. UNFPA‟s mandate, and therefore the agency‟s efforts in Cambodia under CP3, were also closely aligned to the ICPD‟s Programme of Action, focused as they were on the reduction of maternal mortality, the achievement of universal access to reproductive and sexual health services, and the use of population data to inform Cambodia‟s development planning at the national, provincial, and communal levels. In turn, CP3 related closely to three MDGs: promoting gender equality and empowering women; improving maternal health; and combat- ing HIV/AIDS and other diseases. This relevance further highlights CP3‟s suitable incorpora-

21 tion of attention to gender and equity. Conversely, CP3 did not incorporate any notable or particular attention to human rights, which are broader than reproductive rights 15 On the whole, CP3 was also relevant and well aligned with the UNDAF for 2006– 2010. The results and resources framework in the CPAP first identified a national priority and then the associated UNDAF outcome, such as “improved health, nutritional, and educational status and gender equity of the rural poor and vulnerable groups.” The UNFPA‟s intended country programme outcomes and outputs were then listed. While many of CP3‟s outcomes and outputs were arguably related to the UNDAF‟s objectives, others were not. To illustrate, one desired UNDAF outcome was “significant progress toward effective participation of citi- zens, accountability, and integrity of government.” The corresponding outcomes in CP3 included: (a) incorporation of key RH, gender, and population issues in the NSDP; and (b) in- clusion of indicators and targets for access to reproductive health in Cambodia‟s MDGs. Both of these outcomes may be desirable, but their relevance to effective participation and accountability is questionable, at least in the opinion of the evaluation team and among several people who provided comments on the draft evaluation report. As one of these people explained, UNFPA was required to adhere to the UNDAF‟s objectives even in the absence of a “neat fit” with UNFPA‟s desired results in Cambodia. Despite the reasonably good alignment with the UNDAF, some anomalies occurred. UNDAFs identify areas of cooperation where the United Nations can collectively make a difference and add value to a country‟s development. Presumably, therefore, an UNDAF identifies the most important elements of the UN‟s efforts in a country. Cambodia conducted a national census in 2008 with UNFPA‟s support, but the census was not mentioned in the UNDAF.16 In contrast, the UNDAF includes several indicators and corresponding targets relevant to UNFPA‟s mandate that were not included in the CPAP:

 The percentage of pregnant women with two or more antenatal care consultations from skilled health professionals; and,  The percentage of pregnant women who delivered by caesarean section.17

The indicators are among Cambodia‟s MDGs and were included in the NSDP for 2005–2010 (and its update for 2009–2013). Although CP3‟s overall objectives were relevant to a wide range of Cambodia‟s needs, this relevance did not ensure that the activities represented the best use of the

15 According to the UN Evaluation Group‟s guidance on Integrating Human Rights and Gender Equality in Evaluation (2011), human rights represent “the civil, cultural, economic, political and social rights inherent to all human beings, regardless of one‟s nationality, place of residence, sex, sexual orientation, national or ethnic origin, colour, disability, religion, language etc.” 16 As one member of the EMC noted, “It is important to highlight that not all the outcomes/outputs of UN agencies are always included in the UNDAF. The UNDAF provides a collective, coherent and integrated United Na- tions system response to national priorities and needs, but does not comprise all actions planned by UN agencies.” The evaluation team agrees with this statement but was nonetheless surprised that a major UNFPA activity, support for the census, was not mentioned in the UNDAF. 17 As the NSDP Update 2009-2013 indicates, the rate of caesarean sections is a proxy indicator for access to emergency obstetric and newborn care. There are concerns about the uneven distribution of caesarean sections and EmONC services in Cambodia, so the NSDP Update noted the importance of monitoring rates of caesarean sections at the subnational level.

UNFPA‟s resources. Consider, for example, that there are many ways to improve access to reproductive health services, to reduce maternal mortality and eliminate unsafe abortions, or to develop individual and institutional capacity. Which of these are the “right” and most effective ways to achieve these objectives? Similarly, how should UNFPA allocate its resources among the three thematic areas it addresses? The Division for Oversight Services identifies several ways to address and answer these questions.18 DOS considers a situational analysis to be an essential antecedent of any decision about which activities to undertake. This situational analysis should, according to DOS, include a consideration of alternatives and, presumably, selection of responses deemed suitable for the situation. In the absence of a consideration of alternatives, DOS believes that country programmes do not meet UNFPA‟s expectations. Furthermore, DOS recommends that evaluations of country programmes assess whether financial allocations among the agency‟s three thematic areas corre- spond to the priorities identified in the situational analysis.19 A review of the CPAP for CP3 did not identify (a) any consideration of alternatives for its choice of activities for RH, PD, or gender or (b) any explicit identification of priorities among the three thematic areas that served as the basis for decisions about how resources could be allocated among the three areas. As a consequence, although the CPAP was developed through a lengthy consultative process with key stakeholders, it is not clear that the activities that UNFPA chose for CP3 were the best ones or were based on evidence of their effectiveness in Cambodia. According to a report commissioned by the Australian Agency for International Development (AusAID), the Second Health Sector Support Programme, to which UNFPA contributes pooled funding,

has supported continuation of a multiplicity of initiatives for improving health service delivery, most initiated and supported by donors….Many of these initiatives began as pi- lots, some more than 10 years ago….There has been no sector-wide evaluation and assessment as a basis for making evidence-based decisions on which of these initiatives to phase out or integrate into other components of the health system.20

One possible example relevant to UNFPA‟s efforts involves the agency‟s attention to capacity building. The agency and many other donors provide large amounts of money and facilitate scores of training activities each year in an effort to develop capacity. As members of the EMC and country office observed, there is a widespread belief that capacity building alone will not make much difference in Cambodia in the face of the low motivation of many government employees who are poorly paid and often underappreciated. Consider as well the situation with the status of women in Cambodia and the widely acknowledged problems in the country with gender inequality, low levels of gender empowerment among women, widespread violence against women, and sexual abuse and trafficking of women and children. According to the CPAP, gender inequalities remain high, “and social atti-

18 DOS, Evaluation Methodology for DOS Oversight Assessments of Country Programmes (2010). 19 As the country office noted, however, a DOS oversight mission to Cambodia during CP3 did not identify this issue as a concern. 20 Synthesis Assessment of Medium-term Issues and Options for Supply and Demand-side Initiatives for Improv- ing Quality and Access to Health Services Supported under HSP2 (2011).

23 tudes and tradition deem women to be of lower status than men.” Policies and strategies for addressing perpetrators of violence against women, the CPAP observes, are virtually nonexistent. The UNDAF for 2011–2015 observes that Cambodia “has the lowest levels of gender equity in Asia.” The inequalities are especially severe for women in rural areas. According to the U.S. Department of State‟s 2011 Trafficking in Persons Report, Cambodia is a source, transit, and destination country for men, women, and children who are subjected to forced labor and sex trafficking. In addition, a significant number of Asian and other foreign men travel to Cambodia to engage in child sex tourism, and some pay thousands of dollars to have sex with virgins. As recently as 2009, the Department of State noted that while Cambodia was making significant efforts to reduce trafficking, the absolute number of victims of severe forms of trafficking was very significant or significantly increasing. Traf- ficking for sexual purposes is widespread. Data from the DHS reveal that over two-thirds of women in Cambodia were aware of the trafficking of women in 2005.21 Compared to gender, the resources devoted to reproductive health are significantly larger (i.e., over 60 percent of total expenditures in CP3 and projected to be 62 percent in CP4). There is clearly a need for attention to RH in Cambodia, and UNFPA plays a shared role in addressing RH issues, but the CPAP for CP3 did not make a compelling case that RH deserved or required significantly more financial support than the combined total devoted to population and gender- based violence. Moreover, there is no shortage of donors in Cambodia‟s health sector, which receives more development assistance than any other sector. A recent government review of aid effectiveness concluded that Cambodia‟s health sector is “well supported,” resource rich, and that it receives more development assistance for the sector than implementation of the NSDP requires.22 The fact that the MoH spent less money than UNFPA had provided in 2008, 2009, and 2010 is consistent with these findings. Indeed, it may be the case that UNFPA and other donors provide more resources to the MoH for RH-related activities than it can absorb or have the capacity to use effectively. The concerns about absorptive capacity are also likely to apply to other government IPs as well. The review of aid effectiveness further observed that pressures from development partners to implement programmes often dominate the “recognized need to place more attention and effort on capacity and systems development in a manner that is consistent” with the RGC‟s longer-term reform objectives. Despite these latter concerns, the evaluation team concludes that the country office should be applauded for the relevance of CP3 to Cambodia‟s needs. This relevance demonstrates a successful and effective effort to align UNFPA‟s contribution to Cambodia‟s needs.

Effectiveness

21 The country office pointed out that trafficking is not currently one of its areas of focus. Nonetheless, the CPAP included an indicator related to trafficking and the previous UNFPA representative chaired the thematic working group on human trafficking. 22 RGC, Cambodian Rehabilitation and Development Board of the Council for the Development of Cambodia, The Cambodia Aid Effectiveness Report 2010; Ministry of Planning, Achieving Cambodia’s Millennium Devel- opment Goals, Update 2010. Underresourced priority sectors include education, transportation, and rural development. If the MoH is “overresourced,” it may be due to the large amount of money it receives from the Global Fund to Fight AIDS, Tuberculosis, and Malaria.

An assessment of effectiveness examines the extent to which targets projected in the three thematic areas were achieved during CP3. There is much that UNFPA did accomplish and to which its efforts contributed. Cambodia can arguably boast of impressive accomplishments over the past decade, especially in the area of reproductive health. The total fertility rate declined to 3.0 births per women in 2010 from 4.0 births per women in 2000. In 2000, only 38 percent of pregnant women had received antenatal care from a trained health professional. By 2010, this percentage had increased to 89. In 2000, the DHS found that 89 percent of deliveries occurred at home. By 2010, 54 percent of births occurred in a health facility (compared to only 9 percent just ten years earlier).

CP3‟s Achievements

Much of CP3 was intended to contribute to the improvements in maternal health. One of CP3‟s objectives was to have trained midwives at 90 percent of 966 health centers. At the start of CP3, approximately 76 percent of these health centers had a trained midwife on their staffs. Although many midwives had been trained, their quality and competence was a major concern. In late 2006, the country office funded a comprehensive midwifery review.23 The authors of the review asked 185 midwives to self-assess their skills in “traditional competencies,” which have been included in midwifery education programmes for decades. These competencies include managing a normal birth and assessing a newborn‟s health at birth. The results of the self-assessment were alarming. Just over 40 percent of the midwives “did not feel confident in conducting a normal birth, and 40 percent reported that they did not feel confident to assess the newborn at birth” using the APGAR scale. With respect to “new competencies,” less than “50 percent of midwives reported feeling confident in their ability to complete a partograph, manually remove a placenta, diagnose and manage infection in the newborn, diagnose and manage postpartum sepsis, manage eclampsia, or resuscitate a newborn.” Through collaboration with the MoH and UNFPA‟s support for the training, recruitment, and deployment of midwives in rural and remote areas, all 977 health centers had trained midwives by the end of 2010. UNFPA‟s target was exceeded. Indeed, by 2009, 55 percent of health centers already had at least two midwives. Senior health officials acknowledged UNFPA‟s contribution to this achievement. In 2008, 335 students had enrolled in training programmes for midwives. This number increased to 830 in 2009 and to 835 in 2010. As one of UNFPA‟s IPs noted, this increase was attributable to government midwifery incentive payments, stipends for some students and teachers, and a newly introduced three-year direct entry midwifery training programme, the Associate Degree Midwifery Curriculum (ADMC). The ADMC was introduced in February 2009 in four regional training centers with an initial annual intake of 60 to 80 students per center. In addition, the Technical School of Medical Care in Phnom Penh accepted 200 midwifery students in 2009 and 139 in 2010 into its direct entry programme. ADMC students ac-

23 Della R. Sherratt, Patricia White, and Chan Chhuong, Report of Comprehensive Midwifery Review in Cambodia (2006).

25 counted for an extra intake of 961 midwifery students in addition to the continuing intakes of one-year primary midwifery students and postbasic midwifery students in 2009 and 2010. Midwives who wish to work in public health centers must pass a civil service examination and then be hired. Not all who apply are hired because the MoH‟s budget cannot accommodate all who apply. Nonetheless, in 2008, 181 midwives (25 secondary and 156 primary midwives) were selected and deployed at public health facilities. These numbers increased to 299 midwives (65 secondary and 234 primary) and 339 midwives (95 secondary and 244 primary) in 2009 and 2010, respectively. In addition to providing support to increase in the number of midwives, the country office also focused its attention on their quality as well as to upgrading the skills of incum- bent midwives. With UNFPA‟s support, an international midwifery specialist and national project professional were hired in 2008 and were in place until March 2010. They provided technical guidance and support to units within the MoH responsible for the training and deployment of midwives. In addition to developing guidelines and the content of all three years of the ADMC, a training curriculum for midwifery preceptors was also developed and approved. Once the curriculum was approved, a training workshop for preceptors was presented, clinical checklists were developed, and guidelines for preceptors‟ use of clinical logbooks were produced. Furthermore, more than a dozen midwifery teachers were provided with in- service training on life-saving skills that focused on antenatal, postnatal, and neonatal care. Based on site visits to several health centers and other medical facilities, the evaluation team concludes that the quality of medical care has also improved. Seventy-one percent of births in 2010 were attended by a skilled health worker compared to 32 percent in 2000. The evaluation team RH specialist, a physician, observed improvements in quality in regard to infection control, hygiene in delivery rooms, and preparation with sterilized delivery kits. Such improvements, though not directly related to UNFPA‟s targets, are arguably due to improvements in the skills and experience of the medical professionals with whom UNFPA works. One of CP3‟s output targets was that 75 percent of Cambodia‟s population would be covered by health equity funds or other financing mechanisms that would cover or subsidize the cost of medical care, especially for the poorest segments of the country‟s population. This target was achieved. UNFPA supported five of the funds. At least eight other development partners and the MoH were responsible for funding the other 52 equity funds. A related target was to have health equity funds “or other financing mechanisms” in place in 21 of the 24 operational districts in UNFPA-supported areas by 2010. This target was exceeded. Based on the data from the MoH, UNFPA supported health equity funds or other financing mechanisms in 22 of the 24 operational districts at the end of 2010. The availability of the health equity funds and a related effort to provide vouchers that subsidize the cost of health services, including reproductive services, led to what one study of three operational districts in Kampong Cham declared to be a “sharp” increase in the number of poor women who delivered their children in health facilities.24 This increase was much

24 Por Ir, Dirk Horemans, Narin Souk, and Wim Van Damme, “Using targeted vouchers and health equity funds to improve access to skilled birth attendants for poor women: a case study in three rural health districts in Cam- bodia,” BMC Pregnancy and Childbirth 2010, 10:1.

26 greater in the three operational districts than in districts without the vouchers and health equity funds. As the study concluded, “the available evidence suggests that the combination of vouchers and HEFs, if carefully designed and implemented, has a strong potential for reducing financial barriers and hence improving access to skilled birth attendants for poor women.” With support from UNFPA and the German government, the Cambodia Health Edu- cation Media Service (CHEMS) initiated a series of radio and television broadcasts to improve the knowledge, attitudes, and behavior of women, their partners, and newly married couples on key reproductive and maternal health issues in Phnom Penh and four provinces, including three priority provinces. UNFPA and CHEMS established measurable targets and conducted baseline and endline surveys to assess the changes that occurred as a result of the media messages. One purpose of the messages was to increase awareness that some traditional practices during delivery are harmful to mothers and their newborn children. Among women of reproductive age, the baseline survey revealed that 49 percent of the respondents agreed that it would be harmful for a woman not to bathe for many days after delivering her child. Only 8 percent agreed that the traditional practice of “roasting” mothers and their infants is harmful.25 The endline survey found 58 and 41 percent of the respondents, respectively, believed that these two practices can be harmful. These percentages compare with the targets of 70 percent (cannot have a bath for many days) and 20 percent (for roasting after delivery). With support from AusAID, UNFPA initiated the first-ever assessment of emergency obstetric and neonatal care in Cambodia. Doing so contributed to one of the CP3‟s key outcomes – improved access to high-quality RH services. The assessment determined the availability, functionality, quality, and levels of utilization of EmONC services to establish a baseline to monitor progress and to identify barriers to availability, functionality, and levels of utilization. Seventy-seven public hospitals, 230 health centers, and 40 private facilities were assessed. The subsequent report identified critical barriers surrounding guidelines and protocols, policy issues, infrastructure, supplies and equipment, staff coverage, provider skills, blood availability, 24/7 services, referral systems, and user fees. Upon completion of the report the MoH developed and approved in 2010 an implementation plan to strengthen its EmONC facilities. The plan‟s implementation has enabled and increased accessibility to injectable anti- biotics, oxytocic injections for preventing hemorrhages during labor, Caesarean sections, which can significantly reduce maternal and neonatal deaths, and other life-saving interventions for mothers and newborns such as the manual removal of placentas. Implementation also increased the availability of magnesium sulfate, which can be used to prevent and manage eclampsia in pregnant women. The EmONC plan has been well supported by UNFPA as a key programme priority for the agency‟s support to the MoH‟s National Maternal and Child Health Centre.

25 According to the University of Washington‟s EthnoMed website, some Cambodians believe that a woman‟s body becomes cold after giving birth. They take steps to heat the body and prevent further cooling. To remedy this situation the mother lies with her baby on a bed above a fire; this is “roasting.” The roasting starts immediately after delivery and normally takes at least a week to complete. In addition to heating the body, roasting is believed to prevent illnesses after the postpartum period. The percentages reported are for all women surveyed, including those who did not hear or see any of the media messages.

The country office has supported the efforts of the Cambodian Association of Parlia- mentarians on Population and Development to bring issues on UNFPA‟s agenda to the attention of Cambodia‟s Senate and National Assembly. These efforts encouraged Cambodia‟s first lady, Bun Rany, to serve as a national champion for the UN Secretary General‟s Joint Plan of Action Plan for Women‟s and Children‟s Health beginning.26 As the national champion, the First Lady will visit health centers across the country to advocate for healthy motherhood with midwives and other health professionals. Although the Bun Rany‟s acceptance of the role occurred after the end of CP3, in February 2011, her willingness to support the Joint Plan so openly provides validation of the programme. To improve national capacity to integrate and implement population and development issues into national and sectoral plans, UNFPA also collaborated successfully with the Minis- try of Planning to develop its first strategic plan and with the National Institute of Statistics to develop is first Statistical Master Plan. The plan acknowledges that Cambodia‟s socio- demographic statistics are almost entirely dependent on donors‟ support and then identifies UNFPA as one of several donors that have been “instrumental in developing and maintaining the socio-demographic programme of statistics for Cambodia.” With other financial and technical support, UNFPA assisted the National Institute of Statistics to improve its classification of rural and urban areas and to conduct the DHS in 2010. Twenty-five ministries established Gender Mainstreaming Action Groups (GMAG) during CP3, with support from several development partners, including UNFPA. Among the 25 groups, 17 had Gender Mainstreaming Action Plans in 2010, and several of these are being implemented, notably in the Ministries of Health and Planning. These two are the ones that UNFPA supports (and the MoH‟s GMAG has operated for six years). Supporting the plans‟ development and implementation contributed to the achievement of one of UNFPA‟s key gender-related outcomes: institutional mechanisms and sociocultural practices promote and protect the rights of women and girls to advance gender equity. Funding for the plans‟ implementation has also increased, and several line ministries have updated their action plans. According to data received from the MoWA, a majority of the ministries with action plans have received funding to implement the plans. Progress in establishing the action groups and in implementing their plans is impressive, but further work is required before victory can be declared. On the positive side, after reviewing a dozen health sector policies, a recent gender analysis concluded that two serve as role models for gender-responsive policies: the National Reproductive and Maternal and Child Health policy and the National Strategic Plan for a Comprehensive and Multisectoral Response to HIV/AIDS.27 Despite having these models of success, the capacity of some members of the action groups is limited, and turnover among their membership seems to be high. As the gender analysis concluded, “Although the [MoH‟s] GMAG has been active in raising awareness of gender, mostly through trainings and workshops, it has not been able to clarify how gender equity issues influence policy development and the health problems and outcomes of the

26 Bun Rany is also the president of the Cambodian Red Cross and National Champion of the Asian Pacific Leadership Forum on HIV and Development. 27 Chean R. Men, et al., Gender as a social determinant of health: Gender analysis of the health sector in Cambodia. A presentation at the World Conference on Social Determinants of Health, Rio de Janeiro, October 19-21, 2011.

28 population.” Interviews during the evaluation process also found that senior managers in some ministries do not consider gender equality to be an important issue and, supposedly, they do not care whether their ministries‟ action plans are implemented. A related report on gender in the health sector confirms the team‟s findings and is perhaps illustrative of the relative importance of gender-based issues within the RGC. The interviews conducted for the study identified “a significant gap in gender responsiveness between the intention of policies and how they are actually put into practice.” With few excep- tions informants, according to the report‟s authors, were generally unaware of the gender content of policies in relation to their specific work areas and had limited gender awareness and gender analysis skills.28 These findings underscore the need for increased attention to gender mainstreaming into public policies as well as their meaningful implementation. UNFPA has achieved some success by working with the National AIDS Authority to mainstream gender issues into the National Strategic Plan for Comprehensive and Multi- sectoral Response to HIV/AIDS III (2011–2015). Similarly, with UNFPA‟s financial support, MoWA developed a strategic plan for women, the girl child, and HIV/AIDS, and the plan is now being implemented. UNFPA‟s attention to HIV/AIDS reflected it efforts to increase access to high quality and gender-sensitive information and services related to the disease. In another instance, UNFPA‟s financial and technical support contributed to the Min- istry‟s completion of a National Action Plan to Prevent Violence on Women in 2009. Not only does the report acknowledge UNFPA‟s support for the plan‟s development but it also identifies the agency as a source of support for several of the activities that the plan‟s implementation will require. The ceremony that launched the report attracted nearly 250 participants from NGOs, line ministries, and development partners. All of these are impressive accomplishments, especially when one places the UNFPA‟s efforts in the context of Cambodia‟s recent history, as the next section attempts to do.

Partial successes

Although UNFPA/Cambodia accomplished much during CP3, less was achieved than had been anticipated or projected. Several reasons exist for this situation. As readers of this report are sure to be aware, Cambodia has had a recent traumatic and unparalleled history of destruction of human capital. This history requires both recognition and appreciation: UNFPA/Cambodia operates in a difficult environment in which few adults remain unscarred. As one such person eloquently explained, this history:

is particularly inconceivable or unfathomable for those who did not live those times. The country is still struggling to recover from the tumultuous periods of war and especially the Khmer Rouge regime in the late 1970s and the years of strife that preced- ed and followed it. As with other social sectors, health, in particular were largely de- stroyed, and many educated Cambodians were killed or fled the country. So mindful-

28 Kate Grace Frieson, et al., A Gender Analysis of the Cambodian Health Sector (2011).

ness of these facts would benefit any attempts at measuring progress of development interventions.

One way to assess CP3‟s effectiveness is to consider the country‟s offices own assessment of its accomplishments and effectiveness. According to the country office annual report (COAR) for 2010, between 75 and 99 percent of all output targets included in that year‟s AWPs were achieved. When asked how these percentages had been determined, the country office was unable to explain the computations used.29 To ascertain the accuracy and reliability of the COAR‟s data, the evaluation team first reviewed the IPs‟ annual work plan monitoring tools (AWPMTs). These documents are supposed to discuss progress toward achieving outputs and outcomes and explain why targets were or were not achieved. The documents were often of limited value; most summarized the completion of activities rather than the achievement of results. In several instances the AWPMTs did not identify output indicators or targets; in other instances the AWPMTs included targets but little or no discussion of whether the targets had been achieved or the reasons they had not been achieved. The CPAP Planning and Tracking Tool proved to be of value, but it was not complete. IPs are supposed to submit reports that assess progress made in achieving results. The evaluation team reviewed these reports when they were available; few IPs had submitted them. In the absence of IPs‟ annual progress reports and with data of uncertain quality from the AWPMTs, the evaluation team asked the country office‟s programme managers to indicate (a) whether each of the targets had been fully achieved for the 60 output indicators noted in Table 3 and (b) to provide the evidence to substantiate the judgment that the targets had been achieved. Based on the responses received from the programme managers, 70 percent of the output targets had been achieved by the end of 2010, with a range of 53 percent for the gender-related targets to 100 percent of the PD-related output targets. The overall percentage is less than what was reported in the 2010 COAR. When asked about the difference, the evaluation team was told that the percentages reported in the COAR may have reflected activities completed rather than targets achieved. If the percentage range reported in the COAR (i.e., 75 to 99 percent) reflects activities completed, it is important to consider the percentage of output targets actually achieved – which is what was supposed to have been reported in the COAR. To do so, the evaluation team made its own judgments about percentage of output targets that had been achieved by the end of 2010. These judgments, shown in Table 4, are based on the team‟s triangulation of information from AWPMTs, the CPAP Planning and Tracking Tool, annual progress reports, and interviews with UNFPA staff, stakeholders, and government officials in Phnom Penh and in the provinces visited. To overcome the problem of vague and imprecise targets and to base judgments on objective standards, the evaluation team relied on widely used criteria whenever possible. As an illustration, to assess the gender responsiveness of various government policies and documents, the team used the World Health Organization‟s Gender Assessment

29 The country office hired a consultant to complete the COAR for 2010.

Tool, which allows determination of the gender responsiveness of a policy or programme.30 The team used the UNFPA‟s definition of reproductive rights to determine whether Cambo- dia‟s sector strategies for health and education incorporated or discussed these rights.

Table 4: CP3 output targets achieved by programmatic area, 2010 RH RH communi- PD Gender Total cation # of indicators 20 20 5 15 60 % of targets achieved 25% 55% 20% 60% 43.3% % of targets not achieved or partially 75% 35% 40% 20% 45.0% achieved % indeterminate* 0% 10% 40% 20% 11.7% * Indeterminate primarily because of a lack of relevant data. Note: The table shows the percentages of targets that either were or were not achieved, thus ensuring comparability with what the country office reported in the 2010 COAR. The COAR does not consider partial achievement. Under ideal circumstances it would be desirable to report the percentage of output targets that had been partially achieved. Doing so is not possible in the face of at least one outdated baseline, the absence of some baseline and endline data in other instances, and problems with computing levels of partial achievement, especially for qualitative indicators.

The judgments of the country office and the evaluation team were in close agreement in the area of RH communication because of the availability of quantitative data on each of the indicators. Less agreement existed in the other areas. The evaluation team’s composite judgment is that fewer than half of the programme’s output targets were achieved by the end of 2010. In several instances, the country office gave itself the benefit of the doubt even when the evidence did not justify that benefit. As an illustration, there were two targets for EmONC facilities, one comprehensive EmONC facility and two basic EmONC facilities per 500,000 populations. The evaluation team was informed that both targets had been achieved. Despite this claim, AWPs for 2011 for the MoH indicated a lower baseline for these facilities than what had supposedly existed at the end of 2010. Moreover, the status of some EmONC facilities is tenuous. Some facilities lack sufficient numbers of qualified staff, skilled midwives, surgeons, and anesthesiologists and suffer from limited infrastructure, a weak referral system, and insufficient medical equipment. In other instances, output targets were achieved, but not between 2006 and 2010. The National Reproductive Health Strategy was supposed to be evaluated by the end of 2010; such an evaluation was planned for 2011. A target in the PD area was that staff from 14 provincial planning departments would be trained and capable of incorporating RH, PD, gender issues into their provincial development plans. The CPAP Planning and Tracking Tool indicated that a census database had been distributed and “in-depth analysis reports were disseminated to all stakeholders.” Based on these activities, the country office concluded that the target had been achieved. The evaluation team did not consider these activities to be evidence of the target‟s achievement. Other UNFPA efforts also did not lead to the results identified in the CPAP. For instance, training workshops on elementary statistics were provided to departments of planning

30 The tool can be accessed at http://www.who.int/gender/mainstreaming/GMH_Participant_GenderAssessmentTool.pdf

31 in 24 provinces in late 2008. Some participants described the workshops as too theoretical and not sufficiently practical.31 One practical application of the training could have been the analysis and use of provincial-level data from 2008 census in the development of provincial investments plans. To promote this objective, each of these departments was given a CD- ROM with census data. A reference tool on population issues was also updated based on results from the 2008 census of population. The tool is designed to assist planners at the commune level to integrate population issues, such as age structure, mortality, migration, youth, and education) into local plans. Among the provincial departments of planning that were visited during the evaluation, however, none had done anything with the census data or mentioned the reference tool. Sev- eral of these departments did use the Commune Database, but UNDP, not UNFPA, had assisted in the development of the database as well as training for its use.32 The evaluation team was informed of concerns about the quality of training in the health sector and the limited follow up, in particular at the provincial and district levels. Is- sues brought to the team‟s attention included poorly prepared instructors, inadequate training materials, and a lack of follow up of preservice and in-service training. Limited follow up and the absence of attention to the outcomes of activities were also identified as problems with respect to UNFPA‟s gender-based activities. A lack of follow up was a common concern among respondents for the evaluation. Coupled with a general lack of evaluation of the quality and effects of the training, much remains that UNFPA can do to enhance its efforts to improve the skills of the Cambodians with whom the agency works. At the commune and provincial levels, DOLA has engaged technical experts from provincial Departments of Health and Women‟s Affairs to provide training to communal focal points and communities. According to responses received during the evaluation team‟s field visits, the institutional linkages between and among the respective ministries and provincial departments were neither systematic nor in any guided form.33 After reviewing the draft evaluation report, one person from the country office confirmed that the lack of linkages is problematic and reflects the ministries‟ vertical working styles. These styles create constraints on UNFPA‟s and other development partners‟ effectiveness. As part of an effort to support the RGC‟s decentralization strategy, UNFPA has also worked with the Ministry of Interior and its DOLA to strengthen 446 communes‟ attention to RH, gender, and population issues. These efforts have met with mixed success, perhaps because the commune structure is relatively new and because success was never envisaged to be achieved in the short term. On the one hand, respondents at national and commune levels

31 The reader should be reminded that the sample of respondents is not representative of all those who partici- pated in the training. 32 The UNDP training provides an example of overlap in the objectives of two UN agencies. Representatives of the same provincial departments also noted that they had not gained any experience or had been provided with any training or capacity building that would allow them to initiate their own collection of data within their provinces. 33 The evaluation team also found weak to nonexistent linkages between UNFPA‟s efforts to promote youth- friendly adolescent sexual and RH services with the agency‟s efforts to promote learning on RH and HIV/AIDS in schools and at community learning centers. In several instances the evaluation team found that commune councils conducted activities on RH and gender without the involvement of or collaboration with the health or education system.

32 were consistent in citing UNFPA‟s budget as supporting their core functions in comparison to other funding sources and partners. At the national level, IPs commended UNFPA on its flexibility in providing support for their core activities, such as production of reports and consultation and workshops with the implementing partners. At the commune level, UNFPA‟s support was reported to be responsive to local needs because the commune council, chief, or the recently created Commune Committees for Women and Children managed the budget and activities directly. On the other hand, information sharing among commune councilors appeared to be minimal. Depending on the commune, UNFPA-funded budget and activities were either managed by the commune‟s chief, clerk, or chairwoman of the committee on women and children. Similar activities on gender mainstreaming, domestic violence, and migration had been supported and managed by NGOs even though commune councilors and chiefs had par- ticipated in decision making in the past. In some communes, a different person, usually another member of the commune council was responsible for such NGO-funded activities. A primary purpose of UNFPA‟s funding of commune councils was to promote their attention to RH, PD, and gender mainstreaming. When asked why the figures in annual commune investment plans were the same from one year to the next, some clerks responded that they had merely copied the numbers from other plans. This seemed to occur, in the opinion of the evaluation team, without the communes having considered the accuracy or appro- priateness of the information they had been given. There was no other reference to activities related to population. When asked, respondents in several communes said that facilitators at district and provincial level had assisted in the preparation of the commune investment plans. This finding is consistent with an evaluation of commune investment plans that the country office initiated. That evaluation found that many of these plans mentioned RH and population because the relevant language had been given to the commune from district or provincial officials. The communal officials had been the willing recipients of the information rather than the motivated initiators of the attention to RH and population. For this reason, the evaluation team questions whether capacity was improved by copying information from one year to another without due consideration of its meaning or implications. Several respondents and evidence from the field visits suggest that UNFPA‟s three thematic areas are not well integrated but have been implemented as separate components, much as if they are individual silos. In addition, within the PD areas, there appears to have been separate “pillars” and not necessarily well-coordinated pillars for each of the agency‟s IPs in that thematic area. Indeed, the team found little evidence of efforts to ensure the complementarity of activities within PD or across thematic areas, even within the country office. The evaluation team and several members of the country office consider this to be a missed opportunity that should be addressed. In other instances, UNFPA‟s claims of successful achievement of outputs and outcomes are open to question. UNFPA‟s targets for RH included the incorporation of “reproductive health, reproductive rights and the needs and rights of youth” in (a) reports on Cam- bodia‟s Millennium Development Goals; (b) the National Strategic Development Plan for 2009–2013; (b) the Education Strategic Plan, 2009–2013; and, (d) the Health Strategic Plan, 2008–2015. The latter document is the highest level strategic document for the health sector. In each instance the country office reported that all the documents addressed the three topics.

A review of each of these documents produced a different conclusion. Although all four documents had references to reproductive health, only the education plan and the Minis- try of Planning‟s Achieving Cambodia’s Millennium Development Goals: Update 2010 report mentioned reproductive rights (e.g., “a rights-based policy that gives couples the right to decide if and when to have children” in the MDG report). Neither the MDG report nor the Health Strategic Plan addressed youth rights as they relate to sexual and reproductive health; in contrast, the NSDP 2009–2013 and the Education Strategic Plan did so (e.g., “enhancing awareness health education focus on sexual, reproductive health and right, including HIV/AIDS, STI...”). The evaluation team considers the absence of attention to reproductive and youth rights to be a major omission, especially given UNFPA‟s mandate and the purpose of its presence in Cambodia. The absence of mention of reproductive rights in the Health Strategic Plan suggests that UNFPA may be insufficiently modest in having claimed repeatedly that its “ongoing involvement...at the highest levels and in [the RGC‟s thematic working groups] and other fora have proved critical to leveraging, advocating and localizing the ICPD Programme of Action.”34 One other issue deserves some discussion. A number of respondents noted CP3‟s attention to quantity (such as increasing the number of midwives, the number of students exposed to life skills in their classrooms, the number of EmONC facilities, and the number of training sessions for government employees) without concurrent attention to quality. The evaluation team shares this concern and encourages the country office to increase its attention to the quality of its outputs. The approach of the Deutsche Gesellschaft für Internationale Zusammenarbeit in Cambodia‟s health sector is illustrative of what the evaluation team suggests for UNFPA‟s efforts in Cambodia. The Gesellschaft has a distinct emphasis on the quality of its interventions, as the following indicators suggest:

 percentage of supported public health facilities that use the quality assessment tools for quality improvement;  percentage of health facilities that use the quality assessment tools that can demonstrate quality improvement; and,  percentage of midwives that can perform delivery in accordance with national training standards.

One of the evaluation team‟s related concerns focuses on the quality and technical competence of the individuals and NGOs that IPs hire to develop and present workshops. There does not appear to be any means within UNFPA/Cambodia to review the technical cre- dentials of the people or NGOs hired or to assess the quality of what is presented in the training sessions. To illustrate, representatives of some organizations that had been hired by the MoWA to provide training pointed out that the budgets allocated to their training activities were typically insufficient as was the time allocated to prepare and present the training mate-

34 UNFPA/Cambodia, Standard Progress Report for Reproductive Health, 2010. Identical statements are also included in earlier years‟ standard progress reports. The CPAP for CP4 notes that the current sectoral plans for health, education, women, HIV, and planning “minimally incorporate population, reproductive health, and gender issues.”

34 rials. Likewise, some respondents noted that the gender-related training on capacity-building training provided by MoWA was too general to be of much value and not sector specific. In the PD area, some assessment of participants‟ satisfaction with training has occurred, but the evaluation team did not learn of any related efforts to assess trainers‟ capacity, the methods used, or the suitability or effectiveness of the materials used. Many respondents noted the value of training but did not consider it to be the most effective way to build their capacity; they suggested that the number of training workshops be reduced. Other people suggested the desirability of using on-the-ground coaches who could assist communes in the management of their projects. The evaluation team appreciates that IPs do the hiring, but UNFPA‟s credibility and reputation are attached to these training events. Perhaps the country office can work with its IPs to develop some tools to ensure the suitability of the trainers used as well as the relevance and effectiveness of the training.

Aid effectiveness

Mention of UNFPA‟s adherence to the principles of the Paris Declaration on Aid Ef- fectiveness is in order. To paraphrase the Health Strategic Plan 2008–2015, there is a strong consensus among development partners, including UNFPA, and the government on the Dec- laration‟s main principles. There is considerable evidence of UNFPA/Cambodia‟s meaningful support for strong country ownership; strategies that are aligned with the government‟s priorities; harmonization of external support to country procedures and systems; and indicators for monitoring; and mutual accountability based on common systems for accountability. To illustrate this point, a 2011 report from the CDC on donors‟ responsiveness to the principles of the Paris Declaration identified UNFPA as one of several development partners that had increased their reliance on the RGC‟s procurement systems in recent years.35 UNFPA is seen as an important partner in Cambodia‟s development and is well and appropriately recognized for its technical capacity which creates respect among other donors and the government. In turn, this respect provides the country office with leverage and political capital. The agency participates in semiannual Joint Annual Performance Reviews in which representatives of relevant donors and the MoH review achievements, examine constraints to implementation, and establish priorities for the future. The country office is also an active and effective member of joint technical working groups on health, HIV/AIDS, education, planning, gender, and decentralization and deconcentration that the RGC established to assist in developing new sectoral plans, to review ongoing ones, to harmonize and coordinate external assistance to programmes and projects, and to monitor their implementation and progress. To support development partners‟ interface with the MoH‟s Second Health Sector Support Programme, the partners that contribute pooled funding created a Joint Partnership Arrangement Development Partners Interface Group (JPIG). Its first chair was UNFPA‟s former country representative. The position of chair rotates among the donors to the pooled funding, and UNFPA‟s country representative is again serving as the JPIG‟s chair. This posi-

35 CDC, Paris Declaration Monitoring Survey 2011, Cambodia Country Report.

35 tion provides an opportunity for the representative to participate in discussions with and po- tentially affect what other, significantly larger donors decide about their work with the MoH. No less important, the chairperson‟s position allows the agency to exercise influence well beyond the relatively small contribution that UNFPA makes to the pooled fund each year.36 UNFPA‟s contribution to the pooled fund ensures that the agency has a place at the table and demonstrates its commitment both to harmonization among donors as well as a shared commitment to country systems. The MoH would like to see further reductions in fragmentation among donors and increased use of pooled funding. In the words of the Health Strategic Plan, nonaligned financial flows from donors disrupt the governance model for the health sector. To illustrate, a report from the World Bank identified 22 donors in the health sector working with over 100 NGOs through more than 100 projects.37 Many of these projects focused on RH, hospitals, capacity development, and policy and planning, and more than a few donors work on the same issues. In addition to UNFPA, at least two other bilateral donors support effort to increase the number and quality of midwives. The evaluation team was informed that the Korea Foundation for International Healthcare will soon begin a project on midwives. This is but one of several examples in which UNFPA and several other development partners, including other UN agencies, share the same policy space. Having many donors increases fragmentation and transaction costs while reducing the government‟s ability to manage the many projects, which typically have different reporting and financial requirements. Another study found that Cambodia‟s health sector is among the most fragmented in terms of the number of separate donors.38 Fragmentation among donors that fund gender-related projects was also brought to the attention of the evaluation team. As one UNFPA standard progress report observed, one of the main constraints affecting achievement of results is the lack of effective coordination among donor partners. There are hopes that UN Women will take the lead in coordinating the gender-related efforts of UN agencies in Cambodia, and there is reason to believe that UNFPA would be a willing participant in efforts to improve coordination. This discussion of aid effectiveness is relevant to UNFPA‟s participation in the pooled funding arrangements in which the agency participates with six others donors to support the RGC‟s Second Health Sector Support Programme. Continuing with the pooled funding would reflect UNFPA/Cambodia‟s endorsement of several principles of the Paris Decla- ration, including donor harmonization, reliance on government systems, and shared accountability for achieving result. Nonetheless, there is also tension between a desire to adhere to these principles and UNFPA‟s corporate mandate that country offices are accountable for demonstrating the effectiveness of their support. If a country office is deemed to be account-

36 UNFPA contributed $310,555 and $620,000 in 2009 and 2010, respectively, to the pooled fund. These amounts represented 2.42 percent of all contributions in 2009 and 2.53 percent in 2010, according to the audit of the Second Health Sector Support Programme. The projected contribution in 2011 is $550,000. In addition to UNFPA/Cambodia‟s contribution to the pooled funds, the country office also provides discrete funding to the MoH for specific tasks included in the Health Strategic Plan. 37 World Bank, Country Assistance Strategy Assistance Progress Report for the Kingdom of Cambodia, Report No. 43330-KH, 2008. Similarly, the Statistical Master Plan of the National Institute of Statistics identifies 15 key stakeholders among the Institute‟s development partners. 38 Ek Chanbureth and Sok Hach, Aid Effectiveness in Cambodia. Wolfensohn Center for Development, 2008.

36 able for achieving effective results, as the agency‟s Results Based Management Policy makes clear, then there must also be an ability to attribute results observed to UNFPA. Pooled funding with multiple donors largely eviscerates attribution of results to a single donor. To paraphrase a report that reviewed the development efforts of the United Kingdom‟s Department for International Development (DFID) in Cambodia, UNFPA cannot and should not be held primarily accountable for achieving results that lie within the domain of government. In recognition of the virtues of donor harmonization and the principles of the Paris Declaration, UNFPA‟s financial rules and regulations explicitly permit and encourage pooled funding.39 The country office is considering an end to further financial contributions to the pooled funding. In the opinion of the evaluation team, such a decision should be taken only after discussion with the other members of the JPIG and after a thorough consideration of the costs and benefits of doing so. Judgments about whether UNFPA should or should not continue its participation in the pooled funding for the Second Health Sector Support Programme are beyond the scope of the TOR for this evaluation, so the evaluation team takes no position as to whether to continue UNFPA‟s contribution to the pooled funding.

Efficiency

Administrative efficiency

There are several ways to assess efficiency, which is a measure of how economically resources and inputs, such as funds, expertise, and time, are converted into results. One as- pect of efficiency is administrative or procedural. The TOR directs the evaluation team to consider the timeliness of inputs. Two measures of the former are available: (a) the dates that IPs submit their AWPs to UNFPA and the dates the agency approves the AWPs and (b) the subsequent date of first disbursal of funds from UNFPA to the IPs. To receive such funds, IPs are required to submit a Funding Authorization and Certificate of Expenditure (FACE) form. Over the course of CP3, the country office‟s review and approval of AWPs received from implementing partners were exemplary. After the programme‟s first year, the country office approved nearly all AWPs before the end of January of each year and many had already been approved in December. Thirty-seven of 40 AWPs for 2007 through 2010 were approved before the end of January of each year. No less important, the time between an IP‟s submission of an AWP and UNFPA‟s approval was minimal, with many AWPs approved the same day they were submitted to UNFPA. In both instances, these are instances of extraordi- nary efficiency for which the country office should be applauded. No less important, several IPs indicated their appreciation of UNFPA‟s administrative efficiency and the timeliness of the agency‟s approval of AWPs and of requests for advance funding. Once AWPs are approved, implementing partners can request advance funding from UNFPA. Here again the country office demonstrated remarkable efficiency. An average of 10 FACE forms was submitted to UNFPA each year requesting an initial disbursement of funds for that year. For about one quarter of the 50 requests received over the five years of CP3,

39 UNFPA, From Policy to Practice, UNFPA’s Role in the Changing Aid and Development Environment: Guid- ance Note on Aid Effectiveness (2009).

UNFPA made the initial payment to the IP the same day that the requests were submitted to UNFPA. Over 80 percent of the requests were approved within seven days (and these seven days included weekends and holidays). Less than 15 percent required more than ten days to be approved. In several instances the explanation for the time required was due to other problems with the FACE forms that the IP had submitted or because UNFPA required clarifica- tion from an IP that prevented immediate approval. For these reasons as well as for the timely and comprehensive assistance that was provided to the evaluation team, the team notes its belief that the country office‟s financial staff provides high-quality support to its colleagues in the office. The team was thoroughly impressed with the staff‟s abilities and knowledge as well as its patience with and attentive- ness to the team‟s requests.

Financial efficiency

Timely approval of AWPs and FACE forms reflects administrative efficiency but a more important issue focuses on the efficient use and management of the funds provided to IPs. How well does UNFPA match its advance payment of funds to IPs with their projected needs for these funds? IPs can decide how much of their annual funding they will request each quarter, but in 2008, 2009, and 2010, the largest amounts requested always fell in the first quarter of each year. UNFPA honored these requests and fully funded the amounts that had been requested in the FACE forms. How realistic were these requests, and did they match the IPs actual needs? The short answer is that the IPs typically overestimated their actual needs and spent far less in the first quarter than they had projected would be the case. As an illustration, Table 5 shows first- quarter expenditures for all six governmental IPs as a percentage of their first-quarter advances they had received for 2008, 2009, and 2010.40 The higher the percentage, the closer the relationship between projected need (as reflected in the first-quarter FACE form) and actual expenditures, as reflected in subsequent FACE forms. There is no benchmark or standard to determine what constitutes an acceptable match between projected and actual expenditures, but low percentages may be an indicator of poor planning. In addition, however, low percentages also reflect a situation in which UNFPA is not using its resources efficiently. Low percentages mean that UNFPA had provided funds to IPs that were not needed in the immediate future. The data in the table are revealing. Across the three years, the six IPs requested and were provided almost 40 percent more advanced funding than they spent in the first quarter. Across the three years the largest discrepancies between advance payments and the first- quarter expenditures were in the MoH; the advances it received were about 65 percent more than was required.41 The Ministry was provided with an initial advance of nearly $700,000 in

40 The six governmental IPs accounted for 95.4, 89.3, and 89.8 percent of all first-quarter advances for all IPs, including NGOs, in 2008, 2009, and 2010, respectively. 41 A possible explanation for the discrepancy is that there were delays in approving the Ministry‟s AWPs and in the date of UNFPA‟s initial advance payments to the Ministry. These possibilities provide only a small part of any explanation. The Ministry‟s AWPs were approved on December 28, 2007 (for 2008), January 26, 2009, and January 6, 2010. The first initial semiannual advance payments for these three years were made to the Ministry on January 19, 2008, February 25, 2009, and January 16, 2010. The MoH has acknowledged the problem of

2009, but it spent less than $100,000 in that quarter (and less than $425,000 through the first two quarters). In 2009 UNFPA provided the Ministry with an advance of almost $800,000. Although only 75 percent of this amount had been spent by the end of the third quarter, UNFPA had provided an additional advance of about $317,000 in the third-quarter seeming- ly without consideration of the UNFPA funds that that the Ministry already had.42

Table 5: First quarter expenditures as a percentage of advance payments, 2008–2010 Three-year Implementing partner 2008 2009 2010 average MoEYS 39.5% 27.1% 81.1% 51.8% MoH* 37.1% 28.6% 37.8% 34.2% Ministry of Interior 51.6% 59.1% 90.2% 66.7% Ministry of Planning 99.7% 77.7% 67.7% 91.0% MoWA 31.1% 71.3% 75.5% 50.3% NCPD 87.5% 18.8% 75.5% 50.3% Annual weighted average 70.1% 40.9% 66.3% 61.2% Source: UNFPA/Cambodia. * Does not reflect pooled funds provided to the MoH. The percentages for the MoH have been adjusted to reflect the fact that it receives semiannual advances from UNFPA rather than quarterly advances as is the practice with the other IPs.

This situation merits attention from UNFPA. When an IP requests an advance, UNFPA should consider how much money the IP still has from previous advances. Discus- sions with country staff indicate that this is not done consistently. For example, if an IP had $10,000 remaining from a previous advance payment and then requested an additional advance of $100,000, the practice has been to provide the full amount requested – without consideration of the funds already available. This is not an efficient use of UNFPA‟s resources and means that IPs have access to more of UNFPA‟s funds than they themselves believe they need. Another way to assess the efficient use of resources is to examine the spread of expenditures over the course of each year. For the same governmental IPs shown above, the data in Table 6 provide data on the ratio of fourth-quarter to first-quarter expenditures. If spend- ing is spread equitably between the two quarters, the ratios should be close to 1.00, which would mean that the same amounts were spent in both the first and last quarter. Ratios slightly above 1.00 would not be unexpected because none of the governmental IPs received their initial advances during the first week in January in any year. In contrast, ratios above 1.50 indicate back-loading of expenditures in the fourth quarter; the higher the ratio, the more pro- nounced the problem. Ratios below 1.00 indicate front-loading of activities and that more money was spent in the first quarter than in the fourth quarter.

underspending in the first quarter, noting in its 2009 Annual Performance Monitoring Report that one challenge is “difficulties in accessing budget, particularly in Quarter 1.” A comment from a senior official in the MoH further explained that the low percentage of first-quarter expenditures is caused by financial regulations and donor-related delays. A representative of the country office concurred, noting that slow disbursement may be constrained by donors; regulations intended to ensure accountability for results. 42 UNFPA requires country offices to “ensure that the government or managing agent is not holding large cash resources, which would indicate poor utilization of cash and increase the risk of fraud.” See UNFPA, From Pol- icy to Practice, UNFPA’s Role in the Changing Aid and Development Environment: Guidance Note on Aid Ef- fectiveness (2009).

The data reveal considerable variation in the ratios from year to year and within each agency. The Ministry of Planning had an unusually low ratio (i.e., 0.20) in 2008 because of its front-loading of expenditures for the national census in that year. The Ministry‟s low ratio and the magnitude of its first-quarter expenditures explain the relatively low annual weighted average. Among the six agencies, the MoH‟s ratios again draw attention. An overwhelming proportion of its expenditures occurred in the last quarter in each of the three years. In 2010, it spent more than five times as much in the final quarter as it did in the first quarter (and thus repeated the same situation that had occurred in 2009). This is not necessarily an indication of inefficiency, but it does raise a concern about the timing of the funding provided.

Table 6: Ratio of fourth-quarter expenditures to first-quarter expenditures, 2008–2010 Three-year Implementing partner 2008 2009 2010 average MoEYS 1.52 1.85 0.43 0.90 MoH* 3.09 5.06 5.43 4.28 Ministry of Interior 2.33 2.91 1.53 2.15 Ministry of Planning 0.20 1.07 3.10 0.71 MoWA 3.08 2.44 1.99 2.39 NCPD 1.23 2.23 2.49 1.93 Annual weighted average .88 2.80 2.33 1.64 Source: UNFPA/Cambodia. * Ratios do not reflect pooled funds provided to the MoH.

Examination of UNFPA‟s operating fund account (OFA) provides a third way to assess financial efficiency. The OFA represents funds provided to IPs each year but not timely spent or liquidated. High OFA balances indicate that IPs received more money than they needed as well as inefficient allocation of UNFPA‟s resources. Across the six government agencies, the median OFA for each year was extremely low (e.g., about $121 in 2009), thus suggesting a good match between resources provided by UNFPA and resources expended by IPs. The MoH is an outlier, as shown in Table 7. Much of the table is self-explanatory. The MoH‟s expenditures represented slightly over one-third of the total expenditures of UNFPA‟s governmental IPs for the three years, but the Ministry accounted for nearly 90 percent of all funds in the OFA. The data reveal that the Ministry has been receiving more money than it can absorb. This situation was particularly acute in 2009 when nearly one quarter of the funds provided to the Ministry were not used in that year. In addition, in that year the Ministry spent only 81 percent of the funds provided to it through the Maternal Health Thematic Fund and only 70 percent of the money provided through a German trust fund. Placing the amounts of OFA in context underscores their size and the problems that exist with large OFA balances. According to a UNFPA-wide assessment of OFA in mid 2010, Cambodia had one of the highest balances of old OFA among all of the agency‟s country offices and the fifth highest balance of “very old” OFA (i.e., OFA more than a year old) among all country offices.43 When IPs are advanced too much money, UNFPA risks financial

43 S. K. Gupta, DOS, UNFPA OFA Analysis, For UNFPA Globally as at end of July 2010 and end of Q2 (Jun) 2010. PowerPoint Presentation to UNFPA‟s Executive Committee, September 2-3, 2010.

40 losses due to fluctuations in foreign exchange and loss of interest that would have accrued if the excess funds had stayed within UNFPA‟s control. IPs are required to report interest income to UNFPA, but government IPs in Cambodia do not earn any interest on the funds advanced to them.

Table 7: Median OFA balances and OFA in the MoH, 2008–2010 2008 2009 2010 Total All government IPs Median OFA $2,733 $121 $1,985 $1,271 Ministry of Health* Percentage of all funding for governmental IPs 32.9% 40.0% 34.4% 35.7% Percentage of all OFA 77.9% 96.5% 79.3% 89.8% OFA $86,587 $322,766 $69,936 $479,289 OFA as a percentage of total expenditures 8.2% 24.2% 5.4% 13.0% Source: UNFPA/Cambodia * Percentages do not reflect pooled funds provided to the MoH. Note: All amounts are based on UNFPA‟s regular resources and thus exclude trust funds.

UNFPA‟s provision of annual grants to communes provides another example where efficiency is in doubt. The agency provided annual grants of $500 to each of 446 communes to promote their attention to RH, PD, and gender-related issues. The process of transferring the funds to the communes was not notably efficient. UNFPA/Cambodia provided the funds to DOLA, which would then use a commercial bank to transfer the funds to the relevant provincial offices. In turn, these offices would transfer $125 per quarter to each commune, usually in person since many of the communes do not have access to banks or a means to access electronic transfers of funds. The meeting point to collect and transfer $125 is usually at a district office where provincial facilitators and commune councilors meet. In some cases, the provincial and district facilitators transfer cash to the communes. Depending on the communes‟ locations and the season of the year, travel from a provincial office to a commune could take several hours and require the hiring of a taxi or even a boat to transfer the cash to the communes. In some instances the amounts allocated for these transportation costs were significantly lower than the actual costs. One district office informed the evaluation team that the cost of renting a boat to deliver the funds to one commune was $80, or almost two-thirds of the cost of the funds delivered. The communes appreciated the funds that UNFPA provided because they were able to manage the funds directly and had considerable freedom in how the money would be spent. In some cases the funds were distributed to participants in appreciation for their attendance at meetings, to cover the costs of travel, or to reflect the opportunity costs associated with lost working time. In other instances funds were used to purchase snacks to attract participants, but doing so often resulted in having many young children attend the sessions. The process of transferring funds to 446 communes can be cumbersome and inefficient, but at least sound financial procedures were in place to track the distribution of the grants. Less favorably, there was no system to report the results of the UNFPA-sponsored activities and no requirement that commune councils do so. UNFPA judged the success of

41 some activities by considering what percentage of budgets had been spent, but this is not a valid indicator of results achieved.

Indirect costs

Still another measure of efficiency is the extent to which UNFPA ensures that most of its resources are directed to programmatic activities as opposed to indirect or overhead costs of the agency‟s implementing partners.44 The agency prohibits the reimbursement of such costs for government agencies that are IPs and limits these costs to 12 percent of direct costs for nongovernmental IPs. As UNFPA‟s standard letter of understating with NGOs declares, this “amount cannot be exceeded.” During CP3 UNFPA/Cambodia did not comply with either limit. A review of multiple AWPs and FACE forms revealed that the country office routinely reimbursed government agencies for such expenses as utilities, including telephone and internet services, office supplies, accounting software, furniture, and the purchase and maintenance of general purpose equipment. A standard progress report for 2009 indicated that the MoH received over $34,000 in indirect costs (and this amount reflected only 71 percent of what had been allocated for these costs). Staff of the country office acknowledged their awareness of the agency‟s prohibition of reimbursement for the indirect costs of government agencies. When asked why the country office did not comply with this prohibition, the evaluation team was informed that government agencies in Cambodia would not cooperate with UNFPA in the absence of the reimbursement and that such reimbursement may be a standard practice of Cambodia‟s development partners. No evidence was provided to substantiate this claim or the assertion that UNFPA‟s rules on indirect costs can be ignored, as they now are. The issue of indirect costs for government agencies is probably one that other donors also face, so collective attention from these donors could be useful. During CP3 the country office collaborated with five NGOs. The letters of understanding between four of the NGOs and UNFPA limited their reimbursement of indirect costs to 7 to 10 percent of direct costs. With UNFPA‟s concurrence, these percentages were routinely exceeded; UNFPA should not have reimbursed these IPs for indirect costs that exceeded 12 percent. The NGOs did not include costs for rent, utilities, cleaning and guard services, and management expenses in their calculations of indirect costs. Instead, they considered their costs for facilities and administration to be their total indirect costs. Table 8 illustrates the extent to which three NGOs exceeded the 12-percent limit. For another NGO, the Khmer Youth Association (KYA), the evaluation team was told that indirect costs are “built into” the association‟s budgets and are thus not separately calcu- lable. KYA‟s letter of understanding does not include any mention of UNFPA‟s reimbursement of indirect costs. As a consequence, UNFPA was unable to ascertain the actual indirect costs of either KYA.

44 DOS has used indirect costs as an indicator of efficiency. See DOS, Assessment of “A Strategic Management Review of the 7th Country Programme in Indonesia (2006-2010).”

To remedy the situation just described, the country office should ensure that all of its staff are aware of and abide by UNFPA‟s Guidance Note on Indirect and Direct Costs. Fur- thermore, UNFPA/Cambodia should (a) not include any indirect costs for government IPs in AWPs; (b) explicitly identify what indirect costs are reimbursable for NGOs in their AWPs; and, (c) ensure that country office staff and these NGOs understand which indirect costs are reimbursable and the limits of this reimbursement.

Table 8: Indirect costs for three NGOs NGO #1 NGO #2 NGO #3 Facilities and administration $3,512 $2,939 $5,600 Rent $600 $2,061 $2,100 Utilities and office supplies $2,616 $693 $1,554 Administrative staff* $4,111 Total indirect costs $6,728 $5,693 $13,365 Total direct costs $40,671 $26,549 $72.235 Indirect cost rate 16.5% 21.4% 18.5% Source: 2007 FACE for NGO #1, 2006 FACE for NGO #2, and 2010 AWP for NGO #3. * Includes finance and IT officers plus a guard and an office cleaner.

Value for money

The TOR asks the evaluation team to consider whether the outputs achieved were reasonable for the resources spent. Addressing this issue requires a subjective judgment because there is no relevant benchmark or standard. From one perspective, however, the outputs achieved were not commensurate with the resources spent. The CPAP outlined what the country office planned to achieve during CP3. The Country Programme Document identified the resources that would be spent to achieve these outputs (see Table 2).45 As discussed above, many of the CP3‟s projected outputs were either not achieved or only partially achieved, so UNFPA/Cambodia delivered less than it had projected. In UNFPA‟s defense, this was not unexpected. In some instances the expected outputs were too ambitious or largely dependent on the agency‟s implementing partners. Similarly, although UNFPA is accountable for achieving the outputs it identifies, it is unreasonable to expect that the country office could know when the Country Programme Document was developed in 2004 and 2005 what was possible or feasible many years later. These are reasonable explanations but they do not fully mitigate the concerns of the evaluation team about value for money, especially in terms of CP3‟s emphasis on capacity building. There is widespread agreement that deficits in capacity exist within the government. Not surprisingly, therefore, a majority of CP3‟s outputs focused on capacity development (see Table 1), and much of UNFPA‟s budget during CP3 was devoted to this development. This emphasis is common among Cambodia development partners. What is and also remains common among these partners, including the UNFPA during CP3, is the absence of efforts to assess the gaps in capacity that exist and then to develop a strategy to address these gaps.46 One assessment of the MoH‟s capacities concluded during CP3 that:

45 Actual expenditures during CP3 represented about 93 percent of expenditures projected in the County Pro- gramme Document for Cambodia. 46 Ek Chanbureth and Sok Hach, Aid Effectiveness in Cambodia. Wolfensohn Center for Development, 2008.

The absence of any kind of articulated capacity development strategy makes it difficult for partners to harmonise and align external support behind a country-led process. It also makes it difficult to engage in an effective dialogue about capacity development, to reach some kind of common understanding on what it involves and on what external partners can do to assist, and to encourage learning.

A useful strategy would identify the organizational gaps that exist within the agency‟s implementing partners as well as the functions to be strengthened. No less important, at least two questions would be asked and answered: capacity for whom and capacity for what? UNFPA‟s efforts in CP3 partially answered the first question but not the second. UNFPA/Cambodia did not conduct a baseline capacity assessment of its implementing partners prior to the start of CP3.47 According to UNFPA‟s DOS, in the absence of baseline assessments of capacity, country programmes “fail to meet UNFPA‟s basic accountability expectations.” DOS contends that there should be “partner capacity assessments” of every IP that receives more than $100,000. Without a strategy for capacity-building efforts and UNFPA‟s uncertainty about how to assess and measure the success of CP3‟s capacity-building efforts, assessing value for money (as well as effectiveness) is problematic. The evaluation team and others have concerns about how much money is devoted to capacity building, which is largely restricted to training of individuals rather than strengthening the capacity of institutions or organizations. In some years more than half of some agencies’ UNFPA-sponsored budgets was devoted to daily subsistence allowances, most of which were for attendance at training workshops and conferences. The percentage of funds devoted to these activities was even higher when one considers expenses for travel and materials. This situation was especially prominent in the MoH and the MoEYS. The evaluation team considers these expenses to be unduly high. Several respondents noted that the allowances are often seen as a means to supple- ment incomes or to increase enrollments in the training events, with the consequence that people are attracted to training events not because of their content but because attendance allows participants to increase their income.48 Other people similarly suggested that the allowances also create incentives to increase the number of training events. Several respondents further noted that some commune councils use funds to provide cash incentives or to purchase snacks to encourage attendance, but the snacks typically mean that parents will bring their children to meetings. These concerns may provide some of the explanation for a letter that the alternate chair of the JPIG sent to the MoH in late 2009. Writing on behalf of the JPIG, the alternate chair indicated the group‟s concerns that cost reductions were:

47In contrast, the country office conducted an assessment of the capacity needs for the National Institute of Statistics during CP3. 48 To paraphrase the comments of one member of the EMC, international consultants bring high capacity to Cambodia but that capacity often exceeds the needs and ability of some government officials being trained. In addition, he observed, some of the staff focus on family income rather than training. They also need simple skill training such as on data collection or data entry rather than complicated training.

achieved by cutting priority programs at the provincial level, rather than reducing and consolidating centrally managed activities (including training) or through greater in- tegration of activities at provincial and district levels (such as training, community outreach, and integrated supervision….We would like to work with the [MoH] in such a way that many important activities including maternal and child health…each continue to receive an adequate budget.

The letter also noted that “many of the issues raised by JPIG around training and supervision in 2009 remain valid.” As the letter emphasized, “We note over-reliance on training activities, which are sometimes unrealistically planned, divert resources from service delivery and do not necessarily lead to better quality of care.” Several respondents also noted UNFPA‟s and other donors‟ heavy reliance on training workshops of a few days‟ duration as the primary means of capacity building without due attention to (a) follow up; (b) other approaches to capacity building, such as mentoring or coaching; or, (c) rigorous efforts to evaluate the outcomes of the training provided. Given the volume of resources devoted to training, which one respondent estimated to be as much as $10 million a year just within the MoH, the need for ongoing and comprehensive evaluation of training and capacity building is imperative. These comments again address the issue of whether what is being done is the right thing and how well UNFPA‟s resources are being used to achieve its objectives. In the case of capacity building, as already implied, the long-term objectives and expected outcomes have not been well articulated and remain to be evaluated. As UNFPA‟s policy on results- based management emphasizes, such management “uses evidence on results to inform decision-making in respect to design, resourcing and delivery of programmes and activities.”

Financial reporting

Transparency is a corollary of efficiency. UNFPA should be able to track IPs‟ use of the funds provided to them. If such tracking does not occur or is not possible, UNFPA cannot be sure how its funds were spent or even whether they were spent on activities identified in AWPs. The CPAP for CP3 makes clear that funds transferred to IPs should be spent only for the purposes of activities as agreed in AWPs. IPs use FACE forms to report expenditures, but in many instances it was not possible to link expenditures during CP3 with activities included in AWPs because of the ways that IPs reported their expenditures. In other instances it was not possible to discern the relation between expenditures and approved activities. UNFPA‟s Policies and Procedures Manual stipulates that sundry or miscellaneous expenses “should not exceed one percent of the total fund disbursed to an implementing partner for an output.” In a few FACE forms, the “sundry expenses” that some government IPs reported exceeded 80 to 85 percent of their total annual expenditures for some activities; in one instance within the MoH, sundry expenses accounted for 99.2 percent of the reported expenditures for an activity. Finally, AWPs represent agreement between UNFPA and its implementing partners about what activities will be implemented and the estimated cost of doing so. The evaluation identified several instances in budgets included and approved in AWPs were not followed and activities that had been approved were not completed. The funds for the approved activi-

45 ties were typically used for other activities, often not having been included in the AWPs or not having been addressed in a budget revision that had received UNFPAs approval.49

Impact

In accordance with international best practices in evaluation, impact refers to “positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended.” As the UN Evaluation Group has observed, “produced by a development intervention” implies consideration of attribution, both in terms of and causality and degree.50 Similarly, the UNFPA‟s Evaluation Guidelines indicate that questions about attribution are typical of country programme evaluations. Not surprisingly, therefore, attribution is discussed in several recent CPEs, including the most recent evaluations of UNFPA‟s programmes in Iran, Myanmar, the Philippines, and Thailand.51 Assessing UNFPA/Cambodia‟s longer-term impact is challenging. UNFPA supports several government ministries and departments and much of the agency‟s success and accomplishments depend on how well these ministries perform their functions. Several factors affect how well these functions are performed, including high staff turnover, limited individual and organizational capacity, persistently low salaries that lead to low motivation and high levels of absenteeism, long-standing cultural norms that are unsympathetic to gender equality, and corruption, which is rampant and widespread in Cambodia. In the face of these challenges, UNFPA‟s resources provide only limited leverage and compromise the ability to attribute any impacts achieved solely to UNFPA‟s efforts. Over the life of CP3, the MoH spent slightly over $6 million that it had received from UNFPA. In the CP3‟s latter years a portion of this amount was provided to the Ministry through pooled funding and the Second Health Sector Support Programme. UNFPA‟s contribution to the pooled funding for this programme compares with contributions of $50 million from DFID and $30 million each from AusAID and the World Bank. Furthermore, these amounts were in addition to other resources that these and other donors provided for nonpooled, stand-alone projects, such as the United Kingdom‟s support for a project on reducing maternal mortality. These donors, typically working in collaboration with government agencies or NGOs, all seek similar impacts. The health sector provides a relevant example. Over the past ten years access to health services has improved considerably with corresponding improvements in most indicators of health, including reproductive health. The government‟s capability and responsiveness have improved because of donors‟ support. In turn, donors‟ support for the principles of the Paris Declaration is a possible explanation for government ministries becom- ing more assertive in coordinating donors and in leading and assuming ownership of the de-

49 UNFPA‟s letters of understanding with its IPs typically declare that: “Any changes to the budget contained in the AWP that would affect the work performed by implementing partners and in particular the financial aspects of the AWP, will be permissible only after consultation with the UNFPA Representative or his or her designated official. AWP budget amendments should be agreed upon between the implementing partner and UNFPA.” 50 UN Evaluation Group, Impact Evaluation Task Force, “Terms of reference for the preparation of guidance material on Impact Evaluation of normative and institutional support work.” 51Furthermore, Furthermore, the DOS review of the CPE in Myanmar rated the evaluation‟s design and methodology as good, noting that the evaluation had made efforts to address the issue of attribution..

46 velopment process in all the areas in which the agency works. As discussed above, UNFPA‟s efforts have clearly contributed to many of the changes observed. Examples include the considerable increase on the number of trained midwives and their deployment throughout the country as well as the expansion of health equity funds that have dramatically increased the number of women who seek and receive reproductive health services. These positive changes are likely to have impacts for many years to come. Nonetheless, UNFPA has not designed its interventions in ways that facilitate assessment of impacts. As noted earlier, as an example, several of CP3‟s indicators are not objectively verifiable or easily measured and counterfactuals, and essential element in assessing impacts, are typically missing. This situation is not unusual in Cambodia – or elsewhere. To illustrate, the evaluation team‟s interviews as well as the documentation reviewed identified several statements about interventions without corresponding information about their impacts. Here are two examples, both of which are relevant to UNFPA‟s efforts in Cambodia:

There has yet to be a systematic process established to study the effectiveness and impact of the [MoH‟s Gender Mainstreaming Action Group] in generating understanding and support for gender concepts and analysis and their application to the MoH systems and program areas and on the work in individual departments.52

There is an implicit assumption that decentralized arrangements are effective and efficient and will contribute to the overall goal of poverty reduction with little clear evidence to demonstrate whether this is the fact or not in Cambodia.53

As just noted, a counterfactual is useful in determining whether impacts have occurred and whether they can be attributed to the intervention. Use of a counterfactual asks this question: “What would have happened in the absence of the UNFPA‟s interventions, all else being equal?” Using data from the 2005 and 2010 DHS, it is possible to address this question, albeit indirectly and imperfectly for several RH-related outcomes. Table 10 shows a series of comparisons between outcomes in the CP3‟s priority provinces and in the other provinces (in which UNFPA did not focus its activities). It would have been desirable to have

52 Kate Grace Frieson, et al., A Gender Analysis of the Cambodian Health Sector (2011). 53 Paul Thornton, et al., Evaluation of DFID Country Programmes: Cambodia (2009). As the country office has noted, the UNFPA‟s decision to support the government decentralization and decentralization initiative was not related to this finding. In the words of one country officer, “The fact is that the government has decided to undertake this reform process and within the next few years there will be a process of functional assignment under which responsibility for delivery of key services, including those crucial to UNFPA‟s mandates, will be with local authorities. Given this reality and the tendency of local authorities to focus on infrastructure at the expense of social issues (and especially those affecting women and children) both UNICEF and UNFPA have decided to engage with local authorities and with the new machinery charged with responsibility for women‟s and children‟s issues at multiple administrative levels.”

Table 10: Comparison of key indicators in priority and other provinces, 2005 and 2010 Change, Net Expected Indicator 2005 2010 2005–2010 Difference Difference Percent distribution of currently married women who cur- Priority provinces 25.2% 33.1% +7.9% rently use a modern method of contraception*†§ 0.2% Negligible Other provinces 28.8% 36.5% +7.7% Percentage of currently married women with unmet need Priority provinces 26.6% 17.3% -9.3% for family planning§ 1.3% Negligible Other provinces 23.9% 15.9% -8.0% Percentage of currently married women whose demand for Priority provinces 58.3% 73.5% +15.2% contraception is satisfied 1.2% Negligible Other provinces 63.2% 77.2% +14.0% Percentage of women who are not using contraception who Priority provinces 11.6% 13.7% +2.1% visited a health facility and discussed family planning in 0.2% Negligible Other provinces 8.1% 10.0% +1.9% the 12 months preceding the survey Percent distribution of women who had a live birth in the Priority provinces 63.5% 85.6% +22.1% five years preceding the survey who received antenatal 5.2% Yes care from a doctor, nurse, or midwife during pregnancy for Other provinces 75.5% 92.4% +16.9% the most recent birth Percent distribution of live births in the five years preced- Priority provinces 33.3% 62.2% +28.9% ing the survey in which the mother received assistance 0.5% Negligible during delivery from a doctor, nurse, or midwife*†§ Other provinces 51.2% 79.6% +28.4% Percent distribution of live births in the five years preced- Priority provinces 1.4% 2.3% +0.9% ing the survey delivered by caesarean section†§ Other provinces 2.2% 3.6% +1.4% 0.5% Negligible Median number of months since preceding (nonfirst) birth Priority provinces 35.6 38.8 +3.2 Other provinces 38.0 42.5 +4.5 1.3 Negligible Percentage of all women age 15-49 years who gave birth Priority provinces 15.1% 40.0% +24.9% in the two years preceding the survey who received HIV 11.0% No counseling during antenatal care for their most recent birth Other provinces 16.5% 52.4% +35.9% Percentage of women age 15-49 with a comprehensive Priority provinces 39.7% 33.0% -6.7% knowledge about AIDS Other provinces 48.2% 46.7% -1.5% 5.2% No Source: 2005 and 2010 DHS for Cambodia. Note: Data from Koh Kong, a priority province, and Preah Sihanouk, a nonpriority province, are excluded from the analysis because the DHS in both years combined the data from the provinces. This exclusion had an insignificant impact on the results because the total number of cases from both provinces represented no more than 2.5 percent of all cases in 2005 and 2010. * One of UNFPA/Cambodia‟s indicators for CP3. † An indicator for the UNDAF, 2006–2010, and Cambodia‟s MDGs. § An indicator for UNFPA/Cambodia‟s CPAP, 2011–2015.

compared before-and-after data across provinces and particularly for the operational districts in which UNFPA focused its efforts and to have considered UNFPA‟s efforts at the national level. Nonetheless, the indicators in Table 10 provide proxies for much of what UNFPA attempted to achieve in CP3. As the numbers in the far-right column of the table reveal, the discouraging conclusion is that CP3 seems to have had an insignificant impact. For seven of the indicators the net or difference between the priority and nonpriority provinces is negligible and probably within the margin of sampling error. For two indicators related to awareness and knowledge of HIV, the changes were greater in the nonpriority areas than in UNFPA‟s priority areas. Only in one instance (i.e., the indicator on antenatal care) is the difference in favor of the priority provinces. One cause for concern is that the percentage of women age 15-49 with a comprehensive knowledge about AIDS declined between 2005 and 2010 in both priority and nonpriority provinces, but most noticeably in UNFPA‟s priority provinces (e.g., to 33 percent in 2010 from 39.7 percent in 2005). These findings, clearly discouraging, should be viewed in the context of the applicable ca- veats. For example, one might contend that the UNFPA-supported provinces fared equally as well as did provinces supported by other development partners. Similarly, women in UNFPA‟s provinces were particularly disadvantaged relative to women in the nonpriority provinces before the interventions that UNFPA supported. In addition to data from the DHS, the CHEMS‟ behavioral change campaign discussed above provides further evidence of the difficulty in attributing changes in attitudes to UNFPA- funded interventions. In addition to measuring changes that occurred between 2008 and 2010, the endline survey compared responses to its questions among women of reproductive age who had been exposed to the campaign with women who had not seen or heard any of the media messages. As the report on the endline survey concluded, “the indicators are not significantly different for the women who have been „exposed‟ to the campaign” and those who had not. In other words, although most changes were in the desired direction, it is not possible to indicate whether the campaign contributed to the attitudinal changes that occurred or whether the explanation lies in whatever factors contributed to the attitudinal changes among women who were not exposed to the campaign. Table 11 shows the responses to several items for those exposed and those not exposed to the media campaign. At least two items are noteworthy in the table. First, a noticeable difference exists in women‟s willingness to use long-term or permanent methods of contraception. Women who had been exposed to the campaign were more likely to say they would use long-term methods compared to those who had not been exposed to the media messages. Second, and in contrast, the latter group was less likely to worry about the side effects of contraception. Of interest, almost half of the women who worried about the side or health effects of contraception were still willing to use contraceptives for family planning.

Table 11: Attitudes of married women of reproductive age who were and were not exposed to CHEMS‟ media campaign on maternal and reproductive health Indicator Baseline Exposed Not exposed Percentage who say they plan to have their future child at a hospital, private clinic, or health center none 97% 95% Percentage who say they will use long-term and permanent methods of contraception none 63% 53% Percentage who plan to have four or more antenatal visits during their future pregnancy 64% 76% 78% Percentage who worry about any type of side effects when using contraceptives for family planning purposes 36% 46% 37% Percentage who believe that a woman should wait at least three years before having another child 90% 96% 96% Number = 895 1,232 414 Source: Eng Peou and Jean-Pierre Depasse, Improving Maternal Health though BCC: KAP Endline Survey, March 2010 and Market Strategy and Development, Ltd., the Cambodian firm that conducted the baseline survey for CHEMS. Note: For the first two items, the questions asked in the baseline and endline surveys were not the same, at least as translated into English. The statements for all items reflect the wording used in the endline survey. Impacts on gender and human rights

The TOR asks the evaluation team to consider CP3‟s impact on gender and human rights and, in particular, impact on government attitudes to gender and gender-based violence. On the first issue, some evidence exists about the programme‟s impact on gender. The government‟s policy statements on gender and gender equality are impressive as is the existence of more than two dozen gender mainstreaming action groups among government agencies. Gender Mainstreaming Action Plans have been developed, and many of them have received budgetary support from the government and from Cambodia‟s development partners. In the words of the UNDP, “All of this has resulted in the emergence of an institutional structure for mainstreaming gender in government activities that is quite exceptional in the developing world.”54 In contrast to the support for gender mainstreaming just noted, one should also be aware that the budgets devoted to the topic within ministries remains insufficient to meet the objectives identified in the action plans. Likewise, based on the evaluation team‟s interviews, the meaning of gender equality (and gender responsive budgeting) is often limited within most ministries‟ senior leadership. MoWA is a notable exception. UNFPA along with many donors, including several UN agencies, promote gender mainstreaming and attention to gender-based violence within the RGC. A recent report that assessed the results of UNDP‟s efforts in Cambodia identified UNFPA as contributing to increased attention to gender in the MoH and the Ministry of Planning. The report could also have mentioned (but did not) UNFPA‟s gender-specific support for the Ministry of Interior (about $275,000 during CP3) and the MoWA (almost $1 million during CP3). The second issue asks about UNFPA/Cambodia‟s impacts on human rights. The country office has clearly focused on and advocated reproductive rights, but interviews with the staff of

54 UNDP, Assessment of Development Results: Cambodia (2010). 50

the country office and a review of documents provides little evidence of attention to or impacts related to human rights. Finally, the evaluation team‟s efforts to apply triangulation did not identify any specific or long-term impacts that UNFPA‟s efforts might have had on minorities, marginalized, or vulnerable groups.UNFPA has demonstrated effort and activity but not impact for minorities or marginalized groups. To UNFPA‟s credit, it has devoted some of its resources for RH to areas within Cambodia (e.g., Mondul Kiri, Pailin, Ratanakiri, and Stung Treng) that have relatively large minority populations. In 2006 and 2007, the country office supported consultations on indigenous issues in Ratanakiri with participants and community representatives from two other provinces with minority populations. The UNFPA country office has also been a core sponsor of the annual national forum on indigenous peoples and issues. One of CP3‟s objectives was to increase access to high-quality, comprehensive, client- oriented, and gender-sensitive RH information and services for vulnerable groups in priority areas, but the documentation provided to the evaluation team did not identify which groups are deemed to be vulnerable in Cambodia. If the country office is interested in knowing what impact CP4 might have on marginalized and vulnerable groups, then an effort should be made to (a) identify which of these groups are supposed to be the beneficiaries of UNFPA‟s efforts in Cambodia and (b) collect, analyze, and report disaggregated data on these groups.

Conclusions

hat can be said in conclusion about the nature of CP3‟s interventions in the three W thematic areas and the evaluative criteria of relevance, effectiveness, efficiency, and impact? Although every country is unique and requires responses that accommodate local conditions, few countries have a recent history as disturbing and traumatic as does Cambodia. Understanding and appreciating this history helps place UNFPA‟s interventions and achievements in perspective. The Khmer Rouge extinguished a cohort of skilled professionals and their leadership throughout the Cambodian government. Others, too young to be in government service at the time, suffered because they were not able to achieve their aspirations for education or the social development that all children deserve. Had they received the education they wanted and deserved, the numbers of highly skilled professionals in Cambodia would be much larger today, and concerns about individual and institutional capacity would be significantly less worrisome than they are today. The concerns about limited capacity and what to do about it govern much of the donor community‟s decisions about how to allocate its resources. UNFPA‟s decision making about its allocation of resources is no exception. The problems it addresses are vast and not easily resolved. Cambodia has had one of the world‟s highest rates of maternal mortality, gender inequality is widespread and, as in many other countries, data on population and development are frequently insufficient and not used as effectively as they could be. Any organization wishing to address these challenges must also contend with civil servants who are poorly paid and often poorly motivated yet expected to perform to standards well beyond their capacity or experience.

What are the implications of this situation for the UNFPA? First, and of the highest priority, choosing the “right” interventions in the “right” areas is essential. CP3 (and CP4) are typical of many of the agency‟s other country programmes; they devote similar percentages of their limited resources to RH, PD, and gender. Cambodia‟s needs for development are not equitably or proportionately distributed among these three areas. With this in mind, what is the best use of UNFPA‟s resources and where can it have the greatest impact within its areas of particular competence and expertise? Trying to do too much, a prevailing characteristic of most donor agencies, is neither feasible nor likely to be productive. A case can be made that RH and gender deserve the most attention. Population and development issues are important, but one would be hard pressed to make a case that PD is more compelling than RH or gender, at least in Cambodia, particularly because of the weak linkages among the three thematic areas within the country programme. A stronger case for PD could be made if efforts were directed to support for RH and gender. In other words, rather than a series of stand-alone initiatives to assist Cambodia‟s statistically inclined agencies, PD could usefully, primarily, and productively support the other two thematic areas. Second, the modes and methods of UNFPA‟s financial support to the RGC deserve attention. For much of the past ten years UNFPA has provided support to the MoH through both discrete funds for specific activities or projects as well as through pooled funding. The evaluation team sensed ambivalence within the country office about the challenges and desirability of both approaches. As it should, the RGC wants to lead the country‟s development and coordinate the efforts of its development partners. Cambodia has chosen a programme-based approach to development, and the RGC‟s preference, at least for the MoH, is pooled funding. Following such a path has required the development community to change or rethink many of their practices and procedures, and UNFPA is no exception. The evaluation team has intentionally avoided any recommendations about which approach to funding the MoH is best or preferred. The evaluation team understands that the evaluation of the previous country programme recommended greater reliance on pooled funding and less on project support. The choice of approaches will have considerable implications for the country office. If its contribution to Cambodia‟s development goals is the measure of success, then pooled funding is an option. Conversely, if the expectation is that attribution to UNFPA‟s interventions must be demonstrated, then discrete funding may be required. As several people in the country office noted, the TOR for the evaluation was better suited for a project-based approach to support rather than for a programme-based approach in which the RGC coordinates the contributions and activities of multiple donors. This may be a fair observation. If it is, then UNFPA‟s Evaluation Guidelines would be benefit from amendment to accommodate evaluations of pooled-funding arrangements. Third, attention to institutional and organizational capacity is essential. UNFPA‟s efforts in each of its three thematic areas recognizes this need, but it is not clear, at least to the evaluation team, that there: (a) is an overall strategy or longer-term plan for developing the capacity required in the three thematic areas; (b) has been identification of a target or desired endline for efforts to build the required capacity; or, (c) is a means to identify what constitutes success and how it can be measured. Within the donor community, this situation is not peculiar to UNFPA; it probably 52

bedevils most or even all members of the development community as well as the Cambodian ministries with whom they collaborate. Fourth, regardless of what results may have been achieved during CP3, a major challenge has been their identification and measurement. All of CP3‟s targets were desirable, but too many were far too ambitious while others were well beyond the ability of the country office to control or influence. This situation posed a challenge to the evaluation team‟s ability to make objective judgments about the effectiveness and impact of CP3. More important, however, the weakness in the results frameworks and its indicators and targets handicapped the country office in its efforts to assess progress, to demonstrate its successes, and to identify instances in which achievements were less than expected or promised. Much can and should be done to ensure that a similar situation does not occur with CP4. Fifth, UNFPA has procedural and financial requirements that may not be suited ideally to Cambodia, but the country office must adhere to them. It did not always do so during CP3. The evaluation team has no doubt about the good intentions of the country office in trying to accommodate both practice and tradition within the Cambodian context, but these intentions must be put aside unless and until the requirements are formally changed. Finally, with respect to the evaluative criteria, the programme‟s relevance to a wide range of Cambodia‟s policies and priorities was one of its major strengths. The country office was con- scious of the need and desire to align its programme with these policies, and it succeeded admira- bly in doing so at the macrolevel as a result of considerable consultation with the RGC and appreciation of its national development plan and sectoral strategies. The country office had mixed results in terms of the effectiveness of its interventions. Notable successes were evident as well as instances in which interventions were less than had been projected (see Table 4). The country office had mixed results with respect to efficiency. Several of the country office‟s procedures were exemplary whereas other practices were less efficient. Impact was the most challenging of the evaluative criteria to assess. This occurred because CP3, like most of UNFPA‟s country programmes, was not designed or implemented in a way that facilitates a valid assessment of impacts.

Key Recommendations

nlike the formative evaluations that occur during the life of a country programme, Utypically during its penultimate year, this evaluation was completed after CP3 had ended. Given this situation, the evaluation team has limited its recommendations to those that it believes can improve the implementation of CP4. The consequence is that many of the recommendations focus on procedural issues that can be addressed easily. On behalf of key stakeholders, all the recommendations were reviewed by several members of the EMC. The following recommendations are considered to be key or priority recommendations and will require attention from both UNFPA/Cambodia and, in one instance, guidance from UNFPA‟s headquarters. The substantive recommendations are deemed to be of the highest priority.

Several of the recommendations, such as those related to indirect costs, the management of OFA, and IPs‟ adherence of UNFPA‟s reporting and procedural requirements, can and should be implemented immediately. Other procedural recommendations will require more time, but are no less important. Nonetheless, the evaluation team recommends attention to the quality of the results rather than the speed with which the latter recommendations are implemented.

Substantive recommendations

1. UNFPA is a well-respected development partner in Cambodia that brings credibility and technical expertise to the thematic areas in which it works. Unfortunately, the resources it has to address these thematic areas are both limited and often dwarfed by other donors. UNFPA‟s influence is minimized when it tries to do much and when its spreads its resources too thinly. The evaluation team thus recommends that UNFPA identify its areas of particular competency and then limit its attention and resources to those few areas in which the country office can demonstrate a comparative advantage over all or most other development partners. Examples of such areas could include EmONC, the development and effective implementation of national policies and strategies related to reproductive health and rights, support for the enhancement of the RGC‟s capability to collect and analyze data related to population and development, and strengthening the role of women in communal decision making. This recommendation is especially germane in an environment in which (a) donors are increasingly requiring UN agencies to demonstrate improvements in people‟s lives and (b) the resources that many donors can provide are increasingly constrained. Wise risk managers consider the possibility that fewer resources will be available in the future. 2. To the extent that capacity development remains a major focus of UNFPA‟s efforts, a strategy for these efforts should be developed in collaboration with the country office‟s key stakeholders. Consideration should be given to the value of capacity building in the Cambo- dian context and whether it is the best, “right” way to achieve shared objectives for RH, PD, and gender. The strategy should be demand driven, based on a rigorous and comprehensive assessment of gaps and needs, and specifically identify the measurable goals and objectives of UNFPA‟s efforts.55 Statements about the need to “build capacity” or to increase the number of people trained are insufficient. The strategy should be explicit in defining what should be achieved in terms of the skills and abilities among the presumed beneficiaries of UNFPA‟s capacity-building efforts. The strategy should also include a way to measure and objectively evaluate the results, including the quality, of any capacity-building initiatives that UNFPA supports either directly or indirectly through its IPs. 3. UNFPA/Cambodia should increase its attention to the evaluation of the initiatives its supports. As this evaluation has noted, some interventions have proceeded without a solid base of evidence to justify them or to measure their results – not just within UNFPA but within much

55 As one member of the EMC noted in his comments on the draft report, “The proposal that UNFPA ought to develop a strategy that is demand driven and also pegged on an actual assessment of the demand in the recipient country is laudable. There should, however, be scope of sufficient flexibility in this, since demands change at short intervals in countries like Cambodia.” 54

of the donor community. UNFPA/Cambodia should consider evaluations of its initiatives for out- of-school youth, the quality and skills of midwives, efforts to improve EmONC facilities, and the consequences and benefits of including mention of gender and population in national and subnational plans and policies. Given the magnitude of the resources devoted to capacity building, an evaluation of capacity building is essential and should be accorded a high priority. Many donors support capacity development in UNFPA‟s three thematic areas, so UNFPA should explore the possibility of joint funding of the retrospective evaluation of capacity development (as well as for any other areas in which there are multiple donors) as well as means to evaluate efforts to develop capacity during CP4. UNFPA should also specify the intended results of each training it sponsors or supports (e.g., consider how the skills and knowledge gained will be used) and then evaluate whether these results have been achieved. Such an assessment should go well beyond reliance on questionnaires distributed at the end of training events.

Procedural recommendations

4. CP3 had a weak results framework, and it became progressively weaker, notably for reproductive health, as the country office added, dropped, and amended indicators for outputs and outcomes. For this reason, the evaluation team recommends that UNFPA /Cambodia revisit and review the results framework for its current country programme. This review should ensure that a credible and logical relationship exists between activities and outputs and between outputs and outcomes. This exercise should: (a) start with a clear understanding of the difference between outputs and outcomes; (b) identify realistic outcomes to be achieved; (c) select the outputs that are necessary to achieve these outcomes; and, (d) select the activities that are necessary to achieve the desired output targets. A related review of CP4 could usefully consider the extent to which the indicators and their targets are specific, measurable, attributable, relevant, and time bound (i.e., SMART). Too many of the indicators and targets in CP3 were neither SMART nor objectively verifiable. When indicators are not SMART, they do not meet the agency‟s expectations or provide a suitable means to monitor progress or to measure results.56 All indicators in CP4 should be objectively verifiable so that different people looking at the same information at the same point in time will reach a similar conclusion. 5. The evaluation team recommends that UNFPA/Cambodia request UNFPA‟s headquarters assistance in identifying a means to verify objectively and empirically the meaning of “high quality reproductive health services.” UNFPA‟s headquarters could usefully provide written guidance on how to measure the existence of such services and an objective means to determine whether such services do or do not exist nationally and within individual health or other facilities that provide reproductive health services. 6. The country office should soon determine and then ensure the availability of the endline data that will be necessary to measure the expected outputs and outcomes for CP4. CP3 used baseline data from the 2005 DHS for several indicators only to find that similar data were

56 DOS, Evaluation Methodology for DOS Oversight Assessments of Country Programmes (2010). 55

not available in the 2010 DHS. If any baseline data for CP4 are based on the 2005 DHS, these baselines (and, if necessary, the corresponding targets) should be updated with data from the 2010 DHS. 7. The country office should ensure IPs‟ compliance with all of the agency‟s substantive and procedural requirements. In particular, the country office should ensure that:  No indirect costs are provided to government departments or ministries;  Indirect costs for NGOs are calculated correctly and based on the UNFPA‟s Guid- ance Note on Indirect and Direct Costs, May 2011);  completed FACE forms allow the agency to relate expenditures to specific activities and that sundry expenses do not exceed 1 percent of total annual expenditures;57  All IPs submit activity progress reports with every FACE form and annual progress reports that assess progress made in achieving results and, when relevant, comment on factors that facilitated or constrained achievement of results included in the AWPs. During CP3, annual work plan monitoring tools from IPs typically focused on activities initiated and completed without a related discussion on how these activities contributed to the outputs in the AWPs. 8. The country office should (a) work with its IPs to improve the processes by which they determine how much advance funding they request to reach an improved balance between these requests and the amounts they actually needs and (b) minimize OFA within the MoH. The country office‟s financial staff is likely to have many ideas about how both objectives can be achieved and should be engaged in the process of addressing these issues.

57 UNFPA‟s draft NEX Audit Guide for UNFPA Staff (November 2011) declares that “budgets in AWPs must include specific activities to be undertaken…with specific budgets for each activity. Using general terms which do not specify the specific activities to be undertaken…is not sufficient and will make it difficult for the auditors to verify that expenditures are related to activities.” 56

Annex 1: Terms of Reference Background:

UNFPA opened its Cambodia Country Office (CO) in 1994 at the request of the Royal Government of Cambodia (RGC) following the UN-sponsored national elections in 1993. Since then UNFPA has steadily increased its technical and financial assistance to Cambodia through its successive programmes of assistance.

UNFPA Cambodia has recently completed the implementation of its third country programme (CP 3) 2006-2010, the goals of which are to support the RGC in its pursuit of meeting the goals of the ICPD POA and MDG's through the implementation of Cambodia’s National Strategic Development Plan (NSDP) and the United Nations Development Framework (UNDAF) 2006-2010.

UNFPA Cambodia focuses its assistance on three main areas: i) Population and Development; ii) Re- productive Health; and iii) Gender. The CP has six outcomes. The Population and Development (PD) component contributes to the national priorities of good governance, including national capacity to use population data for planning and policy making, and the promotion and protection of human rights, while the Gender programme focuses on contributing to institutional mechanisms and socio-cultural practices to promote and protect the rights of women and girls and to advance gender equity. The three outcomes of the Reproductive Health programme are expected to contribute to the national priority of capacity- building and human resource development in the social sectors through (a) a policy environment that promotes reproductive health and reproductive rights; (b) increased access to and utilization of high- quality reproductive services; and (c) increased awareness and empowerment of the population, particularly women and youth, regarding their reproductive rights, including reproductive health services.

UNFPA’s support to Cambodia has increased since 1994. The first country programme (1997-2000) was worth US$16 million, the second country programme (2001-2005) was worth US$26 million and the third/current country programme (2006-2010) was worth US$27 million, nine million of which was mobilized from other resources. The CP3 enjoyed significant financial contributions from the Governments of Australia and Germany toward the two key components of RH and PD. In addition, the UNFPA-initiated Maternal Health Thematic Fund launched in 2008 and UNAIDS’ Unified Budget and Workplan (UBW) support to maximize UN’s response to AIDS also make up part of the CP resources. Implementing partners under CP 3 have included The Ministry of Planning, Ministry of Health, Ministry of Women's Affairs, Ministry of Education, Youth and Sports, Ministry of Interior’s Department of Local Ad- ministration (DOLA), National Committee for Population and Development (Office of The Council of Minis- ters), Khmer Youth Association (KYA), Equal Access (EA), Cambodia Health Education Media Service (CHEMS), Reproductive Health Association of Cambodia (RHAC), CARE, and until 2008, Partners for De- velopment (PFD).

Rationale

The paucity of reliable data in Cambodia has made monitoring and evaluation of previous UNFPA country programmes a challenging task. As a consequence, the monitoring and evaluation of the CP has relied on sectoral or programme reviews rather than a full CP evaluation. In line with the UNFPA Evaluation Policy, which aims to reinforce accountability, oversight and enhance programme effectiveness, this evaluation intends to take stock of performance and actual achievements and provide independent recommendations for adjustments to the five-year Action Plan of the new CP (2011-2015) and basis for future UNFPA support in Cambodia, should such adjustments seem advisable. The evaluation also aims to provide an independent assessment of the future UNFPA country programmes and the continued role for UNFPA support in Cambodia in the context of support the Government in its commit- ments toward attaining the goals of ICPD and the MDGs.

Purpose and scope The evaluation will focus on the outputs and outcomes achieved through the implementation of the CP to date. The evaluation should consider UNFPA’s achievements since January 2006 against intended results and examine the unintended effects of UNFPA’s intervention and the CP’s compliance with UNFPA’s Strategic Plan, as well as its relevance to national priorities and those of the UNDAF.

The evaluation will assess the extent to which the current CP, as implemented, has provided the best possible modalities for reaching the intended objectives, on the basis of results to date. The end- programme evaluation will provide recommendations for the five-year action plan of the CP 4 and subsequent annual work plans and provide support for the design of future UNFPA operations in Cam- bodia.

The scope of the evaluation will include an examination of the effectiveness/coherence, relevance, efficiency, impact and sustainability of the current CP. In addition, the evaluation should provide feedback regarding the contribution of CP to RGC’s efforts to attain the goals of the ICPD and Cambo- dian Millennium Development Goals (CMDGs).

Audience (Use and User) The main audience of the evaluation is the UNFPA CO and RGC counterparts. Both entities will benefit from findings, conclusions and recommendations aimed at improving CP 4 during its planning and implementation, and establishing guidance for future programming. In addition, the UNFPA APRO/DOS will benefit from the evaluation process and resulting report. Ac- cordingly, the report will cover relevant information on progress in terms of stated objectives, feedback on the sustainability of current programme of assistance, and advice on the nature of future programme activities.

Key Cambodia Country Programme Evaluation Issues The evaluation will address UNFPA’s leadership and management in terms of human resources, financial management and systems. The following key questions for the evaluation apply to all areas of UNFPA’s work in Cambodia.

Relevance

The evaluation will assess the CP’s external coherence/relevance with the RGC’s National Strategic De- velopment Plan 2006-2010, the UNDAF and other key planning documents. In terms of internal coherence, the CP will be examined with reference to the CPAP Tracking Tool.

 How relevant was the CP to: a) the needs of target population, b) national needs and priorities of Cambodia; c) focus and outputs and outcomes of the CP supported by UNFPA in line with the organizational and regional strategies manifested in the Strategic Plan, ICPD programme of Ac- tion and Millennium Development Goals?  How well did the CP’s design respond to and reflect the current and future challenges and trends in the Cambodian context?  Were gender, equity and human rights dimensions effectively incorporated into the CP’s design?  Is there synergy or complementarity between UNFPA’s intervention and that of other development partners?

Effectiveness The CP evaluation will examine the degree of achievement of the core programme outputs and progress made toward achieving programme outputs and outcomes given the changes in the global and national policy environment and identify reasons for this progress and/or discrepancies between plans and achievements.  Were the CP’s intended outputs and outcomes achieved? If so, to what degree?  How did inputs and activities lead to outputs and outcomes?  What was the intervention coverage – were the planned geographic areas and target groups successfully reached?  What were the constraining and facilitating factors and the influence of context on the achievement of results?  How complimentary and well-coordinated were the RH, PD and Gender components of the CP?  How adequate and effective were the monitoring and evaluation tools of the programme, including the baseline and end line survey instruments;

Efficiency  Were the outputs achieved reasonable for the resources spent?  What was the quality of output and outcomes achieved in relation to the expenditures incurred and resources used?  What was the timeliness of inputs; timeliness of outputs?

Impact  What was the qualitative and quantitative impact of programme activities, with a particular focus on the impact on the programme beneficiaries?  Has there been an impact on the gender and/or human rights dimensions? In particular, has there been an impact on government attitudes to gender and GBV in particular? What has been the specific impact on minorities, marginalised and vulnerable groups? Was there any unexpected impact, both negative and positive?

Other issues for consideration The Cambodian context is characterised by a plethora of development partners with different agendas and mandates. In this context it is important that the evaluation explore the extent to which the CP has responded to the aid effectiveness agenda and sector reform initiatives and identify future

opportunities and entry points to advance the CP through enhanced harmonization, alignment and national ownership.

Within the current institutional and policy context, advocacy on population issues is a challenging issue and the evaluation should examine the effectiveness of current approaches, focusing on support through the National Committee for Population and Development and make recommendations regarding the appropriate future mechanisms for support to this area of programming.

In addition to the evaluative criteria noted above, the evaluation team shall also assess the extent to which UNFPA’s programs have integrated gender as a cross-cutting theme and promoted gender equity and gender sensitivity.

Methodology The evaluation will be based on a) the review of documents including strategic plan/Multi-year Funding Framework, UNDAF, Country Programme Documents, Country Action Plan, AWPs, standard progress reports, annual country office reports, MTR report, National Strategic Development Plan 2006-2010 and its MTR reports sector plans and their progress reports; b) site visits to UNFPA targeted areas (x number of 14 provinces, to be determined in the inception report); and c) interviews with stakeholders including national counterparts, implementing partners and development partners and target beneficiaries. It is expected that the selection of provinces will be done in consultation with stakeholders and based on the agreed criteria. The collection of evaluation data will be carried out through a variety of techniques that will range from direct observation to informal and semi-structured interviews and focus/reference groups, where feasible. The analysis will build on triangulating information obtained from various stakeholders’ views as well as with secondary data and documentation reviewed by the team.

The evaluation will follow the principles of the UN Evaluation Group’s norms and standards (in particular with regard to independence, objectiveness, impartiality and inclusiveness) and will be guided by the UN ethics guidelines for evaluators. Evaluations for UNFPA are to be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, especially women, children, and members of other vulnerable or disadvantaged groups, and in accordance with the UNEG’s Ethical

Guidelines for Evaluation, at www.unevaluation.org/ethicalguidelines.

Work Schedule and arrangements The evaluation is expected to be completed within three months. The first week will include a briefing of the evaluation team at the CO and internal discussions among the evaluators on methodologies and tools. The evaluation team will undertake background reading and research (including policies, strategy and programme documents, AWP, progress reports annual country office reports), and prepare an inception report to agree on design of the tools and methodology to conduct the evaluation before travel- ling to field.

The in-country evaluation process will commence in the capital, Phnom Penh, for 8-10 days, and then proceed on a field visit lasting 10 days (in selected provinces). The focus will be on visiting health centers, referral hospitals, commune councils, which will provide an opportunity for interaction with target beneficiaries as well as other relevant stakeholders.

The team will return to Phnom Penh for a final 3-4 days for final interviews and for the drafting and presentation of a draft report, which will be disseminated to stakeholders for feedback and comments (approximately 2 weeks). A debriefing meeting will be organized for consultation with relevant 60

stakeholders and disseminations of findings and recommendations. Subsequently, the International Con- sultant/Team Leader will then produce a full final report, with inputs from team members. The report will then be finalized and submitted to UNFPA. The final product should be submitted no later than the 12th week from the commencement of the evaluation.

The evaluation team shall have no conflict of interest or any connection to the design, planning, or implementation of the current or upcoming country programme. Any such conflict should be brought to UNFPA’s attention immediately.

UNFPA Support Overall guidance will be provided by the UNFPA Representative and technical supervision by the As- sistant Representative. UNFPA CO will provide the evaluation team with all the necessary documents and reports and refer it to web-based materials. The Research Assistant and Librarian will be assigned as the evaluation team’s counterpart to provide support in terms of gathering documentation as required. UNFPA will liaise with the UNFPA programme managers to ensure that the thematic component reports are provided to the evaluation team as these are critical inputs to the programme evaluation. UNFPA management and staff will make themselves available for interviews and technical assistance as appropriate. The CO will also provide necessary logistical support in terms of providing space for meetings, assistance in making appointments and arranging travel and site visits. Use of office space and com- puter equipment may be provided if needed but this will need to be discussed further due to limited availability.

The UNFPA Asia and Pacific Regional Office (APRO) will provide support at several stages. At the pre- paratory stage, the tools for assessment will be reviewed and approved by the Country Office (in consultation with APRO). Consultations will be undertaken during the course of the evaluation/key evaluation stages to provide inputs and validate findings; provide inputs during the briefing and debriefing sessions; and review the draft reports. APRO team will provide a combination of on-site and off-site support.

Team composition The evaluation will be carried out by a team consisting of an International Consultant/Team Leader (ICTL) and three national consultants. It is envisaged that the ICTL will have technical expertise in at least one area among the programme component of Reproductive Health, Population and Development, or Gender while the other areas will be covered by the other team members. The national consultants will collect information, conduct desk reviews based on the assessment framework, developed by the ICTL, and undertake/assist with interviews and site visits with the ICTL. The ICTL shall be responsible for ensuring the timely completion of all deliverables, including a final evaluation that meets or exceeds all of UNFPA’s Evaluation Quality Standards, which the country office will provide to the evaluation team.

The ICTL will be an international expert in monitoring and evaluation of development programmes with:  At least 15 years of experience in the field of programme development and management  Familiarity with UNFPA’s work and mandate  Good knowledge of effective capacity development and aid effectiveness/swaps  Knowledge and experience in conducting evaluation  Experience in the Asia and Pacific region preferred  Good management skills and ability to work with multi-disciplinary and multi-cultural teams  Excellent writing skills in English

The national consultants will have the following criteria (Please refer to the ToR for National Con- sultant):  At least five years experience in conducting research and/or evaluation  Good background in at least one of the programme components – RH, population and development or gender)  Familiarity with UNFPA’s work and mandate  Good analytical ability and writing skills in English  Strong interpersonal skills and ability to work in a multi-cultural team

Specific Outputs of the International Consultant (Deliverables) The ICTL will assume overall coordination and evaluation of the UNFPA CP evaluation and will provide appropriate guidance and support to the National Consultant, accompanying her/him to all stakeholder meetings if necessary and providing other support as required. The ICTL in consultation with the Evaluation Manager (UNFPA Assistant Representative) and inputs from the National Consultant will be responsible for the following tangible outputs at an acceptable level of quality and in a timely manner:  Inception Report  Draft the CPE Report and compile inputs from the National Consultants following the standard evaluation approach and methodology for the CPE.  Final CPE report  Evaluation Brief

1. Inception Report: The ICTL shall commence the assignment on 26 September 2011, and within one week of award of contract, the evaluation team shall submit an electronic copy of a draft inception report to UNFPA. The inception report should include a work plan specifying methodological and organizational aspects of its work, including any provisions for needed meetings, interviews, travel, formal events of consultations etc., as well as the necessary working days foreseen for key components of the work plan. The inception report provides an opportunity for UNFPA and the evaluation team to ensure that their interpretations of the TOR are mutually consistent. The manager will review and approve the report, which will serve as an agreement between UNFPA and the evaluation team about how the evaluation will be conducted. This inception report shall:

- Explain the evaluation team’s understanding of what is being evaluated and why;

- Describe the team’s strategy for ensuring the evaluation’s utility and applicability to the needs of UNFPA and those of key stakeholders;*

- Describe the evaluation team’s plans to engage and involve these stakeholders in the design (e.g., questions, objectives, methods, data-collection instruments), data collection, data analysis, and development of recommendations;*

- Explain how the evaluation questions will be addressed with respect to all evaluative criteria indicated above by way of proposed methods, evaluation designs, sampling plans, proposed sources of data, and data-collection procedures;*

Note: The evaluation team is encouraged to suggest refinements to the TOR and to propose creative or cost- or time-saving approaches to the evaluation and explain their anticipated value.

- For each of the evaluative criteria, describe the measurable performance indicators or standards of performance that will be used to assess progress toward the attainment of results, including outcomes;* 62

- Discuss (a) the limitations of the proposed methods and approaches, including sampling, with respect to the ability of the evaluation team to attribute results observed to UNFPA’s efforts especially when there is no consideration of a valid counterfactual and (b) what will be done to minimize the possible biases and effects of these limitations;*

- Explain the team’s procedures for ensuring quality control for all deliverables;

- Explain the team’s procedures to ensure informed consent among all people to be interviewed or surveyed and confidentiality and privacy during and after discussion of sensitive issues with beneficiaries or members of the public;*

- Explain how the evaluation will reflect attention to and mainstreaming of gender concerns and identify the member of the evaluation team who will be responsible for doing so;*

- Indicate familiarity with and agreement to adhere to (a) the requirements of the Standards for Evaluation in the UN System, especially standards 4.1 through 4.18 and (b) UNFPA’s Evaluation Quality Standards, which will be provided to the evaluation team; and,

- Provide a proposed schedule of tasks, activities, and deliverables consistent with this TOR. Note: Items marked with an asterisk should also be discussed in the evaluation report.

2. Draft evaluation report: The ICTL shall draft and submit an electronic copy of a draft evaluation report to UNFPA no later than 21 October 2011. The draft report should be thoroughly copy edited to ensure that comments from the UNFPA and other stakeholders on content, presentation, language, and structure can be reduced to a minimum.

3. Final report: After UNFPA’s and stakeholders’ review of the draft report, the evaluation manager of UNFPA office will provide written comments to the evaluation team. Based on these comments, the team shall correct all factual errors and inaccuracies and make changes related to the report’s structure, consistency, analytical rigor, validity of evidence, and requirements in the TOR. The team will not be required to make changes to conclusions and recommendations unless they are regarded as qualitative improvements. After making the necessary changes, the evaluation team will submit a revised draft evaluation report, which may lead to further comments from UNFPA. After the second round of review and, if necessary, further revision to the draft evaluation report, the evaluation team can then submit the final report pending UNFPA’s approval.

The final report will follow an agreed outline and format based UNFPA Evaluation Guidelines and should not exceed 40 pages, excluding annexes. The report should capture and describe UNFPA involvement of key stakeholders in the CP. It should include a clear, concise, stand-alone executive summary that includes:

- A summary of evaluation findings

- A summary of actionable recommendations

- Recommendations tracking matrix/management response matrix (maximum 2000 words, with the management responses). Ideally, the number of key recommendations should not exceed a dozen, although additional subsidiary recommendations may be contained in the full final report.

At a minimum, the final report shall contain the following annexes:

- List of persons interviewed (if confidentiality permits) and sites visited; - Data-collection instruments (copies of surveys, questionnaires, etc.);

- A bibliography or list of references; and,

- The TOR for the evaluation.

UNFPA will provide the evaluation team with a recommended outline for the final report. All deliverables must be in English.

As noted above, the deadline to receive the final evaluation report is 10 weeks after the commencement of the contract. In addition to the above, a short (max. 5 pages) note should be produced for de-briefing purposes and provided to the UNFPA Representative at debriefing.

4. Evaluation brief: a brief summary report not exceeding two pages shall provide a quick overview of the overall country programme evaluation may contain the following:

- Context/background information

- Development challenges

- Main conclusion: o capturing key choices in programming and strategic focus o micro linkages in programming o implementation modality

- Key recommendations

Tentative evaluation schedule

Activities Timeframe Place Responsible parties

Desk review 26 Sep – 02 Oct 2011 Home-based All team members

Drafting and finalizing the Incep- TBC Home-based Primary responsibility tion Report and developing the lies with the Interna- draft work-plan and schedule tional Consult- ant/Team Leader, but the National Consultants should provide their input and support Designing the detailed evaluation TBC Home-based International scope, methodology, and ap- Consultant/Team proach Leader

Finalizing the field work TBC Cambodia Primary responsibility schedule, evaluation lies with the Interna- methodology, and logistics tional Consult- ant/Team Leader, but the National Consultants should

provide their input and support In-country mission 03 October – 12 Nov Cambodia All team members 2011

Analyzing the consolidated data TBC Cambodia All team members set from key informant interviews Presentation of preliminary TBC Cambodia All team members (with findings and draft recom- presentation to be done mendations to senior man- by the International agement and reference Consultant/Team group Leader) Finalization of TBC Home-based International First draft of full evaluation Consultant/Team report Leader Finalization of second draft, fol- No later than the 3rd Home-based International lowing feedback from UNFPA week of November 2011 Consultant/Team and counterparts Leader

Consultations UNFPA emphasizes meaningful stakeholder involvement in its evaluations. Such involvement can include participating in design (questions/objectives, methods, data collection instruments), collecting data, analyzing data, or developing recommendations, and other roles as appropriate for the evaluation. Participation in surveys or interviews is not the same as meaningful and effective stakeholder involvement.

Payment of Consulting Fees Payment of the Consulting Team will be made in three tranches, as follows:  First Payment (20 percent of total) – Upon UNFPA’s approval of inception report (detailed evaluation design)  Second payment (40 percent of total) – Upon the submission of the first draft; and  Third payment (40 percent of total) – Upon UNFPA’s acceptance of the final evaluation report. Annexes 1. UNFPA Department of Oversight (DOS) evaluation quality standards 2. List of key documents /background materials for CP evaluation

Annex 2: Methodological Framework for the Evaluation

Evaluation Key evaluation questions (drawn from TOR) Performance Indicators Data Sources Data Collection Methods Objective Assess Pro- How relevant was the CP to: (a) the needs of target Degree of concurrence of CP3 CPAP; national plans and Document review; key gramme population; (b) national needs and priorities of with the needs of the target popula- policies; key informants informant interviews Relevance Cambodia; (c) focus and outputs and outcomes of the CP tion, Cambodia‟s national needs supported by UNFPA in line with the organizational and priorities, and the agency‟s and regional strategies manifested in UNFPA‟s Stra- strategic plan, the ICPD Pro- tegic Plan, the ICPD Programme of Action, and the gramme of Action, and the MDGs MDGs? How well did the CP‟s design respond to and re- Degree of concurrence of CPAP CPAP and NSDP Document review flect the current and future challenges and trends in with UNDAF and Cambodia‟s the Cambodian context? NSDP Were gender, equity, and human rights dimensions Analysis of CPAP. Judgment of CPAP Document review effectively incorporated into the CP‟s design? effectiveness will necessarily be subjective; addressing this question requires only a “yes” or “no” answer. Is there synergy or complementarity between Documentation and analysis of key Key stakeholders Key informant interviews UNFPA‟s intervention and that of other develop- stakeholder opinions. Addressing with UNFPA staff and ment partners? this question requires only a “yes” other UN and donor agen- or “no” answer. cies; document review Assess Pro- Were the CP‟s intended outputs and outcomes Level of achievements against tar- CPAP, AWP monitoring Review of programme gramme achieved? If so, to what degree? gets (as outlined in CPAP Results tools, annual standard achievements against Effective- and Resources Framework). progress reports, and CPAP and outcome and ness Degree to which outputs and out- country office annual re- output indicators; key comes meet acceptable quality port; key stakeholders at informant interviews standards. national, and subnational levels How did inputs and activities lead to outputs and out- Documentation of UNFPA imple- Same as above Same as above comes? mentation processes. Documenta- tion of policy and changes in capacity achieved. Documentation and analysis of key stakeholder opinions on implications of changes for CP4 What was the intervention coverage – were the Comparison of priority provinces CPAP, AWPs, and coun- Document review; key planned geographic areas and target groups success- identified in the CPAP with infor- try office annual reports; informant interviews fully reached? mation in AWPs and country of- key stakeholders fice annual reports What were the constraining and facilitating factors Documentation and analysis of Key informants; country Key informant interviews and the influence of context on the achievement of factors affecting implementation. office annual reports, an- results? This question does not require the nual standard progress evaluation to make a judgment reports related to effectiveness but rather to identify and describe factors. How complimentary and well-coordinated were the Documented evidence of comple- CPAP; key stakeholders Key informant interviews, CP3‟s RH, PD and gender components? mentarity between programme including UNFPA staff components. How adequate and effective were the programme‟s Comparison of the country office‟s UNFPA Policy and Pro- Document review monitoring and evaluation, including the baseline and M&E system with UNFPA re- cedures Manual and; an- end line survey instruments? quirements and expectations for nual standard progress results-based management and reports. Interviews with reporting. Consideration of wheth- country office staff

er CPAP indicators are SMART. Adherence to international best practices for data collection instruments. Assess Pro- Were the outputs achieved reasonable for the re- Documentation and analysis of Key stakeholders; Document review; key gramme sources spent? stakeholder opinions on efficiency documentation of pro- informant interviews Efficiency of utilization of programme inputs gramme inputs by (human, financial, technical) What was the quality of output and outcomes achieved Same as above Same as above Same as above in relation to the expenditures incurred and resources used?

What was the timeliness of inputs; timeliness of For inputs, concordance between AWPs Document review outputs? dates of annual approval and funding of AWPs and UNFPA‟s deadline for their approval. For outputs, concordance between dates in AWPs and actual completion dates. Assess Pro- What was the qualitative and quantitative impact of Documentation of key programme Key stakeholders; Key informant interviews; gramme programme activities, with a particular focus on the achievements. Documentation and country office annual re- document review Impact impact on the programme beneficiaries? analysis of stakeholder opinions on ports programme achievements, strengths, shortcomings, intended and unintended outcomes. Has there been an impact on the gender and/or hu- Same as above Same as above Same as above man rights dimensions? In particular, has there been an impact on government attitudes to gender and gender-based violence in particular? What has been the specific impact on minorities, marginalized and vulnerable groups? Was there any unexpected impact, both negative and positive? Assess man- To what extent did management support/hinder the Documentation and analysis of Key stakeholders Key informant interviews agement for progress of the implementation of the country pro- stakeholder opinions on quality of efficient gramme? management support programme What were the strengths and weaknesses of UNFPA‟s Degree of concurrence manage- Key stakeholders; Key informant interviews, delivery programme management? How appropriate was the ment support with management UNFPA country pro- including UNFPA staff; programme management structure? What were the best practice. gramme organogram; job document review major management issues and how were these re- Appropriate allocation of human descriptions of UNFPA solved? How could management have been im- resources for programme needs country staff proved?

Annex 3: Documents Reviewed for the Evaluation

In addition to the reports and documents already mentioned within the report, the evaluation team reviewed the following: UNFPA/Cambodia AWPs AWPMTs CPAP, 2006-2010, 2011-2015 FACE forms for all IPs Letters of Understanding COAR, 2006-2010 Annual Standard Progress Reports UN Resident Coordinator UNDAF Joint Annual Reviews Cambodian National Council for Congress Report, the Achievement of the 2010 and the Women (February 2011) Action Plan for 2011 Council of Ministers (December Cambodia National Population Policy, Review Report 2010) 2010 by National Committee for Population and Devel- opment (NCPD) UNFPA (October 2008) Mid Term Review Final Repot, Third Country Pro- gramme of Assistance to the RGC 2006-2010. UN Resident Coordinator UNDAF Joint Annual Reviews Cambodian National Council for Congress Report, the Achievement of the 2010 and the Women (February 2011) Action Plan for 2011 Council of Ministers (December Cambodia National Population Policy, Review Report 2010) 2010 by National Committee for Population and Devel- opment (NCPD) Damrei Research and Consulting Health Facilities Assessment HSSP1 End-of-Project As- (April 2010) sessment HLSP (Health and Life Sciences Health Sector Review (2003-2007) Cambodia Partnership) (August 2007) MoEYS, Interdepartmental Com- Strategic Plan and Operational Plan for HIV 2008-2012 mittee for HIV and AIDS (May 2008) MoEYS (December 2005) Education Strategic Plan 2006–2010 MoEYS(January 2009) Mid-term Review Report of the Education Strategic Plan and Education Sector Support Program 2006-2010 Im- plementation. MoH (August 2002) Health Strategic Plan 2003–2007 MoH, National Reproductive Safe Motherhood Clinical Management Protocols Health Program (July 2010) MoH Cambodia Emergency, Obstetric and Newborn Care Im- provement Plan (2010-2015) MoH 2010 Annual Performance Monitoring Report MoH 2009 Annual Performance Monitoring Report

MoH (March 2010) Needs Assessment of Cambodian Public Midwifery Training Institutions and Clinical Practice Site. MoH (March 2010) Health Congress Report 2010 MoH (August 2010) Health Sector Support Project 1 – End of Project Assess- ment Report Ministry of Interior (August 2008) Organic Law (2008): Law on Administrative Manage- ment of the Capital, Provinces, Municipalities, Districts and Khans Ministry of Planning (September Training Needs Assessment and Plan for the National In- 2007) stitute of Statistics of Cambodia. Ministry of Planning (July 2007) Ministry of Planning Strategic Plan (MPSP), 2006–2010 Ministry of Planning (January Annual Operational Plan 2011 of the Ministry of Plan- 2011) ning Strategic Plan Update 2009–2013 MoWA (October 2009) 2009 Follow-up Survey, Violence Against Women Final Study Report (2009) MoWA (July 2009) National Action Plan to Prevent Violence on Women MoWA (September 2009) Five Year Strategic Plan 2009–2013, Neary Rattanak III National AIDs Authority National Strategic Plan for a Comprehensive and Multi- sectoral Response to HIV/AIDs 2006–2010 National Institute of Statistics, Cambodia Demographic and Health Survey 2000 Ministry of Planning and Direc- torate General for Health, MoH (January 2001) National Institute of Statistics, Cambodia Demographic and Health Survey 2005 Ministry of Planning and Direc- torate General for Health, MoH (December 2006) National Institute of Statistics, Cambodia Demographic and Health Survey 2010 Ministry of Planning and Direc- torate General for Health, MoH (March 2011) National Institute of Statistics, Statistics Master Plan (SMP) 2006-2015. Ministry of Planning National Institute of Statistics Primary Statistics (December 2008) National Institute of Statistics, General Population Census of Cambodia 1998, Final Ministry of Planning (August Census Results (2nd Edition) 2002) National Institute of Statistics, General Population Census of Cambodia 2008, Final Ministry of Planning (August Census Results 2009) National Institute of Public Health Cambodia Demographic and Health Survey 2010 and National Institute of Statistics (September 2011) National Institute of Public Health Cambodia Demographic and Health Survey 2005 and National Institute of Statistics

(December 2006) National Institute of Public Health Cambodia Demographic and Health Survey 2000 and National Institute of Statistics (June 2001) National Institute of Public National Emergency Obstetric and Newborn Care As- Health, MoH (May 2009) sessment in Cambodia Provincial Development Plans Banteay Mean Chey, Kampong Cham, Kampong (2006-2010) Chhnang, Kampong Thom, Kratie, Siem Reap, Stung Treng, Provincial Development Plans Banteay Mean Chey, Kampong Thom, Kampong Cham, (2011-2015) Kampong Chhnang, Kratie, Rattanakiri, Siem Reap, Stung Treng Provincial Investment Plans Banteay Mean Chey, Kampong Cham, Kampong Thom, (2011-2013) Rattanakiri RGC (May 2010) Fourth and Fifth National Report on the Implementation of the Convention on the Elimination of All Forms of Discrimination Against Women RGC, Prepared by Ministry of Achieving Cambodia‟s Millennium Development Goals Planning (2010) (Update 2010)

RGC, Ministry of Planning (Oc- Achieving the Cambodia Millennium Development Goals tober 2005) Update 2005 RGC (June 2010) National Strategic Development Plan (Update 2009– 2013) RGC National Strategic Development Plan (2006–2013) RGC (draft 2011) Mid-term Review 2011, National Strategic Development Plan, Update 2009–2013 RGC (May 2010) THE CAMBODIA AID EFFECTIVENESS REPORT 2010, Prepared by the Cambodian Rehabilitation and Development Board of the Council for the Development of Cambodia for the Third Cambodia Development Cooperation Forum (CDCF), 2-3 June 2010 VBNK and RBMG (Result-based Cambodia Country Study Report, Phase Two Evaluation Management Group, (October of the Paris Declaration, Submitted to: Chhieng Yanara, 2010), Secretary General of Cambodian Rehabilitation and De- velopment Board, Council for the Development of Cam- bodia Vuthy Horng (November 2010) Review of 2006/2010 CIPs of UNFPA-supported Areas to Assess the Gender Sensitivity and the Extent of the In- tegration of Key Population, Reproductive Health, Gen- der, HIV/AIDS and Youth Issues. Vuthy Horng (December 2007) Review of 2006/2007 of UNFPA Supported Areas to As- sess the Gender Sensitivity and the Extent of the Integra- tion of Key Population, Reproductive Health, Gender, HIV/AIDS and Youth Issues.

Annex 4: List of People Interviewed Members of the EMC are by an * next to their names. Name Position Institution Contact Address Government agencies Ms. Keth Mardi Director of Legal Protection Ministry of Women‟s 012 433 681 Department Affairs (MoWA) Ms. Kim Siphat Deputy Director of Gender MoWA 012 892 497 Equity Ms. Hou Nirmita Director of Women and MoWA 012 563 574 Health Department Ms. Nhean Sochetra Director of Gender Equality MoWA 012 758 986 Department Mr. Pen Sareoun Director of School Health Ministry of Education 011 873337 Department Youth and Sports (MoEYS) Mr. Kim Sanh Deputy Director of School MoEYS 077 383 438 Health Department, and senior programme officer of ICHAD Technical Secretar- iat Dr. Yung Kunthearith* Chief of Technical Bureau MoEYS 012 905800 of School Health Depart- ment and senior programme officer of ICHAD Technical Secretariat Ms. Chhay Sveng Chea President Cambodia Midwifery Ath Association, Ministry of Health (MoH) Dr. Keat Phuon Director MoH Prof.Tung Rathavy* Deputy Director National Maternal Child Health Center, MoH Ms. Ing Rada President MoH Dr. Mey Sambo Director Personnel Department, 012 810 505 MoH Dr. Loveasna Kiri Director Department of Planning 012 824544 and Health Information, MoH Mr. Poch Sovanndy* Deputy Director General, Ministry of Planning 012 931 264 General Directorate of Planning Mr. Toun Thavrak* Director General, General Ministry of Planning 017 466 772 Directorate of Planning Mr. Meng Kimhour Deputy Director General, Ministry of Planning 016 824 238 National Institute of Statis- tics Ms. Chamroeun Katika* Deputy Secretary General, Council Minister 012 984 798 National Committee on Population and Develop- ment Mr. Nhem Maneth Department Director Council of Ministers 017 300 200 Mr. Yin Malyna* Director, DOLA Ministry of Interior 011 682 222

Mr. Yam Mengsean Deputy Director, DOLA Ministry of Interior 011 256 589 Ms. Hang Lina Deputy Director General, National Institute of 012 723 107 National Institute of Statis- Statistics, Ministry of tics Planning H.E. Mr. San Sythan* Director General National Institute of 016 832 762 Statistics, Ministry of Planning H.E. Mr. Ouk Damry Member of Parliament, and National Assembly, 12 9558 Commission of Legislation Phnom Penh & Justice, Secretary General of Cambodia Association of Parliamentarians on Popula- tion and Development Development partners Mr. Timothy Johnston Senior Health Specialist World Bank 023 217 301 Ms. Begona Castro Access to Justice for Wom- Deutsche Gesellschaft 012 916 467 Vaquez en Advisor für Internationale Zusammenarbeit Dr. Chhom Rada Deputy Programme Coordi- Deutsche Gesellschaft nator, Social Health Protec- für Internationale tion Programme, Zusammenarbeit Mr. Ek Thinavuth Local Governance for Child UNICEF 012 809 686 Rights Officer 023 426 214 Ms. Judith Leveillee Chief , Local Governance UNICEF 012 899 641 for Child Rights Mr. Douglas Broderick United Nations Resident UNDP Coordinator Ms. Sok Chanchorvy* Governance Cluster Team UNDP Leader Ms. Sieng Leakhena Programme Manager UNDP Ms. Pen Rany Gender and Human Rights UNDP Specialist Ms. Niamh (Neev) Coordinator and Programme UN Women 077 599 784 O‟Grady Advisor 023 216 167 Dr. Chris Vickery Health Adviser AusAID 023 213 470 078 723 397 Ms. Tara Milani Deputy Director, Office of US Agency for Interna- Public Health and Education tional Development Ms. Monique Mosolf Director, Office of Public US Agency for Interna- Health and Education tional Development Dr. Sek Sopheanarith Development Assistance Office of Public Health Specialist and Education, US Agency for International Development Ms. Kim Sokuntheary Executive Director CHEMS 012 285 678 023 880 724 (office) Ms. Pok Nanda Executive Director Women for Prosperity 012 420 093 Ms. Ros Sopheap Executive Director Gender and Develop- 012 627 857 ment/Cambodia Mr. Seng Sopheap Program Director Equal Access 077 362 958 023 996 828 Mr. Ou Ratanak Executive Director People Health Devel- 017 855 969 opment Association Ms. Khun Sophea Gender Technical Advisor CARE Cambodia 012 892 130

Ms. Eart Pysal Quality Learning Facilitator CARE Cambodia 078 877 456

Dr. Var Chivorn Associate Executive Direc- RHAC 012 98 22 94 tor Ms. Sun Chan Sen President KYA 017 788955 UNFPA Dr. Marc Derveeuw Country Representative UNFPA Ms. Sarah Knibbs Deputy Country Representa- UNFPA tive Mr. Tum May* Assistant Country Repre- UNFPA sentative Ms. Sam Sochea Programme Officer UNFPA Dr. Sok Sokun* Former RH Programme UNFPA 012 992847 Manager (and currently deputy country representative, PSI/Cambodia) Mr. Yi Soktha PD Manager UNFPA 012 809 192 Dr. Vong Sathiarany Reproductive Health Spe- UNFPA 012 331 905 cialist Dr. Muong Sopha Programme Associate for UNFPA 092 777104 EmONC Dr. Chong Vandara Youth and Sexual and Re- UNFPA productive Health,/Life Skills, and HIV Prevention Mr. Pon Rieng Finance Officer UNFPA Subnational level Battambang Province Dr. Hen Sidara Maternal and Child Health Mong Russey Opera- Chief tional District (OD) Referral Hospital Ms. Min Yim Midwife Prek Chik Health Center (HC), OD Mong Rus- sey, Battambang Chheut Vong Dy Midwife Prek Chik HC, OD Mong Russey, Battam- bang Ms. Y Bunreoun Maternal and Child Health Provincial Health De- 012 320992 Chief partment (PHD), Bat- tambang Dr. Kak Seila Provincial Hospital Director Battambang Referral Hospital Ms. Suon Chan Lak Midwives Chief Battambang Referral Hospital Dr. Ngor Sothy Director of Regional Train- Regional Training Cen- 012 859656 ing Center, Battambang ter, Battambang Mr. Top Buntien Deputy Director, Planning Department of Planning Tel: 012 523 647 and Investment Division and Investment, Bat- tambang 1. Ms. Long Vy, Vice Chairman, Commune Sampov Loun Com- 1. Tel: 092 79 0611 2. Set Thay, Committee on Women and mune, Sampov Loun 2. Tel: 012 319 237 3. Muth Sarim, Children (CCWC); Com- District, Battambang 3. Tel: 097 57 89678 mune Clerk CCWC; Mem- ber of CCWC and Village Health Support Group Mr. Khem Sophat former district facilitator Ta Sda Commune, Tel: 092 27 37 37

Sampov Loun District 1. Mr. Deurk El, Commune Chief Ta Sda Commune, 1. Tel: 077 57 87 67 2. Khun Phern, First Vice Chief Sampov Loun district 2. Tel: 017 704767 3. Tep Pheurn, Councilor 3. Tel: 097 77 63 664 4. Ms. Bun Ly, Ta Sda Village Head Tel: 012 27 16 06 5. Ms. Chhe Chamnan Deputy Head of HC Tel: 012 51 56 77

1. Mr. Nop Neang Commune Chief Chrey Seima Commune, 1. Tel: 092 624 647 2. Thean Bunthoeun, Clerk Sampov Loun district 2. Tel: 097 646 4848 3. Ms. Pol Hak, First Deputy Chief 3. 08888 93 608 4. Sin Seun, Councilor 4. 089 69 17 75

1. Mr. Long Kimhang 1st Deputy Commune Chief Chak Krey Commune, 097 45 07 067 2. Mr. Keo Sam Arn Councilor Phnom Preuk district, 3. Ms Pann Yuth Councilor Battambang 097 81 14 914 4. Ms. La Phat Chairwoman of CCWC 088 91 80 633 1. Ms. Um Sokha Chairwoman of CCWC Phnom Preuk commune, 012 41 14 32 2. Mr. Chen Cham Commune Clerk Phnom Preuk District 012 300 438 3. Mr. Men Seb Commune Chief 092 720 787 4. Ms. Matt Thon Councilor 097 85 28 571 5. Mr. Sam Savat 1st Deputy Commune Chief 092 1486 53 1. Mr. Nob Sat Commune Chief Pich Chenda Commune, 092 952 067 2. Ms. Mey Dy First Deputy Chief Phnom Preuk district 012 387 610 3. Mr. De Vanneang Clerk 017 77 33 24 4. Mr. Cheng Sophal 2nd Deputy Chief 012 69 38 62 1. Ms. Siv Seneh Community Development UNICEF, Battambang Tel: 012 680 607 Officer Tel: 012 998 827 2. Mr. Bun Thun Education Officer Banteay Mean Chey Province Ms. Kuy Saveoun Maternal and Child Health PHD, Banteay Mean 012 619670 Officer Chey Dr. Keo Sophak Tra Director PHD, Banteay Mean Chey Dr. Uk Mony OD Director Mongkul Borey OD Banteay Mean Chey Ms. Kuth Veoun Maternal and Child Health Mongkul Borey OD Chief Banteay Mean Chey Ms. Sien Kolap Midwife Oprasath HC, Mongkul Borey OD, Banteay Mean Chey Ms. Pel Sinuon Midwife Oprasath HC, Mongkul Borey OD, Banteay Mean Chey Mr. Chan Borath Health Center Chief Oprasath HC, Mongkul Borey OD, Banteay Mean Chey Ms. Uk Vandy Maternal and Child Health Preah Net Preah OD, Chief Banteay Mean Chey Ms. Tak Sorn Maternal and Child Health Preah Net Preah OD, Officer Banteay Mean Chey Ms. Duong Neay Raksar Maternal and Child Health Kralanh OD, Banteay Chief Mean Chey Chonleas Dai HC, Ms. Oeu Ra Midwife Kralanh OD, Banteay

Mean Chey Ms. Boy Chamreung Pregnant woman Chonleas Dai HC, Kralanh OD, Banteay Mean Chey Kampong Cham Province Ms. Kim Sokunny Maternal and Child Health Thbong Khmum OD, Chief Kampong Cham Ms. Em Simeou CBD Agent Chup 2 village, Chup commune, Thbong Khmum OD, Kampong Cham Mr. Tuon Sokhom Deputy Director Provincial Department 012 735693 of Education, Youth and Sports, Kampong Cham Mr. Prum Chheung Vice Chief of Primary Provincial Department School Office of Education, Youth and Sports, Kampong Cham Ms. Chuong Sokkun CBD Agent Khmuong village, Chup commune, Thbong Khmum OD, Kampong Cham Ms. Yos Sobun Focal Point of CCWC Anh Cheum CC, Thbong Khmum OD, Kampong Cham Ms. Pieng Nara Maternal and Child Health PHD, Kampong Cham 092 927735 Chief 1. Mr. Yin Phon 2nd Deputy Commune Chief Chhouk Commune, Tel: 012 421 145 2. Mr. Out Samoeurn Councilor Kroch Chmar district, Tel: 077 639 509 3. Mr. Chhun Saron Clerk Kampong Cham Tel: 012 604 904

1. Mr. Dul Chheang First Deputy Chief Tuol Snoul, Kroch 1. Tel: 012 651 271 2. Ms. Yan Savann Chairwoman of CCWC Chmar, Kampong Cham 2. Tel: 097 5520 133 3. Mr. Nheb Sreng Councilor 3. Tel: 012 192 076 4. Mr. Ben Nol Councilor 4. Tel: 092 680 667 5. Mr. Thoy Thea Clerk 5. Tel: 012 270 068 Mr. Chim Leav Deputy Director of Planning Department of Planning, Tel: 092 894 358 Deputy Director of Planning Kampong Cham Mr. Ouch Buntha Head of Statistics Office

Mr. Eng Nareth Mr. Em Sokhon Head of Office of Local Division of Planning Tel: 092 29 25 05 Administration Support Unit and Investment, Kam- pong Cham Ms. Sy Vantha Deputy Director of Planning Division of Planning Tel: 012 420 217 and Investment Division, and Investment, Kam- and Provincial Trainer pong Cham 1. Ms. Teamg Ket Village Women Focal Point Sangkat Chub, Tbong 1. Tel: 012 578 951 2. Ms. Phin Sidorn Chairwoman of CCWC Khmum, Kampong 2. Tel: 097 225 1 856 3. Mr. Lach Veng Councilor Cham 3. Tel: 012 696 951 4. Mr. Meng Heang Councilor 4. Tel: 090 912 901 5. Chiv Bunhak Councilor 5. Tel: 012 208 693 6. Ms. Peng Sinang Clerk 6. Tel: 012 519 860 1. Mr. Chhuong Phyrum Sangkat Chief Sangkat Suong, Tbong 1. Tel: 012 335 747 2. Mr. Keo Chheng 1st Deputy Chief Khmum 2. Tel: 089 935 841 3. Mr. Oum Yet 2nd Deputy Chief Kampong Cham 3. No.

4. Ms. Yon Samnang Chairwoman of CCWC 4. No. 5. Ms. Meach Sory Head of HC 5. Tel: 011 862 629

1. Mr. Meung Tum First Deputy Sangkat Chief Sangkat Vihear Luong, 2. Mr. Pich Say Clerk Tbong Khmum, 3. Ms. Chhun Seab Chairwoman of CCWC Kampong Cham 4. Ms. Chhorn Nay Heak Head of HC 5. Mr. Hang se 2nd Deputy Sangkat Chief 1. Mr. Oung Sambor Commune Chief Chikor Commune 1. Tel: 2. Mr. Hean Thoeun Head of HC 2. Tel: 012 990 795 3. Ms. Khat Samart Chairwomen of CCWC 3. No 4. Mr. In Kheng Clerk 4. Tel: 012 415 021 Kampong Chhnang Province 1. Ms. Pal Yoeun Director of Provincial De- Provincial Department 1. 012 260 056/097 2. Ms. Choun Vath- partment of Women‟s Af- of Women‟s Affairs 661 7 115 neary fairs 2. 012 251 303/097 Deputy Chief of Gender 725 1 303 Equality Ms. Khin Chan Director of District Depart- Kampong Tralach Dis- 012 682 262 ment of Women‟s Affairs trict, Kampong Chhnang province Ms. Tep Yarem FCC and member of CCWC Peany commune, Kam- 012 483 921 pong Tralach district, Kampong Chhnang province 1. Mr. Chhem Piseth Commune Chief Chak commune, Boribor 1. 012 731 501 2. Mr. Nuon Somaly Clerk District, Kampong 2. 092 842 506 3. Ms. Sot Phyrum Chairwoman of CCWC Chhnang 3. 089 820 092 Mr. Sovann Aschar Pong Deputy Director, Depart- Planning and Invest- 012 593 266 ment of Planning and In- ment Division, Kam- vestment pong Chhnang Mr. Ouk Noch Deputy Director, Depart- Planning and Invest- Tel: 092 164 081 ment of Planning and In- ment Division, Kam- vestment pong Chhnang Mr. Sambath Mean Provincial Facilitator Division of Planning Tel: 0977000955 and Investment, Kam- pong Chhnang 1. Mr. Dim Dorn Director of Planning Department of Planning, Tel: 012 412609 2. Mr. Sam Sopheap Planning Office Kampong Chhnang Tel: 097 206 472 3. Mr. Khen Phally Tel: 097 259 0815

Kratie Province 1. Ms. Bun Sithout Director of Provincial De- Kratie 1. 012 531 596 2. Ms. Neang Naron partment of Women‟s Af- 2. 017 490 846 3. Ms. Nean Puy Pirun fairs 3. 092 467 114 Director of District Depart- ment of Women‟s Affairs, Chhlong district Chief of office 1. Ms. Ngim Chhaihean Female Commune Counci- Bosleav commune, 1. 011 674 869 2. Mr. Ouy Srun lor responsible for CCWC Chhlong district, Kratie 2. 088 887 917 Second deputy commune chief 1. Mr. Kong Socheat Director of Provincial De- Provincial Department 1. 012 832 245 2. Mr. Sin Kimsea partment of Planning of Planning, Kratie 2. 012 284 003

3. Mr. Kim Phlla Deputy Director 3. 011 749 484 4. Ms. Vong Bophavy Deputy Director 4. 011 640 122 5. Ms. Tin Sreymum Chief of Administrative. 5. 012 825 905 Office Chief of Controlling Office Pailin Province Ms. Prak Sokhon Maternal and Child Health PHD, Pailin 017 568350 Chief Ms. Kieng Linkea Midwife Ochra Health Center, Pailin Mr. Hak Pan Health Center Chief Ochra Health Center, Pailin Ms. Sor Sovannara Focal Point for CCWC Tuol Lovea Sangkat, Pailin Ms. Duch Sam Oeun Maternal Ward Chief Provincial Referral Hospital, Pailin Ms. Luy Chanlea Vice Chief of Emergency Provincial Referral and Obstetric Care service Hospital, Pailin Mr. Luy Sopheap Deputy Director PHD, Pailin Siem Reap Province Mr. Ly Bunna Chief of provincial School Provincial Department Health Committee of Education, Youth and Sport, Siem Reap Mr. Khath Bopha Member of provincial Provincial Department School Health Committee of Education, Youth and Sport, Siem Reap Mr. Y Navy Member of provincial Provincial Department School Health Committee of Education, Youth and Sport, Siem Reap Mr. Sar Peng An Health Center Chief Puok HC, Angkor Chum OD, Siem Reap Ms. Heun Socheat Midwife Chief Puok HC, Angkor Chum OD, Siem Reap Mr. Mang Sambath OD Deputy Director Angkor Chum OD, Si- em Reap Ms. Ky Tithya Rith Maternal and Child Health Angkor Chum OD, Si- Chief em Reap Mr. Sorn Vuthea Chief of Finance Angkor Chum OD, Si- em Reap Dr. Thong Ramy Maternal and Child Health PHD, Siem Reap Chief Others Ms. Hou Vimol Former UNFPA Youth Of- Phnom Penh 012 981 071 ficer Ms. Min Yim Former UNFPA PD Manag- Phnom Penh 012 850 571 er Ms. Chandy Chea Former UNFPA Gender Phnom Penh 012 994 759 Programme Manager

Annex 5: Interview Guide

RELEVANCE To what extent did the programme‟s priorities and objectives align with the needs of the target population? With the government‟s priorities and policies? With the priorities of other development partners? Probe: Did the programme priorities and objectives evolve over time? If so, how? Probe: To what extent did programme activities of UNFPA overlap and duplicate interventions by other development partners? Probe: How could synergies have been enhanced and duplications avoided?

How appropriate and realistic were the programme‟s strategies and interventions, considering the socio-economic and political environment in which the UNFPA operates in Cambodia? Probe: Did the programme transition process respond to the current challenges in the country?

EFFECTIVENESS To what extent were the planned outputs and outcomes achieved as per the CPAP? Probe: What policy changes and improvements in the national indicators were made in reproductive health, population and development and gender during the programme period? Probe: To what extent were these changes attributable to UNFPA‟s efforts and what will be the implications of them in terms of the current UNFPA programme cycle? Probe: What were the main factors that contributed to the realization or nonrealization of the outcomes? Probe: To what extent did the RH, PD, and gender components complement each other to produce the desired outcomes?

What practical difficulties or challenges were faced during implementation? Probe: How did UNFPA attempt to resolve these difficulties and how successful was it? Probe: Can you provide specific examples of how the programme modality worked well and not so well?

What opportunities exist to improve the UNFPA‟s performance? Probe: Do you have any recommendations to improve or strengthen UNFPA‟s effectiveness and efficiency?

Given the many ways to address the challenges that Cambodia faces, did UNFPA‟s do the right things to address these challenges, or there better or more efficient ways to address these challenges?

EFFICIENCY To what extent do the outputs and outcomes meet acceptable standards of quality?

To what extent was the programme designed to be efficient?

Probe: How and where could improvements have been made to improve efficiency without compromising quality?

Do you have comments or thoughts about the timeliness of UNFPA‟s support to your organization? Probe: For example, did UNFPA provide financial resources in a timely manner? If no, what were the consequences?

To what extent have the programme inputs (human, technical, and financial) been used efficiently?

IMPACT

What were the major strengths and weaknesses of the programme and what were its major achievements? Probe: How would things have been different in Cambodia/in your province if the programme hadn‟t existed? Probe: Are there any qualitative or quantitative indicators or verifiable evidence of programme impact?

Has there been any noticeable impact on the government‟s attitudes to gender and gender-based violence? Probe: Are there any measurable indicators or verifiable evidence of this impact?

Has there been any noticeable impact on minorities or vulnerable groups? Probe: Are there any measurable indicators or verifiable evidence of this impact?

What were the shortcomings of the programme, if any? Probe: Were there any unexpected impacts – positive or negative?

GENDER

To what extent and what ways did the programme promote gender equity and sensitivity to gender-related issues?

To what extent did UNFPA use gender-disaggregated data for planning and assessing the programme, and to what extent has the programme promoted gender mainstreaming?

Did the programme ensure that implementation, monitoring, and reporting take proper account of women‟s and men‟s empowerment?

Annex 6: Statement on Confidentiality and Informed Consent

We appreciate your willingness to participate in the evaluation.

We will be complying with the UN‟s norms and standards for evaluation, including ethics and confidentiality. These require us to:  Respect your right to provide information in confidence and ensure that any sensitive information cannot be traced to its source.

 Ensure that any sensitive information you provide will not be shared with anyone outside the evaluation team, except in instances of fraud or wrong-doing.

For nonsensitive information it may be desirable to include some of your statements in the evaluation report, with attribution if you agree. If we wish to do so, we will first share the statement with you and obtain your approval to include it.

If acceptable to you, your name and position will be included in the report in the list of people consulted.

Please feel free to stop the interview at any time, this is your right.

We have a responsibility to be independent, impartial, objective, accurate and as fair as possible in our compiling of the information, analyzing and reporting.

You are encouraged to be frank and objective in your comments and to provide any information you think might be relevant to the team in fulfilling its responsibilities.

The evaluation team‟s task is to identify the UNFPA/Cambodia‟s programme‟s relevance, effectiveness, efficiency, and impact. In short, what worked well, what did not do so and how can activities be improved in the future? We welcome any recommendations you might have about the programme.

If you would like to contact me after the conclusion of our discussion, you can do so at ______. Thank you for your contribution to this process.

NOTE: This statement was translated into Khmer.

Annex 7: DOS Evaluation Quality Criteria

1. Structure and Clarity of Reporting Does the report clearly describe the evaluation, how it was conducted, the findings of the evaluation, and their analysis and subsequent recommendations? Is the structure logical? Is the report comprehensive? Can the information provided be easily understood? 2. Executive Summary Does it read as a stand-alone section, and is a useful resource in its own right? Is it brief yet sufficiently detailed, presenting the main results of the evaluation, and including key elements such as methodology and conclusions and recommendations? 3. Design and Methodology Is the methodology used for the evaluation clearly described and is the rationale for the methodological choice justified? Have cross-cutting issues (vulnerable groups, youth and gender equality) been paid specific attention (when relevant) in the design of the evaluation? Are key processes (tools used, triangulation, consultation with stakeholders) discussed in sufficient detail? Are constraints and limitations made explicit (including limitations applying to interpretations and extrapolations; robustness of data sources, etc.) and discussed? 4. Reliability of Data Are sources of data clearly stated for both primary and secondary data? Is it clear why case studies were selected and what purpose they serve? Are all relevant materials related to case studies, interviews (list of interviewees, questionnaires) etc. annexed to the report? Are the limitations, and methods to address them, discussed? What other data gaps are there and how have these been addressed? 5. Findings and Analysis Findings Is there a clear pathway from data to findings, so that all findings are evidence-based? Are biases stated and discussed? Are unintended findings reported and discussed? Analysis Are interpretations of the findings understandable? Are assumptions clearly stated and extrapolations well explained? Are their limitations (or drawbacks) discussed? Does the analysis respond to all evaluation questions? If not, are omissions (of both evaluation criteria and questions) recognized and explained? Has the analysis examined cause and effect links between an intervention and its end results? Are contextual factors identified and their influence discussed? 6. Conclusions Are the conclusions organized in priority order? Do the conclusions amount to a reasonable judgment of the findings and are their links to evidence made clear? Are there any limitations and are these made clear? Do they present an unbiased judgment by the evaluators of the intervention or have they been influenced by pre- conceptions or assumptions that have not been discussed? 7. Recommendations Is there a logical flow from the conclusions to recommendations? Are they strategic and clearly presented in a priority order which is consistent with the prioritization of conclusions? Are they useful – sufficiently detailed, targeted and likely to be implemented and lead to further action? How have the recommendations incorporated stakeholders‟ views and has this affected their impartiality? 8. Meeting Needs Does the report adequately address the information needs and responds to the requirements stated in the ToRs? In particular does the report respond to the evaluation questions, issues or criteria identified in ToR?

Overall Assessment:

Quality Assessment criteria Very Good Poor Unsatisfactory Good 3 2 1 4 1. Structure and Clarity of Reporting To ensure report is user-friendly, comprehensive, logically Please insert your main comments here structured and drafted in accordance with international standards. Checklist of minimum content and sequence required for structure:  i) Acronyms; ii) Exec Summary; iii) Introduction; iv) Methodology including Approach and Limitations; v) Con- text; vi) Findings/Analysis; vii) Conclusions; viii) Rec- ommendations; ix) Transferable Lessons Learned (where applicable)

 Minimum requirements for Annexes: ToRs; Bibliography List of interviewees; Methodological instruments used.

2. Executive Summary To provide an overview of the evaluation, written as a stand- alone section and presenting main results of the evaluation. Structure (paragraph equates to half page max):  i) Purpose, including intended audience(s); ii) Objec- tives and Brief description of intervention (1 para); iii) Methodology (1 para); iv) Main Conclusions (1 para); v) Recommendations (1 para). Maximum length 3-4 pages

3. Design and Methodology To provide a clear explanation of the following elements/tools Minimum content and sequence:  Explanation of methodological choice, including constraints and limitations;

 Techniques and Tools for data collection provided in a detailed manner;

 Triangulation systematically applied throughout the evaluation;

 Details of participatory stakeholders‟ consultation process are provided.

 Whenever relevant, specific attention to cross-cutting issues (vulnerable groups, youth, gender equality) in the design of the evaluation

4. Reliability of Data To clarify data collection processes and data quality  Sources of qualitative and quantitative data have been identified;

 Credibility of primary (e.g. interviews and focus groups) and secondary (e.g. reports) data established

and limitations made explicit;

5. Findings and Analysis To ensure sound analysis and credible findings Findings  Findings stem from rigorous data analysis;

 Findings are substantiated by evidence;

 Findings are presented in a clear manner

Analysis  Interpretations are based on carefully described assumptions;

 Contextual factors are identified.

 Cause and effect links between an intervention and its end results (including unintended results) are explained.

6. Conclusions To assess the validity of conclusions  Conclusions are based on credible findings;

 Conclusions are organized in priority order;

 Conclusions must convey evaluators‟ unbiased judgment of the intervention.

7. Recommendations To assess the usefulness and clarity of recommendations  Recommendations flow logically from conclusions;

 Recommendations must be strategic, targeted and oper- ationally-feasible;

 Recommendations must take into account stakeholders‟ consultations whilst remaining impartial;

 Recommendations should be presented in priority order

8. Meeting Needs To ensure that Evaluation Report responds to requirements (scope & evaluation questions/issues/DAC criteria) stated in the ToR (ToR must be annexed to the report).