ASIAN DEVELOPMENT BANK Operations Evaluation Department PROGRAM PERFORMANCE EVALATION REPORT IN MONGOLIA In this electronic file, the report is followed by Management’s response. Performance Evaluation Report Project Number: PPE: MON 28451 Loan Numbers: 1568-MON(SF)/1569-MON(SF) February 2008 Mongolia: Health Sector Development Program Operations Evaluation Department CURRENCY EQUIVALENTS Currency Unit – togrog (MNT) At Appraisal At Project Completion At Operations Evaluation (June 1997) (December 2003) (June 2007) MNT1.00 = $0.0012 $0.00087 $0.00086 $1.00 = MNT804 MNT1,150 MNT1,164 ABBREVIATIONS ADB – Asian Development Bank FGP – family group practice GDP – gross domestic product HIF – Health Insurance Fund HMIS – health management information system HSDP – Health Sector Development Program HSMP – Health Sector Master Plan km – kilometer MMR – maternal mortality rate MOH – Ministry of Health MOSWL – Ministry of Social Welfare and Labor NGO – nongovernment organization NSO – National Statistics Office O&M – operation and maintenance OEM – Operations Evaluation Mission PCR – program completion report PHC – primary health care PIU – project implementation unit PPER – program performance evaluation report PPTA – project preparatory technical assistance PSFML – Public Sector Finance and Management Law TA – technical assistance UNICEF – United Nations Children's Fund WHO – World Health Organization Officer-in-Charge Director R. Keith Leonard, Operations Evaluation Division 1, Operations Evaluation Department (OED) Team Leader Jocelyn Tubadeza, Senior Evaluation Officer, Office of the Director General, OED Team Members Hayman Win, Young Professional, Operations Evaluation Division 1, OED Caren Joy Mongcopa, Senior Operations Evaluation Assistant, Operations Evaluation Division 1, OED Operations Evaluation Department, PE-712 GLOSSARY aimag – province bag – subdistrict ger – traditional Mongolian tent dwelling soum – district NOTE In this report, “$” refers to US dollars. KEYWORDS mongolian family group practice, mongolian health care systems, mongolian primary health care, millennium development goals, transit ional economy, adb, evaluation, asian development bank, health policy reform, mongolian projects programs evaluations CONTENTS Page BASIC DATA iii EXECUTIVE SUMMARY v MAP ix I. INTRODUCTION 1 A. Evaluation Purpose and Process 1 B. Expected Results and Program Objectives 1 C. Program Completion Report Rating 2 II. DESIGN AND IMPLEMENTATION 2 A. Formulation 2 B. Rationale 3 C. Cost, Financing, and Executing Arrangements 5 D. Procurement, Construction, and Scheduling 6 E. Design Changes 6 F. Outputs 7 G. Consultants 11 H. Loan Covenants 12 I. Policy Framework 12 III. PERFORMANCE ASSESSMENT 13 A. Overall Assessment 13 B. Relevance 14 C. Effectiveness 15 D. Efficiency 19 E. Sustainability 20 IV. OTHER ASSESSMENTS 21 A. Impacts 21 B. Asian Development Bank Performance 22 C. Borrower Performance 23 D. Technical Assistance 23 V. ISSUES, LESSONS, AND FOLLOW-UP ACTIONS 24 A. Issues 24 B. Lessons 26 C. Follow-up Actions 27 The guidelines formally adopted by the Operations Evaluation Department (OED) on avoiding conflict of interest in its independent evaluations were observed in the preparation of this report. Munkh-Erdene Luvsan, Danzan Naranturaya, and Penelope Schoeffel were the consultants. To the knowledge of the management of OED, there were no conflicts of interest of the persons preparing, reviewing, or approving this report. APPENDIXES 1. Comparison of Appraisal and Actual Program Costs 28 2. Policy Loan Achievement Matrix 29 3. Project Loan Achievement Matrix 41 4. Contextual Dimensions in Which the Health Sector Development Program Was Implemented 47 5. Primary Health Care and Urban Family Group Practice 52 6. Rural, Provincial, and District Hospitals 58 7. Health Facilities and Personnel 68 8. Issues in Health Insurance and Health Care Financing 70 9. Results of a Study of Health Service Quality and Access in Project Locations of the First Health Sector Development Program 75 10. Poverty, Migration, and Access to Health Services in Mongolia 84 Attachment: Management Response BASIC DATA Loans 1568-MON(SF) and1569-MON(SF): Health Sector Development Program in Mongolia Program Preparation/Institution Building TA No. TA Name Type Amount ($) Approval Date 2414 Health Sector Development PPTA 600,000 3 Oct 1995 2731 Health Sector Resources Development PPTA 100,000 23 Dec 1996 2907 Support for Decentralized Health ADTA 600,000 4 Nov 1997 Services As per ADB Key Program Data ($ million) Loan Documents Actual ADB Loan Amount/Utilization 4.0 3.81 ADB Loan Amount/Cancellation 0.0 Key Project Data ($ million) Total Project Cost 14.9 14.3 Foreign Currency Cost 7.9 9.6 Local Currency Cost 7.0 4.7 ADB Loan Amount/Utilization 11.9 11.2 ADB Loan Amount/Cancellation 0.3 Key Dates Expected Actual Appraisal 16 June–12 July 1997 Loan Negotiations 23–25 September 1997 Board Approval 4 November 1997 Loan Signing 12 February 1998 Loan Effectiveness 6 May 1998 20 April 1998 Disbursement Initial 24 June 1998 Final 18 December 2003 Program and Project Completion 31 December 2002 30 September 2003 Loan Closing Program Loan 30 June 2001 18 June 2001 Project Loan 30 June 2003 18 December 2003 Months (Effectiveness to Completion) 56 65 Borrower Government of Mongolia Executing Agencies Ministry of Health Bank of Mongolia Mission Data Type of Mission No. of Missions No. of Persons No. of Person-Days Inception 1 1 11 Review 7 8 76 Midterm Review 1 2 20 Project Completion Review 1 2 11 Operations Evaluation 1 4 73 ADB = Asian Development Bank, ADTA = advisory technical assistance, PPTA = project preparatory technical assistance. 1 TP Actual disbursements in dollars were lower due to the strengthening of the dollar. EXECUTIVE SUMMARY The Health Sector Development Program (HSDP) in Mongolia was selected for evaluation to determine lessons from a sector-wide, reform-based program in a country experiencing the early stages of transition to a market economy. Until the early 1990s, Mongolia’s health system depended almost entirely on centrally managed, hospital-based health services designed to meet the requirements of socialist agricultural and industrial collectives, the state bureaucracy, and other state agencies, supplemented by centralized specialist hospitals. In the early 1990s, the collectives collapsed, the economy moved towards a market system, and the bureaucracy was reorganized; but the state was nearly insolvent. Health services deteriorated and became unable to meet the needs of most of the population at the levels provided under socialism. Without subsidies from the former Soviet Union, the financial resources to sustain this health system were no longer available. About 40% of staff left the state health service, the net reduction being around 25% because new medical graduates continued to enter the system. On average, in 1990–1996, 8.5% of qualified doctors left employment in the health sector each year. In 1995, reforms to the health sector commenced. These included introduction of a health insurance fund, and training in general practice and the principles of primary health care (PHC) within the medical school curriculum. However, health services continued to deteriorate, and it became clear that there was a need for more fundamental reform to develop a three-tier system in which PHC and preventive services would form a foundation, with referral to rationalized secondary and tertiary levels of care, supported by associated training, management assistance, policy advice, new equipment, and the provision or rehabilitation of health service infrastructure. The HSDP was designed to address these needs. The project preparatory technical assistance (PPTA), completed in September 1996, was of high standard. It provided a comprehensive analysis of the health sector, health services, and facilities; and was conducted in a consultative manner. It is noteworthy that it did not specifically recommend the private family group practice (FGP) model that became central to the HSDP, although it emphasized the need to focus on shifting resources from hospital-based services to public health and PHC and to build a comprehensive referral system. The HSDP, as approved on 4 November 1997, consisted of a policy-based loan of SDR2.94 million ($4 million), an investment loan of SDR8.7 million ($11.9 million), and associated technical assistance (TA) of $600,000. It was signed on 12 February 1998 and became effective on 20 April 1998. The HSDP objectives were to (i) ensure financial sustainability of the health system in a market setting, (ii) maintain universal access to quality essential health services, and (iii) improve health services quality. Accordingly, six policy components or reform strategies were supported: (i) promoting PHC, (ii) encouraging private sector participation in health services delivery, (iii) rationalizing health facilities, (iv) rationalizing health personnel, (v) improving health care financing and management, and (vi) protecting the poor and vulnerable groups. The policy loan also financed an information and education campaign to explain the reforms to the general public and to health professionals. The project loan supported implementation of the reform strategies and financed investments and activities in (i) urban health services, (ii) rural health services, and (iii) sector management strengthening. The associated TA aimed to (i) strengthen local governments’ vi capacity to plan and manage local health services, (ii) identify issues