World Bank Document

Home , Provinces of Argentina

Document of The World Bank

Public Disclosure Authorized

Report No: ICR2618

IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-7409)

Public Disclosure Authorized ON A

LOAN IN THE AMOUNT OF US$ 300 MILLION

TO THE

REPUBLIC OF ARGENTINA

FOR THE

PROVINCIAL MATERNAL-CHILD HEALTH INVESTMENT PROJECT IN SUPPORT OF Public Disclosure Authorized THE SECOND PHASE OF THE PROVINCIAL MATERNAL-CHILD HEALTH PROGRAM

June 25, 2013

Human Development Department Argentina, Paraguay and Uruguay Country Management Unit Latin America and Caribbean Region

Public Disclosure Authorized

CURRENCY EQUIVALENTS

(Exchange Rate Effective December 31, 2012)

Currency Unit = Argentine Peso 1.00 = US$0.2035 US$ 1.00 = AR$4.91

FISCAL YEAR January 1 - December 31

ABBREVIATIONS AND ACRONYMS

APGAR Appearance, Pulse, Grimace, Activity, Respiration AUH Universal Child Allowance (Asignación Universal por Hijo) AE Universal Pregnancy Allowance (Asignación Universal por Embaraza) ASM Supervision and Monitoring Group CC Congenital Heart Disease (Cardiopatías Congénitas) CHD Congenital Heart Disease COFESA Federal Health Council (Consejo Federal de Salud) DEIS Directorate of Health Statistics ECA External Concurrent Audit FESP Essential Public Health Functions Project FM Financial Management GAAP Governance and Anticorruption Action Plan GDP Gross Domestic Product HD Human Development HSRP Health Sector Reform Program IBRD International Bank for Reconstruction and Development IE Impact Evaluation IFR Interim Financial Report IMR Infant Mortality Rate IPR Independent Procurement Review ISR Implementation Status Report MCHIP Maternal and Child Health Insurance Program MTR Mid-Term Review M&E Monitoring and Evaluation NEA Northeastern Region NHSPT National Health Service Purchasing Team NOA Northwestern Region NMH National Ministry of Health NPV Net Present Value PAD Project Appraisal Document PDO Project Development Objective PHIP Provincial Health Insurance Project PHSPT Provincial Health Service Purchasing Team PMCH-SAL Provincial Maternal-Child Health Sector Adjustment Loan PMCH-SAL Provincial Maternal-Child Health Adaptable Lending Program

PMH Provincial Ministry of Health PPR Procurement Post Review QAG Quality Assurance Group QALP Quality Assessment of Lending Portfolio QEA Quality at Entry Assessment QSA Quality of Supervision RBF Results-Based Financing SEA Strategic Environmental Assessment UFI-S International Financing Unit for Health – National Ministry of Health VDRL Venereal Disease Research Laboratory

Vice President: Hasan A. Tuluy Country Director: Penelope Brook Sector Manager: Joana Godinho Project and ICR Team Leader: Andrew Sunil Rajkumar ICR Primary Author: Katharina Ferl

iii

ARGENTINA Provincial Maternal-Child Health Investment Project in Support of the Second Phase of the Provincial Maternal-Child Health Program

CONTENTS

Data Sheet ...... v 1. Project Context, Development Objectives and Design ...... 1 2. Key Factors Affecting Implementation and Outcomes ...... 4 3. Assessment of Outcomes ...... 11 4. Assessment of Risk to Development Outcome ...... 15 5. Assessment of Bank and Borrower Performance ...... 15 6. Lessons Learned ...... 18 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ...... 18 Annex 1: Project Costs and Financing ...... 19 Annex 2. Outputs by Component ...... 21

Annex 3. Economic and Financial Analysis ...... 31 Annex 4. Bank Lending and Implementation Support/Supervision Processes...... 43 Annex 5. Beneficiary Survey Results ...... 45 Annex 6. Stakeholder Workshop Report and Results ...... 49 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ...... 50 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ...... 65 Annex 9. List of Supporting Documents ...... 66 Annex 10. Background Information: The Health System in Argentina ...... 68 Annex 11: Quality Assessment of Lending Portfolio (QALP-2) Summary ...... 73 Annex 12. More Details on Preparation and Implementation of APL-2 ...... 82 Annex 13. Impact Evaluation for Plan Nacer: Challenges and Accomplishments ...... 88

Data Sheet

A. Basic Information

AR Provincial Maternal-Child Country: Argentina Project Name: Health Investment APL 2 Project ID: P095515 L/C/TF Number(s): IBRD-74090 ICR Date: 06/25/2013 ICR Type: Core ICR Lending Instrument: APL Borrower: Republic of Argentina Original Total USD 300.00M Disbursed Amount: USD 300.00M Commitment: Revised Amount: USD 300.00M Environmental Category: C Implementing Agencies: National Ministry of Health, Argentina Cofinanciers and Other External Partners: N.A.

B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 05/24/2006 Effectiveness: 05/31/2007 05/31/2007 04/26/2010 Appraisal: 08/28/2006 Restructuring(s): 06/26/2012 Approval: 11/02/2006 Mid-term Review: 08/08/2011 11/07/2011 Closing: 12/31/2012 12/31/2012

C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Satisfactory Risk to Development Outcome: Moderate Bank Performance: Satisfactory Borrower Performance: Satisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Satisfactory Government: Satisfactory Implementing Quality of Supervision: Satisfactory Satisfactory Agency/Agencies: Overall Bank Overall Borrower Satisfactory Satisfactory Performance: Performance:

C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments (if Indicators Rating Performance any) Potential Problem Project Quality at Entry Yes Satisfactory at any time (Yes/No): (QEA): Problem Project at any time Quality of Supervision Yes Moderately Unsatisfactory (Yes/No): (QSA): DO rating before Moderately

Closing/Inactive status: Satisfactory

D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 9 9 Compulsory health finance 50 50 Health 30 30 Sub-national government administration 11 11

Theme Code (as % of total Bank financing) Child health 40 40 Health system performance 20 20 Population and reproductive health 40 40

E. Bank Staff Positions At ICR At Approval Vice President: Hasan A. Tuluy Pamela Cox Country Director: Penelope J. Brook Axel van Trotsenburg Sector Manager: Joana Godinho Keith E. Hansen Project Team Leader: Andrew Sunil Rajkumar Cristian C. Baeza ICR Team Leader: Andrew Sunil Rajkumar ICR Primary Author: Katharina Ferl

F. Results Framework Analysis

Project Development Objectives The Project development objectives are: (a) to increase access by eligible uninsured mothers and children to basic health services; (b) to strengthen the incentive framework for efficiency and focus on results between the national level and the eligible provinces and among eligible provinces and service providers by linking financing to both services actually rendered to the

vi target population and to the achievement of the Maternal and Child Health Insurance Program (MCHIP) results as reflected by the selected ten tracers of the Trazadoras Matrix.

Revised Project Development Objectives Not Applicable.

vii

(a) PDO Indicator(s)

Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised Target approval Completion or Values documents) Target Years Indicator 1 : Proportion of eligible population voluntarily enrolled in the program. Value (quantitative 0% 80% 97% or Qualitative) Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Target surpassed by large margin. achievement) Proportion of eligible pregnant women with first antenatal care visit before 20th week Indicator 2 : of pregnancy. Value (quantitative 23% 70% 67% or Qualitative) Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Target almost achieved. achievement) Proportion of eligible pregnant women who get Venereal Disease Research Laboratory Indicator 3 : (VDRL) test during pregnancy and antitetanic vaccine previous to delivery. Value (quantitative 45% 90% 83% or Qualitative) Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Target almost achieved. achievement) Proportion of eligible children less than 18 months old with coverage of measles Indicator 4 : vaccine or triple viral. Value (quantitative 45% 95% 77% or Qualitative) Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Target not achieved. achievement) Proportion of eligible puerperal woman that received at least one Sexual and Indicator 5 : Reproductive Health Care consultation. Value (quantitative 27% 60% 90% or Qualitative) Date achieved 11/02/2006 12/31/2012 12/31/2012

viii

Comments (incl. % Target surpassed by large margin. achievement) Proportion of eligible children 1 year old or less, with all well child consultations up to Indicator 6 : date (percentile of weight and height). Value (quantitative 13% 50% 45% or Qualitative) Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Target almost achieved. achievement) Indicator 7 : Proportion of newborns from eligible pregnant women weighing more than 2,500 g. Value (quantitative 47% 85% 90% or Qualitative) Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Target surpassed. achievement) Proportion of newborns from eligible pregnant women, with Apgar score higher than Indicator 8 : “6” at minute 5. Value (quantitative 47% 92% 93% or Qualitative) Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Target achieved. achievement) Percentage of National Ministry of Health-Provincial Ministry of Health (NMH-PMH) Indicator 9 : annual performance agreements successfully implemented. Value (quantitative 0% 60% 79% or Qualitative) Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Target surpassed by large margin. achievement) Percentage of authorized providers under annual performance agreements and provider Indicator 10 : payment mechanism with its respective participant province. Value (quantitative 0% 60% 95% or Qualitative) Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Target surpassed by large margin. achievement) Indicator 11 : Percentage of Tracer targets achieved by the participant provinces in last year billing

period. Value (quantitative 0% 70% 94% or Qualitative) Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Target surpassed by large margin. achievement)

(b) Intermediate Outcome Indicator(s)

Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Indicator 1 : Loan disbursements. Value quantitative or 0% 100% 100% Qualitative) Date achieved 11/02/2006 12/31/2012 03/30/2013 Comments (incl. % Achieved. achievement) Capitation payment occurring according to approved enrollment lists and trazadora Indicator 2 : systems. Value In at least 9 eligible Capitation payments quantitative or None provinces in central occurring in 24 out of Qualitative) and southern regions 24 provinces. Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Target surpassed. achievement) Provincial Health Service Purchasing Team (PHSPT) and National Health Service Indicator 3 : Purchasing Team (NHSPT) function effectively according to concurrent and financial audits. NHSPT functions Value In at least 50% of all effectively; PHSPTs quantitative or None participating function effectively Qualitative) provinces in 24 out of 24 provinces. Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Target surpassed. achievement) Number of authorized providers receiving the basic medical equipment / vehicles / Indicator 4 : communication equipment according to the annual performance agreement. Value None At least 50% of 85.3% of authorized

x quantitative or those for whom an providers Qualitative) investment Project was approved. Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Target surpassed. achievement) Number of PHSPTs established and functioning, capable of preparing and negotiating Indicator 5 : NMH-PMH and PMH-authorized providers’ annual performance agreements. Established and Value At least 60% of all functioning PHSPTs quantitative or None participant in 24 out of 24 Qualitative) provinces. participant provinces Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Target surpassed. achievement) NHSPT is established and functioning, capable of preparing and negotiating NMH- Indicator 6 : PMH annual performance agreements. One National Value Direction for NHSPT has been quantitative or Not Applicable purchase of medical established and is Qualitative) services has been operational established. Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Achieved. achievement) Regular information process among stakeholders on maternal-child health care issues in Indicator 7 : general and entitlements in the Plan Nacer in particular. 23 (out of 24) provinces with Annual Plans of Communication Information and executed as planned. dissemination Various information Value campaign launched and dissemination quantitative or Not Applicable. at national level and activities undertaken Qualitative) in at least 80% of all at national level, e.g. participating via television and provinces. production/distributio n of relevant audio- visual and graphic material. Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Target surpassed. achievement) Indicator 8 : Targeted groups increase, knowledge of their entitlements under the program and

participation in Plan Nacer, and report satisfaction with process and results According to the Estudio de Monitoreo de la Satisfacción del Usuario y de la Calidad de Atención del Plan Nacer (a household survey), conducted to date in At least 70% of 13 provinces eligible population (including both Phase reports (in surveys) 1 and Phase 2 knowledge of Plan provinces) between Value Nacer. At least 50% July 2012 and quantitative or Not Applicable. of enrolled December 2012: (i) Qualitative) population reports 84.1% of those (in surveys) that is interviewed knew satisfied with the about Plan Nacer, Plan Nacer and (ii) 72.1% of Program. interviewed beneficiaries indicated that they were either “satisfied” or “very satisfied” with the services provided by the program. Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Target surpassed. achievement) Project implementation reports available as agreed, including financial reports Indicator 9 : supporting the capitation payments. Satisfactory reports from Concurrent and annual financial auditor. Value Satisfactory reports At least 80% of the quantitative or Not Applicable. available in 24 out of reports Qualitative) 24 eligible provinces Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments (incl. % Target surpassed. achievement) Project evaluation implemented: (i) baseline at the end of Year 2; mid term impact Indicator 10 : evaluation at the end of Year 3; and final impact evaluation at the end of last year of APL-1. Value Baseline done, At least 80% of the quantitative or Not Applicable. midterm evaluation reports Qualitative) ongoing. Date achieved 11/02/2006 12/31/2012 12/31/2012 Comments Partially achieved.

xii

(incl. % achievement)

G. Ratings of Project Performance in ISRs

Date ISR Actual Disbursements No. DO IP Archived (USD millions) 1 01/25/2007 Highly Satisfactory Satisfactory 0.00 2 06/20/2007 Highly Satisfactory Satisfactory 0.00 3 12/20/2007 Satisfactory Satisfactory 11.52 4 06/30/2008 Satisfactory Satisfactory 30.96 5 12/12/2008 Satisfactory Satisfactory 42.55 6 06/19/2009 Satisfactory Satisfactory 67.97 7 12/18/2009 Satisfactory Satisfactory 78.75 8 06/28/2010 Moderately Satisfactory Moderately Satisfactory 103.33 9 02/20/2011 Moderately Satisfactory Moderately Satisfactory 164.05 10 07/26/2011 Moderately Satisfactory Moderately Satisfactory 188.32 11 02/05/2012 Moderately Unsatisfactory Moderately Satisfactory 237.84 12 08/20/2012 Moderately Unsatisfactory Moderately Satisfactory 275.67 13 12/25/2012 Moderately Satisfactory Moderately Satisfactory 300.00

H. Restructuring

ISR Ratings at Amount Board Restructuring Restructuring Disbursed at Reason for Restructuring & Key Approved PDO Date(s) Restructuring Changes Made Change DO IP in USD millions Loan proceeds reallocation, modification of financing percentage for Category (4) of disbursement table of Loan Agreement (capitation 04/26/2010 No S S 85.13 payments), inclusion of new provinces as Eligible Provinces, expansion of eligible list of services, closing date extended to December 2012. 06/26/2012 No MU MS 275.67 Loan proceeds reallocation

xiii

I. Disbursement Profile

xiv

1. Project Context, Development Objectives and Design

1.1 Context at Appraisal

1. Despite Argentina's recovery after the deep economic crisis of 1999-2002, many problems persisted at the time of appraisal, including low access to and quality of health services for the poor. This was the case despite sweeping health reforms, and despite the fact that in the early 2000s, Argentina was among the top 20 countries globally in per capita health spending.

2. Immediately after the crisis, the percentage of the population not covered by formal health insurance had risen to 44% in 2002 (from 38% in 1997). The rest of the population had formal health insurance coverage – either by the Obras Sociales (national social health insurance programs) or by private health insurance policies. But those without formal health insurance had to use the public healthcare system, where there were significant shortfalls in quality and sometimes in service availability. This system was financed fully from public funds – since health care must be provided for free at public health facilities – in a traditional manner based on financing of inputs, without any link between health facility revenues and the quality of care provided.

3. In response to the crisis, the Government of Argentina developed the Health Sector Reform Program in 2003, with two main pillars – the first one providing financing for essential public health actions. An innovative modality was used to provide provinces with financial incentives to produce these actions. This pillar was financed by the World Bank under the Essential Public Health Functions Project Phases 1 and 2 (P090993 and P110599), starting in 2006.

4. The other major pillar centered around the Maternal and Child Health Insurance Program (Plan Nacer), complemented by structural changes in the provincial health systems to improve the efficiency of public financing and the delivery of health services. The latter was supported by the World Bank Provincial Maternal-Child Health Sector Adjustment Loan (P072637).

5. Plan Nacer started in 2004 as a provincial public insurance program providing a free basic package of pre-defined cost-effective services, through participating healthcare providers, to women and children without formal health insurance in the country’s nine poorest provinces – those in the Northeast and Northwest. All uninsured pregnant and lactating women (up to 45 days after birth), as well as uninsured children aged under six, were eligible. No copayments or prepayments were (or are) required, in line with the principle of mandatory free health care in the public system in Argentina.

6. Plan Nacer is a pioneering Results-Based Financing (RBF) program in health. It is one of the first large-scale programs worldwide to use an RBF approach in the health sector, and various other RBF programs in health around the world have been modeled on it.

7. Participating health care providers under Plan Nacer – all public providers in the provinces covered – receive payments on a fee-for-service basis (conditional on quality) for the services covered, on top of the financing they receive from the public system on a traditional basis. These payments are only made for services provided to eligible (uninsured) women and

children enrolled in the program, and are conditional on pre-defined protocols being followed (to assure quality). Annex 10 lists the services covered – all maternal and child health services.

8. At the same time, there is a second level of Results-Based Financing (RBF), whereby payments are provided to provinces on an RBF basis, as part of a two-stage incentive mechanism. The National Ministry of Health (NMH) makes performance payments to provinces called capitation payments – based on the number of eligible persons that have been enrolled in the program, and on provincial achievement regarding a list of ten “tracer” indicators (e.g. prenatal care coverage in eligible women, following pre-determined quality protocols – see Table 2 of Annex 10).1 The financing received by the provinces via this mechanism is used to make the payments to the health facilities on a fee-for-service basis. Thus there are two stages of RBF payments: one from NMH to the provinces, and the other from the provinces to the health facilities. The achievement of performance indicators by both provinces and health facilities (including adherence to quality protocols) is verified by an external audit firm (see Annex 10). The goal is to move from a traditional health system based on inputs and fixed budgets to one geared toward outputs and results.

9. The initial phase of Plan Nacer, covering the nine poorer northern provinces of the country, was financed by the first loan of the Provincial Maternal-Child Health Adaptable Lending Program (PMCH-APL, P071025). This APL was approved by the Board for an amount of US$435 million in April 2004. The Project Appraisal Document for the first phase APL (APL-1) had anticipated that the program would be rolled out nationwide over three phases, and the first phase would consist of APL-1 (for US$135.8 million) covering the nine northern provinces.

10. However, it was decided to merge the planned second and third phases of this APL series into one single loan, APL-2 (P095515), financing a rollout of Plan Nacer across all 24 provinces of the country. This decision was taken due to the strong demand from the provinces that were not included in APL-1, as well as the positive assessment on the part of the Government and the Bank of APL-1 implementation in the nine northern provinces. APL-2 was approved by the Board (for US$300 million) in November 2006. Its anticipated closing date was December 2011.

1.2 Original Project Development Objectives (PDO) and Key Indicators

11. The Project’s PDOs were: (a) to increase access by eligible uninsured mothers and children to basic health services; (b) to strengthen the incentive framework for efficiency and focus on results between the national level and the eligible provinces and among eligible provinces and service providers by linking financing to both services actually rendered to the target population and to the achievement of the Maternal and Child Health Insurance Program (MCHIP) results as reflected by the selected ten tracers of the Trazadoras (Tracers) Matrix. The “eligible provinces” were defined in the original Loan Agreement as the 15 provinces not covered by APL-1, but this definition was later amended in April 2010 to also include the nine poorer provinces covered by APL-1, given that APL-1 was about to end in July 2010.

1 A fixed maximum capitation amount is given to each province per eligible person enrolled in the program. Forty percent of this maximum amount is based on provincial performance regarding the ten tracer indicators – 4% for each tracer indicator – while the remaining 60% of this maximum amount is given regardless of tracer indicator performance. Thus, in effect the province receives somewhere between 60% and 100% of the maximum capitation amount – depending on tracer performance – per eligible beneficiary that is enrolled.

12. The Project had eleven PDO indicators (given in Table 3), of which eight were closely linked with the enrolment or “tracer” indicators used to determine provincial performance. These eight PDO indicators were measured at the national level, while the corresponding enrolment or “tracer” indicators were measured at the provincial level, for each province.

1.3 Revised PDO and Key Indicators, and reasons/justification

13. The PDO remained unchanged throughout the life of the Project.

1.4 Main Beneficiaries:

14. The main beneficiaries were pregnant and lactating women (up to 45 days after birth) and children aged under six years, without formal health insurance, in all provinces. The indigenous population as a whole also benefited from the Indigenous Peoples’ Plans (IPPs). Others in the population also benefitted indirectly from improved health services.

1.5 Original Components

15. Table 1 summarizes the activities covered, for each of the five components of the Project.

Table 1: Project Components, Allocations and Activities Com- Initial Allocation from Bank (with Final Expenditure in Parentheses), and Main Activities Covered ponent 1 US$242.7 million (US$249.7 million): Capitation payments to provinces (in turn financing fee-for-service payments to participating health service providers); equipment (medical, transportation, and communications) for participating service providers; technical assistance and training for provinces and service providers; equipment and consultant services to upgrade information systems. 2 US$10.2 million (US$16.0 million): Reorganizing provincial Ministries of Health; improving epidemiological information, financial, and human resource management systems; studies for policy formulation. 3 US$17 million (US$667,000): Dissemination of detailed information about the program among major stakeholder groups; community outreach to increase participation of the eligible population. 4 US$14.6 million (16.8 million): Information technology design services, software and equipment, and related training; external concurrent audits; in-depth evaluations (including surveys and Impact Evaluation). 5 US$1.6 million (2.6 million): Operational expenses (e.g. travel, per diems). Note. Component 1 was co-financed by the Government in two ways. First, the Government financed salaries, infrastructure, etc. at the health facilities. Second, from January 2011 onwards for the Phase 2 provinces (and from August 2010 onwards for the Phase 1 provinces), 30% of each capitation payment (i.e. the monthly payment for each person covered) was paid by the province. The rest (70%) was covered by the Bank loan.

1.6 Revised Components

16. The Project’s components were not revised during implementation.

1.7 Other significant changes

17. Two restructurings took place during implementation, in April 2010 and in June 2012. In both cases, there were changes in the allocation amounts for the different loan categories of disbursement (consultant services, goods, training, operating costs and capitation payments, which consisted of a separate category). The changes were not large, and the allocation for capitation

payments – the heart of the Project – ended up increasing from US$208.5 million to US$212.1 million. No other changes took place in the second restructuring.

18. In the first restructuring, further changes took place: (i) the closing date was extended to December 2012; (ii) the Bank co-financing percentage for the capitation payments category was changed;2 (iii) coverage was expanded from just 15 provinces to all provinces; and (iv) the basic package of services covered by the program was expanded to also include congenital heart disease (CHD) treatment for children and additional complex maternal health services. The latter was due to analysis showing: (a) CHD was a major factor underlying child mortality in the country; and (b) the package needed additional services to better address higher-risk births.

2. Key Factors Affecting Implementation and Outcomes

19. This ICR’s findings for the Project (APL-2) preparation and implementation phase, until June 2010, are based on the findings of a Quality Assessment of the Lending Portfolio (QALP-2) report, done in June 2010 by the World Bank Quality Assurance Group (QAG). Annex 11 provides details of this report, including ratings for different aspects of Project and Bank performance, and a comparison of these ratings with ISR ratings. QAG assessments are considered to be independent and often critical, and this ICR considers QALP-2 to be a largely accurate assessment of the Project at the time. See also Annex 12 for more information.

2.1 Project Preparation, Design and Quality at Entry

20. The ICR for APL-1 noted “a deep understanding of Argentina‘s complex health sector, the lessons learned from the Bank‘s rich health sector portfolio, and the difficult circumstances the country was going through”. This finding remains relevant for APL-2.

21. The lessons drawn upon at the preparation stage were appropriate, and were incorporated appropriately into the Project design. APL-2 took into account not just the lessons from the preparation of APL-1, but also those from implementation of APL-1, featuring in particular: (i) an additional element in the independent concurrent audit to verify the compliance of the provinces with key elements of Plan Nacer; and (ii) expanded technical assistance.

22. The rationale for Bank intervention for APL-2, as for APL-1, was solid. A major rationale was the Bank’s long experience with Argentina’s health sector (more than ten years), resulting in significant comparative advantages in supporting Government efforts that combine policy reform with investments to contribute to longer-term institutional improvements.

23. The QALP-2 report gave an overall rating of Satisfactory for the Quality of Design, highlighting some of the following as key strengths, which this ICR agrees with:

2 Instead of falling over time from 100% to 70%, and then to 40%, it would fall to just 70%. The portion of the capitation payments not financed by the Bank was supposed to be financed by the provinces. The original loan agreement was found to be abrupt and unrealistic in expecting the Bank co-financing percentage to fall to 40% (implying that the provincial contribution would rise to 60%) so rapidly.

• Strategic relevance and appropriateness of Development Objectives: The Project designed a public insurance system that increased access to health care services for an underserved low income and vulnerable population group.

• Adequate attention to technical aspects and financial sustainability: The Project has a solid and internally consistent framework for technical aspects. There should be no undue risk to financial sustainability given the relatively small size of this operation (about 1% on average of total provincial public health expenditures). (See also Section 4 on sustainability.)

• Quality of institutional framework for the Project: The Project was framed to work within the institutional structure in existence at the time of appraisal.

• Extent of integration and quality of financial management and procurement aspects in project design: Adequate measures were in place to support early start-up of procurement activities, and team’s procurement specialists were fully integrated with other Project activities during the design phase. The Project Financial Management (FM) risk assessment was thorough, and took into account the Bank's Country Financial Accountability Assessment of 2002.

• Quality of arrangements for governance and anti-corruption in Project design: The Project avoided altering the political economy ramifications while introducing an empirically driven performance system for allocating resources. The introduction of an external auditing system provided a measure of verification and third party monitoring.

• Extent of integration and quality of environmental aspects in Project design: This was rated as Highly Satisfactory by QALP-2. APL-1 financed a high-quality Strategic Environmental Assessment with detailed analysis of the environmental risks affecting maternal and child health.

• Government ownership of the design process: The QALP-2 report gives a Highly Satisfactory rating for this, saying: “There has been clear political will on the part of the central and provincial authorities. The decision to widen the scope of the Project to cover a wider set of provinces was in response to demand from the provinces to be included….”

24. Other strengths noted in the ICR for APL-1 regarding the quality of design are also relevant in this ICR for APL-2. These include (see Annex 12) high relevance; a pro-poor focus; innovative results-orientated systems of incentives; strong collaboration with various actors; and appropriate choice of lending instrument since the use of an APL allowed the approach to be tried out first in the northern provinces under APL-1 before including the other provinces under APL-2.

25. Some weaknesses in design were also identified by QALP-2, which ultimately did not turn out to be significant shortcomings given the information available after the Project close:

2.1(a) Factors seen by QALP-2 as the Main Reasons for Lagging Achievement in the PDO Indicators:

26. The evolution of several PDO indicators was lagging at the time of the QALP-2 report – which rated achievement of these indicators at Project close as “Moderately Likely” – and the following are the main reasons identified for this (see Annex 12 for full details):

i. Readiness for Implementation: QALP-2 stated: “APL-2 was judged to be ready for implementation based on the successful implementation of the first two years of APL-1 in nine provinces….the addition of 15 provinces and thereby covering the whole country was ambitious and resulted in a slow start up.” At the time of the QALP-2 in June 2010, many of the provinces that had been newly included under APL-2 (the “Phase 2” provinces) still had low rates of coverage by the program, even though these were relatively richer than the “Phase 1” provinces.

ii. Realism of Project Design, and Varying Provincial Capacities: QALP-2 found that “the overall Project design is realistic” but “could have taken into consideration” varying provincial capacities – referring particularly to the Phase 2 provinces where performance was lagging. iii. The role of municipalities in the larger Central Region provinces (Buenos Aires, Santa Fe and Cordoba): QALP-2 noted the “institutional complexities” here because in these provinces, oversight over the provision of health care services was handled not by the provinces but by the municipalities, which represented an additional layer of management in between the provinces and the providers. But the municipalities did not receive performance payments (nor any other payments) under the program. Providing them with such performance payments would have enhanced their incentive to facilitate good performance by providers. iv. Measuring and auditing performance requires a strong information system that was initially not present in some provinces. The depth of existing information systems in various provinces was initially insufficient in order to follow the fast pace of required reporting to obtain financial incentives. However, this improved over time, during implementation, as improvements were made to the existing information systems.

2.1(b) Other factors:

v. Procurement aspects in Project design: QALP-2 rated this as Moderately Satisfactory (MS), explaining that this rating (rather than Satisfactory) was because of an INT investigation that had just been launched (see below), indicating that the initial procurement risk assessment now appeared “naïve”. The Bank Team explained that the Minister herself initiated the investigation, indicating Government pro-activeness to resolve any potential corruption issues. vi. Social aspects in Project design: The QALP-2 report gave an MS rating for integration of social issues in Project design, citing “a sound basic orientation on essential concepts and approaches, coupled with an insufficient attention to the mechanisms of implementation”.

27. Despite the weaknesses in design mentioned in QALP-2 (listed above), QALP-2 still rated the Project design as Satisfactory, and anyway these weaknesses were later addressed successfully and turned out to be at most minor shortcomings when assessed at Project close:

• The first four of the six weaknesses listed above were seen by QALP-2 as the main reasons for the then-lagging PDO indicators. These would have been significant shortcomings if the PDO indicators had continued to progress at a slow rate. However, as described below, an Action Plan implemented in the last year of the Project led to substantially improved performance (even in the large Central Region provinces). Hence, these four weaknesses ultimately did not significantly hamper performance when assessed at Project close.

• The weakness noted by QALP-2 on procurement (fifth in the above list) was due to the ongoing INT investigation. But ultimately the investigation found no evidence of wrongdoing. • Regarding the last (sixth) identified weakness in the above list, performance in social aspects and social safeguards improved substantially later during implementation (see Section 2.4).

2.2 Implementation

28. QALP-2 found implementation progress overall to be Moderately Satisfactory (MS) in June 2010 (see Annex 11). High ratings were given for several aspects of implementation (notably Government ownership and commitment), while some weaknesses were identified (see Table 2).

29. During the subsequent Midterm Review (MTR) of November 2011, the PDO rating was downgraded from MS to MU since at that time only three of the eleven PDO outcome indicators were on track to be fully achieved (based on end targets) – but performance was good overall in the poorer “Phase 1” provinces (i.e. the provinces that had entered earlier in the program under APL-1.) The lagging indicators were mostly those related to the tracers in the 15 “Phase 2” provinces that had been included later in the program – under APL-2 – and was especially low in the provinces of the Central Region (see Table 1 of Annex 2).

30. The relatively good performance of the tracer indicators in the Phase 1 provinces was matched by good performance regarding various indicators of implementation, in particular the provincial “resource transfer rate” to the health service providers. The “resource transfer rate” in a province is the percentage of the total performance payments transferred under Plan Nacer to the province that has, in turn, been transferred onwards (on a fee-for-service basis) to providers. If the system is working well in a province, this percentage should be more than 80%. A low “resource transfer rate” implies a large unutilized balance in a provincial Bank account. At the time of the MTR, the “resource transfer rate” exceeded 80% in all Phase 1 provinces. Conversely, the poor overall performance of the tracer indicators in the Phase 2 provinces was matched by low provincial “resource transfer rates”. Of the 15 Phase 2 provinces, only four had a “resource transfer rate” exceeding 80%. In three provinces, the “resource transfer rate” was less than 40%.

31. The reasons for the poor tracer performance and poor implementation in many Phase 2 provinces were attributed mainly to a few major factors (which were later addressed), including weaknesses in information systems, overestimation of the capacity of the provinces to engage in a new results-oriented approach and issues with Component 3 (see Table 2).

32. After the MTR the Government conducted an in-depth analysis of the factors that hindered the performance and based on the results, initiated an Action Plan to address them (see also Annex 7). A special focus was given to the lagging Phase 2 provinces. A major focus of the Action Plan was on addressing the reporting, billing and information capture issues, including substantial improvements in existing information systems. The national Project team substantially increased the frequency of supervision visits to the provinces, and regional meetings were held regularly. Different selected individuals in the national team were each “assigned” to one province. As part of the Annual Performance Agreements signed between the national team and the provincial teams, a list of the larger health providers was identified in each province. The provincial teams committed to closely following these providers, and health coverage targets were included in the Annual Performance Agreements signed between the provincial teams and the providers.

Table 2: Implementation Weaknesses Identified in QALP-2 Report (June 2010) and MTR (Nov. 2011) Weakness/Challenge, and Source of Findings (QALP-2 and/or MTR) Mitigation Measures, Negative Impacts (If Any), Steps Taken to Address Weakness The provinces are Constitutionally autonomous. The Central Government was thus As part of the project design and implementation, innovative ways were found to address not able to easily control the actions of the provinces, and instead had to incentivize this issue and to incentivize the provinces. them to act appropriately regarding implementation (QALP-2, MTR). The QALP-2 provided a rating of MU for implementation by the Government on Ultimately, the INT investigation did not result in any proven corruption. But a procurement issues, mainly due to an INT investigation that had just been launched at Governance and Anti-corruption (GAC) Plan was developed in early 2011, and agreed the request of the Minister of the Health (which was seen as positive by the QALP-2) between the Government and the Bank. In June 2012 a Bank supervision mission found (QALP-2). that most of the activities included in the GAC Plan had been completed, except for four. Good progress was noted on these four actions in the last ISR (see Annex 12). Execution of the amount allocated for Component 3 (Communications and Execution for this component was 4% at Project close. But communication activities were Community Outreach) was very low – 2% at the time of the MTR (QALP-2, MTR). carried out and financed by other funding sources (mainly Government own revenues), at the national and provincial levels. Annual Plans of Communication were executed for 23 (out of 24) provinces, as planned. Various information and dissemination activities were undertaken at national level, e.g. via television and production of audio-visual and graphic materials. Also, this component accounts for just 5.6% of the original Project allocation. Political changes at the highest levels slowing down the Government’s response to Project units at the central and provincial levels were reinforced, and this was a key factor implementation problems (QALP-2). behind the substantially improved performance in the last year of the Project. Challenges in changing a medical culture to accept and follow through on working These issues were successfully addressed as part of the Action Plan developed after the under an incentive system; underestimation of the capacity of the provinces to engage MTR, which required close engagement between the national Project unit and the in a new results-oriented approach for the health care system (QALP-2, MTR). provinces, and between the provinces and health providers. Incomplete or late reporting, incomplete capturing of information, incomplete or A major focus of the Action Plan was on addressing the reporting, billing and information faulty health service billing and bottlenecks during the registration processes at the capture issues. Substantial improvements were made in existing information systems, and provider level, lack of appropriate information systems in many provinces – and lack this was key to the acceleration in Project indicators towards the Project close. of harmonization of the systems across provinces (due to challenges posed by the Federal structure – see above) (QALP-2 and MTR, but much more in-depth in MTR). Delays in Impact Evaluation (IE) activities and in the MTR (QALP-2). Although IE activities were delayed, the Bank and Government teams showed a great degree of adaptability, and solid results have now been obtained despite the challenges (see Annex 13). The MTR took place later than expected, but as part of the MTR an in- depth diagnosis was undertaken of the causes of the lagging tracer (and PDO) indicators, and this was instrumental for the development after that of a successful Action Plan. The MTR found that the PDO indicator targets were too ambitious, given the timeline The high degree of achievement of the PDO indicators (see Section 3) suggests that most for the Project – especially considering the number of new provinces (15) and the of the targets were in fact not too ambitious. The close engagement with provinces that size of the additional target population (70% of the total target population across all took place as part of the post-MTR Action Plan addressed the weaknesses in provincial 24 provinces). The MTR also identified weaknesses in provincial target setting. target setting. There are additional complexities in the larger Central Region provinces, since Ultimately the steps taken under the post-MTR Action Plan led to much-improved oversight over the provision of health care services is handled not by the provinces performance even in these larger Central Region provinces. but by the municipalities (see Section 2.1 for more details) (QALP-2, MTR). There was room for some potential efficiency gains from pricing of services (MTR). This was addressed as part of the close engagement with the provincial level under the post-MTR Action Plan. Towards the end of the Project, some provinces had problems in making their The national Government required that under the new follow-on Plan Sumar program, mandatory contribution of 30% of all capitation payments. Provinces would capitation payments (in 2013) would not be made to any province that has not paid its sometimes not make their payments, but they would typically repay in full the arrears debt (if any) under Plan Nacer. At this point, all provinces have paid their debt under Plan of each year at the start of the following year. (See Annex 12 for details.) Nacer. Note: MS = Moderately Satisfactory; MU = Moderately Unsatisfactory.

33. The Action Plan addressed the major weaknesses identified (see Table 2) and had a positive impact on the performance of the tracers, especially for the Phase 2 provinces – and hence also on the PDO indicators. Between the January to April 2011 “cuatrimestre” (latest data at the time of the MTR) and the May-August 2012 “cuatrimestre”, the improvement in tracer- related indicators for the Phase 2 provinces ranged from 22% to 43% of the target population within a period of 16 months (see Table 1 of Annex 2). Improvements were also made in several indicators for Phase 1 provinces, but less so since they had started at higher levels to begin with. Overall performance for the PDO indicators improved significantly, since seven of the 11 PDO outcome indicators were linked closely with provincial tracer indicators.

34. The disbursement rate and the provincial “resource transfer rates” also improved substantially. The disbursement rate rose sharply, from 59% at the time of the MTR to 100% just after Project close (see Annex 1). And by the Project close, the average “resource transfer rate” of the 15 lagging “Phase 1” provinces was 83.4%, as compared to 62.5% at the time of the MTR.

35. A key feature of implementation was the constant interaction between the Bank and Government teams on the ground. Most of the Bank team consisted of country-based staff with strong skills who engaged constantly with the Government team. This close engagement, as well as the Government team’s pro-active efforts, was key in addressing the many emerging challenges.

36. The following are some ratings changes that occurred towards the end of the Project: (a) The Project Management rating was raised from MS to S in the last two ISRs, given the steps taken by the Government to develop and implement an Action Plan successfully, after the MTR; and (b) in the last ISR, the social safeguards rating was raised from MS to S, and the PDO rating was raised from MU to MS (see Sections 2.4 and 3.2 for respective explanations).

2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization

37. Design of M&E: This included: i) internal oversight; ii) external concurrent audits; and iii) Impact Evaluations. The Project tracked results indicators for level of coverage and achievement of outcomes included in the tracer indicators, and monitored the quantity of services delivered.

38. Project Results Framework and Linkages at Different Levels: Eight of the Project’s 11 PDO outcome indicators were closely linked to provincial enrolment or tracer indicators (which were used to determine the level of the performance payments to the provinces). Furthermore, the enrolment and tracer indicators in each province were closely linked to the services in the Nomenclador (the package of services whereby health service providers received a fixed fee per unit of service provided). 3 Thus, provincial performance – which was linked to the PDO outcome indicators – was in turn closely linked to the performance of service providers. This alignment of performance indicators at different levels represents good design, except for the larger Central Region provinces where municipalities were key but received no performance payments.

3 For example, one of the provincial tracer indicators was the proportion of eligible pregnant women with the first antenatal care visit before 20th week of pregnancy. At the same time, antenatal care visits are services for which health facilities receive fee-for-service payments (subject to the required protocol being followed). So, stimulating production of these services at the health facility level in turn stimulates overall achievement of this particular tracer indicator.

39. Implementation and Utilization of M&E: The Project’s outcomes and outputs were adequately monitored throughout implementation. The provincial tracer targets were not always closely linked to the PDO indicator targets, but this linkage was emphasized towards the end of the Project under the Action Plan developed after the MTR. The External Concurrent Audits (ECAs) generally worked well, despite procurement-related delays occurring twice (see Annex 12).

40. Evaluation Agenda Including Impact Evaluation (IE): The Project had a broad evaluation agenda, including in-depth IE, but this in turn led to initial challenges and delays. For example, the baseline IE survey for the Phase 2 provinces was conducted in 2008/09 instead of in 2007 as originally planned. There were other challenges with the surveys and survey data. But the Government and Bank teams have been innovative and adaptable in responding to unanticipated changes on the ground, and were ultimately able to use sources of information other than those originally anticipated, to adapt the evaluation strategy and generate solid results (see Annex 13).

41. Importantly, the Government shifted towards a more comprehensive evaluation agenda, going beyond IE. Quantitative and qualitative studies outside of the IE exercise were implemented to assess critical aspects of the program that could not be covered under the IE sub- program, including, among others: (i) a study examining how health workers’ motivation is affected by an RBF program; (ii) an assessment of users' satisfaction; (iii) an analysis of the synergy between the Universal Child Allowance program and Plan Nacer; and (iv) a review of the results linked to the implementation of a CHD Surgery Network. This evaluation agenda was financed mostly from the Government’s own funds – a sign of direct involvement and leadership on the part of the Government in institutionalizing a broad evaluation agenda for the program.

2.4 Safeguard and Fiduciary Compliance

42. Environmental and Social Safeguards: APL-1 financed a high-quality Strategic Environmental Assessment on environmental risks affecting maternal and child health. QALP-2 provided an MS rating for social safeguards in the quality of design, mentioning insufficient attention to the mechanisms of implementation (see Annex 11). It also accordingly provided an MU rating for implementation and Bank supervision on social aspects, based on information available as of March 2009. The rating for compliance with the Indigenous Peoples policy (OP 4.10) had been downgraded to MS in June 2009, not long before QALP-2. In November 2012, it was upgraded to Satisfactory again (in the last ISR), based on much-improved implementation at the later stages, including: (a) good overall progress on implementation of the Indigenous Peoples Plans (IPPs) of 2010/2011; (b) the activities in the 2010/2011 IPPs that were not completed were discussed in detail as part of consultations for a new round of IPPs, provided to the Bank for 10 provinces in October 2012; (c) for the 5 new provinces that triggered OP 4.10, 4 social assessments had been undertaken.

43. Financial Management (FM): The rating for FM was Satisfactory until December 2009 and was then downgraded to MS (due to moderate shortcomings at the provincial level identified by the concurrent Auditor). It remained that way until the Project closed. QALP-2 also noted delays in some of the audit reports and unaudited IFRs, but these delays were rapidly addressed by the client.

44. Procurement: (see also Annex 12): QALP-2 provided a rating of MU for implementation by the Government on procurement issues, due mainly to an INT investigation that ultimately did not result in any proven corruption (see Table 2). A Governance and Anti-Corruption (GAC) Plan

was developed in early 2011, and progress on this has been good (also in Table 2). The MTR of November 2011 found that overall performance for procurement implementation had improved. Just before the MTR, the procurement rating in the ISRs was upgraded from MU to MS. An action plan was anyway developed at the time of the MTR based on a recent review of procurement in the health sector program. Performance regarding this plan has been satisfactory. In May 2012, an external consultant conducted an Independent Procurement Review (IPR), and the Government then produced an Action Plan in late 2012 to address its findings (agreed to by the Bank).

2.5 Post-completion Operation/Next Phase

45. A new project, the Provincial Public Health Insurance Development Project, PHIP, Sumar (for US$400 million, P106735), was approved on April 28, 2011 to support similar goals as APL-2. But aside from uninsured under-six children and pregnant women, it also includes additional population groups: children and youth up to the age of 19, and women between the age of 20 and 64, that are uninsured (9.4 million instead of 2 million people covered).

3. Assessment of Outcomes

3.1 Relevance of Objectives, Design and Implementation

46. The objectives, design and implementation of this flagship RBF Project were – and still remain – highly relevant for the Argentine health sector, for the country as a whole and for the many RBF initiatives in other countries. The national government and the provinces have demonstrated continuous support and strong interest throughout APL-1 and APL-2. The design of the operation included a highly innovative finance mechanism and linked the national level with the provincial and provider level.

47. This Project is considered one of the most important Projects in Argentina’s portfolio in the Human Development sector in the past decade, and is mentioned as a key Project in the 2010-2012 Country Partnership Strategy. It started an RBF approach in the health sector that has been followed now in many other countries, in Latin America and elsewhere. Design and implementation aspects are highly relevant for other countries with RBF initiatives, which look to the Argentina Plan Nacer experience to draw lessons. International RBF seminars often invariably feature the experience of Plan Nacer, discussing design as well as implementation aspects.

48. The Project is thus rated by this ICR as having High relevance of objectives, design and implementation.

3.2 Achievement of Project Development Objectives

49. After the implementation of the Action Plan formulated after the MTR (see above), the program’s performance improved substantially, and ten of the eleven Project outcome indicators were found to be achieved or almost achieved – with five targets being surpassed by a large margin – at the Project close. This can be seen from Table 3. (See Annex 2 for details of data sources and related calculations.) Note that “surpassed by large margin” indicates a final value that exceeds the end target by more than 20%, while “almost achieved” indicates a final value that is within 90% of the end target.

Table 3: Achievement Level of PDO Outcome Indicators (Summary) Ratio of Description of (A) (C) (B) End (C) to (A) Achievement PDO Indicator Baseline Final Target (where With Respect to Value Value applicable) End Target PDO Sub-Objective (a): to increase access by eligible uninsured mothers and children to basic health services (1) % of eligible population voluntarily Surpassed By 0% 80% 97% enrolled in program. Large Margin (2) % of eligible pregnant women with first antenatal care visit before 20th week of 23% 70% 67% 2.9 Almost Achieved pregnancy (3) % of eligible pregnant women who get VRDL during pregnancy and antitetanic 45% 90% 83% 1.8 Almost Achieved vaccine previous to delivery. (4) % of eligible children < 18 months old with 45% 95% 77% 1.7 Not Achieved coverage of measles vaccine or triple viral. (5) % of eligible puerperal women that Surpassed By received at least one Sexual and Reproductive 27% 60% 90% 3.3 Large Margin Health Care consultation. (6) % of eligible children 1 year old or less, with all well child consultations up to date 12% 50% 45% 3.5 Almost Achieved (percentile of weight and height). (7) % of newborns from enrolled pregnant 47% 85% 90% 1.9 Surpassed women weighing more than 2,500 g. (8) % of newborns, from eligible pregnant 47% 92% 93% 2.0 Achieved women, with Apgar score > “6” at minute 5. Achievement Index for Sub-Objective (a) 105 Surpassed PDO Sub-Objective (b): to strengthen the incentive framework for efficiency and focus on results between the national level and the eligible provinces and among eligible provinces and service providers….. (9) % of NMH-PHM annual performance Surpassed By 0% 60% 79% agreements successfully implemented. Large Margin (10) % of authorized providers under annual performance agreements and provider payment Surpassed By 0% 50% 95% mechanism with its respective participant Large Margin province. (11) % of Tracer targets achieved by the Surpassed By 0% 70% 94% participant provinces in last year billing period. Large Margin Surpassed By Achievement Index for Sub-Objective (b) 152 Large Margin

50. An Index of Achievement – a simple measure of the extent to which final values exceed or fall short of end targets, on average, for different groups of PDO Indicators – shows that there was over-achievement for both PDO sub-objectives, but especially for Sub-Objective (b) where the final values far exceeded the end targets for all indicators.4 An Achievement Index value more than 100 indicates that the extent to which final values have surpassed the end targets is larger than the extent to which final values have fallen short of the end targets. Table 3 shows that the Achievement Index of PDO Sub-Objective (a) slightly exceeded 100, while the Achievement Index of PDO Sub-Objective (b) exceeded 100 by a very large margin.

4 This Index is calculated by taking the simple average of the ratio of the achievement levels (final value divided by end target) of each Indicator included in the calculation (and scaling to 100). For example, the Index value for Sub- Objective (b) is the result of the following calculation: [(79/60) + (95/50) + (94/70)]/3 x 100.

51. The final values far exceed the baseline values for all Project indicators – in most cases there has been at least a doubling in indicator values – indicating substantial progress during the Project period. For PDO Indicators 2 to 8 which are the most closely linked to final health outcomes, the ratio of the final value to the baseline value ranges between 1.7 and 3.5 (see Table 3).

52. Performance has also been good for the intermediate indicators. Out of ten intermediate outcome indicators, nine have been fully achieved, and the targets have been surpassed for seven of these (see Table 2 of Annex 2). The remaining indicator has been partially achieved.

53. An indication of the substantial impact of the program on targeted health outcomes can be seen from results of a recent rigorous Impact Evaluation (IE) of Plan Nacer. While this evaluation is still continuing, preliminary results indicate, for example, that – controlling for other factors and for selection bias – the number of infants with low birth weight (out of every 1,000 live births) are reduced by 28% for beneficiaries of Plan Nacer, as compared with non-beneficiaries who are eligible but not enrolled in the program. (See Annex 3 for more).

54. A further indication of the program’s impact is the strong downward trends in child and neonatal mortality in “Phase 1” and “Phase 2” provinces, and also in maternal mortality (MM) in the areas where it was highest (“Phase 1” provinces). This can be seen from graphs in Annex 3. MM has not shown a clear rising or falling trend in the “Phase 2” provinces. But it was much lower to begin with in these provinces, which also had a lower percentage of all pregnant women that were uninsured (30%) as compared to the “Phase 1” provinces (41%). So the program lowered MM in the areas where the need was greatest.

55. According to survey data, beneficiaries and health workers are highly satisfied with the program. A survey of beneficiaries (see Annex 5) found their satisfaction level (based on an Index) to be almost 7 on a scale of 10, with higher satisfaction levels among beneficiaries with higher utilization levels under the program. Satisfaction was especially high (9.1) among mothers of children benefitting from CHD treatment paid for by the program. A separate health facilities survey found health workers to have a satisfaction level of 7 (on a scale of 10), when asked about the program’s design regarding the use of funds at the facility level.

56. Taken together, the above information indicates that the Project rating for efficacy (achievement of objectives) should be Substantial. Although the final value was less than the end target in some cases, the shortfall was small for all but one PDO Indicator, and the final values far exceeded the baseline values for all PDO indicators. Even in the one indicator where the final value was substantially less than the end target, the final value was 1.7 times the baseline value. The extent to which the final values exceeded the end targets is larger than the extent to which the final values fell short of the end targets, for both PDO Sub-Objectives – especially (b). Additional justification for the Substantial rating comes from the information available on the Intermediate Indicators, the IE results, and recent trends in child mortality, neonatal mortality and MM.

3.3 Efficiency

57. Economic Efficiency: The Project’s internal (financial) rate of return was estimated at between 8.2% and 8.5% if one takes into account only the benefits from averted neonatal deaths

(see Annex 3 for details). The other benefits not factored into the calculations include benefits from averted deaths of older infants (after the neonatal period); higher productivity due to reduced incidence of low birth weight; reduced maternal mortality (at least in the areas with higher maternal mortality to begin with); and reduced morbidity among infants and mothers. If these additional benefits had been included in the calculations, the internal rate of return would have been far higher.

58. Institutional Efficiency: This ICR agrees with the following statement from QALP-2, which indicates a high degree of institutional efficiency: “Within the general framework of a public sector with considerable discretionary power, and associated vested interests, the Project constitutes a serious attempt at introducing a system of performance indicators and associated resource allocation mechanisms, while leaving intact the institutional structure, thereby introducing an element of incentives for improved performance, which hitherto has been absent”.

59. Minor shortcomings in design and implementation: As discussed in Section 2.1, QALP-2 rated the quality of the Project design as Satisfactory, highlighting both strengths and weaknesses but the weaknesses turned out to be only minor shortcomings when assessed at Project close (see explanations above). QALP-2 rated implementation in June 2010 as MS, and both QALP-2 as well as the subsequent MTR in November 2011 found a range of implementation weaknesses (see Table 2). But as Table 2 shows, the Action Plan developed just after the MTR addressed these weaknesses successfully, resulting in accelerated progress of the Project indicators. The result was an excellent level of achievement of the PDO indicators at Project close (see Table 3). Hence, the implementation weaknesses identified turned out to be only minor shortcomings.

60. The Project is thus rated by this ICR as having Substantial efficiency, since: (i) the internal rate of return is high even when taking into account just one category of benefits (those from averted neonatal deaths); (ii) institutional efficiency was high; and (iii) shortcomings in design and implementation were minor.

3.4 Justification of Overall Outcome Rating

61. This ICR assigns a Satisfactory rating for the Overall Outcome for the Project, based on relevance, achievement of PDOs, and efficiency. The ratings given by this ICR for relevance, achievement of PDOs (efficacy) and efficiency are High, Substantial and Substantial respectively (as explained in the previous three sub-sections). Hence, the Overall Outcome rating is Satisfactory.

3.5 Overarching Themes, Other Outcomes and Impacts

(a) Poverty Impacts, Gender Aspects, and Social Development

62. Poverty Impacts: Those covered by Plan Nacer were from the uninsured population, and consisted of vulnerable population subgroups, which – if not enrolled in the program – did not have access to adequate basic health care. The strategy of starting with the poorest part of the country under APL-1 – the poorer northern provinces – was appropriate.

63. Gender Aspects and Social Development: The Project had a strong gender emphasis, given its special focus on pregnant women, women post-partum (up to 45 days after birth or miscarriage) and their young children. The last ISR refers to good implementation on IPPs (see section 2.4 above), which is an important contributing factor to social development.

(b) Institutional Change/Strengthening

64. It is clear that the reforms and changes brought about by the program are having a lasting impact. It is hard to see how they can be easily reversed.

4. Assessment of Risk to Development Outcome

65. The risk to the Development Outcome is rated as Moderate, more due to challenges in institutional sustainability than in financial sustainability. As mentioned in section 3.5 (b), the program has led to institutional changes and reforms that cannot be easily reversed. Furthermore, the program costs only about 1% of total provincial public health spending on average, and thus financial sustainability does not appear to be an issue, even at the provincial level.

66. The program (Plan Nacer and its successor Plan Sumar) is on its way to being fully mainstreamed at the provincial level, and provincial ownership is growing but will take time to be fully entrenched. There are clear indications of growing provincial ownership, such as a reduction over time in the scope of the problem (noted in Table 2) of delayed provincial Counterpart Funding contributions. At this time, all the provincial debt accumulated under Plan Nacer has been paid – a big achievement in Argentina’s Federal system. Under Plan Sumar, provinces have also begun to finance the salaries of some members of the provincial Project teams.

5. Assessment of Bank and Borrower Performance

5.1 Bank Performance

67. This ICR rates Bank performance on Quality at Entry as Satisfactory, in line with the Satisfactory rating on this issue in the QALP-2. As QALP-2 noted, the Project designed a social insurance system that increased access to health care services for an underserved low income and vulnerable population group of uninsured children and pregnant women. The Project introduced a system of incentives for improved performance in a difficult political context, with vested interests. It was based on an innovative APL approach, which showed to be a well-functioning model for the nine Phase 1 provinces. The Bank’s preparation team consisted of experts with long experience in the field of health, and the Bank team worked in close collaboration with the Government in the preparation process. QALP-2 gave a Satisfactory rating for the following, among others: relevance and quality of the Project indicators; attention given to technical, financial and economic aspects; quality of institutional framework; effectiveness of the Bank’s management of the preparation process; and overall quality of design (see Annex 11). The shortcomings in design noted by the QALP-2 turned out to be minor shortcomings when assessed at Project close (see Section 2.1).

68. This ICR rates Bank performance on Quality of Supervision as Satisfactory (S), for reasons which are now explained. The QALP-2 report of June 2010 rated overall Bank supervision as MU. This rating was based on the following sub-ratings (see Annex 11): (a) S for Focus on Development Effectiveness; (b) MU for Fiduciary, Safeguards and GAC aspects; (c) S for Supervision Inputs and Processes; (d) MS for Candor and Realism of ISRs.

69. Thus, the main problems identified by QALP-2 were with the sub-category of Fiduciary, Safeguards and GAC aspects, but the Bank team addressed these aspects well. Within this sub-category, Satisfactory ratings were given by QALP-2 for FM and procurement. But performance was found to be MS for GAC aspects and MU for social aspects. On social aspects, which relate in particular to poor implementation at the time of QALP-2 on issues relate to the IPPs, implementation improved markedly towards the latter phase of the Project (see section 2.4). This was facilitated by much closer supervision on the part of the Bank on social aspects, including on the part of a Bank social safeguards expert. On GAC aspects, the Bank supported the Government in the development in early 2011 of a solid GAC plan to address governance and corruption issues in the health sector, and implementation of the plan has been good (see section 2.4).

70. The candor of ISR ratings – which had been rated as MS in the QALP-2 – improved substantially after QALP-2, and indeed some of the subsequent ratings were lower than those in the QALP-2 (see Table 2 of Annex 11). In particular, at the time of the MTR of November 2011, the PDO rating was downgraded to MU as compared to the MS rating given by QALP-2. (The ISR PDO rating previously has been S). The end-of-Project rating in the ISRs for the PDO was MS.

71. A competent task team is in place in the country and at headquarters with all relevant skills, as noted in the QALP-2, and this helped the Bank to provide technical assistance towards the production of an effective Action Plan to substantially improve Project performance after the MTR. The TTL in place at the time of the QALP-2 (June 2010) had been in place for two years, and remained until just before the Project close, indicating good continuity.

72. The successful interactions on the Action Plan were made possible by the constant and close engagement by the Bank team on the ground. Aside from the TTL, the core Bank team consisted of committed and capable county-based staff. Discussions focused on key implementation issues, and adaptations had to be made in response to often-changing circumstances on the ground.

73. The extension of the Project closing date by one year (to December 2012) was a sound move, allowing additional time to take the actions needed to improve the performance of the Phase 2 provinces. This was agreed to by the Bank and Government teams.

74. Since the ratings for Quality at Entry and Supervision are both Satisfactory, the overall rating for Bank performance is Satisfactory.

5.2 Borrower Performance

75. This ICR rates the performance of the Implementing Agencies (the National and Provincial Ministries of Health) as Satisfactory for the following reasons: The overall quality of the design process was rated as Satisfactory by QALP-2, and in this ICR. The reasons for this are explained in detail in Section 2. Furthermore, this is certainly attributable in very large part to the Government, since as mentioned in QALP-2: “There has been clear political will on the part of the central and provincial authorities. The decision to widen the scope of the Project to cover a wider set of provinces was in response to demand from the provinces to be included in the program.” In fact, Government ownership of the design process was rated as Highly Satisfactory by QALP-2.

76. Implementation progress was rated as MS by the QALP-2 report for the reasons mentioned above in Section 2.2. Implementation problems led to poor progress in many of the tracer indicators and the related PDO indicators, in particular in the “Phase 2” provinces, and this led to a downgrading of the PDO rating to MU at the MTR of November 2011.

77. However, during the MTR an in-depth diagnosis was undertaken of the reasons for the poor implementation and indicator performance, and the Government subsequently developed an in-depth Action Plan to address the issues raised. The Action Plan was implemented intensively, and the Government was pro-active in expanding the size of the provincial Project teams in several “Phase 2” provinces, to help address local complexities that had not been anticipated in the design phase. Performance improved substantially since then, especially in the lagging “Phase 2” provinces. As a result, tracer indicator performance improved substantially, and so did the performance of the seven PDO indicators linked to the provincial tracer indicators.

78. The Government team had strong skills and showed flexibility in making adaptations a number of times as needed, responding to new information or circumstances. For example, in 2010 the package of services covered was expanded to also include CHD treatment and additional complex maternal health services, due to an analysis showing the importance of these additional services. In November 2009 and May 2011, cash transfer programs were started first for disadvantaged children and then for disadvantaged mothers. In both cases, the transfers were made conditional on the child/mother being enrolled in Plan Nacer, as well as fulfillment of certain requirements regarding health checkups and immunizations. The Government team also adopted diverse strategies to deal with the delays in enrolment in the larger Central Region provinces, such as a contingency plan to improve performance in Buenos Aires province within 100 days.

79. There were sometimes delays in provincial counterpart funding contributions (due to issues such as inflexible provincial administrative procedures, cash flow constraints and financial planning difficulties), leading to several provinces owing debt under Plan Nacer at the time of Project closing. The national Government responded by making capitation payments under the follow-on Plan Sumar program (in 2013) conditional on full repayment of Plan Nacer debt on the part of each province. At this time, as a result, provincial debt under Plan Nacer has been paid in full, for all provinces. Despite the challenges with IE, the Government and Bank teams showed a great degree of adaptability, and solid results have now been obtained (see Annex 13). Finally, the agreement to extend the Project closing date was key, allowing additional time needed for the Phase 2 provinces.

80. All of this justifies an overall Satisfactory (S) rating for the Implementing Agencies for implementation. The overall rating for the Implementing Agencies’ performance is thus S, in line with an S rating for both the quality at entry (preparation) and implementation phases.

81. Performance of the Ministry of Economy and Public Finance (MEPF), with which the Loan Agreement was signed, is also rated as Satisfactory (S), and thus overall Borrower performance is also rated S. The Project was implemented by the Ministries of Health, who played the major role in its implementation. But MPEF played a suitable support role, allowing the NMH to take the lead in putting in place an innovative, transformative and effective program that was important for the country. MEPF’s strong support for Bank-financed health projects is in line with its recognition of the health sector as key to the Bank’s portfolio in Argentina. Since the performance by both the Government and Implementing Agencies is S, overall Borrower performance should also be rated S.

6. Lessons Learned

82. Differences in local institutional capabilities need to be analyzed and taken into account: For Argentina, and other countries with a federal institutional framework, it is critical to take diversity in terms of local institutional capacity into account and adjust the Project accordingly. An in-depth analysis is required to identify differences in local capabilities and develop a strategy on how to address the specific circumstances of different provinces or localities.

83. RBF can work well even in a Federal system, but all major actors at different levels need to receive performance payments, and the different sets of performance indicators at different levels need to be aligned. Alignment of province-level performance indicators and health facility-level performance indicators was key to overall performance of the program, and worked well in most cases. The exception is with the larger Central region provinces where the municipalities had direct oversight over the health facilities but did not receive performance payments. This clearly slowed down progress in these provinces.

84. Close engagement with actors at lower levels – including health providers – can make a large difference to overall performance of an RBF program. This type of close engagement was a key component of the successful Action Plan developed after the MTR. This Plan involved frequent supervision visits by national Project team members to the provinces, regular regional meetings and a commitment by provinces to work closely with larger health providers.

85. A strong focus on information systems (including for reporting and billing) is key for the success of a larger-scale RBF program. A major reason for the substantial improvement in performance resulting from the Action Plan after the MTR was the Plan’s emphasis on strengthening information systems.

86. Institutional changes/strengthening takes time: Another reason for the better performance of Phase 1 provinces could have been that they had more time to implement the Project. The time frame for Phase 2 provinces was likely too short and better results could have been achieved if the Project period was longer. If the Project time frame could not be changed, then it would have been better to be less ambitious with the indicator targets. More broadly, provincial ownership and institutional sustainability have been growing but will take time to be fully entrenched.

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners

(a) Borrower/implementing agencies

87. Through Plan Nacer, the Nation and the Provinces began to implement a policy of prioritization, public insurance and explicit health coverage for the most vulnerable members of the population. The program has achieved results along several different dimensions, many of which are not reflected in the PDO or Intermediate Indicators. See a summary of the Government’s contribution to this ICR, in Annex 7, for more details.

(b) Cofinanciers: Not Applicable.

Annex 1: Project Costs and Financing

Table 1: Project Cost by Component (in USD Million equivalent)

Appraisal Actual/Latest Percentage of Components Estimate Estimate Appraisal (USD millions) (USD millions) 1. Implementation of 589.00 727.56 the Maternal-Child (of which (of which 124% Health Insurance 242.70 from 249.66 from Program IBRD) IBRD) 2. Strengthening National and Provincial 10.20 16.02 157% Ministries of Health Stewardship Capacity 3. Communications and 17.00 0.67 3.9% Community Outreach 4. Program Monitoring, Evaluation and 14.60 16.79 115% Concurrent Auditing Systems 5. Project Management 1.60 2.58 161% and Administration Total Baseline Cost 286.10 285.73 99.8% Physical Contingencies Price Contingencies Total Project Costs Front-end fee PPF - - - Front-end fee IBRD Total Financing 300.00 285.73 95.24% Required

Table 2: Financing Appraisal Actual/Latest Type of Estimate Estimate Percentage of Source of Funds Cofinancing (USD (USD Appraisal millions) millions) Borrower 346.3 477.9 138% International Bank for Reconstruction 300.00 300.0 100.00 and Development

Figure 1: Disbursement Over Time

Table 3: Disbursement per Expenditure Category1 Cate- Category Description Allocated USD Disbursed USD Undisbursed USD Level of gory Exe- cution % 1 Consultants Services including Auditing 39,900,000.00 40,305,812.60 -405,812.60 101% Services

2 Goods 45,150,000.00 41,828,657.08 3,321,342.92 93% 3 Training 2,100,000.00 1,932,321.14 167,678.86 92% 5 Capitation Payments - Northern Region 39,000,000.00 36,128,582.09 2,871,417.91 93% (Phase 1 Provinces) 6 Capitation Payments - Central, Patag, Cuyo 120,000,000.00 118,561,245.16 1,438,754.84 99% Region (Phase 2 Provinces) 7 Capitation Payments - Heart Surgery, 53,100,000.00 47,612,131.32 5,487,868.68 90% Maternal Care Services 8 Operating Costs 750,000.00 440,663.34 309,336.66 59% 9 Advances to 12,802,539.21 -12,802,539.21 Designated Account A 10 Advances to 388,048.06 -388,048.06 Designated Account B Total 300,000,000.00 300,000,000.00 0.00 1. As of the date of this ICR, the final Interim Financial Report (IFR) still has not been processed, hence the above figures are not the final figures.

Annex 2. Outputs by Component

A. Evolution of Project PDO and Intermediate Outcome Indicators

1. Table 1 shows the values of the PDO Indicators of the Project, over time. The table reports the baseline values (column A), values at the time of the Midterm Review (column E), and latest known values/estimates (columns F, G and H).

2. For PDO Indicators 2 to 8 – which are linked to the provincial tracer indicators – two sets of “final” values are reported, since the results from the final external concurrent audit are not yet available for all provinces. For the tracer indicators, the procedure is that: (a) data are first obtained, and values calculated, providing “pre-audit” values; and then (b) the external concurrent audit is carried out, producing “post-audit” values for the tracer indicators. The audit always leads to a “downward adjustment” of the results, since some observations are rejected (i.e. the post-audit levels of achievement are lower than the pre-audit levels of achievement). The data are considered finalized for the tracer indicators only after the post-audit values are obtained.

3. At this time, the external concurrent audit for the Phase 2 provinces for the last “cuatrimestre” (September-December 2012) is still ongoing, and its results will not be known in time to be used for this ICR. The results from the external concurrent audit for the Phase 1 provinces for the same “cuatrimestre” are also not yet known, at present. The only period for which post-audit values are available for all provinces at this time is the second “cuatrimestre” of 2012 (May-August 2012).

4. Columns F and G report values for PDO Indicators 2 to 8 for the second “cuatrimestre” of 2012 (May-August 2012) – the last “cuatrimestre” for which post-audit values for the tracer indicators are available for all provinces. Column F reports these values for the Phase 1 and Phase 2 provinces separately, while column G reports these values for all provinces as a whole.

5. Column H reports estimates for PDO Indicators 2 to 8 for the last “cuatrimestre” covered by the Project (September-December 2012), based on estimates of post-audit tracer values (calculated using known pre-audit values). Before the results of the external concurrent audit are known, post-audit values will not be available. But the pre-audit results are available for all provinces. Estimates of the post-audit results were calculated for all provinces, using the following procedure: (i) For each province, data on pre-audit and post-audit levels of achievement in the first and second “cuatrimestre” of 2012 (January-April 2012 and May-August 2012) were used, to obtain an estimate of the typical percentage “downward adjustment” applied as a result of the audits (for each tracer indicator); and then: (ii) this estimated percentage “downward adjustment” was applied to the known pre-audit levels of achievement for each tracer indicator for the last “cuatrimestre” (for each province), to obtain estimates for the post-audit levels of achievement. Finally: (iii) the estimated post-audit achievement levels for each tracer indicator for the last “cuatrimestre” were used to compute estimates for the final values of PDO Indicators, as of December 31, 2012. These are the values reported in column H.

6. The values reported in the Datasheet in front of the ICR are from column (H). However, the final values in column (H) are actually not so different from those of Column (G). It

was decided to use the values of column (H) because for PDO Indicators 2 to 8, these provide data for the last “cuatrimestre” of the Project (even though they are based on estimates of post-audit data, rather than actual post-audit data).

7. Key findings from the results shown in Table 1 are:

• Based on the latest known data (see columns F, G and H), seven out of eleven PDO indicators have been fully achieved or surpassed, while three more have been almost achieved (the latter is based on column H). • Out of the seven indicators that were fully achieved or surpassed, the targets have been surpassed by large margins for all except two indicators. • The Achievement Index for PDO Sub-Objectives (a) and (b) – see Note 4 below the Table for an explanation on this – shows that the extent to which the final values exceeded the end targets more than compensated for the extent to which final values fell short of the end targets. This was true for both PDO Sub-Objectives, but was especially true for Sub-Objective (b). • Towards the end of the Project (after the Midterm Review), performance increased substantially for the Phase 2 provinces in particular, that had been lagging.

8. Note that the values reported in columns (E), (F), (G) and (H) are based on data coming in from the program, i.e. based on services invoiced and paid for as part of the program. Services rendered to the eligible populations that were not paid for under Plan Nacer are not included in the figures. So, the figures under-report the reality, e.g. the percentage of eligible pregnant women that had the first antenatal care visit before the 20th week of pregnancy (Indicator 2) is more than what is reported in columns F, G and H since many eligible pregnant women did indeed have a first antenatal care visit before the 20th week of pregnancy, but without this visit being recorded under Plan Nacer records at the time.

9. Thus, a large part of the improvements seen between the Column E values and the Column F (or G or H) values come from a large increase in the number of women and children registered in the program, and an increase in the number of services paid for and recorded under the program. They do not all reflect an increase in the number of services rendered to the eligible population per se (if one also counts the services that were not paid for and recorded under the program).

10. However, the baseline values (column B) – taken from the Project Appraisal Document – reflect all services given to the eligible population, not just services that were invoiced, paid for and recorded under the program. (The program was not even operational at the time of appraisal in the Phase 2 provinces).

Table 1: Performance of the Project PDO Outcome Indicators (H) Achievement as (F) Achievement of Dec 31, 2012 for as of Aug 31, 2012 Indicators 2 to 8 (E) Achievement for Indicators 2 to (based on estimated (I) at time of 8 (using known post-audit values for Achievement (B) Midterm Review post-audit values (G) Same as tracer indicators, at Project Baseline (by Apr 30, 2011 for tracer in column calculated using close – Value (D) for Tracer indicators),2 and (F) but at known pre-audit description (Natio- (C) Phase 1 indicators; by 31 as of Dec 31, 2012 National values),2 and as of based on nal) in End or 2 Pro- Oct, 2011 for for other level (for all Dec 31, 2012 for values in (A) PDO Indicator PAD1 Target vinces? other indicators) Indicators provinces)2 other Indicators (H)3 (1) Proportion of eligible Phase 1 86.3% 99% Surpassed by population voluntarily 0% 80% 97% 97% Large Margin enrolled in the program. Phase 2 83.3% 96% PDO Sub-Objective (a): to increase access by eligible uninsured mothers and children to basic health services (2) Proportion of eligible pregnant women with first Phase 1 64% 69% Almost 23% 70% 67% 67% antenatal care visit before Achieved 20th week of pregnancy Phase 2 31% 65% (3) Proportion of eligible pregnant women who get Phase 1 88% 89% VRDL test during Almost 45% 90% 84% 83% pregnancy and antitetanic Achieved vaccine previous to Phase 2 49% 81% delivery. (4) Proportion of eligible Phase 1 77% 85% children less than 18 months old with coverage 45% 95% 69% 77% Not Achieved of measles vaccine or Phase 2 24% 60% triple viral. (5) Proportion of eligible Phase 1 88% 94% puerperal women that Surpassed by received at least one 27% 60% 91% 90% Large Margin Sexual and Reproductive Phase 2 49% 90% Health Care consultation. (6) Proportion of eligible Phase 1 46% 53% Almost children 1 year old or less, 13% 50% 42% 45% Achieved with all well child Phase 2 15% 35%

(H) Achievement as (F) Achievement of Dec 31, 2012 for as of Aug 31, 2012 Indicators 2 to 8 (E) Achievement for Indicators 2 to (based on estimated (I) at time of 8 (using known post-audit values for Achievement (B) Midterm Review post-audit values (G) Same as tracer indicators, at Project Baseline (by Apr 30, 2011 for tracer in column calculated using close – Value (D) for Tracer indicators),2 and (F) but at known pre-audit description (Natio- (C) Phase 1 indicators; by 31 as of Dec 31, 2012 National values),2 and as of based on nal) in End or 2 Pro- Oct, 2011 for for other level (for all Dec 31, 2012 for values in (A) PDO Indicator PAD1 Target vinces? other indicators) Indicators provinces)2 other Indicators (H)3 targets achieved by the Large Margin participant provinces in Phase 2 67% 91% last year billing period. Achievement Index for Surpassed by 164 164 PDO Sub-Objective (b) 4 Large Margin Notes: 1. The baseline values are all national values, based on national data (except for Indicators 1, 9, 10 and 11 where the baseline values are only for the Phase 2 provinces since the Phase 1 provinces were not included at the beginning). Since APL-2 was originally meant just for the 15 new Phase 2 provinces, it was planned that the baseline IE study would be used to obtain baseline values for the Phase 2 provinces only (as a group), and these would be used to replace the national baseline values reported in the PAD. But the baseline IE study was delayed substantially, and was ultimately conducted in 2008/09, well after the project started in late 2006. In addition, in mid-2010 the Phase 1 provinces were included in APL-2, in addition to the Phase 2 provinces. At that point, APL-2 became essentially a Project with national coverage. So, there was no longer a need for baseline values for the Phase 2 provinces only, and it is appropriate to use the original national-level baseline values as reported in the PAD. 2. See main text above for explanations regarding pre-audit values, post-audit values and estimations of post-audit values. 3. “Not Achieved” = final value lower than 90% of end target (e.g. lower than 45% if target is 50%). “Almost Achieved” = final value between 90% and 99% of end target (rounded to nearest % point). “Achieved” = final value between 100% and 104% of end target. “Surpassed” = final value between 105% and 119% of end target. “Surpassed by Large Margin” = final value 120% or more of end target. 4. The Overall Achievement Indices were calculated for PDO Sub-Objectives (a) and (b) by taking the simple average of the ratio of the achievement levels (final value / end target) for each PDO Indicator (and scaling to 100). For example, the Index value for Sub-Objective (b) is the result of the following calculation: [(100/60) + (95/50) + (94/70)]/3 x 100. The Overall Achievement Index would have a value of exactly 100 if the final values were exactly equal to the end targets for all indicators. A high Overall Achievement Index indicates that the extent to which final values have surpassed the end targets is larger than the extent to which final values have fallen short of the end targets. 5. The definition of PDO Indicator 9 (percentage of NMH-PHM annual performance agreements successfully implemented) was changed (made more stringent) towards the end of the Project. According to the previous definition, all annual performance agreements that had been signed and were being implemented were counted, regardless of the quality of the implementation (i.e. the word “successfully” had not been given sufficient weight). The level of achievement at the time of the Midterm Review was 100%, based on this previous definition. The final values reported for this indicator in the above table are based on the latest definition agreed to with the Bank which is more stringent. The latest definition is that one should only count performance agreements where: (i) the rate of enrolment of the eligible population is higher than the minimum target agreed to, and also: (ii) at least 70% of all projects in the Strategic Plan of Technical Assistance and Capacity Building (which is a part of the performance agreement) have been executed.

11. Table 2 shows achievement levels of the Project’s ten Intermediate Outcome Indicators. Out of ten intermediate outcome indicators, nine have been fully achieved, and the targets have been substantially surpassed in seven out of these nine indicators. In the next section, achievement of individual indicators linked to each component of the Project is discussed.

Table 2: Achievement for Intermediate Outcome Indicators Achievement as of Level of Intermediate Outcome Indicator End Target December 31, 2012 Achievement (Project Close) (1) Loan disbursements 100% 100% Fully Achieved (2) Capitation payment occurring In at least 9 eligible Capitation payment Fully Achieved, according to approved enrollment lists provinces in central occurring in 24 out of 24 and Surpassed and trazadora systems and southern regions provinces. NHSPT functions (3) PHSPT and NHSPT function In at least 50% of all effectively; PHSPTs Fully Achieved, effectively according to concurrent participating function effectively in 24 and Surpassed and financial audits. provinces out of 24 provinces. (4) Number of authorized providers At least 50% of receiving the basic medical equipment those for whom an 85.3% of authorized Fully Achieved, / vehicles / communication equipment investment project providers and Surpassed according to the annual performance was approved agreement (5) Number of PHSPTs established Established and and functioning, capable of preparing At least 60% of all functioning PHSPTs in Fully Achieved, and negotiating NMH-PMH and participant 24 out of 24 participant and Surpassed PMH-authorized providers’ annual provinces provinces performance agreements. One National (6) NHSPT is established and Direction for NHSPT has been functioning, capable of preparing and purchase of medical established and is Fully Achieved negotiating NMH-PMH annual services has been operational performance agreements established 23 (out of 24) provinces with Annual Plans of Communication Information and executed as planned. (7) Regular information process dissemination Various information and among stakeholders on maternal-child campaign launched dissemination activities Fully Achieved, health care issues in general and at national level and undertaken at national and Surpassed entitlements in the Plan Nacer in in at least 80% of all level, e.g. via television particular participating and provinces production/distribution of relevant audio-visual and graphic material. At least 70% of According to the Estudio eligible population de Monitoreo de la (8) Targeted groups increase, reports (in surveys) Satisfacción del Usuario knowledge of their entitlements under knowledge of Plan y de la Calidad de Fully Achieved, the program and participation in Plan Nacer. At least 50% Atención del Plan Nacer and Surpassed Nacer, and report satisfaction with of enrolled (a household survey), process and results population reports conducted to date in 13 (in surveys) that is provinces (including satisfied with the both Phase 1 and Phase 2

Achievement as of Level of Intermediate Outcome Indicator End Target December 31, 2012 Achievement (Project Close) Plan Nacer provinces) between July Program. 2012 and December 2012: (i) 84.1% of those interviewed knew about Plan Nacer, and (ii) 72.1% of interviewed beneficiaries indicated that they were either “satisfied” or “very satisfied” with the services provided by the program. (9) Project implementation reports available as agreed, including financial Satisfactory reports reports supporting the capitation At least 80% of the Fully Achieved, available in 24 out of 24 payments. Satisfactory reports from reports and Surpassed eligible provinces Concurrent and annual financial auditor (10) Project evaluation implemented: (i) baseline at the end of Year 2; mid Baseline done, midterm term impact evaluation at the end of Partially Achieved evaluation ongoing. Year 3; and final impact evaluation at the end of last year of APL-1 Note: NHSPT = National Health Services Purchasing Team; PHSPT = Provincial Health Services Purchasing Team; NMH = National Ministry of Health; PMH = Provincial Ministry of Health.

B. Discussion of Individual Components, Outputs and Achievements

12. Component 1: Implementation of the Maternal-Child Health Insurance Program (MCHIP) (Original Bank allocation of US$242.7 million; final expenditure of US$249.7 million from IBRD)

13. Description: This component aimed to ensure the sustainable implementation and functioning of the MCHIP. It provided support for: i) capitation payments for MCHIP services by the National Ministry of Health (NMH) to participating provinces covering a share of the costs of the basic service package, calculated on a per capita basis; ii) Equipment (medical, transportation and communications) for basic health care facilities that supply the MCHIP package; iii) Technical assistance and training programs for the provincial Ministries of Health to develop systems, instruments, and skills necessary to implement and run the MCHIP; iv) Health service delivery training for providers delivering basic services under the MCHIP; v) Information technology equipment and consultant services to upgrade and expand information systems for monitoring the implementation of the MCHIP; vi) Technical assistance and training for the management of participating health service providers to strengthen areas including their billing capacity, development and implementation of provider data systems.

14. Achievement of Intermediate Results for this Component (see also Table 2):

• Capitation payment occurring according to approved enrollment lists and trazadora systems (Fully Achieved and Surpassed). • PHSPT and NHSPT function effectively according to concurrent and financial audits (Fully Achieved and Surpassed). • Number of authorized providers receiving the basic medical equipment/vehicles/communications equipment according to the annual performance agreement (Fully Achieved and Surpassed).

15. Component 2: Strengthening the National and Provincial Ministries of Health Stewardship Capacity (Original Bank allocation of US$10.2 million; final expenditure of US$16 million)

16. Description: The aim of this component was to adapt the provincial Ministries of Health to meet the implementation requirements (information, managerial) of the MCHIP. It includes the essential and major structural change of separating the purchasing and provision of services and setting up and training national and provincial health service “purchasing” team. This component provided financial support for: i) reorganizing participating Provincial Ministries of Health (PMHs) in both staffing and interrelationships, as necessary; ii) improving epidemiological information, financial, and human resource management systems; iii) completing studies essential for NMH policy formulation.

17. Achievement of Intermediate Results for this Component (see also Table 2): • Number of PHSPTs established and functioning, capable of preparing and negotiating NMH-PMH and PMH-authorized providers’ annual performance agreements (Fully Achieved and Surpassed). • NHSPT is established and functioning, capable of preparing and negotiating PMH-NMH annual performance agreements (Fully Achieved).

18. Component 3: Communications and Community Outreach (Original Bank allocation of US$17 million; final expenditure of US$667,000)

19. Description: The aim of this component was to ensure the effectiveness of the MCHIP through providing the eligible population, particularly those who have historically been marginalized, with enough knowledge about, and motivation to use, the services being offered. The component was supposed to provide financial support for: Consultant services, incremental ministry operating costs, event organization, and media communication services to support two main lines of communication: i) Dissemination of detailed information about the program among major stakeholder groups (provincial governments and their populations, the Federal Health Council (COFESA), the medical profession and insurance agency managers and staff). ii) Community outreach to increase participation of the eligible population.

20. Achievement of Intermediate Results for this Component (see also Table 2): • Regular information process among stakeholders on maternal-child health care issues in general and entitlements in the Plan Nacer in particular (Fully Achieved and Surpassed).

• Targeted groups increase, knowledge of their entitlements under the program and participation in Plan Nacer, and report satisfaction with process and results (Fully Achieved and Surpassed).

21. Component 4: Program Monitoring, Evaluation and Concurrent Auditing Systems (Original Bank allocation of US$14.6 million; final expenditure of US$16.8 million)

22. Description: This component provided financial support for: i) Information technology design services, software and equipment, and training for NMH and PHMs staff to upgrade the monitoring of health provider performance in collecting and reporting information; ii) External concurrent auditing of key elements underlying transfers of capitation payments; iii) In-depth Project evaluation, including the completion of the baseline for impact indicators and Project impact evaluations at mid-term and closing.

23. Achievement of Intermediate Results for this Component (see also Table 2): • Project implementation reports available as agreed, including financial reports supporting the capitation payments. Satisfactory reports from concurrent and annual financial auditor. (Fully Achieved and Surpassed). • Project evaluation implemented: baseline at the end of PY2; mid-term impact evaluation at the end of PY3; and final impact evaluation at the end of last year of APL-1. (Partially Achieved – see Table 2 for more details).

24. Component 5: Project Management and Administration (Original Bank allocation of US$1.6 million; final expenditure of US$2.6 million)

25. Description: This component provided financial support for operational expenses such as travel costs for the provinces travel and per diem costs for coordination meetings, mainly for the National Health Services Purchasing Team (NHSPT) activities.

Discussion of Achievement of Individual Components

26. All intermediate outcome indicators were achieved, for all Components, except for two indicators, for which achievement was partial. One of these indicators with only partial achievement was for Component 3 (Communications and Community Outreach), which also had a very low level of expenditure of just US$667,000 out of a total allocation of US$17 million.

27. This component had some difficulties, as raised in the ISRs. The Communication Strategy was ultimately only partially implemented, and disbursement of the component was just 4% at Project close. This component had a rating of MU at Project close. However, the government stated that first, communication campaigns were absorbed by the Government Communication Unit without the direct participation of the National Ministry of Health, and second, the government wanted to avoid an increase in the demand for health services in urban areas to an extend more than the health system could deliver. On the other hand, there was a lack of demand side interventions to bring people to health facilities particularly in rural areas were access was low.

28. However, the goals of the component were still partially achieved, as shown by the achievement levels of the two Intermediate Outcome Indicators for this component (one was fully achieved and the other partially achieved). This was because many of the activities were still carried out, financed from Government own sources. The Government also noted that communication campaigns were absorbed by the Government Communication Unit, outside of the National Ministry of Health.

29. The partial achievement of one of the indicators for Component 4 (Program Monitoring, Evaluation and Concurrent Auditing Systems) was due to challenges with the planned Impact Evaluation for the program. See Annex 13 for a full discussion of this.

Annex 3. Economic and Financial Analysis

A. Trends in Enrolment 1. Enrolment in Plan Nacer has tended to rise rapidly from the time a province is newly enrolled, and at this time close to 100% of eligible children are enrolled. But the enrolment rate among women has been lagging to some extent (although it is still high). Figures 1 through 4 show basic trends in enrolment of eligible women and children in Plan Nacer, over time, separately for the Phase 1 and the Phase 2 provinces.

160,000 Figure 1. Number of women enrolled 140,000

120,000

100,000 104,094 Phase 2 102,113 81,699 80,000 provinces Phase 1 59,587 provinces 60,000 39,672 24,888

40,000 0

0 43,977 43,773 42,655 44,245 48,262 43,202 20,000 35,863 22,450 0 3,2670 2004 2005 2006 2007 2008 2009 2010 2011 2012

2,000,000 Figure 2. Number of children enrolled 1,800,000

1,600,000

1,400,000

1,200,000 Phase 2 1,240,090 provinces 1,078,986 1,000,000 Phase 1 860,952 provinces 800,000 495,494 364,118 600,000 215,183 400,000 0 0 572,002 476,532 484,885 481,800 496,567 513,091 200,000 394,629 309,512 0 30,5020 2004 2005 2006 2007 2008 2009 2010 2011 2012

80 Figure 3. Percentage of women registered, out of eligible 70 population

Phase 1 provinces 20

Phase 2 provinces 10

0 2004 2005 2006 2007 2008 2009 2010 2011 2012

Figure 4. Percentage of children registered,

100 out of eligible population

60 % 0f eligible population

Phase 1 provinces 20 Phase 2 provinces

0 2004 2005 2006 2007 2008 2009 2010 2011 2012

2. These figures show that enrolment has initially moved sharply upwards for both Phase 1 and Phase 2 provinces, but has then plateaued or even fallen to some extent once it has hit the 60% to 70% range for women. However, for children enrolment rates have continued to move up to 100% for both Phases 1 and Phase 2 provinces.

3. The reason for the lower enrolment rates for women is that it has proved challenging, at times, to attract early registration (enrolment) on the part of women. This would need to occur via health centers, and yet it has proved harder to get health centers to participate actively in the program, than hospitals. The reasons for this include higher capacity on the part of hospitals, as well as better record-keeping and registration systems, and better ability to invoice properly.

Thus, often, a woman is registered in Plan Nacer only when she goes to a hospital for her birth. Registration rates for children are high, on the other hand, because children get enrolled from the time the mother goes to the hospital to give birth, and they generally remain enrolled afterwards until they reach age six.

B. Trends in Infant Mortality, Neonatal Mortality and Maternal Mortality

4. Plan Nacer aimed to address basic health care needs of uninsured pregnant/lactating women and young children. Thus it is illustrative to look at trends in infant, neonatal and maternal mortality, with the important caveat that there are many factors that affect these indicators aside from Plan Nacer and the characteristics of this program. Economic circumstances (at the national and local level) affect these indicators, for example, and so movements (or lack of movement) in these indicators cannot always be attributed to Plan Nacer.

5. With that caveat, the trends in infant, neonatal and maternal mortality are now analyzed.

Figure 5. Infant Mortality Rate National, Phase 1 Provinces and Phase 2 Provinces

19 19

16 16 16 16

Rate per 1,000 live 1,000 births Rate per 15 14.4 14 14 14 14

13.3 13.3 13 13 12.9 12 12.5 12 12 12.1 12 12 11.9 11.7 11 11 11 11

10 2004 2005 2006 2007 2008 2009 2010 2011

Phase 2 Phase 1 National

Figure 6. Infant Mortality Rate (IMR) - Ratio of IMR of Phase 1 Provinces to IMR of Phase 2 Provinces

1.5

1.4

1.3

1.3 1.3 1.3 1.3 2004 2005 2006 2007 2008 2009 2010 2011

Figure 7. Neonatal Mortality Rate National, Phase 1 Provinces and Phase 2 Provinces 16

13 13 12

11 11 11 10 10 10 9.7 10 9 9 9 9 8.9 Rate per 1,000 live births 8.5 8.5 8.3 8 8 8 8 8 8.0 7.9 8 7.6 7 7 7

6 2004 2005 2006 2007 2008 2009 2010 2011 Phase 2 Phase 1 National

Figure 8. Neonatal Mortality Rate (NMR) Ratio of NMR of Phase 1 Provinces to NMR of Phase 2 Provinces

1.5

1.4

1.3 1.3 1.3

1.3

1.2 2004 2005 2006 2007 2008 2009 2010 2011

Figure 9. Maternal Mortality Rate National, Phase 1 Provinces and Phase 2 Provinces

Rate per 1,000 live births 1,000Rate per 5

2 2004 2005 2006 2007 2008 2009 2010 2011 Phase 1 Phase 2 National Linear (Phase 1)

6. The Figures (5 to 9) indicate the following:

7. The infant mortality rate (IMR) and neonatal mortality rate (NMR) have been falling in both the Phase 1 and Phase 2 provinces, but the fall has been faster in provinces included in Plan Nacer. Figure 5 (for the IMR) and Figure 7 (for NMR) show the falling trends in both groups of provinces. Figure 6 shows that the ratio of the IMR in the Phase 1 Provinces to the IMR in the Phase 2 Provinces fell substantially between 2004 and 2007 – years in which the Phase 1 provinces were in Plan Nacer (but not the Phase 2 provinces). The Phase 2 provinces entered the program in 2007, and from then on the ratio of the two IMRs remained roughly constant. In short:

• The IMRs of the Phase 1 Provinces and Phase 2 Provinces converged to each other when Plan Nacer was operational in Phase 1 provinces (from 2004 to 2007). • Once the Phase 2 provinces were also included (from 2007 onwards), the ratio of the two IMRs remained roughly constant.

8. Similar observations can be made for the NMR. As Figure 8 shows, the NMRs of the two provinces converged from 2004 to 2007, when only Phase 1 provinces were included in the program. Once the Phase 2 provinces were included from 2007 onwards, the divergence in the two NMRs ceased. In fact, the NMR of the Phase 2 provinces actually fell faster than the NMR of the Phase 1 provinces between 2009 and 2011. (This can be seen from the slight rise after 2009 in the line tracking the ratio of the two NMRs in Figure 8, after falling quite sharply for several years.) The improved relative performance in the Phase 2 provinces regarding the NMR may be due, in turn, to the faltering in the previously rising trend in the enrolment rate of eligible women from 2009 onwards in the Phase 1 provinces (see Figure 3).

9. Maternal mortality (MM) has shown a falling trend in the Phase 1 provinces, where it is highest to begin with. However, there is no obvious trend – rising or falling – in the Phase 2 provinces since they joined the program. This can be seen from Figure 9. While the lack of a falling trend in the Phase 2 provinces is not ideal, these provinces do start out with a relatively low maternal mortality rate already.

10. A recent analysis of MM in Argentina shows that there are many factors affecting it that need to be modified outside of a program like Plan Nacer, and by itself such a program cannot lower MM rates below a certain point. This was one of the findings underlying the recent Operational Plan for Reducing Maternal Mortality in Argentina, developed by the National Ministry of Health in Argentina. To reduce MM below levels already seen in the Phase 2 provinces, additional steps are needed, including, among others: (a) redefining the roles of hospitals, especially the ones that currently do not comply with all of the conditions for providing adequate maternal care; (b) strengthening the referral network, and assigning appropriate roles to health facilities in the network; and (c) providing additional technical assistance to hospitals and provinces to that they can maximize use of (and invoicing of) the new sub-package of additional complex maternal health services that were included in 2010 into the “Nomenclador” of Plan Nacer (see main text of this document). Utilization of and invoicing for these additional complex services, designed to address high-risk births, was not high in 2011 and much of 2012, but is now picking up.

11. The maternal mortality rate of the Phase 2 provinces should fall under the follow-on program Plan Sumar, as the new sub-package of additional complex maternal health services becomes entrenched, and as other steps are taken outside of Plan Sumar. Plan Sumar is the follow-on program, of Plan Nacer. It will support similar services as Plan Nacer, and will add additional population subgroups in addition to pregnant/lactating women and under- six children. As the steps mentioned in the previous paragraph and others are undertaken, in line with the Operational Plan for Reducing Maternal Mortality in Argentina, the chances of falls in MM even below the levels seen in the Phase 2 provinces are good.

12. Further analysis shows that there is a particularly close relationship between health care coverage rates of pregnant women – either by formal health insurance or by Plan Nacer – and the neonatal mortality rate (NMR). For this analysis, an indicator was created called “Health Coverage Gap (HCG) Among Pregnant Women”. Values of this indicator were calculated separately over time for the Phase 1 Provinces and for the Phase 2 Provinces. This indicator is defined as the percentage of all pregnant women in the entire population (either within the Phase 1 or the Phase 2 provinces) that are not covered by either formal health insurance or by Plan Nacer (which is conceptualized as a public health insurance program).

Figure 10. Health Coverage Gap (HCG) Among Pregnant Women Versus Neonatal Mortality Rate (Phase 1 provinces)

60.00 11.00

50.00 10.50

40.00 10.00 30.00 9.50 20.00

9.00 Rate Mortality Neonatal 10.00

0.00 8.50 2006 2007 2008 2009 2010 2011

year in preceding women, pregnant HCG among Health Coverage Gap Among Pregnant Women, in Preceding Year

Neonatal Mortality Rate

Figure 11. Health Coverage Gap (HCG) Among Pregnant Women Versus Neonatal Mortality Rate (Phase 2 provinces) 35 8.0

30 7.8

25 7.6

20 7.4

15 7.2

10 7.0 Rate Mortaluity Neonatal 5 6.8

0 6.6 2008 2009 2010 2011 HCG among pregnant women, in preceding year Health Coverage Gap Among Pregnant Women, in Preceding Year

Neonatal Mortality Rate

13. The “HCG Among Pregnant Women” was lagged by one year and then plotted on the same graph as the NMR, for the Phase 1 Provinces (see Figure 10). The same was done for the Phase 2 Provinces (see Figure 11). The figures show that the neonatal mortality rate (NMR) closely tracks the value of the HCG Among Pregnant Women in the preceding year. It makes sense that a fall (or rise) in health care coverage among pregnant women would lead to a rise (or fall) in the NMR in the following year, because: (i) in the case of Plan Nacer, as discussed above, a low enrolment rate among pregnant women is usually due to low registration of newly pregnant women during the earlier trimesters, rather than low registration of women at birth (when they are easily registered); and: (ii) low registration during the earlier trimesters may have an impact on NMR, but that would occur at the time of birth (or immediately afterwards), which would often.

14. Remarkably, the relationship between the two variables (HCG Among Pregnant Women in the previous year and the NMR) remains close even when the HCG Among Pregnant Women rose in 2010, for Phase 1 provinces. Furthermore, the “proportionality” of the relationship between the two variables – i.e. the ratio of the change in one to the change in the other – is roughly similar for the Phase 1 provinces as in the Phase 2 provinces.

15. The spike upwards in the HCG Among Pregnant Women in the Phase 1 provinces in 2009 (the year preceding 2010) is attributable to the fall in the enrolment rate of women in Plan Nacer in 2009 – see Figure 3. The HCG Among Pregnant Women rose by 8.4% – that is, by 84 for every 1,000 pregnant women – in 2009 (the year before 2010), and the NMR rose by 0.32

(per 1,000 live births) in 2010. This implies that for every X additional women enrolled in Plan Nacer, the number of neonatal deaths saved will be X multiplied by the factor [0.23 / 84] = 0.27. This value is used in the Benefit-Cost Analysis conducted below.

16. This analysis is not as rigorous as one that controls statistically for other “confounding factors” – exogenous factors that lead to changes in the Plan Nacer enrolment rate and at the same time to changes in the NMR, resulting in spurious correlation between these two variables. It is hard to see what these exogenous “confounding factors” could be, in this case. The exogenous “confounding factors” would need to explain the fall in the Plan Nacer enrolment rate in 2009, and at the same time the rise in the NMR in 2010.

17. Nevertheless, the magnitude of the impact on neonatal deaths, of being enrolled in Plan Nacer, as estimated using the above exercise, is remarkably similar to that found from the results of Impact Evaluation (IE) activities under Plan Nacer. Great care was taken to correct statistically for confounding factors in the IE analysis, which is discussed in the next section. The similarity of the findings from the two different sets of analyses can be seen from the similar estimates of the number of neonatal deaths averted, the Benefit-to-Cost ratio and the internal rate of return from the two different exercises in Section D.

C. Results of Impact Evaluation for Plan Nacer

18. As detailed in Annex 13, while the Impact Evaluation (IE) activities under Plan Nacer had their challenges, reliable results have so far been obtained from two IE exercises involving analysis of administrative data: (A) an analysis of data from medical records at health centers and hospitals (as well as from Plan Nacer records) at Tucuman and Misiones provinces covering the years 2006 to 2009; and (B) an analysis of data from medical records at public maternity wards (as well as from Plan Nacer records) at 13 provinces – the nine northern Phase 1 provinces as well as Cordoba, Entre Rios, La Rioja and Santa Fe.

19. The data and results from (A) have been finalized and are fully available, while only preliminary results are available at this time from (B). As discussed in Annex 13, results from survey data are not yet available, although data from the follow-up IE survey of the Phase 2 provinces are now being cleaned and analyzed and also be available in the next few months.

20. A summary of key results from (A) and some preliminary results from (B) are given below. The analyses only looked at the eligible population, i.e. only pregnant/lactating women and under-six children without formal health insurance. Among the eligible population, both Plan Nacer beneficiaries as well as those that chose not to be enrolled in the program were included in the analyses. In all cases, statistical techniques have been used to correct for possible bias, such as selection bias.

21. Table 1 presents the key results from these two IE exercises. All of the results shown are statistically significant, and in most cases highly statistically significant (significant at the 1% level). These results are very strong, and much stronger than what has been found in most other similar IE exercises.

Table 1. Implications of Results of Impact Evaluation (IE) Exercises for Plan Nacer Variable of Focus Implied change due to Plan Nacer2 First IE Exercise for Tucuman and Misiones Provinces Number of Babies Born with Very Low Birth Weight1 Fall of 26% (per 1,000 live births among eligible population) Number of prenatal consultations (per 1,000 pregnant Rise of 16% women among eligible population) Number of women receiving antitetanus vaccine (per Rise of 10.1% 1,000 pregnant women among eligible population) Number of consultations for infants aged 45 to 70 days Rise of 32.7% (per 1,000 infants among eligible population) Number of consultations for infants aged 70 to 120 Rise of 21.5% days (per 1,000 infants among eligible population) Number of consultations for infants aged 120 to 200 Rise of 18.2% days (per 1,000 infants among eligible population) Number of neonatal deaths (per 1,000 live births among Fall of 25% eligible population) Second IE Exercise for 13 Provinces (9 “Phase 1” Provinces and 4 Other Control Provinces) Number of Babies Born with Very Low Birth Weight1 Fall of 15.9% (per 1,000 live births among eligible population) Number of Babies Born with Low Birth Weight1 (per Fall of 27.9% 1,000 live births among eligible population) Notes: 1. Very low birth weight means less than 1500 grams, while low birth weight means less than 2500 grams. 2. The implied change due to Plan Nacer is the change that is estimated conditional on enrolment, e.g. the first row indicates that out of every 1000 live births, the number of babies born with low birth weight would be reduced by 26% if all of these mothers were enrolled in Plan Nacer (as compared to the number of babies born with low birth weight if none of the mothers were enrolled in Plan Nacer).

D. Benefit Cost Analysis and Calculation of Internal Rate of Return

22. As part of a Benefit-Cost Analysis (BCA), the benefits from reduced neonatal mortality as a result of Plan Nacer were estimated using two different methodologies:

• Methodology 1: Using the finding emerging from the graphical analysis of Section B above, that for every X additional women enrolled in Plan Nacer, the number of neonatal deaths saved was X multiplied by the factor [0.23 / 84]. • Methodology 2: Using the finding of the first IE exercise described in Section C above, that the number of neonatal deaths was reduced by 25% (in the eligible population) as a result of being a beneficiary of Plan Nacer.

23. In the case of Methodology 2, it was necessary to estimate the number of neonatal deaths occurring in the eligible population in the first place, since the available data do not give this figure directly. This estimation was made by: (i) taking the number of neonatal deaths as a whole in each province (per year) which is a known figure from administrative data records; and (ii) multiplying this by the assumed ratio of the NMR of the eligible population to the NMR of the population as a whole. The latter ratio was assumed to be the same in all provinces (in all years) as the ratio found in the IE sample dataset drawn from Tucuman and Misiones provinces that was used for the first IE exercise mentioned in Section C.

Table 2. Benefit-Cost Analysis: Number of Neonatal Deaths Averted 2007 2008 2009 2010 2011 2012 Total Registered 24.9 39.7 60.0 102.7 152.4 145.3 499.8 mothers (under APL-2) (thousands) No. neonatal deaths saved using Methodology 1 93 149 214 385 571 545 1957 No. neonatal deaths saved using Methodology 2 101 156 216 383 549 523 1928

24. The value of each neonatal life saved was estimated using the Human Capital Approach, i.e. by calculating the discounted stream of estimated lifetime earnings. The latter was estimated using the following assumptions and parameters: (a) per-capita GDP growth rate of 2% (which is conservative compared to recent trends); (b) discount rate of 3%; (c) income will be earned every year from age 18 to 62, at a level of 70% of the per-capita GDP level (given that these are disadvantaged members of the population). With these assumptions, the value of each life saved was calculated as US$246,964.

25. This value was applied to the stream of benefits as calculated using each of the two methodologies. Benefit-to-cost ratios (BCRs) and internal rates of return were computed under the assumption that all benefits other than those from averted neonatal deaths are zero. This assumption is clearly not correct, but was nevertheless made for the purposes of obtaining a lower bound for the BCR and the internal rate of return. The cost figure used in the calculations was the cost of the IBRD contribution of US$300 million, together with the cost of the Provincial contributions to the capitation payments (US$39 million).

26. The results are given in Table 3 below. Both methods show similar results, that the Internal Rate of Return from the Project was either 8.2% or 8.5%. The BCR was calculated as around 1.4 in both cases.

Table 3: Estimated Benefit-to-Cost Ratios and Rates of Return (Taking into account only benefits from averted neonatal deaths) Benefit-to-Cost Ratio Internal Rate of Return Methodology 1 1.43 8.5% Methodology 2 1.40 8.2%

27. In summary, the internal rate of return from the Project was calculated as either 8.2% or 8.5%, from considering the benefits from averted neonatal deaths alone, without taking into account other benefits. The other benefits not factored into the calculations include the benefits from averted deaths of infants (after the neonatal period); higher productivity due to reduced incidence of low birth weight; reduced maternal mortality (at least in the areas with higher maternal mortality to begin with); and reduced morbidity among infants and mothers, among others. If these additional benefits had been included into the calculations, the internal rate of return would have been much higher.

Annex 4. Bank Lending and Implementation Support/Supervision Processes

Task Team Members Names Title Unit Lending Cristian C. Baeza Task Team Leader LCSHH Luis Orlando Perez Sr. Health Specialist LCSHH Monique Francine Mrazek Sr. Public Health Specialist LCSHH Gaston Mariano Blanco Operations Officer LCSHH Jose Pablo Gomez Senior Economist LCSHH Marta Molares-Halberg Lead Counsel LEGLA Natalia Moncada Program Assistant LCSHD Martha P. Vargas Team Assistant LCSHD Paul Gertler Chief Economist HDNVP Jorge Uquilias Senior Sociologies LCSEO Emiliana Vargas Education Economist LCSHE Sr. Financial Management Antonio Blasco LCSFM Specialist Maria Lucy Giraldo Sr. Procurement Specialist LCOPR Xiomara Morel Sr. Finance Officer LOAG1 David Peters Sr. Public Health Specialist HDNHE Sr. Economist (Health) Pablo Gottret HDNHE (Peer Reviewer) Sr. Economist (Health) April Harding LCSHD (Peer Reviewer) Alejandro Solanot Consultant LCSFM Juan Sanguinetti Consultant Isabel Tomadin Consultant Supervision / ICR Task Team Leader, Senior Health Rafael Cortez LCSHH Economist (TTL) Senior Health Economist (TTL and Andrew Sunil Rajkumar LCSHH ICR TTL) Katharina Ferl ICR Author LCSHH Luis Orlando Perez Sr Public Health Spec. LCSHH Jose Pablo Gomez Senior Economist (Health) LCSHH Alexandre Arrobbio Lead Public Sector Specialist AFTP4 Gaston Mariano Blanco Sr. Social Protection Specialist LCSHS-DPT Vanina Camporeale Operations Officer LCSHH Daniela Romero Operations Analyst LCSHH Keisgner De Jesus Alfaro Senior Procurement Specialist LCSPT

Ana Maria Grofsmacht Procurement Specialist LCSPT Alvaro Larrea Senior Procurement Specialist LCSPT Alejandro Roger Solanot Sr Financial Management Specialist LCSFM Alejandro Alcala Gerez Senior Caunsel LEGES Efraim Jimenez Consultant LCSUW Rony A. Lenz Consultant MNSHD Luz Maria Meyer E T Consultant LCSFM Juan Luis Sanguinetti Consultant LCSHH Marcos Miranda Consultant LCSHH Paula Giovagnoli ET Consultant LCSHH Paul Gertler Consultant HDNCE Isabel Tomadin Consultant LCSSD Santiago Scialabba Program Assistant LCC7C Sarah Bailey Junior Professional Associate LCSHS Geraldine Beneitez Team Assistant LCSHH Natalia Moncada Senior Executive Assistant LCSHD Gabriela Moreno-Zevallos Program Assistant LCSHH Silvestre Rios Centeno Team Assistant LCC7C

Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY06 18.96 120.73 FY07 20.67 96.01

Total: 39.63 216.74 Supervision/ICR FY07 4.98 17.86 FY08 32.18 107.60 FY09 34.04 143.23 FY10 32.01 120.45 FY11 32.71 111.77 FY12 68.72 284.03 FY13 29.75 106.55

Total: 234.39 891.49

Annex 5. Beneficiary Survey Results

A. Recent Survey of Satisfaction of Beneficiaries

1. Between July 2012 and May 2013, data were collected from 9633 users of Plan Nacer services as part of a household survey called the “Estudio de Monitoreo de la Satisfacción del Usuario y de la Calidad de Atención del Plan Nacer” (Study of Monitoring of Satisfaction of Users and of the Quality of Care Given Under Plan Nacer). The data were collected at the household level, from households in all 24 provinces, in 3 phases, as follows:

Table 1. Number of Cases in Each Province Province Number of Cases Total in Phase Buenos Aires 600 San Juan 400 Phase 1 (July to La Rioja 400 2200 September 2012) San Luis 400 La Pampa 400 Chaco 400 Cordoba 400 Mendoza 400 Phase 2 (October to Neuquen 371 3171 December 2012) Tucuman 400 Santa Fe 400 Santiago Del Estero 400 Rio Negro 400 City of Buenos Aires 400 Catamarca 311 Corrientes 400 Entre Rios 400 Jujuy 400 Phase 3 (March to Salta 400 4262 May 2013) Chubut 400 Formosa 400 Misiones 400 Santa Cruz 400 Tierra Del Fuego 351 TOTAL 9633 9633

2. Data were collected from households with children aged between 1 and 2 years, since these were considered to have the greatest potential in terms of using health services under Plan Nacer. A strategy of stratification was applied, with 3 different strata as follows:

• Stratum 1: Children with high utilization of services (children aged between 1 and 2 years, with more than 3 consultations invoiced and paid for in the previous year). • Stratum 2: Children with low utilization of services (children aged between 1 and 2 years, with between 1 and 3 consultations invoiced and paid for in the previous year). • Stratum 3: Children with no utilization of services (children aged between 1 and 2 years, with no consultations invoiced and paid for in the previous year).

3. Interviews were conducted in users’ homes because, unlike in the case of exit surveys at health facilities, one would not have contamination due to possible temporary effects produced by an immediate feeling (positive or negative) generated by contact with the health system just before the interview.

4. At this time, results from the first and second phases are available. These are from both Phase 1 and Phase 2 provinces, as can be seen from Table 1.

5. A number of questions were asked at each interview – some focusing on services received by mothers during the periods of pregnancy, childbirth and postpartum, and some focusing on services received by children up to two years of age. The answers allowed the formulation of an overall User Satisfaction Index, which is a conceptual and statistically consistent measure that accounts for the various factors that affect the satisfaction of beneficiaries with respect to public services provided under Plan Nacer. The methodology used for the construction of this Index is based on statistical tools such as factor analysis and binary logistic regression.

6. As is expressed in the Table 2, the User Satisfaction Index level on the whole, regarding use of Plan Nacer services, is 66 points (out of a possible 100) on average, reflecting a very high level of satisfaction.

Table 2. Index of User Satisfaction Regarding Plan Nacer Services, Average Global Value Average Value – on Scale of 0 to 100 Index of User Satisfaction – Global Value (for all 66.0 users)

7. When the average value of the User Satisfaction Index is calculated separately for those in large urban centers, versus those in smaller urban centers, the value of this Index is found to be higher in larger urban areas (see Table 3).

Table 3. Index of User Satisfaction, by Size of Urban Area Average Value – on Scale of 0 to 100 Index of User Satisfaction – Large Urban Centers 67.8 Index of User Satisfaction – Smaller Urban Centers 65.4

8. The “degree of entrenchment” of Plan Nacer, for a particular health facility, can be measured in different ways. One simple way is to look at the proportion of services that are eligible under Plan Nacer, that are in fact invoiced and paid for by the program. If Plan Nacer is working as intended, then a higher “degree of entrenchment” implies more financing under the program, better motivated health workers, better provision of services, and so on.

9. Table 4 presents the values of the User Satisfaction Index for different groups of users, according to whether they use predominantly health facilities with a high, medium or low “degree of entrenchment” of Plan Nacer. Clearly, there is a positive relationship between the “degree of entrenchment” of Plan Nacer at a health facility and the degree of satisfaction of the users of health services at that facility.

Table 4. Index of User Satisfaction, by “Degree of Entrenchment” of Plan Nacer Average Value – on Scale of 0 to 100 Index of User Satisfaction – Among Users of Health Facilities With High “Degree of Entrenchment” of Plan 67,3 Nacer Index of User Satisfaction – Among Users of Health Facilities With Medium “Degree of Entrenchment” of 65,8 Plan Nacer Index of User Satisfaction – Among Users of Health Facilities With Low “Degree of Entrenchment” of Plan 61,4 Nacer

10. Table 5 presents the values of the User Satisfaction Index for different groups of users, according to level of utilization. Clearly, there is a positive relationship between the User Satisfaction Index and the level of utilization of Plan Nacer services, which is a sign of the positive impact of Plan Nacer in terms of the satisfaction of users, particularly those that are very familiar with the program (i.e. among those with a high degree of utilization of the services under the program).

Table 5. Index of User Satisfaction, by Level of Utilization of Plan Nacer Services Average Value – on Scale of 0 to 100 Index of User Satisfaction, Among Those With High 68,9 Level of Utilization of Services Index of User Satisfaction, Among Those With Low Level 64,1 of Utilization of Services Index of User Satisfaction, Among Those With No 63,9 Utilization of Services

B. Survey of Satisfaction Regarding Congenital Heart Disease (CHD) Treatment

11. CHD treatment services were included in the benefits package during the period of implementation of Plan Nacer. Over the period February to April 2013, a telephone survey was conducted of households with beneficiaries of CHD treatment under Plan Nacer. Out of 1698 such households, it was ultimately possible to include 723 (with adequate information gathered) in the survey.

12. Similar to the case of the broader household survey mentioned above, in this case a Satisfaction Index was constructed using a statistical model. The average value of this Index among all households interviewed was 91.3 out of a scale of 10, which is an extremely high value for such studies (see Table below).

Table 6. Index of Satisfaction Regarding CHD Services Under Plan Nacer, Average Global Value Average Value – on Scale of 0 to 100 Index of Satisfaction – Global Value 91.3

Annex 6. Stakeholder Workshop Report and Results

Not Applicable.

Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR

Evaluation and Dissemination of Experiences in Implementing Public Health Policies to Support Sustained Progress in Achieving Social Goals: the Final Evaluation of Plan Nacer

1. Plan Nacer is the Federal program of the National Ministry of Health through which, since late 2004, the Nation and the Provinces began to implement a policy of prioritization, public insurance and explicit health coverage for the most vulnerable members of the population. The program’s immediate objective was to contribute to the reduction of maternal and child morbidity and mortality, in line with the Millennium Development Goals. In addition, the program aimed to strengthen the public health sector in terms of implementing Provincial Maternal-Child Health Insurance programs intended to provide health coverage to pregnant women, lactating women and children under 6 years without explicit (formal) health coverage as a mechanism to ensure, from an equity perspective, more and better access to health services of adequate quality.

2. Throughout its implementation, the strategy of generating Provincial Health Insurance schemes, initiated by Plan Nacer, was characterized by a continuous process of evaluation and dissemination of lessons learned. This enabled a critical analysis of the decisions implemented and the establishment of a strong foundation to face the challenges ahead.

3. This document is a part of the Final Evaluation of the implementation of Plan Nacer in the provinces of Argentina. This year marks the completion of the second phase of Plan Nacer (Phase 2). Phase 2 incorporated the provinces of the Cuyo, Centro and Patagonia regions, starting in mid-2007. The provinces of Northeast and Northwest Argentina were a part of Phase 1 starting in 2004. Phase 2 of Plan Nacer consists of a geographical expansion of a public health insurance program. This program aims to ensure a common set of prioritized health services, intended initially for mothers and children. Consequently, it will be evaluated as part of a process that seeks to promote universal coverage which is a key principle mentioned in the Constitution, through Provincial Health Insurance schemes.

4. This evaluation is intended to determine the degree of compliance with the objectives and targets in the program’s original design and to identify key lessons facilitating the continuation and expansion of the strategy of public health insurance, featuring a set of prioritized services, to a broader population group and addressing more complex lines of care. In order to enable a comprehensive assessment of the implementation of Plan Nacer, in terms of both intended and unintended results at the time of the of design IBRD loan 7409-AR APL II, we propose the following principles that will enable good use of this document in the pursuit of learning, addressing gaps and building a strategic vision of the policy initiated through the program.

A. Proposed Principles for the Final Evaluation of Plan Nacer

5. Prior to the Final Evaluation of Plan Nacer, it is necessary to establish a set of principles that will be useful for this final evaluation – in particular, for better interpretation of the results and conclusions of this exercise.

• Originally, Plan Nacer was expected to be rolled out in three phases, but then a decision was made to include in the program in 2007 all provinces that were not already included. This implied significant challenges on the part of the Nation (central government) in providing support to complex and diverse processes throughout the country. • Phase 2 of Plan Nacer should not be evaluated only as a financing mechanism in isolation, but rather as a step in the process of rolling out the strategy of Provincial Health Insurance (SPS) schemes, in order to provide to the population full access to health care, in line with their rights as enshrined in the Constitution.

“It must be said that this is a priority today for the National Government. I am not referring here only to Plan Nacer – which will continue with its original conception as planned, and which has proved to have very positive results – but rather, I am talking about going beyond the original Plan Nacer and expanding its coverage.

Dr. Juan Manzur National Minister Health

• Phase 2 of Plan Nacer was designed and began, in terms of objectives and implementation mechanisms, at a time when Phase 1 had not completed implementation of all mechanisms that were important for the generation of health results. (In June 2007, for example, health providers under Phase 1 were able to use only 54% of the resources received). • Phase 1 was implemented in provinces with high rates of maternal and child morbidity and mortality, with few differences between provinces in institutional complexity. By contrast, the provinces of Phase 2 have a wide range of diversity in key aspects such as the size of the eligible population, the number of health facilities, health indicators (before implementation), and the size of the additional resources (from Plan Nacer) relative to the local budget, among others. • The more advanced level of institutional development of the provinces of Phase 2 and better health outcomes at the outset led to the formulation of Project indicators with ambitious goals given the time period of implementation, without fully taking into account possible significant barriers to effective implementation. (But, most of the challenging targets were achieved or were close to being achieved at the Project close. This reflects the strong commitment to fulfill the Development Objectives on the part of the Government team and the provinces.) • The implementation strategy of the Provincial Health Insurance schemes (SPSs) in Argentina, in particular Phase 2 of Plan Nacer, is characterized by the simultaneous search of progress in explicit universal coverage in various areas: population, health services and financial protection. This simultaneity was possible thanks to the program’s Monitoring and Evaluation system, which provided relevant and timely information to support decisions made at each step. • The ability to successfully implement Phase 2 of Plan Nacer was favored by the flexibility and adaptability of the design to local realities. • No design can provide everything needed to ensure successful execution and be free of undesirable occurrences. Thus, in the first phase of implementation of the SPSs, more

emphasis was given to overall performance, while in the later phases of implementation of the SPSs, emphasis will also be given to enhancing equitable outcomes. • The SPSs come with a set of tools and a framework to help translate the “Nomenclador” (services package) into effective access to health:  Alliances with others responsible for good health outcomes (Maternal and Child Health Units of Ministries of Health, Municipalities, Social Security Schemes, Health Zones, etc.)  Training and communication to health teams and to the public.  Search of consensus with health establishments, definition of explicit targets and ongoing discussion with them of performance.  Package of services that focuses on priority health services, whose prices are determined according to an assessment and prioritization of services within each Province.  Adequate use of funds received via the program, on the part of the health establishments.  Medical audits  Application of financial penalties to health establishments for failure to follow the rules of the program.

B. Implementation of the Different Axes of Universal Coverage: Results as Reflected in the Project Development Objectives (PDO) Indicators, And Other Results Not Reflected in the PDO Indicators

Results Reflected in the PDO Indicators

6. In terms of insurance coverage under the program, 95% of the eligible population was “nominalized” (identified with ID – i.e. Identification Number – and names recorded and entered in a database, etc.) in the Phase 1 provinces, and 94% of the eligible population was “nominalized” in the Phase 2 provinces, in December 2012. This result was very much helped in the Phase 2 provinces by the Universal Child Allowance (Asignación Universal por Hijo) and Universal Allowance for Pregnant Women (Asignación por Embarazo) policies.

7. The program was able to identify a manageable number of performance indicators (Tracers) that allowed the institutionalization of the agreements between different levels of government, working towards improved health results via the use of a system of results-based transfers.

8. Among the tracers, one can distinguish between those measuring final health outcomes (e.g. birth weight and APGAR score) versus those measuring the processes for generating results. The former are measured at the time of birth, mainly in hospitals and maternity wards, and had high levels of achievement even before the start of Plan Nacer, according to baseline data for the Phase 1 and Phase 2 provinces. Other tracers like the percentage of pregnant women having prenatal checkups early in the pregnancy, and the percentage of children having well-child consultations, are measured (and administered) mainly at the level of the Centers of Primary Attention (health centers). These had lower levels of achievement before the start of Plan Nacer.

9. Measuring health outcomes using reported data, to track tracer achievement levels, is difficult because of deficiencies in reporting leading to an under-statement of the true levels of achievement. The reported levels of achievement in most provinces are well below the corresponding figures obtained from various statistically representative surveys (e.g. Surveys of Living Conditions, baseline surveys, the National Nutrition and Health Survey).

10. An analysis of the deficiencies in this capacity for reporting, done by the Plan Nacer national Government team (the “Unidad Ejecutora Central” or UEC) and the provinces at the time of the Mid-Term Review of the second phase of implementation of the program, identified the possibility of substantially improving this reporting for many of the tracers.

11. Plan Nacer, during its implementation, institutionalized formal agreements between the Federal government and the provinces, and between provinces and health facilities – via Annual Performance Agreements (Compromisos Anuales de Gestión) signed between the Federal government and the provinces, and via Performance Agreements (Compromisos de Gestión) signed between the SPSs and the health facilities. As of December 2012 all provinces had signed Annual Performance Agreements and had succeeded in signing Performance Agreements with more than 90% of their health facilities.

12. The Mid-Term Review was a welcome opportunity to assess the determinants of the gaps in reporting and in results that led to the then-reported levels of achievement (vis-à-vis the targets). A set of actions was identified (Action Plan) relating to different aspects of the implementation of Plan Nacer, aiming to have a substantial impact on performance in the short term. A subset of these actions is presented in the following table.

Table 1. Subset of Strategic Actions Implemented to Improve Performance in the PDO Indicators Actions in Original Action Plan Actions Implemented Definition and Identification of Health Facilities of The national and provincial Plan Nacer teams closely High Impact (i.e. contributing to a large extent to and regularly monitored the performance of every one provincial performance because of size, etc.): of these health facilities, and actions were agreed to in order to maximize the chances of attaining the targets • Setting targets for these facilities agreed to for each facility. • Monitoring and close follow-up of performance of these health facilities

Deepening of analysis and Technical Assistance to the These trainings took place on October 10 and 11, provinces with respect to the role of service fees 2011, with the participation of the technical units of (prices) in the implementation of the provincial fee-for- the provincial Plan Nacer teams, the technical unit of service insurance schemes: the national Plan Nacer team (UEC), and leading academic experts who evaluated different aspects of • Training with respect to the strategic role of the the design and implementation of the program, and service fees (prices) in the definition of the policy made valuable recommendations. applied under the program. • Training on production planning by health facilities, and its coordination with higher institutional structures.

Actions in Original Action Plan Actions Implemented Strengthened engagement with the provinces, vis-a-vis • Took place in the second semester of 2011 via the application and exploitation of the Plan of Video-Conference with the participation of all Production and Application of Funds (PPAF): provinces. • Tutors (persons to provide the training needed) in • Training on the strategic role of this tool as a the use of funds were appointed for the provinces. channel of communication between different • Implemented through the Performance institutional levels and as a planning and Agreements of 2012. monitoring tool. • Training for health facilities on the use of this tool. • Agreement on targets by health facilities regarding the use of the PPAF by 2012. • Visits to the provinces: joint search for strategies to • Between January and August 2012, 22 provinces improve performance, and joint agreement on were visited. plans of action. • Between June and September 2012, 5 Regional • Regional Encounters: modalities of joint work with Encounters took place, with the participation of selected sub-groups from the provinces. the provincial Project teams from all the • Telephone conferences with provincial teams: provinces. modalities of joint work between a UEC team and • Between July and August telephone conferences provincial teams. took place with 9 provinces.

13. The identification of these issues that were critical for the generation of results, and for the planning and implementation of strategic actions, resulted in significant improvements in the reported levels of the provincial performance indicators related to the Project PDO Indicators.

14. However, it is necessary to make an observation about the periods used to evaluate the results reflected by the Project PDO Indicators. Periodically measuring (e.g. monthly and 4- monthly) progress in performance requires the measurement of a reference parameter that permits a normative judgment regarding the result, i.e. it requires measurement of a parameter that represents the potential outcome. The accurate measurement of this parameter requires the development and implementation of a consistent methodology in the context of the information available. In practice, whatever efforts one might make in terms of methodology, the available information may place limits on what can be done.

15. Under the framework of Plan Nacer, even with technically solid methodologies, the data do not permit estimations with intra-annual variation, due to a lack of reliable information on population and social phenomena leading to seasonality patterns. Moreover, the nature of the required information would make it necessary to make projections of variables taking into account a large number of socio-economic determinants where – even in cases where these can be measured – there are important data limitations (e.g. in the case of the percentage of the population without formal insurance coverage).

16. These limitations become especially relevant in the context of targets for intra-year periods (e.g. months or “cuatrimestres”), making it impossible to adequately distinguish between variation in performance on the one hand and variation in seasonal phenomena (and measurement errors) on the other hand. To illustrate this with a concrete example, the Phase 1 provinces have reported, on average, a performance level exceeding the target for the proportion

of live births (among eligible mothers) weighing more than 2,500 grams (Tracer III). But these levels are significantly lower in the 3rd “cuatrimestre” (September-December) of each year, i.e. there is a seasonality pattern.

17. Below we describe the performance of the Project PDO Indicators as of December 2012. Note that for the last “cuatrimestre” (September-December 2012), the results of the External Concurrent Audit were not available for all provinces, and so the reported values incorporate estimates of the post-audit values (based on pre-audit and post-audit performance data from the first two “cuatrimestres” of 2012):

• Regarding the PDO indicators designed to monitor the sub-objective of establishing a “new incentive scheme for provinces and provincial providers" (PDO Indicators 9, 10 and 11 of Table 3 of main text):

 Target of 60% substantially exceeded for “percentage of Annual Performance Agreements successfully implemented”. In 2012, 79% of the provinces achieved the agreed actions (relating to key dimensions of program management and implementation). The performance levels of the Phase 1 and Phase 2 provinces were 89% and 73%, respectively.  Target of 50% substantially exceeded for “percentage of authorized providers under Performance Agreements and provider payment mechanism with its respective participant province”. Performance for this indicator reached 97% for the Phase 1 provinces and 94% for the Phase 2 provinces. It should be noted that all of the provinces succeeded in exceeding their provincial targets for this indicator, and the global achievement level was 95%.  Target of 70% substantially exceeded for “percentage of Tracer targets achieved by the participant provinces in last year billing period”. Performance for this indicator reached 98% for the Phase 1 provinces and 91% for the Phase 2 provinces. The global achievement level was 94%.

• Regarding the PDO indicators designed to monitor the sub-objective of “improving critical intermediary outcomes”:

 Target of 85% exceeded for “proportion of newborns from enrolled pregnant women weighing more than 2,500 grams” (linked to Tracer Indicator III), with this indicator reaching 90% in the last “cuatrimestre” of 2012. The provinces incorporated in Phase 2 reached 94% for this indicator, while the provinces of Phase 1 achieved a performance equivalent to 98% of the target (84%). It should be mentioned that since the 2nd “cuatrimestre” of 2010, the Phase 1 provinces have been able to report, on average, performance levels exceeding the targets, but with dips in performance in the 3rd “cuatrimestre”. Furthermore, the last “cuatrimestre” of Plan Nacer coincided with a period of renegotiation of salaries and strikes in the health sector, adversely affecting reported levels of results.  Target of 92% exceeded for “proportion of newborns, from eligible pregnant women, with Apgar score > “6” at minute 5” (linked to Tracer Indicator II), with this indicator reaching 93% in the last “cuatrimestre” of 2012. The provinces incorporated in Phase

2 reached 96% for this indicator, while the provinces of Phase 1 achieved a performance equivalent to 95% of the target (87%). Similar to the situation for the previous indicator, the Phase 1 provinces, on average, succeeded in attaining much higher levels of 94% and 93% for the first and second “cuatrimestre” (January-April and May-August) of 2012, respectively, exceeding the target in each of those cases, before seeing a dip in the third “cuatrimestre”.

• Regarding the PDO indicators designed to monitor the sub-objective of “increasing access to basic health services of the target population”:

 Target of 80% exceeded substantially for “proportion of eligible population voluntarily enrolled in program” – at the level of the country as a whole (97%), as well as for the Phase 1 and Phase 2 provinces individually (99% and 96% respectively).  For the “proportion of eligible pregnant women with first antenatal care visit before 20th week of pregnancy” (linked to Tracer Indicator I), the achievement level at the national level in the last “cuatrimestre” of 2012 was 67%, or 96% of the target of 70%. On average, the Phase 1 provinces attained an achievement level of 95% of the target (67%), while the Phase 2 provinces attained an achievement level of 96% of the target (68%).  For the “proportion of eligible pregnant women who get VRDL during pregnancy and antitetanic vaccine previous to delivery” (linked to Tracer Indicator IV), the achievement level at the national level in the last “cuatrimestre” of 2012 was 93% of the target of 90%, reaching 83% in the last “cuatrimestre”. On average, the Phase 1 provinces attained an achievement level of 92% of the target (82%), while the Phase 2 provinces attained an achievement level of 93% of the target (84%). It is important to emphasize the value of this result, given the complexity of coordination in real time and in a “nominalized” manner, for this indicator. The reporting for this indicator requires the reporting of two linked sets of results where the verification needs to take place at the level of hospitals as well as Centers of Primary Attention (health centers).  For the “proportion of eligible children less than 18 months old with coverage of measles vaccine or triple viral pregnant women with first antenatal care visit before 20th week of pregnancy” (linked to Tracer Indicator VI), the achievement level at the national level in the last “cuatrimestre” of 2012 was 77%, or 80% of the target of 95%. On average, the Phase 1 provinces attained an achievement level of 98% of the target (93%), while the Phase 2 provinces attained an achievement level of 67% on average, with 5 provinces reaching 100% for this indicator.  The target of 60% was substantially exceeded for the “proportion of eligible puerperal women that received at least one Sexual and Reproductive Health Care consultation” (linked to Tracer Indicator VII), with the achievement level at the national level in the last “cuatrimestre” of 2012 reaching 90%. This target was exceeded substantially for the Phase 1 provinces (86%) as well as the Phase 2 provinces (92%).  For the “proportion of eligible children 1 year old or less, with all well child consultations up to date (percentile of weight and height)” (linked to Tracer Indicator VIII), the achievement level at the national level in the last “cuatrimestre” of 2012

was 45%, or 91% of the target of 50%. Eight of nine Phase 1 provinces reached or exceeded the target for this indicator. The same was true for five Phase 2 provinces.

18. In summary, the first group of indicators (for “new incentive scheme for provinces and provincial providers”) substantially exceeded the targets set, for all indicators. In the case of the second group of indicators (for “improving critical intermediary outcomes”), the targets were exceeded for both indicators. Finally, for the third group (for “increasing access to basic health services of the target population”), the targets were exceeded for three of the eight indicators, while for the other indicators, the level achieved varied between 80% and 96% of the targets originally set.

Results not Reflected in the PDO Indicators

Institutional

• The incorporation of the provinces of the Northeast and Northwest into IBRD Loan 7409-AR (APL-2) provided timely financing to accompany and support the progress in the implementation of the Provincial Health Insurances in those provinces. • The key elements of Plan Nacer – identification of the target population, prioritization of services for these populations, design of targets and consensual search for results among key actors, financing based on results and clear mechanisms for supervision as well as internal and external auditing – constituted critical tools for the provincial health systems seeking to move towards better and more equitable health outcomes. • Plan Nacer has gained strong recognition that is based on the support provided by the health teams and provincial health ministries, which led to Plan Nacer being the only national program where participation is a requirement for the most relevant Social Policies of the National Government: the Universal Child Allowance (Asignación Universal por Hijo) and the Universal Allowance for Pregnant Women (Asignación por Embarazo). • Plan Nacer participated jointly with ANSES (the National Social Security Administration) in the design of the Universal Allowance for Pregnant Women (Asignación por Embarazo) policy. This coordination, in planning as well as implementation of policy actions, represents an unprecedented step forward in the path of institutionalization of the lessons of the program.

“With the Universal Child Allowance policy – which will necessitate enrolling in Plan Nacer and having all the needed checkups on the part of mothers – we are making a very strong commitment to life and also to the objectives of lowering infant mortality and lowering maternal mortality, and I am sure we will be able to succeed.”

Cristina Fernández de Kirchner President of the Nation Opening of the 129th Period of Ordinary Sessions of the National Congress of Argentina. 1 March 2011

“Thanks to the presence of Plan Nacer at the national level, the State has ceased to be seen as a threat and has become a facilitator, a creator of opportunities, enhancing creativity in the management of resources. This translates into a relationship between the Nation and the Provinces where each party contributes and both gain.”

Cristian Baeza Director of Health, Nutrition and Population, Human Development Network, World Bank. National Meeting featuring results of implementation of Plan Nacer as an innovative strategy in health financing. March 2011

“I was impressed by how Plan Nacer helps to improve the health system. UNICEF will continue to monitor this policy which puts children in the first place.

Nils Kastberg Regional Director for Latin America and the Carribbean, UNICEF.. August 2008, Capital Online Journal of Misiones Province.

• The ability of the national government and provincial governments to implement, gradually but steadily, a program of financing for results in a middle-income Federal country like Argentina, as well as the design of the program and the initial results from its impact evaluation, have gained strong international recognition. • The characteristics of the design, implementation and execution of the program, as well as the results noted, were considered sufficient for the program to be recognized by IBRD over other projects, and to be awarded a "Good Practice Award". • Advances in the process of establishing health insurance for the population were based on information obtained from unpublished studies, health service provider capacity, the costs of services, and utilization levels observed and desired for priority services – all making possible the formulation of agreements between the National and Provincial ministry teams, based on this evidence gathered. • The program generated transfer mechanisms between the Federal government and the provinces with very low administrative burden, allowing a flow of resources almost in real time.

ISO Certification of the Processes of Nation-to-Province Transfers

Plan Nacer implemented, from July 2012 onwards, a System of Quality Management following international standards. Thus allowed the process (termed "Transfer for Health Results Under the Sumar Program Involving the Nation and the Provinces") to be certified under ISO 9001: 2008 by IRAM (Instituto Argentino de Normalización y Certificación), with registration Number 4959 in January 2013. IRAM is an institution recognized for its role in standardization and quality certification, locally and internationally.

• Resource transfers from the Provincial Health Insurances to health facilities initially did not have a predetermined circuit, generating large administrative burdens and delays in payments, which was detrimental to the notion of an incentive scheme as envisaged under the program. This led to the need for the program to ensure the existence of specific, formal and transparent payment circuits, involving extra-budgetary transfers from the accounts of the Provincial Health Insurances to health facilities, which for the first time started to receive and administer funds in an autonomous manner.

• The national government and the provinces co-financed the capitation payment paid to the Provincial Health Insurances to cover all services in the benefits package of the program. The Nation contributed 70% of the capitation value and the Provincial Governments the remaining 30%. This integrated funding mechanism is unprecedented in the public health sector of the country. • The ability by the health facilities to decide on the use of the funds received led to changes in the levels of satisfaction on the part of the health workers, independently of whether or not the funds could be used for monetary incentives to individuals.

Study on the Financing and Use of Resources by Health Facilities in the Provinces of Northern Argentina

Given the diversity of institutional settings and the different relationships established by different province with their health facilities, a study was carried out to:

• Assess the model implemented for the process of decisionmaking regarding the use of funds, applied in different provinces. • Investigate the extent to which the policy regarding the use of funds has had an impact on the motivation of health workers. • Identify deficiencies in monitoring of the use of funds in each province and allow for more effective supervision.

Main Results

The level of satisfaction with the model implemented for the process of decisionmaking regarding the use of funds under Plan Nacer was given a score of nearly 7 out of 10 according to the managers of the health facilities. Among the health workers, the satisfaction level was 7.2 out of 10.

Identification of Principal Determinants

• Among the reasons mentioned for the above, 8 out of 10 health workers highlighted the direct benefits to the population that are generated by the program. • Since the implementation of Plan Nacer started, health personnel at the facilities covered mention the enhanced value placed on teamwork and the increased possibilities of professional and personal fulfillment, as a result of the program. • The application of the model for the use of funds designed under Plan Nacer was found to improve the organization of work:

“Plan Nacer encourages one to be more organized since there is more supervision, and it is a task that we all do together.” (Testimony from a health team at a Primary Attention Center – i.e. health center – in Chacos Province.)

• Optimizes professional practice due to the availability of more supplies and equipment, an increase in the coverage of the target population and an improvement of the physical working environment. • Approximately 65% of Directors/Heads of the health establishments noted that funds received under Plan Nacer were distributed across the institution – contributing to supplies, instruments and medical equipment to enhance care for the entire population.

• The flexibility of the program’s design to be able to allow for provincial realities provided a lot of latitude in the determination of coordination mechanisms between the

responsible parties at the ministerial levels and the health facilities, in the search for enhanced results. UEC provided technical assistance to the provinces, and put at their disposition the “Plan of Use of Funds” tool. • In its design, the program gives the provinces the power to set the prices (fees) for the services in the package of services (the “Nomenclador”). This characterizes the design of the program as one that strengthens the Federal nature of the Argentine government, and provides flexibility in dealing with the heterogeneity of health conditions among provinces. However, the use of prices for services in the public sector is a tool never before used. Thus, the technical unit of UEC conceptualized and proposed to the provinces an assessment tool to value the services in the “Nomenclador”. The tool assisted the provinces to assess the contribution of each service to the national and provincial health objectives. On their part, the valuation of the services allowed the provinces to incorporate in their reasoning aspects related to resources needed for service provision, and the related budgetary constraints, allowing the formulation of a consistent set of prices (fees) for the services in the “Nomenclador”. • The mechanisms for contracting health facilities in other provinces were refined. Framework Agreements between provinces were promoted, and later a contracted Provincial Health Insurance was allowed to pay a different set of prices for health facilities outside of its own province. An example of this is an Agreement signed between Salta and Jujuy Provinces, promoted by UEC.

Monitoring and Evaluation

“Plan Nacer is characterized by implementation of a system of records, audits and evaluation, which ensure accountability and the establishment of clearly defined responsibilities. It also promotes and guarantees the use of funds in line with agreements made under a scheme where results achieved are documentated. Unlike other plans, it aims at improving the management of health system resources and promotes transparency in the provision of health care."

Philip Musgrove Adjunct Editor, Health Affairs. “Provincial Maternal and Child Health Insurance Using Results-Based Financing (RBF).” March 2011

“Plan Nacer is a dream come true for any child. To enable all children to have coverage and, furthermore, that such coverage is evaluated. It is no use having coverage without monitoring.”

Fina Rodriguez Executive Medical Director of Garrahan Hospital

• The program, from the information generated as well as the analysis of that information and the dissemination of the results of this analysis, has allowed the implementation of mechanisms to implement public policies based on evidence (evidence-based decisions). • The monitoring and evaluation (M&E) component of the program included the completion of an impact evaluation (IE), both for Phase 1 and Phase 2 of the Project. The IE strategy for Phase I included the use of quasi-experimental methods with the provinces of La Rioja, Córdoba, Santa Fe and Entre Rios as control provinces. This initial strategy

had to be abandoned as Plan Nacer was expanded throughout the country ahead of schedule. • The joint work of the Plan Nacer (Government) and World Bank technical teams allowed the exploration and identification of possible new IE strategies: experimental methods focusing on exogenous interventions at the health facility level, exploitation of the administrative databases of the Provincial Health Insurance schemes, exploitation of the data from clinical birth records from 2004 to 2008 for Phase 1 provinces, and other quasi- experimental methods using instrumental variables, among others. Impact Evaluation Using Perinatal Data: Provinces of Northeast and Northwest

The exercise conducted to put together the data for this IE was done using perinatal data from public maternity wards, through an operation that involved teams visiting health facilities, and digitalizing the perinatal data, at thirteen provinces (Catamarca, Chaco, Córdoba, Corrientes, Entre Rios, Formosa, Jujuy, La Rioja, Misiones, Salta, Santa Fe, Santiago del Estero and Tucumán). Between November 2010 and November 2011, on-site digitalization took place for 467,281 birth records at 287 health facilities. More than 170 consultants participated in this operation.

A team of specialists led by Professor Paul Gertler (of UC Berkeley) is working on the databases to identify program impacts in terms of the variable available in the database. While research is ongoing, preliminary results show significant impacts of the program on relevant health variables (see Annex 4, especially Table 1).

• IE should not be seen as the most important source of information for the design of new operations, but rather as a means of providing ex-post empirical validation of some dimensions of the existing operation. • The program already has established IE results in terms of the impact on service utilization levels and the health status among the eligible population in Misiones and Tucuman provinces. This IE exercise used data from provincial information systems in order to identify the impact on (among others) utilization levels of health services prioritized by the program, the quality of prenatal care and of well child care, children's health status at birth and decreases in neonatal mortality (see Annex 3 for some results). • Recognition of the limitations of quantitative assessment tools in monitoring results along all dimensions of interest promoted the search for other assessment strategies such as qualitative ones. This included, for example, a study of the use of funds (received under the program) by health facilities finance in the northern provinces. • A distinctive aspect of the program is its measurement of the levels of satisfaction of users of prioritized health services. This is a necessary first step in the incorporation of civil society in the process of prioritizing public policy actions.

Among the major challenges facing Plan Nacer is enhancing transparency vis-à-vis beneficiaries – making them take on a central protagonist-type role in the program. Because, after all, Plan Nacer is not just a payment mechanism for provincial health facilities, but also a mechanism to empower people. So that they know their rights, and have (and act on) expectations regarding service delivery. I think this challenge is very important and the Plan has all the elements needed to address it." Amanda Glassman Center for Global Development National Meeting featuring results of implementation of Plan Nacer as an innovative strategy in health financing. March 2011

Study of Satisfaction of Users of Plan Nacer (2012)

In 2012, a study was conducted to measure user satisfaction, with a methodology that determined the level of overall user satisfaction with the basic services package under Plan Nacer to be 66 out of a maximum score of 100. Additionally, there was evidence that the level of satisfaction is higher when the level of utilization of services is higher (63.9, 64.1 and 68.9 for users with very low use, low use and high use, respectively). Also, the higher the level of integration (insertion) of Plan Nacer in a health facility, the higher was the level of user satisfaction (measured at 61.4, 65.8 and 67.3 respectively for health facilities with a low, medium and high level of integration, respectively).

In turn, there is also evidence that levels of user satisfaction are sensitive to the level of complexity of services. Among users of the congenital heart disease services that were a part of the Plan Nacer benefits package, the level of user satisfaction was 91.3 out of 100.

The methodology applied provided evidence that Plan Nacer achieved higher levels of satisfaction among users, as compared to the situation without the program. This can be seen from the positive relationship between satisfaction levels and the level of integration of Plan Nacer in health facilities, and also from the positive relationship between satisfaction levels and service utilization levels (higher levels of the latter indicating greater knowledge of the services and their quality).

• The External Concurrent Audit was an important management tool that facilitated, among other things:

 Better monitoring of the processes and results on the ground  Warnings regarding key errors and limitations  Possibility of guiding behavior towards what was desired via effective monetary penalties  Training in processes for key personnel responsible for implementation of these processes  Greater transparency in the process of using the financial resources of the program

Expansion of Program Coverage

Among the virtues and the challenges of the Plan is its continuity. Plan Nacer has had more continuity than many public policies in Argentina. The challenge in order to achieve positive results lies in the continuity of its implementation." Ariel Fiszbein Chief Economist of Human Development Network, World Bank National Meeting featuring results of implementation of Plan Nacer as an innovative strategy in health financing. March 2011

“Plan Nacer is a centralized and serious attempt to promote greater equity.”

Mirta Roses Director of the Pan-American Health Organization April 2007, “La Razón” Journal

• As occurred in Phase 1, and also for the provinces within Phase 2, Plan Nacer was a program that allowed people without health insurance coverage by the “Obras Sociales” (Social Security schemes) in these provinces to be – for the first time – “nominalized”

within the health system. Furthermore, the program provides an explicit list of services for those without formal health insurance coverage. • A mechanism was developed that allowed the incorporation into the benefits package of new health services with a substantial impact on hard-to-reduce factors affecting infant and maternal mortality, such as treatment for children with congenital heart disease and maternal/neonatal services of high complexity, among others. • The incorporation into the Health Services Plan (benefits package) of cost-effective services allowed one to make explicit the rights to health contained in the newest regulations. This advance in the area of health coverage was facilitated by resource availability which also facilitated an acceleration in its implementation. • The introduction of congenital heart diseases services for children in the benefits package constitutes a successful and model experience for the Argentinian health sector. It marks the first time that catastrophic care interventions have been included in a public health insurance scheme in the country, and also marks the first Federal network of health facilities providing specialized complex care in the country, and consisting of facilities whose quality has been rigorously pre-certified. • The incorporation of more complex services into the benefits package required the definition and implementation of strategies for strengthening and improving existing health networks at the provincial and national level. • The need to strengthen and improve existing health networks required the formulation of appropriate coordination structures – both provincial and interprovincial. These structures were institutionalized by creating administrative circuits that combine the necessity of providing timely responses with the ability to document the whole process. • The inclusion of the more complex services led to a significant increase in financial protection offered by the program, prompting some provinces to show greater interest in participating more effectively in the implementation of the strategy. • The exercise of prioritization of the services under Plan Nacer took place in a context of limited resources and, for this reason, aimed to address fewer “lines of care” but in a comprehensive manner in order to increase the chances of organizational changes and improvements in performance. • The program has made available to the Ministry and to the Provinces information on costs and coverage gaps for services that form part of a “line of care” under Plan Nacer.

Training

• Plan Nacer, through efforts on the ground, has identified provincial counterparts for indigenous health issues and for work from an intercultural perspective, advancing significantly from 2009 in terms of participation by indigenous people. Indigenous Peoples’ Plans have been agreed on and validated locally. In cases where participation was achieved not only in planning but also in the implementation of these activities, empowerment was enhanced among this population.

• The Strategic Plan for Technical Assistance and Training has been recognized by the provinces as a management tool suitable for generating proposals for training and technical assistance tailored to their particularities. It is also valued because it adapts to the dynamics and heterogeneity of each province, allowing the contracting of additional personnel. • There has been a deepening in the training of personnel, shifting to a greater focus on processes, with the generation and dissemination of tools such as the Distance Learning Program (e-learning). This was aimed at promoting the growth of skills that enable better implementation of the program and the achievement of better health outcomes.

From 2011 onwards, Plan Nacer incorporated a new modality of training, e-learning, through the development of the National Distance Learning Program. The objective of this Program is to allow more people to be incorporated in the health system, and to gain knowledge regarding the management of Plan Nacer and of public health.

The National Distance Learning Program is linked with other programs of the Ministry of Health: the National Programmes for Control of Vaccine-Preventable Diseases, Comprehensive Adolescent Health, Oral Health, Tobacco Control, Hearing, Maternal and Child Health, and Sexual and Reproductive Health. A special platform, with assistance in its utilization, was put in place to facilitate with these linkages. This program represents one of the options to address the need for continuous education on a large scale, promoting equity and inclusion through access to education.

The program is free, and exponentially expands the access of people to training and retraining regarding the health system, allowing users to choose their own time schedules vis-à-vis their learning, and to access the platform from any location with an Internet connection. The program supports learning and allows new concepts to be added, while strengthening existing knowledge and skills.

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders

Not Applicable.

Annex 9. List of Supporting Documents

National Ministry of Health. Various Documents. Buenos Aires, Argentina.

National Ministry of Health. October 2011. “Informe Final: Estudio de Financiamiento y Uso de Recursos de Efectores del Plan nacer en las Provincias de la Fase 1.” Buenos Aires, Argentina.

National Ministry of Health. 2011. “Evaluación de la Implementación de la Segunda Fase del Proyecto de Inversión en Salud Materno-Infantil Provincial en Argentina: Desafíos, Retos y Aprendizajes. Documento de Base para la Evaluación de Medio Termino Préstamo BIRF 7409- AR APL II”. Buenos Aires, Argentina.

National Ministry of Health. December 2012. “Informe de Desempeño de los indicadores de los Objetivos de Desarrollo del Proyecto”. Buenos Aires, Argentina.

National Ministry of Health. 2013. “Informe Ejecutivo: Monitoreo de la Satisfacción del Usuario y de la Calidad de Atención del Plan Nacer.” Buenos Aires, Argentina.

National Ministry of Health. 2013. “Informe Ejecutivo: Monitoreo de la Satisfacción de Beneficiarios de la Cobertura para la Corrección Quirúrgica de Cardiopatías Congénitas.” Buenos Aires, Argentina.

National Ministry of Health. June 2013. “Evaluation Final de la Implementation del Proyecto de Inversión Materno-Infantil Provincial en Argentina: Documento de Base para la Evaluación Final del Préstamo BIRF 7409.” Buenos Aires, Argentina.

World Bank. Various Aide Memoires, ISRs, Independent Procurement Reviews and Post- Procurement Reviews, Loan Agreements and Amendments, and Other Documents. Washington DC.

World Bank. September 2003. Project Appraisal Document for the Provincial Maternal-Child Health Sector Adjustment Loan. Washington DC.

World Bank. March 2004. Project Appraisal Document for the Provincial Maternal-Child Health Investment Project. Washington DC.

World Bank. May 2006. Country Assistance Strategy 2006-2008. Washington DC.

World Bank. October 2006. Project Appraisal Document for the Provincial Maternal-Child Health Investment Project in Support of the Second Phase of the Provincial Maternal-Child Health Adaptable Program Loan. Washington DC.

World Bank. October 2006. Project Appraisal Document for the Essential Public Health Functions and Programs Project. Washington DC.

World Bank. June 2009. Country Partnership Strategy 2010-2012. Washington DC.

World Bank. June 2010. Quality Assessment of Lending Portfolio (QALP-2) for Provincial Maternal-Child Health Project. Washington DC.

World Bank. November 2010. Project Appraisal Document for the Essential Public Health Functions and Programs II Project. Washington DC.

World Bank. January 2011. Implementation Completion and Results Report for the Provincial Maternal-Child Health Investment Project in Support of the First Phase of the Provincial Maternal-Child Health Adaptable Program Loan. Washington DC.

World Bank. March 2011. Project Appraisal Document for the Provincial Public Health Insurance Development Project. Washington DC.

World Bank. March 2011. “Impact of Plan Nacer on the Use of Services and Health Outcomes: Intermediate Results Using Administrative Data from Misiones and Tucuman Provinces.” Washington DC.

Annex 10. Background Information: The Health System in Argentina

1. Argentina is a federal country with 23 provinces and the city of Buenos Aires. While the National Ministry of Health is mainly responsible for general health policies and regulation, the 23 provinces and the city of Buenos Aires are responsible for the provision of health care. The Argentine health sector can be divided into three sub-sectors: public, social security and private. The combination of Federalism and sectoral fragmentation make it a complex system.

2. Formal insurance typically covers only public employees, not the large population that does not belong to an Obra Social5or have private insurance or the financial ability to pay the full cost of care. The publicly-funded services in each province operated in the traditional Latin American manner, with investment and recurrent costs paid out of fixed budgets. Public providers often did not bill for services and had little or no autonomy in the use of the resources transferred to them from the provincial Ministries of Health. These budgets were often insufficient to provide all the care needed by the uninsured population, and the rigid budgeting process meant that the necessary inputs were not necessarily available in the right proportions and substitution among them was difficult or impossible. These features continue to characterize most of the Argentine health system and required some flexibility in the use of resources from the introduction of a new insurance scheme. Since the provinces differ greatly in per capita income, disease burden, insurance coverage and public funding capacity, care for the uninsured was not only inadequate on average but varied inequitably from province to province. In 2001, for example, the highest provincial infant mortality rate was more than twice as high as the lowest, and the variation in maternal mortality was much greater, more than 10 to 1.

3. The new program Plan Nacer intended to increase access to health care, reduce inequalities and speed the recovery in health following the recession especially for the most vulnerable part of the uninsured population, pregnant women, newborns and young children.

4. For women, the coverage is from the start of pregnancy (or at the time the pregnancy is confirmed) until 45 days after birth or the loss of the fetus if that occurs—a maximum, that is, of 10-1/2 months. Each new pregnancy starts a new interval of coverage; even health problems that are caused by having been pregnant, such as complications from cesarean surgery, are not covered if they occur after 45 days. Women needing the corresponding services for such problems are referred to the province's reproductive health program. For children, in contrast, coverage lasts up to age six, with a concentration on interventions during the first year of life.

5Obras Sociales are national or provincial social health insurance organizations which administer funds contributed by employers and employees and provide health care to their members and their direct families. The obras sociales are organized by occupation or trade Penchaszadeh V. et al, Italian Journal of Public Health, Vol. 7. Nr 4, 2010)

Basic Design of Plan Nacer

5. Plan Nacer is an innovate approach and includes several features: an investment in improved capacity to deliver services; a formal insurance scheme for people who have no access to employment-based coverage and cannot afford private insurance; a pair of legally binding agreements, one between levels of government and one between a provincial government and providers; a scheme of voluntary affiliation by providers; and a system of record- keeping, auditing and evaluation to ensure accountability and that funds are used as intended and results documented.

6. The program includes two results-based incentive mechanisms for the inclusion of the target population and the improvement in the quality of services. The first incentive is in terms of eligible population and rewards provinces when eligible beneficiaries get enrolled since they receive a capitation payment for each pregnant woman and child who joins the program. The second incentive is linked to the providers who deliver services, since they bill and are paid for these services (fee for service) that are covered by Plan Nacer. The services that are included, the benefit package, is known as the “Nomenclador”, and is the same in every province. The “Nomenclador” contains over eighty services. Table 1 summarizes the services in the “Nomenclador” in the first few years of the Project. (In 2010, as mentioned in the main text of this report, two new sub-packages of services were included: Congenital Health Disease treatment services, and additional complex maternal health services to better address high-risk births).

Table 1: Services in “Nomenclador” of 2008 Group or Function Services Pregnant (low risk) Consultations for healthy pregnancy Pap test for cervical dysplasia or cancer Tetanus immunization Women Pregnant (high risk) Consultations for control of pregnancy risks Women Ambulatory treatment for HIV infection Childbirth Childbirth Care in childbirth and for the newborn Following childbirth or Rubella immunization fetal loss Consultation on postnatal health Newborn Immunization (including BCG and hepatitis B) Care in incubator for up to 48 hours Children Immediate treatment for vertical transmission of HIV Up to age six Eye examination Consultations on mouth and dental health Laboratory tests and procedures Pregnancy test Colposcopy following positive Pap test Taking blood samples Blood tests Imaging in pregnancy Sonogram Chest x-ray Community-level services Active recruitment of pregnant women in first trimester Complete rounds in rural areas by health agent Socio-epidemiological diagnosis of population at risk Group meetings to promote healthy nutrition Group meetings to promote child development Transportation Emergency transport of newborn as needed

7. Ten indicators (tracers, trazadoras) are defined to measure the program output and health outcomes, and are used by the National Ministry of Health to determine the financing to the provinces (see Table 2). Targets for each tracer are negotiated annually with each province. This is an essential feature for the program because it allows a province with a weaker health system to aim for a lower target than better-off provinces. In terms of achieving each tracer the program used initially an “all-or-nothing” approach. However, this approach was then changed to a threshold approach with minimum, medium and maximum level. Depending on which target was achieved, a certain percentage was paid. The full 4 percent are only being paid for reaching a higher target while 1 to 3 percent is being paid for lower targets. Therefore, the provinces have a strong incentive to achieve each target for a given tracer (but no marginal incentive to exceed it). This scheme aims to motivate providers to increase their efforts to deliver better care even after a target was met.

Table 2: The Ten Tracers (Trazadoras) in Plan Nacer No. Health objective Corresponding tracer: number of ____as a share of all the eligible women, newborns, or children 1 Early enrollment of Pregnant women with first prenatal consultation before pregnant women the 20th week 2 Effectiveness of childbirth Newborns with Apgar score of 6 or better, five minutes after care and care for the newborn Delivery 3 Effectiveness of prenatal care Women with newborn weighing at least 2,500 grams (2.5 kg) and prevention of prematurity 4 Effectiveness of prenatal care Women with VDRL (tests for STDs) and tetanus immunization and care in childbirth during pregnancy 5 Attention in cases of maternal Cases of maternal death or death of infant under one year of or infant death age, fully evaluated as to cause 6 Immunization coverage Children under the age of 18 months with measles or measles- mumps-rubella (MMR) vaccination 7 Sexual and reproductive health Women receiving sexual and reproductive health consultation within 45 days after giving birth 8 Care of healthy child to age Children under age one with complete record of consultations one and height, weight, and head diameter 9 Care of healthy child from age Children between one and six in age with complete record of one to age six consultations and height and weight 10 Inclusion (coverage) of the Providers who deliver services to indigenous populations, with indigenous population personnel trained in that group's specific culture and health needs

Figure 1: Comparison of Input Based and Results-Based Financing Model

Source: Ministry of Health of Argentina, 2009

Finance Mechanism

8. The Ministry of Health transfers the funds on a per-capita basis to the provincial ministries in two steps: 60 percent of the financing is provided only upon verification of enrollment and 40 percent after accomplishment of the tracers. Since provinces differ in their capacity and resources to deliver the services listed in the plan, the target levels for the ten tracers are negotiated between the national and provincial government and differ between the provinces of Phase I and Phase II.

9. The funds flow from the provinces to the contracted healthcare providers (public or private), who can use up to half of the funds to pay incentives to staff to improve productivity and quality of services. The program’s incentive scheme is based on two major relationships: the relationship between the national and provincial governments, and the relationship between the provincial government and healthcare service providers. The national government periodically provides capitation transfers to the provincial governments. The capitation-transfer is based on the established per capita payment multiplied by the number of beneficiaries registered in the databases that the province maintains. Sixty percent of the total amount is transferred monthly based on enrollment statistics, and forty percent is added to the capitation-transfer once output of outcome results (accomplishment of the ten health indicator targets or tracers) is presented. The second relevant relationship is between the province and each of the health facilities. Provinces

pay health care providers for services rendered to the target population. Prices of services in the “Nomenclador” can be reviewed twice a year. The combination of a fixed capitation payment for enrollment that goes to the provinces, and the Fee For Service (FFS) payments to providers, which vary according to the type and the amount of care delivered, presents a financing scheme that aims to avoid the negative effects that the capitation or FFS payment alone might cause. Table 1 demonstrates financing flow as described above.

Figure 2: Relationship between National Government, Provinces and Providers

Supervision and Audits

10. In addition to the financial incentives, management and auditing mechanisms have been instituted to monitor the program’s progress. Legally binding management agreements signed between the National Ministry of Health and the provincial government, and between the provincial government and healthcare providers outline their respective roles and responsibilities and hold parties accountable. Annual Performance Agreements between the National Ministry of Health and provincial governments are especially important to ensure compliance to specific targets on enrollment and tracers. Internal audits verify the work performed, while independent auditors complete the process through regular (every four months) detailed reports to the National Ministry of Health, monitoring the progress of the program. The feedback from the audits and management reports is used to correct any mismanagement, break bottlenecks, and improve the program’s functioning. The result is an innovative management model for a results- based financing program with an effective control and monitoring system that aims to deliver substantially better results in the health sector.

Annex 11: Quality Assessment of Lending Portfolio (QALP-2) Summary

1. This Annex provides a summary of QALP-2, an assessment of the Project that was carried out in June 2010 by the independent Quality Assurance Group (QAG). The first Table provides a summary of the findings, together with ratings given by the QALP-2 for different aspects of performance (and overall ratings for Quality of Design, Implementation Progress and Quality of Bank Supervision).

2. The second table provides ratings from the QALP-2 using the same framework that is used for ISRs, and also provides ISR ratings – for the ISRs just before QALP-2 as well as the ones that came after it.

Table 1: Summary of Findings from QALP-2 1. QUALITY OF DESIGN S 1.1 Strategic Relevance and Approach S Comment: The Government of Argentina set in motion an innovative Maternal-Child Health Insurance Program in 2003, known as Plan Nacer, to improve child and maternal health care. Plan Nacer was supported by the first phase (APL-1) of an adaptable loan and is a provincial social insurance program providing a free basic package of cost-effective services to uninsured pregnant women until 45 days after delivery and children under six in the country’s nine poorest provinces. The program was expanded to the remaining 15 provinces in 2006 with a loan of US$ 300 million and Government financing of about US$346 million. The expansion merged the two subsequent phases of the adaptable loan into one (APL-2). Outwardly there was a consensus to combine the two phases but provinces exerted pressure to be included in an apparently successful program that used performance incentives. APL-1 and APL-2 have introduced performance payments and a resource allocation system by changing institutional relationships but without altering the existing institutional setup. The Program uses an incentive mechanism and redefines the roles and relationships between the national Ministry of Health and the Provincial Governments, and between the Provincial Government and the healthcare providers to enhance quality and accountability in health service provision. The Bank sought a multiplier effect of its financial support as its contribution averaged only US$4 million annually per province. 1.1.1. Adequacy of enabling environment? S Comment: Within the general framework of a public sector with considerable discretionary power, and associated vested interests, the Project constitutes a serious attempt at introducing a system of performance indicators and associated resource allocation mechanisms, while leaving intact the institutional structure, thereby introducing an element of incentives for improved performance, which hitherto has been absent. This approach enables the maternal/child health investment to take place in a more disciplined way that had been possible so far, and operate within the policy and institutional constraints prevailing in the country. 1.1.2 Strategic Relevance and appropriateness of DOs? S Comment: The Project designed a social insurance system that increased access to health care services for an underserved low income and vulnerable population group of uninsured children and pregnant women with unacceptable high morbidity and mortality rates. 1.1.3 Quality of the results framework to achieve the DOs? S Comment: The results framework consisted of ten measurable tracer conditions and a set of component-specific intermediary outcomes. The ten tracer interventions were the same for all participating provinces and served to measure provincial and national progress towards achieving the DOs. The tracer conditions and the intermediary outcomes focus on process interventions that address the main causes of under-five and maternal mortality. 1.1.4 Quality of arrangements for monitoring and evaluation? S Comment: The arrangements that have been put in place are solid and practical, and accordingly

are operationally meaningful. Impact evaluations are part of these arrangements. 1.2 Realism of Project Design and Risks MS Comment: The overall Project design is realistic but could have taken into consideration the capacity of different Provinces to deal with the new system being introduced. The capacities and complexities among different Provinces vary significantly and the targeted population is not always well enough informed to use the new system. A more differentiated approach, reflecting those varying provincial capabilities, and a greater emphasis on communication and community outreach could have improved the absorptive capacity and included more beneficiaries. Several risks identified during preparation materialized, e.g. failure to reach most intended beneficiaries, but the proposed mitigation had no effect, e.g. outreach campaigns were delayed. Furthermore, the institutional complexities associated with working with the Municipalities were not identified as risks at the time the Project was prepared. 1.2.1 Appropriateness of Project design and complexity given past track record and MS absorptive capacity? Comment: The Project design was appropriate but should have anticipated the variation among the 15 provinces and their capacity and willingness to organize the health care providers to provide health care and the difficulties in enrolling a low-income population. 1.2.2 Adequate attention to technical, financial and economic aspects? S Comment: The Project has a solid and internally consistent framework to deal with the technical aspects. Given the small size of this operation in the context of budgetary resources available at the Federal and Provincial levels, the financial and economic aspects play a subsidiary role, and in themselves should not pose an undue risk on sustainability, once Bank resources are no longer available for the continuation of the program. That notwithstanding, the issue will need to be addressed and increasing attention will have to be focused to the subject as the Project nears completion. 1.2.3 Quality of institutional framework for the Project? S Comment: The Project has been framed to work within the institutional structure as it was in existence at the time of appraisal. With greater understanding of the different capabilities among Provinces, the institutional strengthening elements could have been enhanced to reflect the weaker jurisdictions, particularly those that involve Municipalities, in the conduct of the Project. Operationally, a more extensive institutional audit and greater familiarization with the different provincial and municipal capabilities would have enhanced the design of the implementation support. 1.2.4 Quality of risk assessment and mitigation measures? MS Comment: Risks to the country are minimal and so are sector risks. The social and cultural aspects for reaching the intended beneficiaries of the Project were underestimated and the mitigation measures of community-based information and outreach campaigns were only moderately satisfactory. 1.2.5 Readiness for implementation? MU Comment: The APL-2 was judged to be ready for implementation based on the successful implementation of the first two years of APL-1 in nine provinces. Despite several measures adopted by the Bank to ensure readiness for implementation, the addition of 15 provinces and thereby covering the whole country was ambitious and resulted in a slow start up. The fact that APL-1 so far had a successful start should not have been taken as proof of a readiness that all provinces, including the largest and most economically advanced ones, would be capable of replicating this early success.

In its response the task team challenged this judgment but the panel believes that the 19 months disbursement lag for a phase 2 of an APL is ample proof of lack of readiness. The panel also notes the low rate of coverage that is explained by the poor performance of the Central Region Provinces (Buenos Aires, Santa Fe and City of Buenos Aires). This region concentrates 80% of the target population and had only reached 20.3% on average of coverage by to September 2009. 1.2.6 Government ownership of the design process? HS Comment: There has been clear political will on the part of the central and provincial authorities. The decision to widen the scope of the Project to cover a wider set of provinces was in response to

demand from the provinces to be included in the program. 1.2.7 Effectiveness of the Bank’s management of the preparation/appraisal process? S Comment: An internal QER was enriched with comments by three peer reviewers and served to frame the issues more sharply. However management could have addressed the readiness of the Project to absorb all 15 remaining provinces and considered the implications of scaling-up so rapidly without an adequate institutional audit. The task team included relevant operational skills with good country and sector knowledge. Resources were more than adequate given the short time (six months) between concept review and Board approval. Argentina is an important Bank client and has a large health sector portfolio of US$1.2 billion and management obviously paid attention to this operation. 1.3 Fiduciary, Safeguards and GAC Aspects S 1.3.1 Extent of integration and quality of procurement aspects in Project design? MS Comment: The key issues and risks concerning procurement that were identified during the assessment of the first phase of the Project (inadequate procurement information system and omission of specifics on procurement procedures in the operational Manual) had already been implemented at appraisal. External audits and procurement reviews also provided evidence of adequate capacity without serious deviations but with some common deficiencies that are present in other Projects in Argentina. The rating here would have been Satisfactory, except for later events. In the ISR dated 12/18/2009, the Sector Manager refers to “…the ongoing INT investigations increasingly suggests a pattern of integrity problems in the implementation unit, especially pertaining to procurement.” The Team explained that the Minister herself initiated the investigation, the results of which are still not in the public domain. However, the Bank and the Ministry agreed on a plan to improve the procurement function in general, with positive steps to reduce corruption opportunities (web disclosure, open bidding, clear evaluations, web based records, etc.).

As of now the government has been slow to implement the agreed plan and this led to the Team rating procurement in the ISR as MU (although Project implementation is proceeding well). However, the team expressed optimism that next ISR will show marked improvements. In any case the initial procurement risk assessment, particularly regarding corruption seems now to have been rather naïve. In reality, capacity itself was not an issue, and the measures to improve it were adequate (training, systems, manual, etc.). Given the APL nature of the Project, and the good early pace of implementation, the reviewer considers that adequate measures were in place to support early startup of procurement activities. The plans were prepared and the provinces engaged reasonably early. The documentation indicates that the team’s procurement specialists were fully integrated with other Project activities during the design phase. 1.3.2 Extent of integration and quality of financial management aspects in Project design? S Comment: The Project risk assessment included in the PAD took into account the Bank's Country Financial Accountability Assessment, issued in September 2002, and identified the country risk and Project risk as Moderate. The assessment adequately covered the implementing unit’s arrangements for budgeting, accounting, internal control and funds flow, including disbursement, financial reporting and auditing. Its preliminary conclusion is that acceptable FM arrangements at NMH, through its International Financing Unit (UFI-S) continued in place and would remain the same for the proposed Project; the Project control framework was adequate and had been functioning effectively through the first year and a half of implementation of the then current project. Implementation risks were recognized from the experience of previous Projects and were taken into account in the design of this one. These relate principally to the weak institutional capacity of some provincial health ministries compared with the demanding technical requirements for the implementation of the Project. Appropriate mitigating measures, including an operational concurrent audit under TOR and by audit firm acceptable to the Bank, were identified and included in an action plan. 1.3.3 Quality of arrangements for governance and anti-corruption in Project design? S Comment: By designing the program around the existing setup, the Project avoided altering the political economy ramifications while introducing an empirically driven performance system for allocating resources. The introduction of an external auditing system (third party monitoring) provided at the same time a measure of verification, and with it a method to deal with fiduciary as

well as performance assessments. A greater emphasis on disclosure and beneficiaries’ outreach could have in time also strengthened the demand side of governance, particularly to hold the local Authorities accountable to the beneficiaries themselves. 1.3.4 Extent of integration and quality of social aspects in Project design? MS Comment: The overall assessment of integration of social issues in Project design is Moderately Satisfactory, reflecting a sound basic orientation on essential concepts and approaches, coupled with an insufficient attention to the mechanisms of implementation, particularly on the social side. This judgment applies both to component 3 (i) and (ii) (Communication and Community outreach), and also to social safeguards, where the Indigenous Peoples policy was triggered for certain provinces.

Program goals had a strong intrinsic focus on poverty alleviation and gender issues. The strong performance of APL1, and enthusiasm on the government side led to high expectations and ambitious promises for stage 2, and perhaps an underestimation of risks to performance on the social aspects.

Essentially, in sub-component 3 (ii) in particular, the Project promised to deliver, through the provincial work program agreements, increased participation of the target population, organized community groups, and even an element of citizen oversight. There does not seem to have been any prior assessment of the will and capacity of the provinces to do this, and the M&E indicator (people who knew of the program, and percentage of enrollees who were satisfied) was insufficiently challenging. Subsequently, the Bank agreed that this component had been under- resourced. It also seems to have been under-designed.

Safeguards policy required that Indigenous Peoples Plans (IPPs) had to be prepared for identified provinces. This would ensure inclusion, consultation, cultural and linguistic adaptation and “participative evaluation”. For this, there was a policy framework (summarized in the PAD) and a promise of institutional cooperation between the Ministry of Health and the National Institute of Indigenous Affairs. Despite detailed guidance at QER, there is little evidence of specific preparatory work to ensure that institutional capacity, political will, technical and human resources and an acceptable time-table would be in place at provincial level, not just to prepare the plans, but to implement them. IPPs were just listed as tasks in the provincial work agreement.

The problems of underperformance in the communications and outreach, and the nonperformance of the OP 4.10 requirements, became apparent through supervision two years into implementation. However, the origins lie in inadequate design. 1.3.5 Extent of integration and quality of environmental aspects in Project design? HS Comment: The Project is classified as a category "C" Project and has no safeguards aspects. However, APL-1 has financed a Strategic Environmental Assessment (SEA) which analyses in depth those environmental risks affecting maternal and child health. The SE identifies environmental causes relating to mother and child mortality and morbidity; covers markets, regulations and institutional capacity; analyzes strategic options for improving environmental policies relating to maternal and child health; and relates all of the above to the Plan Nacer. As such, it is a well-timed and high-quality piece of analytical work to inform government policies and programs in APL-2.

2. IMPLEMENTATION PROGRESS MS

Comment: The task team and the panel concurred on most major issues that have affected to date Project implementation. These are: (i) weaknesses in performance by some provincial and municipal public and private health care providers; (ii) changing a medical culture to accept and follow through on working under an incentive system; (iii) uneven provincial managerial capacities; (iv) political changes at the highest levels that slowed down the Government’s response to implementation problems; and (v) measuring and auditing performance requires a strong IT capacity that is not present at sufficient depth in some provinces. In addition the panel notes the lesser commitment of the larger provinces and their slow acceptance of the new

institutional arrangements and performance incentives. The task team groups the 15 provinces in three categories that reflect their commitment and contribution to the DOs. The third category that includes the three largest provinces has the largest number of potential eligible women and children and their performance affects the overall Project implementation. 2.1 Continued Government ownership and commitment to development objectives? HS Comment: The Government commitment in responding to the number of Provinces wanting to be associated with the program is one of the hallmarks of the program. The Federal Health Council (COFESA) has played an important role in ensuring this commitment to the DOs. The PCU at the national level provides the proper Government stewardship commensurate with the decentralized federal system in place in Argentina. 2.2 Extent to which the implementing agencies are able to ensure high quality MS implementation? Comment: Project implementation is the responsibility of the participating provinces and in some cases local Municipalities. Their capabilities vary greatly and a major challenge is how best to support and motivate the less performing provinces. Mechanisms in place to correct these weaknesses include meetings of the provincial PIUs every three months to exchange experiences and to learn from each other; active support from the central level PCU present at the provincial meetings; and targeting implementation support. 2.3 Borrower’s effectiveness in resolving implementation issues in a timely manner? MS Comment: General Issues: The country's constitution defines the authority of the central level as what is delegated to it by the constituent provinces. This dispersed authority makes it difficult at times to assign clear general implementation responsibilities. The Project does not include civil works. Legal covenants are in place but their compliance could not be ascertained by the panel as the ISRs do not report on them.

Procurement aspects: The new Minister herself raised issues of fraud and corruption prompting an INT investigation still underway. A plan was developed between the Bank and the Borrower to improve the transparency and efficiency of the procurement function, but it is being implemented too slowly. So, the borrower gets a good rating for raising the issue, and a bad one for slow resolution, thus explaining the MU rating. The procurement process and related risks has not been very well managed by the borrower, although Project implementation is proceeding reasonably well. As pointed out by the task team in its response to the initial draft, the implementation of the Action Plan has shown important advances in recent months as reported in the last ISR. While taking note of this development the panel supports the Sector Manager decision to retain the MU rating for procurement.

Financial Management: Per aide memoirs related to supervision missions in 2007, 2008 and 2009, the Task Team was generally satisfied with the capacity of the entity relating to financial management arrangements; however, due to the complexities of the Project prompt corrective action was not always taken timely. The Project has been audited by a government auditor who identified a number of internal control issues during the audits of the Project financial statements. Some of the issues had already been identified by the FMS and discussed with the Borrower. Some of the audit reports and unaudited IFRs were delayed. 2.4 Effectiveness in addressing environmental and social aspects (including safeguards)? MU Comment: Despite optimistic language in the PAD, it appears that Government commitment was lacking or inconsistent on implementing social safeguards, and on implementing the communications and outreach components. This situation, had it been correctly assessed, would have implied greater and earlier emphasis during supervision on social issues. 2.5 Timely production and utilization of M&E data? MS Comment: There has been on the whole a slow response to the delays in the startup of the Project. Now that it is at mid-point of the implementation period, there is an opportunity to reset the targets and reenergize implementation, particularly among lagging Provinces through more focused attention and support.

2.6 Effective coordination between executing agency and implementing agencies, S stakeholders and donors? Comment: There are several coordination vehicles, including the Federal Health Council (COFESA) and the National Health Service Purchasing Team (NHSPT), which have been useful for exchanges of experiences, and taking stock among all concerned. Provincial PIUs meet quarterly and meetings are attended by the national PCU. There are no other donors involved in the Project, and stakeholders have hitherto had a minor voice in Project execution

3. QUALITY OF BANK SUPERVISION MU 3.1 Focus on Development Effectiveness S 3.1.1 Extent to which the Bank identified and responded in a timely manner to S implementation problems? Comment: The Bank has identified implementation issues correctly and has discussed these issues with national authorities but political changes at the highest levels have slowed down the Government’s response to implementation problems. 3.1.2Appropriateness of Bank advice and proposed solutions to the Borrower? S Comment: Bank advice and proposed solutions are mostly appropriate but are not always politically acceptable. A greater specificity in the advice and solutions to enable a better-graduated response to the varying performance of the participating provinces would require a focus on specific provincial-level issues. However the sheer size of this operation makes it logistically difficult to work with all provinces and the task team therefore must rely on the technical assistance provided by the central level PCU to the provincial PIUs. 3.1.3 Adequacy of Bank’s efforts to address significant risks during supervision? S Comment: The scope of the Project, gaps in Project preparation, staff changes after elections have posed constraints. The task team now has a good understanding of the major risks to DO achievement. The delayed Mid Term Review provides an opportunity to seek corrective actions in the lagging Provinces. 3.1.4 Extent to which Bank’s supervision efforts focused on sustainability issues? S Comment: Programmatic sustainability is judged to be favorable as the institutional arrangements and the performance incentive payments introduced by this operation are likely to stay in place after Project closure. Financial sustainability once Bank funding has been used up and provinces have to rely on their own resources is less certain but would vary among provinces as interprovincial economic disparities are significant and the willingness and financial capability of the central government to help the poorer provinces is uncertain. Both these issues are being addressed by the task team. 3.1.5 Extent to which the Bank’s supervision efforts made adequate use of quantitative S and/or qualitative performance indicators: Comment: Data generated at the local level and subsequently aggregated at the provincial level are used to decide on changes in service provision and on targeting beneficiaries. The ten health interventions used as tracers are a useful guide for assessing implementation progress across provinces. The task team has identified the major implementation obstacles that would limit achieving the DOs and conditions are now ready to poise the Government and relevant Provinces to act on the issues that have been identified. Baseline data and impact evaluations will be useful to adjust resource allocation and manage identified shortcomings as the Project continues to be implemented. 3.2 Fiduciary, Safeguards and GAC Aspects MU 3.2.1 Quality of Procurement Oversight? S Comment: From a corruption point of view, the procurement risks identified at the design stage were underestimated, as proven by the INT investigation still underway with the full support of the Bank Management Team, initiated at the request of the new Minister (see 1.3.1.b). The Bank team reacted proactively and effectively working with the Borrower to develop an action plan to improve the transparency of the procurement function. The Borrower has been rather slow to implement the plan and the Team has been putting pressure to accelerate it. It can be concluded that Fraud and Corruption issues have been identified and are being addressed. Two very good annual procurement reviews covering ex-ante, ex-post, and the procurement function in general were carried out. Both reviews indicate staffing problems, lack of progress in agreed plans to

improve the procurement function, and slow implementation of procurement activities in general, proposing a MU rating (2009 review). The reviewer believes the Bank Team has acted effectively and proactively in discovering and explaining these issues to the Borrower. Although the ISRs and reviews indicate that the procurement plan has not been updated in a satisfactory manner, disbursements are not really an issue (as explained by the TTL). The reports indicate that there have been no complaints. The ISRs and ex-post reports indicate that there has been an excellent integration of procurement supervision activities with the overall supervision effort (especially integration with FM activities to ensure full coverage of fiduciary issues) and these are reflected in the documentation. 3.2.2 Quality of Financial Management Oversight? S Comment: The task team performed nine supervision and technical missions between June 2007 and December 2009, and has included the FMS or an FM consultant in four of them. The intensity and coverage of those missions corresponded to a Moderate level of risk. The FMS/Consultant has reviewed relevant aspects of the Project and has proposed corrective actions, verifying that mitigating measures were implemented and in most cases were working satisfactorily. In the case of annual audit reports, the Bank provided comments in a timely manner after receipt of the report.

Most critical records were made available for the QAG review, but some documents were missing such as audit reports. The task team spent an inordinate amount of time searching for them in IRIS and finally provided QAG with the needed documents. This appears to be a system's deficiency and does not reflect on the task team. 3.2.3 Quality of Oversight of Governance and Anti-Corruption Aspects (including MS procurement, financial management and other fraud and corruption issues that affect Project quality)? Comment: Oversight quality of GAC is mixed; although according to the QALP procurement review the supervision reports indicate that there have been no complaints, the panel believes that the upcoming MTR should address GAC issue up front. 3.2.4 Quality of Oversight of Social Aspects (including safeguards)? MU Comment: On social matters, the level of effort during supervision seems to have been low. This would have identified issues with the communications program, with monitoring the behavioral “story line” on beneficiary enrollment, with ensuring effective communications and community outreach, and to ensure that policy requirements with safeguards were complied with. Although the PAD suggested in places that some measure of social accountability might be introduced, there is no evidence that during implementation this was ever under consideration.

The team is commended for prompt and focused response to IP safeguards compliance once the issue was identified. However, matters should not have been allowed to reach this stage, as an out of compliance operation poses reputational risk. No information is available on how the matter was followed up after March 2009 and in its absence, it has to be assumed that the situation remains unsatisfactory.

This assessment is based on the evidence made available so far and may be modified if satisfactory evidence is forthcoming to show that meaningful provincial level IPPs were developed and implemented. 3.2.5 Quality of Oversight of Environmental Aspects (including safeguards)? NA 3.3 Supervision Inputs and Processes S 3.3.1 Adequacy of staffing arrangements? S Comment: There is a competent task team in place in country and at headquarters with all relevant skills to address the issues at stake. Although seemingly unavoidable, three TTLs were responsible for the Project over three and half years. The present TTL has been in place now for two years suggesting an improved continuity in task leadership. 3.3.2 Adequacy and timeliness of country and sector management attention and actions S (including follow-up letters to the borrower and comments in the ISR)? Comment: The Project had the support of both the country and sector managers, with the latter

making relevant comments at each ISR, and been “on top” of the effort, including his joining a supervision mission. 3.3.3 Adequacy and effectiveness of budget allocation and use? S Comment: All indications are that the TTL has received the resources needed to undertake supervision work and those resources were used appropriately. Supervising implementation in 15 provinces requires a larger and stronger team in order to be able to focus more selectively on provincial-level problem areas and to manage issues with greater selectivity 3.4 Candor and Realism of ISR MS 3.4.1 Candor and realism of ISR ratings for DOs? MS Comment: Achieving DOs has been consistently rated satisfactory. The panel recommends a more cautious rating approach given the lower than expected enrolment of eligible children and women. A moderate satisfactory DO rating would be a more realistic reflection of these concerns until some critical milestones are achieved.

In its response to the panel's comments above the task team expressed the intention to thoroughly assess this aspect during the October 2010 MTR. 3.4.2 Candor and realism of ISR ratings for IP? MS Comment: Implementation progress is affected by a number of factors enumerated under Implementation Progress. Notwithstanding these issues the ISRs consistently assign a satisfactory rating to the IP; the panel considers a moderately satisfactory rating to have been more realistic. The panel also considers that the risk flags for FM and safeguards (indigenous people) should have been used. Use of these flags would have placed the Project in the “potential problem Project” category in FY09 and FY10 and would have signaled to management the nature of the implementation problems 3.4.3 Quality of supervision reporting? MS Comment: The reporting in the ISRs is rather light. It could be enhanced with comments on the RF indicators, more specificity on agreed actions, reporting on legal covenants (no covenants are included in the ISR) and a more convincing explanation why DO and IP are rated satisfactory.

In responding to these comments the task team points out that RF indicators and agreed actions are being commented upon in the ISRs under the “tracers system” name. Regarding the Legal covenants, they had been already met, and they are informed under the Intermediate Results Indicators section in the ISR. 3.4.4 Consistency between supervision reporting and supervision ratings? MS Comment: ISR ratings in general are mostly favorable but are not fully supported by the narrative in the ISRs.

In responding to the comment above the task team points out that although some indicators targets are underperforming, the team considers the overall execution of the Project to be satisfactory while the functioning of the incentive scheme is a valuable result. Furthermore, the team is reviewing with the Government how to address the underreporting related with the information source that provides inputs for the indicators. While the Project indicator targets were set using the national statistics systems, the Government is informing the targets using the tracer systems, based on nominalized population. Note: HS = Highly Satisfactory; S = Satisfactory; MS = Moderately Satisfactory; MU = Moderately Unsatisfactory; U = Unsatisfactory; HU = Highly Unsatisfactory. (Uses 6-point scale H, S, MS, MU, U, HU.)

Table 2: Ratings Comparison of QALP-2 and ISRs DO/IP/Risk End End Mid–FY 10 4/ End FY 11 FY 12 FY 13 Factor 1/ FY 08 FY 09 FY 10 ISR Panel ISR Panel ISR Panel June Aug June Dec Aug Nov (ISR 8) (ISR 9) (ISR 10) (ISR 11) (ISR 12) (ISR13) Achievement S S S MS S MS MS MS MS MU MU MS Towards PDOs Implementation S MS S MS S MS MS MS MS MS MS MS Progress Project Components 1/ Provincial S S S MS S MS S MS MS MU MS MS Maternal-Child Health Stewardship S S S MS S MS S S S S S S Ministries of Health Community MS MS MS MS MS MS MU MU MU MU MU MU Outreach M&E, Auditing S S S S S MS MS MS MS MS MS MS Management and S S S S S S S S S MS S S Administration Discretionary Flags2/ Counterpart No Yes No Yes No Yes No No No No No No Funds Financial No Yes No Yes No No No No No No No No Management Legal Covenants No No No No No No No No No No No No Monitoring and No No No No No No No No No No No No Evaluation Project No No No No No No No No No No No No Management Procurement Yes Yes Yes Yes Yes Yes Yes Yes No No No No Safeguards No Yes No Yes No Yes No No No No No No Sub-Total 1 4 1 4 1 3 1 1 0 0 0 0 Exogenous Flags3/ Disbursements No No No No No No No No No No No No Delays Effectiveness No No No No No No No No No No No No Delays Country Yes Yes Yes Yes Yes Yes Yes No No No No Yes Environment Country Record Yes Yes No No No No No No No No No No Long-Term Risk No No No No No No No No No No No No Flag Notes: 1. HS = Highly Satisfactory; S = Satisfactory; MS = Moderately Satisfactory; MU = Moderately Unsatisfactory; U = Unsatisfactory; HU = Highly Unsatisfactory. (Uses 6-point scale H, S, MS, MU, U, HU.) 2. No means rating is HS, S or MS. Yes means rating is MU, U or HU. 3. Exogenous flags are system generated flags and have the same ratings for both ISR and Panel. 4. As of February 7th, 2010.

Annex 12. More Details on Preparation and Implementation of APL-2

1. This annex provides additional details of preparation and implementation aspects of this APL-2 Project, to complement what is already in Section 2 of the main body of this report.

A. Project Preparation and Design

A-1. Lessons Drawn Upon at Preparation Stage, and Rationale for Bank Intervention

2. The lessons drawn upon at the preparation stage were appropriate, and were incorporated appropriately into the Project design. The PAD for APL-2 mentions three major lessons that were drawn on for APL-1 and APL-2: (a) The focus on reform of the national social health insurance system during the 1990s proved insufficient to address the health problems of the poor, and thus coverage of the uninsured did not expand significantly; (b) previous projects and reforms insufficiently reached the poorest provinces where most of the uninsured live; and (c) the Bank’s experience in supporting Argentina’s health sector indicates that Bank’s financing should be closely linked to results. The first and third lessons were especially relevant for APL- 2.

3. The PAD for APL-2 notes two additional lessons learned from the implementation of APL-1. (1) The reforms introduced by Plan Nacer and APL-1 are demanding, both technically and institutionally, requiring substantial technical assistance and intense policy dialogue for successful implementation; and (2) The introduction of an additional element in the independent concurrent audit to verify the compliance of the provinces with key elements of Plan Nacer has proven instrumental in the program’s success. The design of APL-2 did incorporate expanded technical assistance and a continuation of the independent concurrent audits following the modality introduced under APL-1.

4. The rationale for Bank intervention for APL-2, as for APL-1, was solid. A major rationale for the Bank’s intervention, according to the PAD of APL-2, was its long experience with Argentina’s health sector (more than ten years), resulting in significant comparative advantages in supporting government efforts that combine policy reform with investments to contribute to longer-term improvements in institutional performance. The ICR for APL-1 found this rationale to be appropriate, and this present ICR for APL-2 finds the same.

A-2. Design and Preparation

5. In addition to the strengths at the design and preparation stage identified by the QALP-2 report, this ICR finds the following to be strengths regarding the quality of design, in line with the ICR for APL-1:

• High relevance: The issues addressed under APL-2, as for APL-1, were still highly relevant for the country. The poor access and quality of health services was an issue for the Phase 2 provinces, as they were for the Phase 1 provinces under APL-1 earlier. As for

the Phase 1 provinces, this was having a negative impact on pregnant/lactating women and young children without formal health insurance.

• Pro-poor focus: The Project addressed the root cause of inequity in healthcare by focusing on the most vulnerable populations: uninsured children, pregnant women and mothers.

• Importance of Provincial Governments taken into account: APL-2 contributed to further strengthening the stewardship function of the national government in a federal context while also recognizing the importance of provincial governments in the delivery of these services. The Project strengthened the linkage between the national government and the provinces by establishing an incentive system between the two of them in order to improve the quality and quantity of specific health services.

• Innovative results-orientated systems of incentives: The Project incorporated a strong results-oriented system where provinces as well as health service providers were incentivized to improve performance, bringing a critical change to the system and orienting it to an approach focused on results.

• Strong collaboration with various actors: Close coordination of activities in the health sector was maintained with the Inter-American Development Bank (IDB) which supported the government’s pharmaceutical program (free essential medicines for the poor) through REMEDIAR. To ensure close coordination responsibility for both programs, Plan Nacer and REMEDIAR were placed under the same Secretary of Health (Secretary of Sanitary Programs). The Project was also closely coordinated with the Pan American Health Organization.

• Choice of Lending Instrument: The choice of an the Adaptable Program Investment Loan (APL) instrument allowed testing of the design and implementation of Plan Nacer in the Northern provinces under APL-1 so that preliminary lessons could be derived from ongoing experience. The lessons that had been learned were built into practices and norms to guide service delivery, reforms in the financing regime, and institution building.

B. Implementation

B-1. Procurement, and Governance and Anti-Corruption Plan

6. Procurement in the QALP-2 report: Prior to the QALP-2 report of June 2010, the rating for procurement in the ISRs had been downgraded to Moderately Unsatisfactory (MU) after a post-procurement review in May 2008 found moderate deficiencies in procurement management (for the entire health portfolio) at the International Financing Unit for Health (UFIS). UFIS is an operational unit within the Ministry of Health in charge of procurement in Bank projects. The Bank and UFIS agreed on an action plan in order to improve the situation. The action plan was also meant to improve the transparency and efficiency of the procurement function, to help address issues related to an INT investigation that was started because the Minister herself raised issues of fraud and corruption. However, at the time the plan was being

implemented too slowly, which explains the rating that was still MU in the last ISR done before the QALP-2 report.

7. The QALP-2 report provided a rating of MU for implementation by the Government on procurement issues, linked partly to the INT investigation that had just been launched. The fact that the investigation was launched at the request of the Minister of the Health was seen as positive by the QALP-2 report.

8. Governance and Anti-Corruption (GAC) Plan: Ultimately, the INT investigation did not result in any proven corruption under this Project or others in the health sector in Argentina. However, a GAC Plan was developed in early 2011, and agreed between the Government and the Bank. In June 2012 a supervision mission conducted by the Bank found that most of the activities included in the GAC had been completed, except for four. Decent progress was noted on these four actions in the last ISR in November 2012.

9. Specifically, the following was noted for each of the four pending actions:

(1) Revision of the system of independent alerts regarding internal controls against fraud and corruption: the Bank will hire an external consultant to advise on this, and this will be done around the time of the Mid-Term Review of the FESP II project (in early 2014). (2) Evaluation and finalization of contracts within the agreed timelines: A Plan of Action has been formulated to strengthen the Procurement and Contracting Unit of UFIS, and periodic followup will take place regarding this plan. (3) An exhaustive analysis regarding adequate via the external technical audit of two provinces selected by the Bank, under the FESP II project: The two provinces have been selected, and the analysis will be done when the FESP II project is at a more advanced stage (this requires inputs from an external consultant hired by the Bank). (4) UFI-S will integrate the functions of financial management and procurement by adopting the interface of UEPEX-SEPA (Unidad Ejecutora de Proyectos Externos–Sistema de Ejecución de Planes de Adquisiciones) with technical assistance from the Bank: An external consultant has been hired for this, and the work will soon be in progress.

10. Procurement performance later in the Project: The Mid-Term Review (MTR) of November 2011 found that overall performance in procurement implementation had improved. This had justified (a few months earlier) an upgrade in the procurement rating in the ISRs from MU to MS in June 2011. An action plan was anyway developed at the time of the MTR based on a review of procurement in the health sector program that had recently been conducted. Performance regarding this plan has been satisfactory.

11. Finally, in May 2012, an external consultant conducted an Independent Procurement Review (IPR). The Government then produced an Action Plan in late 2012 to address its findings (agreed to by the Bank).

B-2. Changes to the Benefits Package

12. In early 2010, the benefits package was broadened to include two additional groups of services: (a) a sub-package of congenital heart disease (CHD) treatment services for children; and (b) a “perinatal sub-package” of highly complex maternal health services. This was the result of analysis that showed that: (a) CHD was a major factor underlying child mortality in the country (accounting for about 8% of under-five deaths); and (b) additional services were found to be needed in the package to address more complex birth problems and higher-risk births, and to increase the impact on maternal mortality. A loan agreement amendment was approved in April 2010, in support of this. The inclusion of these additional services led to a small increase in the monthly capitation payment to be paid to the provinces.

13. A separate Fund was set up at the central level for the CHD services, but the pace of invoicing for the CHD services was initially low, leading to a large accumulation of unused resources in this Fund. The Fund was set up separately because in the case of the CHD services, fee-for-service payments would go directly from this Fund to the hospitals providing the CHD services (which were not many to begin with), instead of being first transferred to provincial Bank accounts. (In the case of the original basic package of services, the modality of operation continued as before, i.e. all capitation payments continued to be made to the provinces and to flow into provincial Bank accounts, before being used for fee-for-service payments to the health facilities.) Thus, in effect the CHD Fund was managed at the central level, although on behalf of the provinces.

14. But due to various reasons, in particular the complexity of the new sub-package of CHD services, invoicing at the level of the health facilities was initially slow. By the time of the MTR in November 2011, capitation transfers related to Congenital Heart Disease (CHD) Interventions had accumulated to approximately US$40 million of which only 10% had been used for fee-for- service payments to health facilities. The remainder of the funds remained in the CHD account.

15. For this reason and to avoid any additional increase of accumulated funds, the Government and the Bank team agreed to suspend further accrual of the capitation payments related to CHD services until the execution of these proceeds had reached an acceptable level. By Project close, the amount of accumulated funds in the CHD account had fallen to around half at about US$20 million, due to a much higher rate of invoicing and fee-for- service payments for these services at the health facility level. The reminder of the funds will be used for CHD payments under the follow-on Plan Sumar project.

16. The introduction of CHD services for children in the benefits package nevertheless constitutes a successful and model experience for the Argentinian health sector. It marks the first time that catastrophic care interventions have been included in a public health insurance scheme in the country, resulting in a significant impact on infant and neonatal mortality, and enhancing financial protection among beneficiary families. A well-functioning network of establishments providing CHD treatments was created – the first Federal network of health facilities providing specialized complex care in the country. As a result, the number of surgeries to treat CHD rose by 97%, and the waiting list for patients with CHD fell by 80%. As noted in

Annex 5, a survey of beneficiaries showed that the level of satisfaction of mothers of beneficiaries of this scheme was found to be very high, at 9.1 on a scale of 10.

B-3. Provincial Counterpart Funding

17. Provincial Counterpart Funding challenges and positive steps taken by the Government towards resolution: From January 2011 onwards, “Phase 2” provinces started to make their required contributions of 30% to their capitation payments, with Project loan (IBRD) funds used to pay the remaining 70%. (Previously, 100% of the capitation payments had been financed by the project.). “Phase 1” provinces had already had to make their 30% contributions since 2009, under APL-1, before being transferred to APL-2 starting in August 2010. Provinces had 15 days to make their deposit for the co-payments into the Plan Nacer provincial accounts. If there was a delay of more than 15 days, provinces were charged a fine. If later than 45 days, the national government suspended its payment (financed by the Project loan) of 70% of the total.

18. This process didn’t take into account the particularity of administration processes and provincial constraints in each province. A pattern was seen whereby towards the end of the calendar year, provinces would sometimes not make their payments, but they would typically repay in full the arrears of each year at the start of the following year.

19. At Project close in December 2012, nine provinces (five from Phase I and four from Phase II) had been temporarily suspended from receiving payments, since they had not deposited the full amount of their counterpart contributions within the agreed time period. The national Government responded by stating that capitation payments under the Sumar program for January 2013 onwards would not be made to any province that had not paid its debt (if any) in full under the Plan Nacer program. This was in order to incentivize the provinces to pay their debts owed under Plan Nacer. The strategy worked, since all provinces have now paid their debt in full under Plan Nacer.

B-4. Impact Evaluation (IE) and Baseline Survey

20. A broad evaluation agenda existed under the Project, including in-depth IE, and this in turn led to its own challenges and delays. This included a delay in the baseline IE survey for the Phase 2 provinces, which was conducted in 2008/09 instead of in the first year of the Project (2007) as originally planned (see Annex 13 for full details).

21. Because of the delay in the baseline IE survey, PDO Indicator baseline values could not be obtained for individual provinces or groups of provinces. The baseline values for the seven PDO Indicators that are linked to the provincial “tracer” indicators – PDO Indicators 2 to 8 in Table 3 of the main text of this ICR – are reported in Annex 3 of the PAD of APL-2. These values are all national values (and are reported as such), based on national data, since provincial- level values were not available at the time. Since APL-2 was originally meant just for the 15 new Phase 2 provinces, it was planned that the baseline IE study would be used to obtain baseline values for the Phase 2 provinces only (as a group), and these would be used to replace the national baseline values reported in the PAD.

22. But then two unforeseen events occurred. First, the baseline IE study was delayed substantially, and was ultimately conducted in 2008/09, well after the Project started in late 2006. Second, in mid-2010 the Phase 1 provinces were included in APL-2, in addition to the Phase 2 provinces. At that point, APL-2 became essentially a Project with national coverage. So, there was no longer a need for baseline values for the Phase 2 provinces only, and the original national-level baseline values were valid after all.

Annex 13. Impact Evaluation for Plan Nacer: Challenges and Accomplishments

A. Impact Evaluation (IE) Activities Under APL-2 and Plan Nacer – An Overview of Events

1. Impact Evaluation (IE) is an essential part of Component 4 of APL-2, and was designed for the fifteen Phase 2 provinces, just as an IE exercise had been designed separately under APL-1 for the “Phase 1” provinces. The Phase 2 provinces are those that entered into Plan Nacer for the first time under APL-2, while the Phase 1 provinces had entered under APL-1. Both IE studies – the IE for the Phase 2 provinces and that for Phase 1 provinces – were envisaged to generate evidence to evaluate the design and implementation of Plan Nacer. The primary objectives of the IE in both cases were the following:

i. Generate evidence on the overall impacts of Plan Nacer program on access to care, quality of care and health outcomes. ii. Evaluate design features of Plan Nacer in order to determine how the existing RBF model can be strengthened in order to improve priority health outcomes. iii. Disseminate Knowledge based on the evaluation Program; publications and Seminars/BBLs.

2. The IE for the Phase 2 provinces under APL-2 experienced major challenges and therefore significant delays and modifications, as has been the case for the IE for the Phase 1 provinces. At the appraisal stage for APL-2, it was anticipate that a baseline study for the 15 “Phase 2” provinces (that were newly included under APL-2) would be completed within the first year of implementation of APL-2 which was approved in late 2006. But data collection for this baseline survey was ultimately completed in 2009 (see below for factors underlying this). Similarly, the IE exercise for the nine “Phase 1” provinces (that were newly included under APL-1) had its challenges and experienced delays. A description of events in each case is given below.

3. The teams working on the IE for both studies (for the Phase 1 and the Phase 2 provinces), have shown a high degree of adaptability, strengthening their efforts on IE and producing a series of credible results with work still ongoing. At this point, the credible results are from the IE of the Phase 1 provinces, with work still ongoing for the IE of the Phase 2 provinces (see below for more details). This was accomplished, mostly before 2011, via exploring new sources of information, particularly administrative data of the Program for the provinces of Misiones and Tucumán. Early in 2011 both teams identified the need to digitalize clinical records of births occurring during the period 2004-2008, for the Phase 1 provinces (and some control provinces). This information was deemed suitable as a sample frame for the follow- up survey (for the Phase 1 provinces) and to replicate the evaluation techniques used for Misiones and Tucuman. At the end of 2011, both teams recognized the need to increase the number of staff working on the IE agenda, and both teams soon did so.

4. Most importantly, the Government team has broadened the evaluation perspective, shifting from IE, with a quantitative and causal approach, towards a more comprehensive evaluation agenda. The Government team (National Ministry of Health) identified critical

aspects and results of the Program that cannot be captured by means of a quantitative-causal approach. Thus, other quantitative and qualitative studies were implemented. These studies include the following, among others: (i) an examination of how motivation of health facilities’ staff is affected by the presence of an RBF program; (ii) an assessment of users' satisfaction; (iii) an analysis of the synergy between the Universal Child Allowance policy and Plan Nacer; and (iv) a review of the results linked to the implementation of a CHD Surgery Network in the country. This evaluation agenda was funded mainly from the Government’s own funds, as a sign of direct involvement and leadership by the Government in institutionalizing a broad evaluation agenda, going beyond IE.

B. IE for Phase 1 Provinces in Detail

5. The IE for the nine Phase 1 provinces was conceptualized at the start of APL-1. The original evaluation design – the matched double-difference method – planned to use 4 provinces adjoining the northern region (where the nine Phase 1 provinces were located) as controls. These were Córdoba, Entre Ríos, La Rioja and Santa Fe provinces. The nine Phase 1 provinces would be the “treatment” group.

6. During 2006, a baseline survey was carried out in the nine provinces in the “treatment” group, as well as the four provinces in the control group. The survey collected data on the eligible population in the health service providers’ area of influence, on patients from exit interviews, and on health facilities.

7. However, in 2007 the original evaluation design was invalidated because all provinces in the country rapidly implemented the Program, under APL-2. (As noted elsewhere in this document, it had originally been envisaged that Plan Nacer would be rolled out in three phases, but a decision was taken to roll it out rapidly nationwide under APL-2). Thus, the control group provinces of Córdoba, Entre Ríos, La Rioja, and Santa Fe became “contaminated”, and could no longer serve as controls.

8. The strategy to evaluate the impact of Plan Nacer in the Phase 1 provinces thus had to be modified. Among others, a follow-up household survey for the Phase 1 provinces was conducted in 2011/12 but the composition of the sample of households in this follow-up survey had to be modified (compared to what had been originally planned and compared to the baseline survey). In particular, the new survey re-defined the sample of visited children born between July 2006 and August 2007 in order to include a high proportion of children who had been exposed to the program since birth in treatment provinces but not in control provinces. With these data, an “intention to treat” analysis will be carried out using a “difference in differences model” with fixed effects at the provider level, comparing changes in outcomes in areas under the program compared to areas not under the program. This analysis will shed light on the average effect of the program on providers and on the eligible population, controlling for baseline characteristics. In a second analysis, the team will estimate the effect of the program on the beneficiaries themselves (“treatment on the treated estimates”), controlling for fixed effects at the provider level and instrumenting beneficiary status using the eligible population at baseline and date of providers’ incorporations. The data from this follow-up survey are still being cleaned and analyzed, and results are expected in the next few months.

9. These efforts focusing on surveys were complemented by a number of very fruitful efforts focused on the collection and use of administrative data. First, in 2009/10, administrative datasets were put together and analyzed for the two northern provinces of Tucuman and Misiones. These data cover the periods 2005/06 and 2008 for Tucuman and 2007 to 2009 for Misiones, and consist of data from the Plan Nacer records as well as from medical records at health centers and hospitals.

10. Later, in 2011, an even larger (and more laborious) effort took place, focusing on digitalizing clinical records at maternity wards. A large administrative dataset was assembled using data from the Plan Nacer records as well as from medical records collected from public maternity wards, for births that had taken place between 2004 and 2008 in 9 provinces (all of the nine Phase 1 provinces) as well as the control provinces La Rioja, Córdoba, Santa Fe and Entre Ríos. A dataset of more than 400,000 observations was put together, in this manner.

11. The data from the abovementioned follow-up survey of 2011/12 are still being cleaned and analyzed, and results are not yet ready. But results from the analysis of the administrative data from Tucuman and Misiones provinces are available, albeit with the caveat that they cover only two provinces. Key results are reported in Annex 3.

12. Some initial results from the analysis of the larger administrative dataset with more than 400,000 observations (assembled in 2011/12) are available. These data are still being analyzed, and final results are expected to come in during the next few months. Some preliminary results are reported in Annex 3.

13. These results from the analysis of administrative data show that administrative data can serve as an effective tool for generating solid results regarding program impact. Administrative data are also useful, of course, for the purposes of ongoing adjustments during implementation, i.e. for monitoring purposes. In summary, strengthening administrative data collection can potentially lead to an effective approach for generating evaluation results and for monitoring program implementation, partially complementing survey data.

C. IE for Phase 2 Provinces in Detail

14. The IE exercise for the Phase 2 provinces, conceptualized at the start of APL-2, was complicated by the fact that all provinces would be rapidly included in the program. Hence, it would no longer be possible to have a true experimental design with a “treatment” group and a “control group”, where the intervention is applied only to the treatment group but not to the control group.

15. An alternative quasi-experimental design was thus proposed. The idea was to generate exogenous variation in the intensity of treatment, and then to assess whether higher intensity of treatment leads to better outcomes. The exogenous variation in this case was supposed to come from information campaigns administered to eligible persons in randomly selected localities. These campaigns were meant to increase program enrolment. This design – known in the

literature as randomized promotion – would produce exogenous increase in the registration level in the Program, creating a valid instrumental variable to mitigate the selection bias inherent in a household’s decision to enter the program.

16. By the end of 2007, a pilot test was implemented in order to identify the most effective promotion strategy. The pilot test was carried out in 4 provinces – Catamarca and Misiones from the Phase 1 province group and Cordoba and Chubut from the Phase 2 province group – showing highly satisfactory results in terms of registration rates.

17. The baseline IE survey for the Phase 2 provinces took place finally, in 2008/09. The most effective promotion strategies were implemented in a representative sample of localities in Phase 2 provinces. But after 15 months of promotion implementation, results showed that on average, there was no effect on registration rates in these provinces. The exceptions were the 2 provinces that participated during the pilot test (Cordoba and Chubut), suggesting that randomized promotion requires detailed design and an intensive process of supervision to be successful.

18. A second round of promotion strategies including additional activities to increase participation and to incorporate new localities was scheduled for 2011, and a follow-up IE survey for the Phase 2 provinces was planned for later. However, in 2009 the national Government introduced the “Asignación Universal por Hijo” program. This is cash subsidy program where families with vulnerable children can receive a cash subsidy (per child) if they are enrolled in Plan Nacer. The introduction of this cash subsidy program created a strong incentive for eligible children to be enrolled in Plan Nacer across the country, weakening the randomization promotion evaluation design.

19. Due to these unplanned events, the randomization promotion evaluation design approach was abandoned. During 2011, alternative sources of information were explored in order to obtain the fastest results without compromising methodological rigorousness. The follow up data collection originally planned for Phase 2 provinces did not take place due to the introduction of this cash subsidy program and the contamination of the control group.

D. Adaptability on Part of Government and Bank Teams

20. Despite the changing circumstances on the ground and the challenges faced (as detailed above), the Bank and Government teams showed great adaptability and flexibility, and solid results have been generated using administrative data despite the challenges (see Section B). This required a large and laborious effort, as mentioned above, to put together an administrative dataset using data from the Plan Nacer records as well as from medical records collected from public maternity wards. This is in addition to the results generated using administrative data from Tucuman and Misiones provinces.

21. Work using the data from the follow-up household survey for the Phase 1 provinces (conducted in 2011/12) is ongoing, and results are expected soon. This is another example of adaptability on the part of the Bank and Government teams, with a drive to generate results despite challenges faced.

91 IBRD 33362R 70°W BOLIVIA To Tarija 60°W 50°W

PilcomayoPi JUJUYJUJUY lco ma To yo PARAGUAY Calama SanSan SalvadorSalvador DeDe JujuyJujuy

To SALTAS A L TA SaltaSalta FORMOSAFORMOSA San Pedro s To S CHILE a y Cascavel l a a u d á g n o CHACOCHACO FormosaFormosa a a e r a r TUCUMÁNTUCUMÁN ParaguayP a To SanSan MiguelMiguel DeDe TucumánTucumán ParanáP Copiapó S d SANTIAGOSANTIAGO PosadasPosadas E BRAZIL ResistenciaResistencia ON CATAMARCACATAMARCA SantiagoSantiago DelDel EsteroEstero CorrientesCorrientes ISI MISIONESM y ua n ug DELDEL ESTEROESTERO Ur CatamarcaCatamarca CORRIENTESCORRIENTES To LaLa RiojaRioja Santa Rosa To A La Serena SANTAS A N TA LALA To LagunaLaguna FEF E Santa Maria 30°S MarMar ChiquitaChiquita á RIOJARIOJA n SANSAN ra CordobaCordoba a To ParanáP Artigas JUANJUAN SanSan JuanJuan SantaSanta FeFe ParanáParaná 30°S CerroCerro CORDOBACORDOBA D To AconcaguaAconcagua e s ENTREE N T R E Tacuarembó (6960(6960 mm)) a g

PACIFIC u MendozaMendoza a RÍOSR Í O S d To e SanSan LuisLuis RosarioRosario r Valparaíso o To URUGUAY OCEAN s Montevideo MENDOZAMENDOZA SANSAN a LUISLUIS BUENOSBUENOS AIRESAIRES

p LaLa PPlatalata To Talca m 0 100 200 300 400 500 Kilometers S a BUENOSB U E N O S

l a a

d SantaSanta RRosaosa

o o o ColoradoColo AIRESA I R E S rado LAL A P 0 100 200 300 Miles PAMPAPA M PA MarMar deldel PlataPlata To Los Ángeles NEUQUENNEUQUEN

BahíaBahía BlancaBlanca

NeuquénNeuquén N eg ro A RIORIO NEGRONEGRO 40°S

n To ViedmaViedma Puerto Montt

d 40°S

C e h u b CHUBUTC H U B U T u t ARGENTINA

RawsonRawson ATLANTIC s

OCEAN a

o ic h i C

To n Coihaique ComodoroComodoro RRivadaviaivadavia

o sea De do

CHILE g SANTAS A N TA a CRUZC R U Z

t C hi ca

a ARGENTINA FALKLAND ISLANDS (ISLAS MALVINAS) A DISPUTE CONCERNING SOVEREIGNTY OVER THE

P ISLANDS EXISTS BETWEEN ARGENTINA WHICH CLAIMS 50°S 50°S THIS SOVEREIGNTY AND THE U.K. WHICH ADMINISTERS SELECTED CITIES AND TOWNS THE ISLANDS. RíoRío GallegosGallegos PROVINCE CAPITALS StanleyStanley NATIONAL CAPITAL RIVERS

TIERRATIERRA MAIN ROADS DELDEL FFUEGOUEGO This map was produced by the Map Design Unit of The World Bank. RAILROADS The boundaries, colors, denominations and any other information UshuaiaUshuaia shown on this map do not imply, on the part of The World Bank Group, any judgment on the legal status of any territory, or any PROVINCE BOUNDARIES endorsement or acceptance of such boundaries. INTERNATIONAL BOUNDARIES 80°W 70°W 60°W 50°W AUGUST 2008