» EVALUATION REPORT MARCH 2019
EXTERNAL EVALUATION OF PHASE I OF THE PBF SHOWCASE PROJECT IN BORENA
Workie Mitiku Eshete Yilma
February 18, 2019 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA Acknowledgements
ACKNOWLEDGEMENTS
The evaluation team would like to acknowledge several colleagues from Cordaid Head Office and Cordaid Addis Ababa Office for their inputs, support and guidance throughout the evaluation process. In particular, the team is grateful to Inge and Maarten from Cordaid head Office and Fikremariam Gezahegn from Cordaid Addis Ababa Office for their valuable comments on the evaluation design, tools, and report preparation, all of which improved the quality of this evaluation report. We also acknowledge Nurky Ibrahim and Gelgelo Hallke from Cordaid Yabelo Borena Office. Without their support, vital appointments with key informants at zonal, woreda, facility and community levels and data collection would not have been successful. Finally, we are very grateful to all the key informants and focus group discussion participants at the federal, regional, zonal, woreda, facility, and community levels for their open and honest views on the performance of the project.
2 MARCH 2019 © CORDAID contents EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
CONTENTS
acknowledgments...... 2 acronyms...... 5 executive summary...... 6
1. introduction...... 8
2. purpose and scope of evaluation ...... 9
3. methodology ...... 10
4. findings...... 12 4.1 Qualitative findings (Evaluation Question No.1)...... 12 4.2 Quantitative Findings (Evaluation Question No.2)...... 16
5. conclusions and lessons learnt...... 23
6. recommendations...... 25 annexes
Annex A: Figures on quality scores by Health Facility...... 26 Annex B: Tor for an external evaluation of phase I PBF showcase Project in Borena Zone...... 28 Annex C : Questionnaire for the evaluation...... 32 Annex D: List of people consulted...... 37 Annex E : References...... 41
MARCH 2019 © CORDAID 3 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA contents
list of tables
Table 1: Evaluation Design Matrix...... 10 Table 2: List of Woreda Health offices and Health Facilities Visited...... 11 list of figures
Figure 1: Outpatient consultations including under-five children for all 8 health centers, 2008-2010. . . . 17 Figure 2: Consultations including under-five children by health center, 2008-2010...... 17 Figure 3: First ANC visit within 16 weeks of Pregnancy for all 8 health Centers, 2008-2010...... 17 Figure 4: First ANC visit within 16 weeks of Pregnancy by health Center, 2008-2010...... 17 Figure 5: ANC Visits (4 times) for all 8 health centers, 2008-2010...... 18 Figure 6: ANC visits (4 times) by Health center, 2008-2010...... 18 Figure 7: Skilled Deliveries excluding Caesarean section for all 8 Health Centers, 2008-2010...... 18 Figure 8: Skilled Deliveries excluding Caesarean section by health center, 2008-2010 ...... 18
Figure 9: Postnatal care visit (1) for all 8 health centers, 2008-2010 ...... 19 Figure 10: Postnatal care visit (1) by health center, 2008-2010...... 19 Figure 11: Beneficiaries of long term Family Planning Methods for all 8 Health Centers, 2008-2010. . . . 19 Figure 12: Beneficiaries of Long term Family Planning Methods by Health Center, 2008-2010...... 19 Figure 13: Cases of Sexually Transmitted Infections treated for all 8 Health Centers, 2008-2010...... 20 Figure 14: Cases of Sexually Transmitted Infections treated by health center, 2008-2010...... 20
4 MARCH 2019 © CORDAID Acronyms EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
ACRONYMS
ANC: Antenatal Care CBO: Community Based Organizations CBHI: Community Based Health Insurance Cordaid: Catholic Organisation for Relief and Development Aid DHIS: District Health Information System FMOH: Federal ministry of Health FDG: Focus Group Discussion FGB: Facility Governance Board HEP: Health Extension Program HMIS: Health Management Information System HSTP: Health Sector Transformation Plan ORHB: Oromia Regional Health Bureau PBF: Performance Based Financing WHOs: Woreda Health Offices ZHO: Zonal Health Office
MARCH 2019 © CORDAID 5 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA Executive Summary
EXECUTIVE SUMMARY
Background equity through remoteness bonus. The evaluation team is Cordaid has been financing and implementing Phase I of the impressed with the extraordinary achievements of the Performance Based Financing (PBF) showcase project in project and found out that the PBF approach was effective Borena, Ethiopia, starting on May 2015 running through June and game changing. Because of the dependency of more 2018 in 9 health facilities (Yabelo hospital and 8 health centers) payment to increased results in health outcomes, the in the four woredas of the Borena zone. The main objectives of financial incentives to the staff, the autonomy given to the PBF showcase project were improving the quantity (cover- health mangers to make decisions on the use of the subsidy age), quality of health services and the health information of the project significantly improved the behaviors of the system through the implementation of a new approach, and health workers and managers motivating them to achieve introducing output/performance-based financing, shifting more health outcomes. away from the conventional input-based financing. The project has also created effective health information Evaluation Purpose system through the establishment of an independent body The general objectives of the external evaluation was to for the verification of reported data, with the consequences document lessons from the implementation of the showcase of health facilities losing money for reporting incorrect data, project and generate evidences / findings for the scale-up of the creating an accurate health information system in the program to other zones and regions in Ethiopia. Specifically, project area. the evaluation assessed: b. Extent of progress on quantity and quality of health 1. the consistency and effectiveness of the PBF approach/six services/Quantitative Findings: elements and 2. to what extent Phase I of the PBF showcase project has Relevance: The evaluation found out that the PBF project and improved the quantity, quality, equity of health services its approach is highly relevant to the implementation of three and health information system. transformation agenda of the health sector transformation plan (HSTP): Transformation in equity and quality of health care since it Evaluation Methodology works in hard to reach area and enhances quality, Information Revolution The external evaluation used a mixed methodology including: by putting in place checks and balance system through its a. Review of existing project documents; sample of the independent verification process ,Caring, respectful and compas- monthly quantitative reports, verification reports, quarterly sionate health workforce through the promotion of autonomy and quality assessment reports, sample of contracts, sample of financial incentive mechanisms. health facility business plan, the Project Implementation Manual, baseline data and internal review reports and other Effectiveness: The project uses 21 indicators for health centers relevant PBF documents; and 18 indicators for the hospital to measure the progress b. Key informant interview (KII) with Cordaid Addis and Yabelo in quantity of preventive and curative services over time. Office, FMOH, Borena Zone Health Office, Oromia Regional The evidences show that at the health center level, significant Health Bureau, Federal Ministry of Health, four Woreda progress has been made in quantity of health services in Health Offices, Yabelo Hospital, seven health centers and all of the eight health centers covered under the program. CBOs; For example, in some of the health centers such as Did - Yabelo c. Focus Group Discussions (FGDs) with direct beneficiaries of and Dikale, health service utilization in outpatient consulta- the target population; and tion including children under five has increased seven-fold d. Quantitative data analysis on PBF quantitative data using compared to the baseline data. El Waye health center has the PBF open data base portal. registered fourfold increase while Yabelo town and Surupha Health centers increased the service in three folds. In other Key Findings health centers such as Haro Wayu, Adi Gulchat and Yabelo a. Consistency and Effectiveness of the PBF Approach/ general hospital, OP consultation has increased two fold while Qualitative Findings: The evaluation has found out that the it increased about two fold in Chari-Rufa health center in three six elements/principles upon which the approach of PBF are years’ time. based are strictly applied. The separation of roles/responsi- bilities, formalizing the same with contract agreements PBF assessment also includes progress in technical quality (accountability) and linking payments with verified results, indicators. 16 quality indicator categories are used for the vetting performance reports through an independent assessment of health centers and 13 quality indicator categories verifying body, and increased autonomy (decision space) to for the assessment of Yabelo hospital. The achievement in the health facility management teams on the use of the average quality score for all indicators varies across health subsidy earned through their performance created dynamic facilities. It has increased more than three fold in Dikale, Haro health system and improved its functionality in the pilot Wayu, Surupha, Adi Gulchat and Chari-Rufa health centers area. In addition, the system has enabled to empower while it doubled in Yabelo Town, Did Yabelo and El Wayu community through the involvement of CBOs and promoted Health Centers. However, outreach service is weak at all health
6 MARCH 2019 © CORDAID Executive Summary EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
relation of the project with the FMOH and ORHB is also found to be inadequate as these key institutions were not involved in the monitoring of the project and provision of feedbacks.
Conclusions and Lessons Learnt The program is in line with the government strategies (Quality and Equity of health services, Information Revolution and Caring, Respectful and Compassionate health workforce) of the transformation agenda of the HSTP. The project has positively affected all of the six health system building blocks. All the six elements upon which the approach of PBF is based are properly applied and found to be consistent, effective, and game changing.
The project has met its objective of increasing the quantity and quality of health services although there is a room to further improve on quality of health services. In addition, the project has significantly improved the quality of data. As stated by the health workers in El Waye Health center, “The project has given life to the health facilities.”
However, the Zonal PBF Steering Committee was only func- tional at the beginning of the program but as it evolves there was no periodic review meeting to learn about the progress of the project and provide feedbacks. In addition, well designed refresher trainings on technical quality assessment, develop- 2017, Petterik Ethiopia, Wiggers 2017, ment of business plan, concepts on PBF are missing. Although
Photo effort has been made by Cordaid, the engagement of the ORHB and FMOH particularly in the planning, implementation and Loko Goba (20) in the health center in Diida. monitoring to sustain the project results and scale-it up to other areas was inadequate. centers. The average score for all quality indicators in Yabelo Recommendations hospital has increased three-fold. While the performance in ▪▪ The project needs to consider development of exit strategy general appearance, safety and infection control and waste plan with the relevant stakeholders which constitutes management fluctuates in all health centers, there is no institutional arrangement for purchaser and fundholder improvement at Yabelo hospital. The reliability of HMIS data that will takeover the responsibility after the completion of has significantly improved as reflected by the low level of the project and create synergy between the activities of the variation between declared and verified data at health facility project with other ongoing initiatives including health level. financing and establishment of CBHI for possible financial sustainability. Efficiency: The project has incurred about 12.98 million Birr ▪▪ Strengthen the relationship with ORHB and involve FMOH as a subsidy for the nine health facilities, four woreda health in the implementation of PBF in the current project areas offices and for the zonal health office during the last three and design of the project in the scale-up areas. The project years. On the other hand it has recorded remarkable achieve- needs to increase the visibility of the program to wider ment in quantity and quality of health services and improved government stakeholders and development partners data reliability. Although, it was not possible to compare the through site visits. In addition more consultation is needed cost of the project and the output achieved against other on the design of the scale-up strategy, review of programs, the achievements gained worth the costs incurred performance and getting feedbacks periodically. for the project because of the output approach.
Sustainability: The program has established mechanisms of sustaining the project results through the direct involvement of local institutions (zonal health office, woreda heath offices, CBOs and health facilities) in the implementation of the project activities. In addition, the quantity and qualitative indicators that are used for verification are aligned with government indicators. The program, however, has limitations in terms of identifying relevant institutions to serve as Fund holder and Purchaser and engage them to create institutional and financial capacity to take over after the completion of the project. In spite of all the efforts made by the Cordaid, the
MARCH 2019 © CORDAID 7 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA 1. Introduction
1. INTRODUCTION
Like that of most underserved areas, health service delivery With the implementation of the PBF project, health facilities both in terms of coverage and quality in Borena zone were develop business plan to identify their priorities, provide below acceptable standard by all measures. According to a health services, and receive subsidies based on their verified baseline study that was performed before the start of the performance-in terms of the provision of a broad set of health Phase-I PBF project, the health facilities in Borena zone were services, quality of care and level of patient satisfaction. The faced with the following challenges: health facilities use the funds to improve the coverage and ▪▪ Shortage of supplies and human resources; quality of health services in their respective facility. This ▪▪ Low quality of services showing an average quality score approach promotes entrepreneurship, leading to better below 30%; management of resources and higher staff motivation, while at ▪▪ a very low average HC utilization of less than 2 OPD per day the same time reinforcing local health authorities in their on average per health center; supervisory role. Communities are empowered by introducing ▪▪ Poor continuity of service delivery. proper patient feedback mechanisms.
To address these challenges, Cordaid started a PBF showcase The Cordaid office in Yabelo performs the PBF-function of project in 2015, covering 9 health facilities in the drylands of Purchasing Agency while the Cordaid Ethiopia Addis Ababa the Borena Zone (Oromia region): one primary hospital (in office executes s the PBF function of Fund Holder. Borena Zonal Yabelo) and 8 health centers. This Phase-I of the PBF showcase Health Office and the woreda health offices play the role of project was designed in close collaboration with the Oromia regulator. Phase I of the showcase project had a running period Regional Health Bureau, the Borena Zonal Health Department starting from the 1st of May 2015 up to the 30th of June 2018. and the targeted 4 woreda health offices. Borena Zone has 16 districts of which Phase-I project covered the following four woredas: El Wayye, Gomole, Yabelo Rural The main objective of the PBF showcase Project is improving and Yabelo town with a total catchment population of 125,918. the quantity (coverage), quality and equity of health services Yabelo Primary Hospital with its catchment population of through health systems strengthening. The project focused on 387,000 is also covered by the project. enhancing functionality of decentralized health systems through the introduction of the six elements of PBF which includes: separation of functions among different actors, introduction of contracting mechanisms, linking payments and results, promoting autonomy, reinforcing community empowerment and ensuring equity.
8 MARCH 2019 © CORDAID 2. Purpose and Scope of Evaluation EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
2. PURPOSE AND SCOPE OF EVALUATION
The general objective of the external evaluation of Phase I was services provided by the contracted health facilities to the to distil lessons from its implementation and disseminate the population? In addition, it assesses to what extent it has findings to stakeholders as an input for the scale-up of the strengthened the WHOs and ZHO in their regulatory role program to other zones and regions in Ethiopia. of supporting the health facilities? In this connection, it also looks into the relevance of the PBF approach to the Specifically the evaluation assessed: existing government policy, other stakeholders and its 1. The consistency and effectiveness of the PBF approach/six viability to sustain the program. elements i.e. separation of functions, autonomy, contracting, linking payment with results, empowering The evaluation period covers phase I of the project that spanned community and equity; and from May 2015 up to June 2018 in 9 health facilities (Yabelo 2. To what extent Phase I of the PBF showcase project has hospital and 8 health centers) in the four woredas of the Borena contributed to improving the health system in Borena and zone. to improving the quantity, quality and equity of health 2017, Petterik Ethiopia, Wiggers 2017, Photo
Health center in Diida.
MARCH 2019 © CORDAID 9 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA 3. Methodology
3. METHODOLOGY
3.1 Evaluation Design stakeholders, Focus Group Discussion (FGD) with community One of the key evaluation questions dealt with was the members who have frequently used the health service, collec- consistency and effectiveness of the PBF approach. The PBF tion and analysis of the quantitative data from the PBF open approach has six elements/principles to implement a project. data base portal created for the project. In addition during the These elements are: separation of functions mainly between field visit in the project area, health facilities covered by the regulator and purchaser and between provider and purchaser; project were visited. The evaluation team, however, has not contracting, autonomy of facilities to utilize the subsidy; visited health facilities that are not covered by the project since linking payment with results after due verification process there was no baseline for control facilities to make comparison involving regulators, purchaser and CBOs; community empow- overtime. The data collection methods are briefly discussed as erment; and ensuring equity. The evaluation team assessed follows: whether these elements are put into practice and their functionality to achieve the targets of the project in terms of Document Review: The evaluation team looked into a sample quantity, quality of health services and improvement in HMIS of the monthly quantitative reports of the health facilities and data recording and reporting. also the verification reports that are used for the calculation of subsidies, quarterly quality assessment reports both by the The second most important aspect of the evaluation question regulators (Zonal Health Department for the case of Yabelo was to assess the achievements of the project in terms of the hospital and Woreda health offices for the case of health progress in coverage of health service delivery and also quality centers), verification reports by CBOs, sample of contracts of health services based on the agreed up on quantity and between the purchaser (Cordaid Yabelo office and health quality indicators by the providers (Yabelo hospital and eight facilities), and a sample of health facility business plan, the health centers). In addition, an assessment was made at the Project Implementation Manual, baseline data and report, level of health offices (zonal health office and four woreda power point presentation on the findings of the internal review health offices) whether they have been undertaking the of the PBF phase 1 project and power point presentation of the required support to the health facilities operating under them zonal health department on their findings of the PBF project. and also prepared their own quarterly plan regularly. The In addition, literatures on the international experience in the evaluation, in addition to the analysis of the effectiveness of implementation of PBF were reviewed. the interventions i.e. achievements in quantity and quality of health service delivery stated above, looked into the relevance Key Informant Interview: Structured questionnaires were of the program to the stakeholders as reflected in providers developed to undertake key informant interviews with the questionnaires, efficiency/cost-effectiveness reflected in the relevant stakeholders. In this regard, the FMOH particularly fund holder questionnaire and also sustainability covered in the Partnership and Coordination Directorate, Oromia the regulators (FMOH, ORHB, Borena ZHD, Fund holder and Regional Health Burea, the Fund Holder (Cordaid Addis Ababa Purchaser) questionnaires. The major evaluation questions, Office, The Purchaser (Cordaid Yabelo Office), Borena Zonal the method of data generation, source of information and data Health Office, four Woreda Health Offices, Yabelo General analysis are presented below, in Table 1. Hospital, seven health centers covered in the PBF project were interviewed. In addition, CBOs in Surupha and Did Yabelo 3.2 Data Collection Methods Health Centers that have been involved in the perceived quality The evaluation team used mixed approach i.e. both quantita- verification were interviewed. These interviews generated tive and qualitative data collection methods which includes qualitative information on the effectiveness of the PBF document review, key informant interview with the relevant approach (contracting, payment, separation of functions,
Table 1: Evaluation Design Matrix EVALUATION ISSUE OF INTEREST DATA COLLECTION SOURCE OF DATA ANALYSIS QUESTION INSTRUMENT INFORMATION METHOD
1. To what extent the PBF To what extent the six elements Key informant interview FMOH, ORHB, WoHO, Comparison between approach is consistent (separation of functions, autonomy, at different levels Fund holder, Purchaser, the current and baseline and effective contracting, linking payment with CBOs, Providers and institutional structure results, empowerment of commu- Document Review and functionality. nity and equity) of PBF approach are in place and are effective to achieve the set targets of the project
2. What achievements Progress in quantity and quality of PBF portal database, Key www.pbfethiopia.org Percentage increase and have been made health services as per the set targets Informant Interview and document review, trends by type of health focusing on quantity, and agreed upon indicators i.e. fund holder, purchaser, services and disaggre- quality of health effectiveness, and also relevance, and regulators gated by quarter services and HMIS data efficiency and sustainability of the reporting project results.
10 MARCH 2019 © CORDAID 3. Methodology EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
autonomy, community empowerment, and equity) implemented In addition, the qualitative data obtained from the key inform- in the zone and informed the strength and weakness of the ant interviews of the stakeholders (regulators, providers, fund approach and provided inputs for recommendation. holder, purchaser and CBOs) were analyzed to assess the consistency and effectiveness of the PBF approach. The team sat Focus Group Discussion: FGDs were conducted at community together each day of the field visit to identify the key findings level in El Waye Health Center, Yabelo General Hospital, Did for each elements of PBF (separation of functions, contracting, Yabelo Health Center and Adi-Gulchat Health center to gather autonomy, payment linked to results, empowerment of information on PBF from the community’s perspective. The community and equity) and took note of this as an input for the discussion focused on the relevance of the PBF, the benefits it overall findings and conclusion of the evaluation report. Some offered to improve health service coverage and quality and any quantitative data obtained from such interviews were also suggestion to improve the program. Each FGD was conducted used to strengthen the findings of the data from the PBF portal constituting more than 10 people that have used the health database. service frequently during the last three years. 3.4 Sampling Methods and Team Deployment FGD at El Waye Health Center The evaluation team covered all of the four woredas, eight health facilities (one primary hospital and seven health centers) where the PBF showcase project was implemented in Borena zone. The names of woredas and health facilities that were covered in the evaluation are stated below in Table 2.
Table 2: List of Woreda Health offices and Health Facilities Visited NAME OF WOREDA NAME OF HEALTH FACILITIES
1. El Waye Adi-Galchet Health Center
El Waye Health Center 2018, Workie Mitiku & Eshete Yilma 2. Gomole Surupha Health Center
Photo 3. Yabelo Rural Dikale Health Center
Did-Yabelo Health Center Quantitative Data Analysis: Selected quantitative data on the Haro Wayu Health Center 21 indicators in the case of health centers and 18 indicators in the case of Yabelo primary/General hospital were used to 4. Yabelo Town Yabelo Health Center analyze the trend during the project period and determine the Yabelo Primary/General Hospital increase over time in terms of health service coverage by type of health services. The PBF open data base portal was used to generate the data and make the necessary analysis to establish The evaluation team was grouped into two teams and the first whether the project has improved quantity and quality of team covered Elwaye and Gomole woreda Offices; and El Waye health service coverage. In addition, technical quality assess- health center, Surupha health center, and Haro Wayu health ments and perceived quality surveys conducted by the Zonal center while the second team covered Yabelo town Woreda health office/Woreda health offices and CBOs respectively were Health office, Adi-Galchet health center, Did-Yabelo health analysed. center and yabelo town health center. Yableo Primary/General Hospital, Dikale Health center, Yabelo rural Woreda Health Health Facility Visit: The evaluation team visited the health office, FGD in Yabelo hospital, Borena Zonal health office, and facilities (Yabelo hospital and seven health centers out of the Yabelo Cordaid office were covered jointly by the evaluation eight health centers) to observe the effects of the PBF project team. and see the actual benefits of the PBF project. Chari-Rufa health center was the only health center which was not visited 3.5 Limitations due to logistic problem. The field visit was carried out from The design of the project has not included control health December 2-14, 2018. facilities that are not covered in the PBF project. As a result, there is no data on control health facilities at the beginning of 3.3 Data Analysis the PBF project. This has limited the possibility to compare the Both quantitative and qualitative data analysis were carried achievements of health facilities with PBF and without PBF out. The quantitative data analysis used trends of the quantita- and to single out the contribution of the project. tive health services, average quality score of the data obtained from the PBF portal data base that contains both quantity and quality of health service delivered by health facilities over the project period. In addition the performance of the zonal and woreda health offices were analysed against the expected targets. The analysis of trends was on quarterly basis and disaggregated by the type of health services delivered.
MARCH 2019 © CORDAID 11 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA 4. Findings
4. FINDINGS
4.1 Qualitative findings (Evaluation Question No.1) Achievements 1. Yabelo Cordaid Office was found to be strictly independent 4.1.1 Separation of functions (between regulator and purchaser; organization from all other actors, (counter) verifying and between provider and purchaser) performance reports while the health providers (health The PBF PIM states that the PBF approach is based on the centers and Yabelo Hospital) provide services as per their principle of clear separation of functions/roles of actors business plans and report on their performance to the wherein the purchaser of the services, the provider of the regulators. The Cordaid office takes the declared reports services, the regulators and fund holders are entrusted with and verifies the data at the health facility levels while the different roles to create dynamic/functional health systems Cordaid Addis office settles the payment. This separation of and avoid conflict of interest. The key element, separation of function and its reinforcement has enabled the purchaser functions, upon which all other elements are built, was to strictly verify the correctness of the performance of expected to be applied in the showcase project. The purchaser health facilities by the purchaser without any conflict of of the services and verifier is Yabelo Cordaid office, health interest. The regulators (Zonal health Department and service providers are the health centers and Yabelo Primary Woreda Health Offices) are also different from the Hospital, the regulators are WHOs and ZHD. CBOs, represent- purchaser working on a specific task of supervisions and ing the community, are also verifiers. In addition, PBF Steering quality assessment. committee provides overall strategic direction and make 2. The Evaluation Team has noted the separation of functions decisions while PBF TWG provide technical support to the effectively working: the provider producing services as per project. The PIM also states the need for an independent the plan, the purchaser verifying data, the regulator external counter verifying team to be contracted by PPA to supervising, and the payment done on time. The system is counter verify the reports approved by the verifier (Yabelo found to be consistent and effectively established with Cordaid Office). checks and balance systems in place.
2017, Petterik Ethiopia, Wiggers 2017, Photo
Karsi Doyo (18) in the health center in Diida.
12 MARCH 2019 © CORDAID 4. Findings EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
Areas for Improvement items, giving no room for flexibility. The offices identify ▪▪ PBF Zonal Steering Committee (SC) which was supposed to their priorities and use the subsidy for those priorities. provide strategic directions, oversight to the performance of the project and conduct promotion of the project was Areas for Improvement functional only at the beginning of the program but later it The implementation of the Health Financing Reform is based stopped to meet and review the performance of the project. . and supported by legal framework (Proclamation and As per the PIM, the SC is the decision-making body and also Regulation), which was endorsed by the Oromiya Regional plays arbitration role in case of disputes. The decisions, Parliament and Cabinet. The legal frameworks gives authority to strategic directions, arbitration, promotion of PBF the public hospitals and health centers to retain and use the fees approaches and activities are missing functions from the they collected and implement also other reform elements PBF process. (formation of facility governance body, outsourcing of non-clini- ▪▪ External Verification; The PIM states that “an external cal services, creation of private wings in hospitals, etc.). These verification team will counter-verify quantity verification laws give authority to facility governance body (Hospital boards done by the PPA , the verification done by the CBOs and the and health center management committee) to manage inter- quality assessment done by the WHO and ZHD. It is an nally generated resources. In principle, these legal frameworks independent quantity and quality performance audit “. provided authority to public hospitals and health centers to This important function was not contracted out and never enter agreements with other entities (for example insurance) to happened. provide services, buy services and mobilize resources. The autonomy promoted through PBF element should take into 4.1.2 Autonomy consideration and create synergy with the health care financing Through the application of autonomy, PBF project makes reform components stated above including facility autonomy. available additional resources to health service providers, where they make decisions on those resources. It is one 4.1.3 Contracting modality of decentralization of authority to the point of service Contracting is one of the key PBF elements where agreements delivery. The application of this key element is based on the between the purchaser and health facilities, purchaser and assumption that through increased decision space, the health regulators and purchaser and CBOs are signed to formalize workers are able to increase health outcomes through changed obligations of the actors stating the conditions associated to behaviors and the right motivation. “Health workers will work deliverables, time, reporting, payment per indicators and better in a setting where they have a better autonomy”. others. The contract agreement clearly outlines the activities and obligations of the parties entering agreement. Achievements 1. All health facilities in the project area have the authority to Achievements use the subsidy based on the collective decision of the 1. Contractual agreements between Cordaid Yabelo office and department represented management committee of the each health facility are signed on bi-annual basis health facilities. The priorities of the health facilities will be containing the quarterly business plan and the agreement. determined by the management committee of the health In the case of health centers, the contractual agreement is facilities and included in their quarterly business plans and signed after the approval by the woreda health office. the subsidy from PBF will be used for these priorities. 70% of 2. The process of contract signing happened regularly without the subsidy is used for those priorities. This is the norm in all any failure. This binding agreement is effective, clearly health centers and Yabelo Hospital. This financial autonomy outlining the responsibilities of each party and reinforcing given to the health facilities has increased the sense of implementations accordingly. It has created an accountable ownership and generated commitment to properly identify system where responsibilities are undertaken as per the the gaps in their facilities and consequently address them to agreement. The evaluation team has witnessed that failure improve health outcomes. The priorities vary between health to fulfill obligations has serious consequences resulting in centers, for some health centers it might be solar energy establishing functional and accountable systems. while for others it is transportation. This makes the application of PBF element, autonomy, relevant and effective. Areas for Improvement 2. The remaining 30% of the subsidy is used for staff financial ▪▪ Because of staff turnover, English language problems and motivation. The 30 % staff incentive is divided among the other factors, there is inadequate capacity to develop staff based on points (45 to 100) assigned for each staff business plans. based on their level of effort and profession. The staff feels ▪▪ Some indicators (quantitative) included in the contract empowered, motivated and task oriented. needs further revision per the DHIS2 as requested from 3. The Evaluation Team has observed that the woreda offices Surupha Health Center. in Borera Zone are not getting enough government ▪▪ Quality indicators are mostly based on infrastructural operational budget to properly implement their activities. standards (For example buildings, availability of lab The woreda offices also get subsidies to increase their services) instead of focusing on operational achievements operational budget. The PBF subsidy which gives full taking into consideration of the reality on ground. For authority to the offices on the allocation of the resources example, a health center which doesn’t have a lab facilities helped them to invest on the priorities of their respective gets nothing (doesn’t get points for PBF payment ) although office. The small amount they get from the government is it does screening on TB patients and send the patient to not only insufficient, but also earmarked to certain line health facilities that have lab service to get the lab services.
MARCH 2019 © CORDAID 13 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA 4. Findings
4.1.4 Linking payment with results limited to) waste management and sanitation, mainte- The core element of the PBF is linking payment with results. nance and repair, fencing, motorbikes, solar energy, Linking payment to results shifts away from input-based laboratory services, office furniture, computers, medical financing to output based financing. This paradigm shift is equipment, family folders, supplies, etc. In all facilities based the assumption that payments based on performance visited the impact of the subsidy is visible and the staff are will reduce the inefficiency of the health delivery systems, energized by the conducive working environment and reduce wastage, motivate health taskforce and enhance facilities made available to them. These inputs have integrated approach. Incentives are correlated to performance; improved the quality of health services which in turn has more performance more incentive. It will make service increased service utilization. providers active actors rather than passive elements. The Zehara Leky, Laboratory Technician at Yabelo Health payment in Borena PBF is linked to outputs of predetermined Center, talks very proudly about the improvement of the quantity, quality indicators and taking into consideration of laboratory services because of the support of Cordaid/PBF. reliability of data. The subsidy from Cordaid was used to procure critical lab equipment by the decision of the health center manage- Achievements ment team. The types of lab tests that could be taken by the 1. The evaluation team has noted that in all health facilities, health center which were only 4 types three years back payments are made after due verification done by Cordaid, have currently increased to 21 types of tests. Accordingly, woreda health office and CBO reports. The payments the number of the patients using the lab has increased to increased as the performance in the indicators improved approximately 120 per month from 15-20 patients per overtime. Because of the dependency of more payment to month three years back. She said that “ I love my job k increased results in health outcomes, the health managers because of the readiness of the laboratory services and and the staff are committed to improve their performance. highly motivated by the top-up I get s from the subsidy”. The head of Haro Wayu Health center, Gezahegn Chofero, in 5. The health center staff considers the top up (30%) as this regard said that “ideas can emanate from different meaningful incentive. In all health centers, we have corners for the improvement of health services. However if observed highly motivated staff supporting and caring for the idea doesn’t have implementation strategy, it will patients and working to get more patients. A staff member remain as an idea. Cordaid through its PBF program has at El Waye said that “The PBF has created their motivation both the idea and the strategy to implement its idea to to undertake their tasks since it has improved the avail- practice. This has enabled to undertake many tasks in the ability of supplies at the health center. The staff are facility.” grateful to the program. 2. Some professionals argue that one of the key preconditions to commence PBF in a given locality /place, is the existence of reliable health data (health management information system). Otherwise, they say, PBF may encourage “cooked data” and may distort the financing system. The routine health information system in Ethiopia is not yet reliable and even worst in Borena Zone. However, the PBF approach instituted through the showcase Project has clearly disproved this argument, by creating reliable health information through the PBF system itself. The primary source of data for PBF is the HMIS from the zones and woreda offices. This declared report goes through the verification process by the independent Cordaid Office. The verification process is based on data in the patient registry books and medical records of health facilities. The process of checks and balance is established through the comparison of declared and verified data with the consequences of losing money for reporting incorrect data. Consistent and independent verification process through the Cordaid Yabelo Office, with zero tolerance, resulted in creating reliable health information systems. Molu Dima, Deputy Head of Borena Zone Health Department, said that “Cordaid, through its PBF approach has helped us achieve notable improvement not only in increasing health services, but also achieve significant improvement in reliability of routine health data/ reporting as compared to non-PBF
facilities. It is because of the verification process”. 2018, Workie Mitiku & Eshete Yilma 3. The accuracy, timeliness and completeness of the health
reports have significantly improved in the pilot facilities. Photo 4. The facilities use the subsidy to fill the critical gaps that affect their performance. These inputs include (but not Zehara Leky, Laboratory Technician at Yabelo Health Center.
14 MARCH 2019 © CORDAID 4. Findings EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
6. Through focus group discussion, the community has confirmed that the patients are extremely happy and appreciate the services in all HCs and Yabelo hospital. They have observed significant improvement during the last three years. 7. All evidences on the ground show that linking payment with result has shifted the frustration of the health professionals working in hard to reach areas, uplifting their motivation for work and significantly improving facility readiness with increased health outcomes. The head of the El Waye Health center has clearly stated that “PBF is the backbone of the health center” as the revenue retained through health care financing is not adequate to run the facility. And staff member of El Waye health center said that “PBF has given life to the activities of the facility.” Molu Tahule, Adii-Galchet Health Center Director also goes far and said that “No Cordaid, no health center” to reflect the importance of the program. The timely and regular payments of subsidy have also created a trusted partner- ship between purchaser and provider. The process is consistent and effective.
Areas for improvement ▪▪ Health extension program (workers), the key part of the primary health care, is not included in the subsidy of the PBF. The evaluation team didn’t get explanation for the exclusion of HEP workers from the PBF.
▪▪ The knowledge of the HCs staff regarding the broader HMIS 2018, Workie Mitiku & Eshete Yilma is found to be very low. They cannot explain the current Photo status of health service coverage in their catchment as expected outside the domain of PBF reporting. ▪▪ The share of 30% is small, specially at the Yabelo Hospital. The picture above depicts the CBO secretariat at Did-Yabelo It is insignificant. The staff incentive at Yabelo Hospital is explaining how she traces patients and fill patient extremely low to have meaningful impact on behaviors. satisfaction format going door to door. She loves her ▪▪ The support from woreda in the preparation of Business Plan assignment. is low; 2. The CBOs are paid flat rate for the reports submitted on ▪▪ The feedback from the CBOs is not available for the HCs; time and additional bonus for number of questionnaires ▪▪ Limited availability of drugs and lab services; and poor completed correctly. sanitation of latrines and shortage of water supply were 3. The feedback from the patient satisfaction survey is reflected in FGD at Yabelo Hospital. summarized and incorporated in the calculation of the quality of health services. 4.1.5 Empowering community 4. The evaluation team discussed with the CBOs (Surupa, and The PIM clearly outlines the role of CBOs, community network- Did –Yabelo Health Centers) and find out that they verify ing, to assess community perception of quality of health the patients and conduct patient satisfaction survey and services. The CBOs are contracted to undertake community provide their findings to Cordaid regularly. verification and patient satisfaction. PPA staff takes sample of patients from registers with information necessary to trace the Areas for improvement patient and provide to the CBO. When the patient is traced by ▪▪ The detail findings of the CBOs on patient’s satisfaction the CBO, the same patient is asked about his/her satisfaction survey are not provided to the health centers. Many health with the health services. The CBOs are paid for such undertak- centers do not engage with the community to improve their ing in a performance based manner. health services; The project uses the CBOs feedback to only provide the subsidy; Achievements ▪▪ CBOs do not coordinate with the health extension workers, 1. The CBOs are actively engaged in the PBF process, patients in undertaking the patient satisfaction survey which could verified, patient satisfaction surveys are conducted every have been easier for them to locate the patients; quarter using 22-25 patients sampled from different ▪▪ Other additional tools which are cost effective patient departments of health centers. The community recognizes satisfaction survey tools such as suggestion boxes in the the role of the CBOs, consequently, provide them feedback health facilities and CBHI forums need to be explored on the health services. The system is well organized ▪▪ Lack of transportation and long distance to undertake the without conflict of interest and compromising survey by the CBOs (as stated in the conversation with CBOs confidentially of the health data of the patients. at Surupha and Did-Yabelo Health Centers.
MARCH 2019 © CORDAID 15 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA 4. Findings
The program also contributes on improving data quality through its strict verification process to effect payment after performance by the health facilities which is also in line with another transformation agenda-Information Revolution- of HSTP. Motivation of health facility staff is also part of the subsidy program to the health facilities and as revealed during the various discussions at health facilities and woreda health offices, this has contributed to improve the working environ- ment of the health workforce which again supports the third 2017, Petterik Ethiopia, Wiggers 2017, transformation agenda of HSTP – Caring, respectful and compas- sionate health workforce. As stated above, the program is relevant Photo to the three transformation agendas of the four transforma- Health center in Surupha. tion agendas of HSTP. The program is also relevant to the community as it helped to decrease home delivery and in- creased facility level delivery through the increased awareness 4.1.6 Equity program using logistics purchased by the program and The showcase Project initiated remoteness bonus to promote purchase of food stuff for mothers waiting for delivery at the equity. Factors for eligibility for remoteness bonus include, health centers. It is also relevant to the health facilities as it distance of the facility, non-availability of communication, improved facility readiness to provide services through access to road, non-availability of public transport and distance provision of subsidies, introducing development of quarterly from the nearest referral health facility. business plan to prioritize their needs and adhere to its implementation. The relevance of the program to the zonal and Achievements woreda health offices is reflected in its support to undertake 1. Without considering the remoteness bonus, the whole area the regular supportive supervision at facility levels as a result selected for interventions: pastoral areas/ hard to reach of the subsidies to these offices based on their performance. and low population density, and low health coverage are appropriate to ensure equity in access to health services. 4.2.2 Effectiveness The project is equity centered and a model for universal This section addresses the achievements made at providers health coverage. (Yabelo hospital and the 8 health centers) and regulators (Borena 2. Remoteness Bonus: The application of additional bonus zonal office and four woreda health offices) during the last three (20%) for HC in remote areas (Adii-Gulchet, Haro Wayo, years. At the providers level, the achievements in quantity of and Chari-Rufa) is found to be rational and relevant. health services, improvement in quality scores and reliability of data will be evaluated while at the regulators level the evaluation 4.2 Quantitative Findings (Evaluation Question No.2) focuses on the extent of the support provided to health facilities This section presents the quantitative findings focusing on the in terms of quarterly supportive supervision, quality assessment, relevance of the program to different stakeholders; effective- development of business plan and also conducting coordination ness of the program in achieving improvement in health meeting on PBF with relevant stakeholders to assess the perfor- service coverage, quality of health service and also decreasing mance of PBF and undertake corrective measures.1 the variation between declared and verified HMIS data; efficiency; and sustainability and scalability of the program. a. Effectiveness- Achievement in Quantity of preventive and curative The findings are based on the PBF database and the field visits health services conducted in the project area. As per the program, 21 quantity indicators in the case of health 4.2.1 Relevance of the Program centers and 18 quantity indicators in the case of hospital are The relevance of the program was assessed from the perspec- identified and used to measure the progress in quantity of tive of government health policy, health facilities/health offices preventive and curative services over time. Accordingly, at the and beneficiaries/community. The evaluation found that the health center level, significant progress has been made in program interventions are relevant to government policy, quantity of health services provided in all of the eight health health facilities and community. The program has contributed centers covered under the program. For example, in some of the to improving per capita health service utilization as it in- health centers such as Did Yabelo and Dikale, health service creased the availability of drugs and supplies thereby increas- utilization in outpatient consultation including children under ing health service utilization. In addition, it has also contrib- five has increased seven fold compared to the baseline number of uted to the objective of equity in health service utilization as it outpatient consultation in these health centers. El Waye health contributed in increasing health service in hard to reach area / center has registered fourfold increase while Yabelo town and pastoral areas as Borena is a pastoral area. Further, the Surupha Health centers increased the service in three folds. In program is contributing to quality improvement as both other health centers such as Haro Wayu, Adi Gulchat and Yabelo technical and perceived quality are criteria used to provide general hospital, OP consultation has increased two fold while it subsidy to health facilities in addition tothe quantity of health increased less than two fold in Chari-Rufa health center. services delivered. The program is therefore in line with one of the transformation agenda of the health sector transformation 1 Please note that the dates stated in the figures on the next page are in Ethiopian plan (HSTP) i.e. Transformation in equity and quality of health care. fiscal calendar, as this was the standard for data registration for the project.
16 MARCH 2019 © CORDAID 4. Findings EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
As it is displayed in Figure 1 and 2 below, there is a continuous increase in outpatient consultation including children under five in all health centers aggregated with the exception of Quarter III of 2009 Ethiopian fiscal year. This increase was reflected in most of the health centers. However, the fluctuation in the case of Did Yabelo and Haro Wayu health centers were observed due to absence of staff during verification day which lead to get nil for that particular month.
Figure 1 : Outpatient consultations including under-five Figure 2: Consultations including under-five children by children for all 8 health centers, 2008-2010 health center, 2008-2010
9000 2500
8000 2000 7000
6000 1500
5000 1000 4000
3000 500 2000
1000 0 Q1 Q2 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 0 2008 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010
Q1 Q2 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Yabelo Town Chari-Rufa Surupha Dikale 2008 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010 Haro Wayu Did Yabelo El Waye Adi Gulchat
In addition to outpatient consultation services, first antenatal care (ANC) visit within 16 weeks of pregnancy has also increased in all health centers aggregated with the exception of the last quarter, Sene 2010 EFY. The decrease in the last quarter is because of a shift from HMIS to DHIS2 which caused change of definition and shortage of registry book. The increase has been manifested mainly in Dikale, El Waye, Adi Gulchat and Haro Wayu. However, it didn’t increase in Chari-Rufa and the increase in Yabelo Town Heath Center has not be been maintained as expected (See below in Figure 3 and 4).
Figure 3: First ANC visit within 16 weeks of Pregnancy for all Figure 4: First ANC visit within 16 weeks of Pregnancy by 8 health Centers, 2008-2010 health Center, 2008-2010
500 180
450 160
400 140
350 120 300 100 250 80 200 60 150 40 100 20 50 0 Q1 Q2 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 0 2008 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010
Q1 Q2 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Yabelo Town Chari-Rufa Surupha Dikale 2008 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010 Haro Wayu Did Yabelo El Waye Adi Gulchat
MARCH 2019 © CORDAID 17 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA 4. Findings
In addition to the first antenatal care visit within 16 weeks of pregnancy, ANC visits (i.e. four times) has also increased continu- ously with the exception of the last quarter of 2010 EFY. The decrease in the last quarter is, like that of first ANC, because of a shift from HMIS to DHIS2 which caused change of definition and shortage of the new registry book. This continuous increment was the result of the increase in all health centers with the exception of Chari-Rufa health center like first ANC visit as depicted in Figure 5 and 6. The increment in Dikale, Adi Gulchat, El Waye and Haro Wayu was remarkable while it was not in Yabelo Town, Surupha and Did Yabelo.
Figure 5: ANC Visits (4 times) for all 8 health centers, Figure 6: ANC visits (4 times) by Health center, 2008-2010 2008-2010
600 200 180 500 160 140 400 120 100 300 80 60 200 40 20 100 0 Q1 Q2 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 0 2008 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010
Q1 Q2 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Yabelo Town Chari-Rufa Surupha Dikale 2008 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010 Haro Wayu Did Yabelo El Waye Adi Gulchat
Skilled deliveries excluding caesarean section has doubled within three years for all health centers aggregated. This achievement was the result of the increment in most of the health centers (Did Yabelo, Dikale, Surupha, El Waye and Adi Gulchat). However, it has decreased in the case of Chari-Rufa and the increase in Yabelo Town is minimal as shown in Figure 7 and 8 below. It has doubled in Yabelo Hospital from 195 to 400 deliveries in a single quarter.
Figure 7: Skilled Deliveries excluding Caesarean section for Figure 8: Skilled Deliveries excluding Caesarean section by all 8 Health Centers, 2008-2010 health center, 2008-2010
500 160
450 140
400 120
350 100 300 80 250 60 200 40 150 20 100 0 50 Q1 Q2 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 0 2008 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010
Q1 Q2 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Yabelo Town Chari-Rufa Surupha Dikale 2008 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010 Haro Wayu Did Yabelo El Waye Adi Gulchat
18 MARCH 2019 © CORDAID 4. Findings EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
Post natal care visits wasn’t practiced in most of the health centers at the beginning of the program. But after a couple of quarters the total visits for the 8 health centers peaked and reached its highest level and stabilized then after as shown in Figure 9 below. On individual health centers, the achievement in post natal care visits, like that of skilled deliveries, was remarkable except Yabelo town and Chari-Rufa Health centers. Actually, it has decreased in Chari-Rufa health center and the increase in Yabelo town is low as shown in Figure 10 below.
Figure 9: Postnatal care visit (1) for all 8 health centers, Figure 10: Postnatal care visit (1) by health center, 2008-2010 2008-2010
500 160
450 140
400 120
350 100 300 80 250 60 200 40 150 20 100 0 50 Q1 Q2 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 0 2008 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010 Yabelo Town Chari-Rufa Surupha Dikale Q1 Q2 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 2008 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010 Haro Wayu Did Yabelo El Waye Adi Gulchat
The achievements in long term family planning methods for all health centers has increased three fold. However on the achieve- ments on individual health centers were mixed. It has increased significantly in El Waye and Dikale Health centers and moderate increase in Adi Gulchat and Chari-Rufa but decreased in Yabelo town, Surupha and Did Yabelo Health Centers (See Figure 11 and 12 below). It has also almost doubled in Yabelo hospital.
Figure 11: Beneficiaries of long term Family Planning Figure 12: Beneficiaries of Long term Family Planning Methods for all 8 Health Centers, 2008-2010 Methods by Health Center, 2008-2010
400 160
140 350 120 300 100 250 80 200 60
150 40
100 20
50 0 Q1 Q2 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 0 2008 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010
Q1 Q2 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Yabelo Town Chari-Rufa Surupha Dikale 2008 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010 Haro Wayu Did Yabelo El Waye Adi Gulchat
MARCH 2019 © CORDAID 19 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA 4. Findings
Cases of sexually transmitted infections treated have significantly increased for all 8 health centers aggregated. This was the result of at least doubled increment in half of the health centers ( Yabelo Town, Dikale, Did Yabelo and Surupha) while it has stabilized in the other remaining four health centers ( Adi Gulchat, Chari-Rufa, El Waye and Haro Wayu) as shown in Figure 13 and 14 below.
Figure 13: Cases of Sexually Transmitted Infections treated Figure 14: Cases of Sexually Transmitted Infections treated for all 8 Health Centers, 2008-2010 by health center, 2008-2010
300 120
250 100
80 200
60 150 40 100 20
50 0 Q1 Q2 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 0 2008 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010
Q1 Q2 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Yabelo Town Chari-Rufa Surupha Dikale 2008 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010 Haro Wayu Did Yabelo El Waye Adi Gulchat
b. Effectiveness-Technical Quality infection control and waste management fluctuate and at the hospital it remained where it began. Generally, there is Assessment of the progress in technical quality indicators are improvement in technical quality score. However, it fluctuates part of the verification process to provide subsidies to the and has not reached the highest possible score. This might health facilities. In this regard, 16 quality indicator-categories necessitate to look into the nature of indicators (as some are identified for the assessment of health centers and 13 indicators focus on complete availability of infrastructure) and quality indicator-categories for the assessment of Yabelo also the support provided by the regulators in improving hospital by the Woreda health offices and the Zonal health quality of health services and the attention given to health office respectively. facilities. The progress in quality score for selected quality indicators for each health facility is annexed (Annex A). The average quality score for all health centers has more than doubled over the last three years. It has increased from its level c. Effectiveness: Reliability of Data of 21.56%in Quarter III of 2008 EFY to 59.38% in Quarter IV of 2010 EFY. However, the achievement in average quality score Improving data quality through decreasing the variation for all indicators varies across health facilities. It has increased between declared data and verified data at health facility level more than three fold in the case of Dikale, Haro Wayu, is one of the objectives of the project. The evaluation found that Surupha, Adi Gulchat and Chari-Rufa health centers while it variation between declared and verified data was high at the doubled in Yabelo Town, Did Yabelo and El Wayu Health beginning of the program but it has reduced overtime and Centers. Of those quality indicators, general appearance and improved the reliability of data at health facilities. For example safety and infection control and waste management has the variation between declared and verified data in Yabelo increased three fold in Yabelo Town Health Center and more town health center was 52% at the beginning of the program than 5 fold Did- Yabelo health center. Similarly, availability of (first quarter of 2008 EFY) but reduced to zero at the last phase drug has increased three fold in Dikale and more than 10 fold of the program (fourth quarter of 2010EFY). However, the in Haro Wayu and Lab service has improved in all heath centers average variation between declared and verified data for the that provide lab services. However, outreach service is poor in whole program period is different among health facilities. For all health centers. example, variation in Adi Gulchat was .3%, .12% in Chari-Rufa, 2.5% in El Waye, and 8% in Yabelo hospital,therefore, within the In the case of Yabelo hospital, the average quality score for all scope of acceptable variation to effect payment. However, the indicators has increased from its level of 42 % to 63% but its rate average variation in Did Yabelo and Haro Wayu health centers of increment is less than what is registered in most of the was significant due to their absence during verification for two health centers. This less rate of increment could be attributed months in the case of Did Yabelo Health center and a month in to the difficult of increasing quality like that of health centers the case of Haro Wayu Health Centers. The improvement in which started at a lower level. Quality indicators such as lab variation between declared and verified data was partly service, availability of medical supplies, maternity, radiological attributed to the strict verification carried out by the verifier- services have improved but general appearance and safety and Purchaser/Cordaid Yabelo Office.
20 MARCH 2019 © CORDAID 4. Findings EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
d. Effectiveness: Regulators and Community Based Organizations (CBOs) of the amount of quarterly subsidy for the health facilities. However, there is a gap in sharing the results of the CBOs Similar to f the providers, regulators (Borena zonal health survey to the health facilities by the purchaser. office and the four woreda health offices) are part of the PBF program. The regulators are responsible to undertake about The support in the development of quarterly business plan by six tasks: Conduct quarterly supportive supervision, undertake the zonal health office to Yabelo hospital and the four woreda technical quality assessment, support development of business health offices to the health centers is being performed well plan at facility level, receive monthly HMIS reports from either through direct involvement in the preparation or review facilities, Prepare their own quarterly action plan and conduct of the draft business plan and provision of feedbacks. However, coordinating meeting on PBF biannually specially by the zonal the support was not provided for four quarters by the Yabelo health office. town woreda health office and for two quarters by each of the three remaining woreda health offices. The evaluation has found that supportive supervisions are periodically undertaken by the zonal health office and the Regarding the preparation of their own quarterly plan and woreda health offices using the checklist such as integrated receiving of monthly HMIS reports from health facilities, both supportive supervision and Ethiopian Health Center Reform the zonal health office and the four woreda health offices have Implementation Guideline. The subsidy from PBF has helped done without any interruptions for the evaluation period. The them to undertake the supportive supervision as subsidies are gap noted is s in conducting PBF steering Committee meeting used to buy motor bicycles and to pay per diem and fuel cost. biannually by the zonal health office. The steering committee However, key informant interview with the woreda health was functional at the beginning of the project but later on it offices and health centers has revealed that the ownership of was not functional and this has limited for sharing experi- quality assessment by the woreda health offices was found to ences and to address emerging issues encountered during be weak. Quality assessments are usually undertaken together implementation. with the purchaser in most of the cases. This low ownership and follow up by the woreda health offices of quality indicators 4.2.3 Efficiency might have contributed for the slower increase of the quality The project has incurred Birr 12.98 million during the last three score compared to the high improvement in quantity of health years for all health facilities (Yabelo hospital and eight health service coverage. CBOs , mostly saving and credit associations centers) and all Regulators (Borena zonal health Office and four also undertook quarterly patient satisfaction survey and woreda health offices) together. Of these 12.98 million birr, Birr provided their feedbacks /scores on the perceived quality to the 3.02 million is for Yabelo hospital, 8.41million birr for purchaser to take into account the feedback in the calculation all health centers and 1.55 million birr for the regulators. 2017, Petterik Ethiopia, Wiggers 2017, Photo
Loko Goba (20) visiting the health center in Diida.
MARCH 2019 © CORDAID 21 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA 4. Findings
The achievement gained with this investment during the last The program, however, has limitations in terms of identifying three years in terms of improving quantity and quality of relevant institutions to serve as Fund holder and Purchaser health services, improvement in data quality, and motivation with the concerned organs (Federal Ministry of Health and of staff and establishment of the culture of paying based on Oromia Regional Health Bureau) and engage them to create performance is remarkable by any standard. Although it was the capacity and readiness to take over such activities after the not possible to compare the costs and benefits of the program completion of the project. In addition, the duration of the with other program, the achievements gained from the project supported by Cordaid as source of the subsidy and also program are worthy of the investments made both in terms of as a fund holder and purchaser is not specified. These and other money and effort. elements of exit strategy were not conceived in the project design will affect the sustainability of the results achieved by 4.2.4 Sustainability and scalability of the project results project and scalability when the current financer and imple- The program has established mechanisms of sustaining the menter of the project-Cordaid departs. project results through the direct involvement of local institu- tions in the implementation of the project activities. The zonal In addition, the engagement of the Oromia Regional Health health office and the four woreda health offices are the ones Bureau (ORHB) is not adequate in the implementation of the that undertake technical qualitative assessment in the Yabelo project. They have been involved in the planning stage of the hospital and the eight health centers respectively. In addition, program but less in the implementation and monitoring of the these regulators also support the health facilities in the project activities. They didn’t receive periodic reports on the development of business plan and also strengthening of the implementation of the project nor had the opportunity to quarterly supportive supervision. Community based organiza- provide feedbacks. There was no any single review meeting tions based in the project area are also responsible to undertake (quarterly or biannually) conducted involving the ORHB and patient satisfaction surveys to measure perceived quality of the the implementers (woreda health office and health facilities) community. The health facilities which are the direct benefi- during the project period. The involvement and linkage with ciaries of the program are passing through all the processes of the FMOH is weaker than the ORHB. They didn’t participate in the PBF which includes development of business plan, con- the design and implementation of the program although they tracting, verification, and payment based on performance. In have participated in the trainings and workshops on the addition alignment of the quantity and qualitative indicators results achieved almost at the end of the first phase (March that are used for verification with government indicators is 2018). These higher level regulators should be part of the also another dimension which avoids parallel system and implementation i.e. monitoring and provision of feedbacks on ensures sustainability of the program. All these activities the program enhance ownership and provide strategic leader- develop capacity and significantly contribute for the sustain- ship to the program, consequently refinement of the design for ability of the project results and also enable to draw lessons to the scale-up. Cordaid has undertaken several efforts to engage scale-up in other areas. these stakeholders but the engagement of these stakeholders to lead and support the program was not adequate.
22 MARCH 2019 © CORDAID 5. Conclusions and Lessons learnt EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
5. CONCLUSIONS AND LESSONS LEARNT
The program is in line with the government policy as it is related to three (Quality and Equity of health services, Information Revolution and Caring, respectful and compas- sionate health workforce) of the four transformation agenda of the HSTP. It is contributing to the improvement of health equity in pastoral area, one of the targets set in HSTP. It is also relevant to the community as it has improved the readiness of the facilities to provide health services through availing the necessary supplies and creation of staff motivation. Health facilities and regulators have directly benefited from the program not only financially but also through the development of capacity to undertake activities and instituting the culture Petterik Ethiopia, Wiggers 2017, of payment based on performance. Photo
The relevance of the project could also be assessed from the Health center in Diida. perspective of the six building blocks of the health system. The project has positively affected all of the six building blocks. It has affected service delivery through improving quantity and Payment based on results through the rigorous verification quality of health services, Health information system through process made the leadership and staff in the facilities, zonal the data verification process, health work force through 30% and woreda health offices more responsible, task oriented, subsidy to health facility staff based on performance, access to motivated and created conducive working environment in the essential medicines through allocation of 70% of the subsidy to health facilities. Because of this the health workers are purchase drug and other supplies, financing through the empowered, motivated and caring. subsidy program to fill the gaps of the health facilities and regulators, and leadership/governance through strengthening The project has met its objective of increasing the quantity and development of business plan i.e. prioritization of activities and quality of health services although there is a room to further resources. improve on quality of health services. In addition, the project has significantly improved the quality of data, one of its The PBF showcase in Borena has demonstrated with evidences objectives, as it is demonstrated in low variation between how the application of the principles of PBF can improve the declared and verified data from health facilities through its performance of health systems and achieve better health strict verification process (zero tolerance of inaccurate data outcomes. The elements upon which the approach of PBF is reporting). This system enabled to create accountable perfor- based i.e. separation of functions, autonomy, contracting, mance reporting system and is a lesson to be shared and linking payment with results, community empowerment and scaled-up in the expansion of the program. As stated by the equity are properly applied. The separation of roles/responsibili- health workers in El Waye Health center, “The project has given life ties, formalizing the same with contract agreements (account- to the health facilities.” ability) and linking payments with verified results, vetting performance reports through an independent verifying body, However, of the structures created in the PBF, the Zonal and increased autonomy (decision space) to the health facility Steering Committee was not functional and there was no management teams on the use of the subsidy earned through periodic review meeting to learn about the progress of the their performance created dynamic health system and im- project and provide feedbacks. There is no forum to resolve proved its functionality in the pilot area. The evaluation team implementation challenges and celebrate successes. is impressed with unexpected achievements of the project and In addition, well designed refresher trainings on technical found out that the PBF approach was relevant, consistent, quality assessment, development of business plan, concepts effective, and game changing. on PBF are missing.
In terms of creating reliable routine health information Although a number of efforts has been made by the project, the system, the project has created a commendable model. While engagement and linkage with the ORHB and FMOH particu- the primary source of data for the PBF is the HMIS from the larly in the planning, implementation and monitoring to zones and woreda offices, these reports should go through a sustain the project results and scale-it up to other areas was verification process by the independent body, Cordaid Office, inadequate. The ORHB is involved in the design of the project using the patient registry books of the health facilities. The and got capacity building trainings on PBF. However, the establishment of an independent body, for the verification of follow-up in the implementation of the project through reported data with the consequences of losing money for periodic site visits of the project area, monitoring of the reporting incorrect data, resulted in creating reliable health progress made and provision of feedbacks by ORHB is weak. information systems in the project area. This situation is severe in the case of FMOH as it was not
MARCH 2019 © CORDAID 23 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA 5. Conclusions and Lessons learnt 2017, Petterik Ethiopia, Wiggers 2017, Photo
Patient in the hospital in Yabelo.
involved in the design of the project and didn’t conduct any site visits of the project area and follow-up of the implementation with exception of being involved in training and participation in the workshop at Bishoftu to learn about the project outcomes at the end of the first phase in March 2018.
In addition, the project has to develop and implement exit strategy stipulating the process to identify institutions to take over the purchaser and fund holder responsibility, creating their capacity and also the period when the project will be completed and government takes over fully the responsibility. Such exit strategy should also cover the possible areas of finance for the subsidy including the use of the retained revenue at the facilities level particularly to cover the subsidy allocated to the staff.
24 MARCH 2019 © CORDAID 6. Recommendations EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
6. RECOMMENDATIONS
The evaluation findings and conclusions dictates that the ▪▪ Review the relevance of the quality indicators and adjust project needs to undertake specific actions to susutain the based on the lessons learnt so far as the progress in quality gains achieved so far and scale it up to other areas in Oromia fluctuated during the last three years; region and other parts of Ethiopia. These actions, among ▪▪ Provide detailed feedbacks from the CBOs to the respective others, include: health facilities for their action; ▪▪ Develop exit strategy which constitutes institutional ▪▪ Provide periodic capacity building on business development arrangement for purchaser and fundholder that will because of high turnover of staff at woreda health office and takeover the responsibility after the completion of the health center levels; project with the relevant stakeholders at regional and ▪▪ Ensure the ownership of the woreda health office in federal levels; undertaking quality assessment at health center level; ▪▪ Strengthen the relationship with ORHB and involve FMOH ▪▪ Include the Health extension workers in the staff subsidy in the implementation of PBF in the current project areas program to expedite outreach program and increase and design of the project in the scale-up areas; coverage of health services; ▪▪ Continue working in the most hard to reach and neglected ▪▪ Conduct quarterly review meeting involving stakeholders low land areas and health facilities given the current low (zonal health office, woreda health offices, health facilities, health outcomes in the lowland areas like Borena , the Cordaid, and CBOs) to address emerging issues; expansion should preferably be to low land areas, creating ▪▪ Ensure synergy between the activities of the project with “Center of Excellency” in Borena from which the government, other ongoing initiatives including health financing and development partners, and others can learn and expand to establishment of CBHI; other low land areas; ▪▪ Increase the visibility of the program to wider government ▪▪ Review and further align the quantitative and qualitative stakeholders and development partners by presenting the indicators with the DHIS 2 and other government reforms design in the scale-up area and getting feedbacks and such as Ethiopian Health Center Reform Implementation implementation results periodically. Guidelines;
MARCH 2019 © CORDAID 25 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA annexes
ANNEX A: FIGURES ON QUALITY SCORES BY HEALTH FACILITY
Technical Quality Score, El Waye HC, megabit 2008 - Technical Quality Score, Chari-Rufa HC, megabit 2008 - Sene 2010 Sene 2010
100 90 90 80 80 70 70 60 60 50 50 40 40 30 30 20 20 10 10 0 0 Megabit Sene Meskrem Tehesase Megabit Sene Meskerem Tahesase Megabit Sene Megabit Sene Meskrem Tehesase Megabit Sene Meskerem Tahesase Megabit Sene 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010
General Appearance and Safety Average score General Appearance and Safety Average score Laboratory Services Infection control and Laboratory Services Infection control and Logistics, Medicines and Supplies waste management Logistics, Medicines and Supplies waste management
Technical Quality Score, Haro Wayu HC, megabit 2008 - Technical Quality Score, Surupha HC, megabit 2008 - Sene 2010 Sene 2010
100 100 90 90 80 80 70 70 60 60 50 50 40 40 30 30 20 20 10 10 0 0 Megabit Sene Meskrem Tehesase Megabit Sene Meskerem Tahesase Megabit Sene Megabit Sene Meskrem Tehesase Megabit Sene Meskerem Tahesase Megabit Sene 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010
General Appearance and Safety Average score General Appearance and Safety Average score Laboratory Services Infection control and Laboratory Services Infection control and Logistics, Medicines and Supplies waste management Logistics, Medicines and Supplies waste management
26 MARCH 2019 © CORDAID annexes EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
Technical Quality Score, Dikale H, megabit 2008 - Technical Quality Score, Adi Gulchat HC, megabit 2008 - Sene 2010 Sene 2010
100 100 90 90 80 80 70 70 60 60 50 50 40 40 30 30 20 20 10 10 0 0 Megabit Sene Meskrem Tehesase Megabit Sene Meskerem Tahesase Megabit Sene Megabit Sene Meskrem Tehesase Megabit Sene Meskerem Tahesase Megabit Sene 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010
General Appearance and Safety Average score General Appearance and Safety Average score Laboratory Services Infection control and Laboratory Services Infection control and Logistics, Medicines and Supplies waste management Logistics, Medicines and Supplies waste management
Technical Quality Score, Did Yabelo HC, megabit 2008 - Technical Quality Score, Yabelo Town HC, megabit 2008 - Sene 2010 Sene 2010
100 100 90 90 80 80 70 70 60 60 50 50 40 40 30 30 20 20 10 10 0 0 Megabit Sene Meskrem Tehesase Megabit Sene Meskerem Tahesase Megabit Sene Megabit Sene Meskrem Tehesase Megabit Sene Meskerem Tahesase Megabit Sene 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010
General Appearance and Safety Average score General Appearance and Safety Average score Laboratory Services Infection control and Laboratory Services Infection control and Logistics, Medicines and Supplies waste management Logistics, Medicines and Supplies waste management
Technical Quality Score, Yabelo hospital, megabit 2008 - Sene 2010 100 90 80 70 60 50 40 30 20 10 0 Megabit Sene Meskrem Tehesase Megabit Sene Meskerem Tahesase Megabit Sene 2008 2008 2009 2009 2009 2009 2010 2010 2010 2010
General Appearance and Safety Average score Laboratory Services Infection control and Logistics, Medicines and Supplies waste management
MARCH 2019 © CORDAID 27 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA annexes
ANNEX B: TOR FOR AN EXTERNAL EVALUATION OF PHASE I PBF SHOWCASE PROJECT IN BORENA ZONE
1. Introduction to the PBF showcase project During the first half of 2018 Phase-II of the showcase project After a preparation and designing phase, Cordaid in 2015 was designed. Phase II will run for 18 months and has started started a PBF showcase project, covering 9 health facilities in on the 1st of July 2018. During a consultative review process the drylands of the Borena Zone (Oromia region): one primary held in April 2018 in which representatives of the Borena Zonal hospital (in Yabelo) and 8 health centers. This Phase-I of the PBF Health Department participated, it was decided to make the showcase project was designed in close collaboration with the following changes for the PBF project for Phase-II: Oromia Regional Health Bureau, the Borena Zonal Health ▪▪ Upscaling from 9 facilities to a maximum of 25 facilities Department and the 4 woreda health offices which it targeted. (total catchment population of 488,556 (or 793,365, if we take It had a running period starting from the 1st of May 2015 up to the total catchment population of the two hospitals the 30th of June 2018. included in Phase II). ▪▪ Adapt and sharpen the quantity indicator package for HCs In total the Borena zone has 16 districts of which the project and DH level. Phase-I covered the following four: El Waye, Gomole, Yabelo ▪▪ Slightly modify the prices for the quantity indicators used Rural and Yabelo town with a total catchment population of for the hospitals and health centers. 125,918. According to a baseline study that was performed ▪▪ Finetune the Equity bonus so that more differentiation is before the start of the Phase-I project the health facilities were possible compensating even the remotest Health Facilities. facing the following challenges: ▪▪ Integrate the FMOH’s Community Score Card question ▪▪ Poor continuity of service delivery (indicators) in the Patient Satisfaction surveys ▪▪ Shortage of supplies and human resources ▪▪ Include a CBHI enrollment indicator in the Score Card used ▪▪ Low quality of services showing an average quality score for the Woreda Health Offices. below 30% ▪▪ Include an indicator focusing on organizing quarterly PBF ▪▪ A very low average HC utilization of less than 2 OPD per day review meetings at Zonal level. on average per health center ▪▪ Rewrite the verification guidelines for the HCs and Hospital indicators so that they are aligned to the newly adapted Under the PBF approach, a direct link is established between Ethiopian HMIS, fitting it to DHIS-2. payments and performance. The health facilities receive ▪▪ Include the newly targeted Woreda Health Offices and the subsidies based on their verified performance – in terms of the 16 Health facilities in the Open database system. utilization of a broad set of services, quality of care and patient Both Phase I and Phase II of the PBF project are financed with satisfaction. Contrary to traditional health financing, they the Memisa funds of Cordaid. receive funds directly in their own bank accounts. The health facilities develop business plans, and identify their priorities, The long term vision: Institutionalisation of PBF within and use the funds to improve the quality of their own facility. Ethiopian Health policies: This approach promotes entrepreneurship, leading to better The PBF project from the start has been designed as a showcase management of resources and higher staff motivation, while at project, meaning that its intention is to demonstrate (within the same time reinforcing local health authorities in their the Ethiopian context), what a PBF Health Systems supervisory role. Communities are engaged by introducing Strengthening approach could achieve in terms of improving proper patient feedback mechanisms. For a more detailed the uptake and quality of primary health care services. This description of the set-up, design and tools of the PBF Phase-I with the long term objective (and of course in close consulta- showcase project is referred to the Project Implementation tion with the (F)MOH) to see if and how PBF could become part Manual (PIM) of Phase I. of the Ethiopian health policy and institutional set up, such as inclusion within the HSTP. The Cordaid office in Yabelo performs the PBF-function of Purchasing Agency (contracting, verification and support), while the Cordaid Ethiopia Addis Ababa office holds the PBF function of Fund Holder (FHA: making the actual payments to the contracted parties). The PBF experts from Cordaid HQ provide the necessary technical PBF support. The different levels of the MoH, perform the role of regulator. For the project an online open database system (following the Ethiopian fiscal year) has been designed and is used to collect declare and verified performance data, to monitor the project, to provide data for analysis of performance at different levels and to generate the quarterly invoices per facility (www.pbfethiopia.org).
28 MARCH 2019 © CORDAID annexes EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
2. Objectives of the external evaluation 3. Building block 3: Separation of functions, the set-up used in the showcase project The external evaluation of the PBF showcase project Phase I ▪▪ Was the separation of functions chosen for the design of this The external evaluation of Phase I (from May 2015 up to June project sufficient and effective: if so: why? 2018) will be performed for learning and dissemination ▪▪ How did the separation of functions contribute to improving purposes. Also in light of gathering evidence on PBF that could the quality of services at each level of care? contribute to Health systems strengthening, further PBF ▪▪ Is there room for improvement in relation to the separation scale-up and to Ethiopian institutionalisation of PBF). The of functions? evaluation is also performed as an accountability measure. The evaluation will partly be a “generic” evaluation and partly 4. Building block 4: Autonomy of the contracted health facilities (as a specifically be focused on the type of project: PBF. condition for PBF to function) It will focus on the key overall question: if (and to what extent) ▪▪ In what way has receiving PBF incentives on the bank Phase I of the PBF showcase project has contributed to improving the accounts of the Health Facilities, coaching the facilities to health system in Borena and to improving the quantity, quality and equity formulate their business plans and prioritising where/how of health services provided by the contracted health facilities to the to invest the earned quarterly PBF subsidies helped them to population? improve the quality of care and to attract more patients? ▪▪ Note that the evaluation is expected to provide information that is credible, reliable and useful and will make recommendations in relation 5. Building block 5: Community empowerment to the process of achieving sustainability and institutionalisation of PBF ▪▪ Did the perception, attitude and behaviour of the as an (Ethiopian) Health Systems approach. community change in relation to the health facilities in ▪▪ Note that in the Borena zone other Health related policies and their catchment area? interventions took place during the period of implementation of Phase I ▪▪ If so, what were these changes? of the PBF project: it will be challenging to look at “attribution”. Therefore, the evaluation will focus on the contribution“ ” of PBF. 6. Building block: Equity (equal access to services) ▪▪ Has the PBF project been able to improve the quantity and Specific tasks to be undertaken: quality of health services in the most remote contracted Health Centers? a. Consistency and Effectiveness of the PBF Approach meant to ▪▪ Has the project been able to increase the accessibility of the analyse how the project performed on each of the six main health services for the most vulnerable and poorest Borena? building blocks of the PBF Health Systems Strengthening ▪▪ Is there room for improvement and are there recommen approach. Key question is to determine the relevance and dations to increase equity in access to healthcare? effects for each of PBF’s six main building blocks: b. Health Outcomes and Effects on the Health System: assessing the 1. Building block 1: Linking payment and results effects of PBF during phase I per level of care (contracted parties ▪▪ What have been the effects of linking quarterly PBF only) payments to the results achieved by the contracted health 1. Effects at Health Center level: facilities? ▪▪ What have been the effects in terms of quantity and quality ▪▪ What have been the effects of linking quarterly PBF of services at HC level? payments to the results achieved by the contracted WHOs ▪▪ What have been other (positive and negative) effects of the and the ZHD? PBF project at this level of care? ▪▪ Has the PBF project contributed to improved reliability of 2. Yabelo Hospital level: data registration and HMIS reporting? ▪▪ What have been the effects in terms of quantity and quality ▪▪ Has the PBF project been cost effective, i.e. is there a good of services at Yabelo hospital level? balance between health outcomes achieved and total costs ▪▪ What have been other (positive and negative) effects of the incurred? PBF project at this level of care? ▪▪ Was the project implemented in the most efficient way? 3. Woreda Health Office level: ▪▪ What have been the effects (positive and negative) of the PBF 2. Building block 2: The Contracting approach and cycle: from business project at this level of care? plan to verification 4. Borena Zonal Health Department: ▪▪ In what way has contracting the health facilities helped ▪▪ What have been the effects (positive and negative) of the PBF them? project at this level of care? ▪▪ Have the business plans helped the facilities to improve the quality of the services provided by them? c. Governance and Policy Environment ▪▪ In what way have the monthly quantity verifications and ▪▪ Did the PBF project during Phase I contribute to: the quarterly quality verifications helped the facilities to - The improvement of Quantity and Quality of Health increase the quantity and quality of their services? services provided by the contracted facilities? ▪▪ At the level of Woreda Health Office and Zonal health - Improvement of the accuracy and quality of HMIS data (of Department: what effect did contracting them have on their the contracted facilities) performance? - Stronger and improve performance of the (four contracted) Woreda Health Offices and Zonal Health Department
MARCH 2019 © CORDAID 29 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA annexes
- Change the attitudes and ways of working (toward each 3. Methodology other and toward the patients and community) of the staff in the contracted Health facilities. Introduction ▪▪ Based on an analysis of the policy environment, which were The methodology for the external evaluation will consist of the compounding factors (both within the Woredas, Borena 1. Review of project documents, zone and Oromia region and the MOH at Federal level) that 2. Interviews with for the PBF project relevant actors and key may have impacted on the project’s outcomes. stakeholders, ▪▪ To what extent has the institutionalisation strategy 3. Data analysis among other using the PBF open data base contributed to the sustainability of the results achieved portal, and during these first three years of implementation? What 4. Field visits to Health facilities under PBF Phase I. would be advise in relation to further realising Ethiopian 5. For comparation: possibly a number of Borena Health institutionalisation and sustainability at the different levels Facilities Fs not under PBF Phase I could also be visited. of the MOH? ▪▪ Is the PBF project sufficiently tailor made to the Ethiopian Team requirements and composition context and its health policies: recommendations for The team will consist of: improvement ▪▪ Preferably a Health Economist expert with extensive experience in PBF as the Lead Consultant. d. The quality of the verification processes ▪▪ A Public Health expert as a team member. ▪▪ What has been the quality of the different types of ▪▪ If possible and desirable the team could be completed with 2 Verification (quantity by the Verifier, Cordaid Borena, persons as observers to facilitate the exercise: one staff quality verification by the WHOs and the Zonal Health member from the Borane Zonal Health Department and one Department, Patient satisfaction surveys via the CBOs)? from the Oromia Regional Health Burau. ▪▪ How well have these actors done the task of verification? Is there room for improvement? Required skill and experience of the Lead Consultant: ▪▪ Master degree in relevant field, preferably health economics e. Key Success Factors, Lessons Learned and Recommendations or health policy. ▪▪ Based on a thorough data analysis of the performance of the ▪▪ Minimum 5 years of project evaluation experience in project, both in terms of quantity and quality outcomes and relation to Health financing and/or Health Systems its unique institutional setting: is it possible to identify the Strengthening, preferably also experience in evaluating PBF key factors that led to the apparent success of the project? schemes. ▪▪ What have been the main Lessons Learned of the Phase-I of ▪▪ Extensive knowledge of and experience with the Ethiopian the PBF showcase project? Health system and its policies at different levels ▪▪ Recommendations of the evaluation team: how to further ▪▪ Minimum of 5 years of experience in designing and improve the PBF project and how to proceed further (the implementing quantitative impact evaluations latter also in relation to if and how PBF could be further ▪▪ Relevant experience in measurement of health outcomes scaled up in Ethiopia in the future) through household surveys ▪▪ Preferably relevant experience designing and coordinating More general: field work for household surveys and health facility surveys ▪▪ What would be the minimum conditions required for the ▪▪ Relevant experience in analyzing quantitative and implementation of PBF in Ethiopia? qualitative data ▪▪ Recommendations in relation to scalability of the design and implementation of PBF as a government system (f.e. as a Documents needed model for channelling government resources to facilities). Review of documents for the Evaluation: Also, best practises and lessons learned from other African The following documents will be used: countries can feed into these recommendations). a. Monthly declared reports on contracted indicators submitted by the Health Facilities on quantitative data and the verified data that are used for calculations for invoices/ subsidies b. Quarterly quality assessments reports (patients satisfaction reports & quality assessments done by the WHOs and the ZHD) c. Contracts with all the contracted parties under the PBF project d. A sample of the Health Facility business plans e. The Project Implementation Manuel (PIM) for Phase I f. The open data base portal g. Project proposal and budget h. Baseline data information Phase I (also the baseline data for the new Borena facilities under Phase II, recently gathered could be interesting)
30 MARCH 2019 © CORDAID annexes EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
i. The PowerPoint with the findings of the internal review of the PBF project Phase I j. The PowerPoint of the Zonal health Department with their findings on the PBF project presented during a workshop in Debrezeit on PBF for the FMOH
Of course the evaluators can make use of other sets of docu- ments if needed and as found necessary by the them.
Interviews Stakeholders will be interviewed individually or in groups. These may include (but is not limited to) staff of the WHOs, the ZHD, the ORHB and the FMOH, contracted Health Facility staff, members of the community, members of the contracted CBOs, staff of the Cordaid Borena verification team and of Cordaid Addis Ababa office.
4. Scope of the mission
Scope of the work: The assignment includes the following tasks: ▪▪ To develop a simple yet effective Evaluation Design ▪▪ To develop the necessary Questionnaires ▪▪ To select and arrange the interview with key actors to interview ▪▪ To prepare, plan and oversee the evaluation and implement the activities ▪▪ To arrange and manage the needed Data Collection, Documentation, Analysis and Storage ▪▪ To formulate the draft report and, once feedback has been received, prepare the final report and send to Cordaid for approval
5. Expected deliverables
Deliverables The external evaluation will result in the following deliverables: ▪▪ An evaluation report of maximum 25 pages (without annexes and introductory paragraphs) which answers key questions in respect to aforementioned objectives. ▪▪ A validation/feedback presentation/workshop for the partners at the end of the evaluation (Cordaid Ethiopia will organise this workshop. Partners to be present: WHOs, the Borena Zonal Health Department, the ORHB, Cordaid Borena and Cordaid Ethiopia and a selection of representatives of contracted health facilities and CBOs. If possible representatives from the FMOH) ▪▪ Debriefing on the findings and the reports via skype to Cordaid HQ health care staff.
MARCH 2019 © CORDAID 31 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA annexes
ANNEX C : QUESTIONNAIRE FOR THE EVALUATION
Questionnaire for Health Centers 19. Do you have the autonomy to use the subsidy? What are 1. What are the major constraints to improve the quantity the procedures to utilize the fund? Is it different from the and quality of health services in your facility? procedures of using retained revenue from user fees if you 2. Do you have demand–side problems (financial barriers, low have implemented revenue retention and utilization health seeking behavior, etc.) to utilize health services in reform? the area? 20. Do you have the necessary capacity to implement PBF 3. Could you please explain us about PBF? effectively? Development of business plan, contracting, 4. How much money did you get from PBF payments per data reporting, use of the subsidy?) If not what should be annum during the last three years? What was the share of done? the PBF subsidy from the operational budget and total 21. Is there any accountability mechanism on the use of the budget of the facility? subsidies? If there is, could you please explain us? 5. Did the average coverage of curative and preventive 22. How far the feedbacks from CBOs have contributed to services (21 performance indicators) increase from 14% to adjust the health service delivery on your facility? 53% during the pilot period (May 2015-June 2018)? 23. Did the PBF enable you to serve the most vulnerable and 6. Is there an improvement on the average quality score from poorest population in the area? If so, how? 46% to 70%? 24. What are the strengths /key success factors in the 7. Do you think PBF has contributed to the improvement of implementation of PBF project? quantity and quality of health services? How? 25. What are the weakness of the PBF implementation in 8. Do you think the business plan, contractual agreement, terms of contracting, reporting, verification and payment? verification and payment linked to results contribute to 26. Could you please share us your opinion to improve the the increased quantity and quality of health services? weakness of the PBF implementation? What were the mechanisms? 9. How is the progress of improvement of the variation Questionnaire for Yabelo Primary Hospital between registered and verified HMIS data overtime? Is 1. What are the major constraints to improve the quantity the variation less than 10%? and quality of health services in you facility? 10. What improvements should be made on the verification 2. Do you have demand–side problems (financial barriers, low process? health seeking behavior, etc.) to utilize health services in 11. What was the trend on the number of referred and the area? correctly registered deliveries from the health center to 3. Could you please explain us about PBF? the Yabelo primary hospital? 4. How much money did you get from PBF payments per 12. Does the Woreda health office undertake periodic annum during the last three years? What was the share of supportive supervision including support on HMIS? What the PBF subsidy from the operational budget and total was their major support? budget of the facility? 13. How frequent does the woreda health office conduct the 5. Did the average coverage of curative and preventive quality assessment? Is the assessment constructive? services (18 performance indicators) increase from 49% to 14. Have you established PBF team in the facility? How 88% during the pilot period (May 2015-June 2018)? effective are they in supporting the facility to implement 6. Do you think PBF has contributed to the improvement of PBF? quantity and quality of health services? How? 15. Could you please explain us on the utilization of subsidies 7. Do you think the business plan, contractual agreement, from the PBF project? What are the allocation criterions to verification and payment linked to results contribute to use the subsidy? Is there any organ that supports you to the increased quantity and quality of health services? prioritize the utilization of the fund? What were the mechanisms? 16. Does the subsidy have any effect on the attraction and 8. Is there an improvement on the average quality score retention of human resources, attitude of the staff to checklist from 84% to 95%? patients, provision of supplies and medicines and any 9. How is the progress of improvement of the variation other effects? Could you please explain? between registered and verified HMIS data overtime? Is 17. How do you share the top-ups (30% of the subsidy) among the variation less than 10%? the staff? 10. Does the Zonal Health Department undertake periodic 18. Have you implemented health care financing reform supportive supervision including support on HMIS? specifically revenue retention and utilization reform 11. How frequent does the Zonal Health Department conduct component? the quality assessment? Was it constructive and lead to quality improvement?
32 MARCH 2019 © CORDAID annexes EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
12. What improvement should be made to the verification Questionnaire for Oromia Regional Health Bureau process? 1. Did you approve the PBF implementation manual? 13. Have you established PBF team in the facility? How 2. Do you receive performance report of the PBF from Cordaid effective are they in supporting the facility? office and provide suggestions on areas for improvement? 14. Could you please explain us on the utilization of subsidies 3. What is your impression on the performance of the PBF from the PBF project? What are the allocation criterions to project? use the subsidy? Is there any organ that supports you to 4. Do you see the benefit of expanding the PBF in other prioritize the utilization of the fund? zones? If yes, could you please explain? 15. Does the subsidy have any effect on the attraction and 5. If you think expansion of the project is beneficial, what retention of human resources, attitude of the staff to kind of adjustment should be made? patients, provision of supplies and medicines and any 6. What would be the sustainable institutional arrangement other effects? Could you please explain? for PBF? 16. How do you share the top-ups (30% of the subsidy) among 7. What would be the sustainable sources of finance for PBF? staff? 8. What are the potential areas/localities for scaling up PBF? 17. Have you implemented health care financing reform specifically revenue retention and utilization reform Questionnaire for Borena Zonal Health Department component? 1. How do you evaluate the role of PBF incentives to improve 18. Do you have the autonomy to use the subsidy? What are reliability of HMIS data? the procedures to utilize the fund? Is it different from the ▪▪ Do you think that your zone has achieved the target, in procedures of using retained revenue from user fees if you terms of the difference b/n declared and verified data? have implemented revenue retention and utilization ▪▪ What is working (positive) and not working in the reform? verification process? 19. Do you have the necessary capacity to implement PBF 2. Any suggestion for improving the verification process? effectively? Development of business plan, contracting, 3. How is PBF health indicators and reporting mechanisms data reporting, use of the subsidy?) If not what should be align with FMOH HMIS indicators and process? done? 4. How do you compare the performance of PBF woredas with 20. Is there any accountability mechanism on the use of the other non-PBF woredas in Borena Zone in terms of subsidies? If there is, could you please explain us? reliability of reports? 21. How far the feedbacks from CBOs have contributed for the 5. Do you regularly conduct quarterly quality assessment adjustment of the health service delivery? (QQA) 22. Did the PBF enable you to serve the most vulnerable and 6. How do you describe QQA practices before PBF and after? poorest population in the area? If so, how? 7. What are your roles to address the quality gaps? 23. What are the strengths /key success factors in the 8. How do you describe the role of PBF in quality improvement? implementation of PBF project? Does Quarterly quality assessment contribute to the 24. What are the weakness of the PBF implementation in improvement of quality of health services? How? terms of contracting, reporting, verification and payment? 9. What is working and not working in the QQA process? 25. Could you please share us your opinion to improve the What imporvments should be made? weakness of the PBF implementation? 10. Do you provide supportive supervision to Yabelo hospital, please describe the process Questionnaire for the Federal Ministry of Health 11. Do you support Yabelo hospital in preparing operational 1. Are you aware of the PBF project in Borena implemented plans? by Cordaid? 12. What was the practice before PBF? 2. What was your role in the design and implementation of 13. What is working in your support to the hospital? the PBF project? 14. What are the weaknesses in your support to the hospital? 3. Do you get progress reports on the PBF project? 15. Your suggestions for improvement in terms of supporting 4. What are the areas for improvement? the hospital. 5. What would be the sustainable institutional arrangement 16. How do you evaluate the activities and function of the PBF for PBF? Steering Committee? 6. What would be the sustainable sources of finance for PBF? 17. Do you have suggestions to improve the functionality of 7. Do you see the potential for expansion of PBF project in the Steering Committee? other regions? 18. Do you have PBF Technical Working Group, Please describe. 8. What are the potential areas/localities for scaling up PBF? 19. How do you evaluate the role of PBF approach to improve the health status of your Zone? 20. What are the positive aspects of PBF approach and its implementation? what are the weaknesses? Any suggestion for Improvement 21. What are the key lessons from the PBF experiences. 22. How can we ensure sustaining the PBF successes while expanding to other woredas? 23. Do you report on the performance of the PBF to ORHB and get feedback?
MARCH 2019 © CORDAID 33 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA annexes
Questionnaire Tools for Woreda Health Office Questionnaire for the Fund holder (Cordaid, Addis 1. How do you evaluate the role of PBF incentives to improve Ababa Office) reliability of HMIS data? 1. As a fund holder, how much did you pay to zonal health 2. Do you think that your woreda has achieved the target, in office, woreda health offices and health facilities (Yabelo terms of the difference b/n declared and verified data? Primary hospital) and for the 8 health centers up to the ▪▪ What is working (positive) in the verification process? end of the project ( June 30, 2018)? ▪▪ What is not working in the verification process? 2. What is the achievement in utilization of the project fund ▪▪ Any suggestion for improving the verification process ? compared to the plan? 3. How do you evaluate improvement in the attitude and 3. What is your relation with the FMOH and RHB? Do you motivation of health workers in improving reliability of report on the progress of the project periodically? What data at health center levels and your office? were your feedbacks? 4. How is PBF health indicators and reporting mechanisms 4. Do you approve the contractual agreements between the align with FMOH HMIS Indicators and process? purchaser (Cordaid Yabelo office) and health facilities, 5. Any general suggestions to improve HMIS through PBF . zonal health Department and woreda health offices? If not 6. Is monthly supportive supervision (SS) integrated into your not, why? regular duty? 5. What is your assessment of the PBF project at Borena in 7. How do you describe the added value of PBF in improving improving the quantity and quality of health services at your SS duty? health facilities levels and health offices? 8. Are there irregularity in conducting monthly SS? why? 6. Are the results achieved in quantity and quality of health 9. Have you faced interruption in the regular SS during PBF? service cost-effective? why? 7. What was the contribution of the project on the six 10. Are there consequences for not conducting monthly SS building blocks of the health system, (leadership/ before and during PBF? governance, health work force, access to essential 11. What is working well and not working in SS process? Any medicines, financing, HMIS, and service Delivery)? How? suggestion for improvement? 8. What other roles do you play for the smooth operation of 12. Do you regularly conduct QQA at the health centers? the project? 13. How do you describe QQA practices before PBF and after? 9. What are the major factors that are enhancing the smooth 14. What are the advantages of QQA? implementation of the project? 15. What are your roles to address the quality gaps? 10. What are the major constraints for the implementation of 16. Do you support the HCs to include the quality gaps in their the project? quarterly plans? 11. What do you recommend for the ease of implementation of 17. How do you describe the role of PBF in all these processes? the project? Does Quarterly quality assessment contribute to the 12. What is your opinion on the sustainability and scalability improvement of quality of health services? How? of the project? What should be the institutional 18. What is working well and not working well in the QQA arrangement? Who should be the fund holder and process? what improvements should be made? purchaser within the government structure at different 19. Do you regularly support HCs in their Quarterly plans (QPs)? levels of administration? 20. How do you describe the process? 13. What are your suggestions on the sustainable source of 21. What was the practice before PBF? finance given the current limited fiscal space of the 22. What are your strengthens and weakness in the process? Government 23. Have many PBF stakeholders meetings have you organized? Questionnaire for Purchaser (Cordaid- Yabelo Office) 24. Who participated on those meetings? 1. What consideration do you take into account to develop 25. What were the agenda items of the meeting? the business plan of the health facilities? Do you consider 26. How do you evaluate the importance of those meetings? the woreda based plan targets? Can we have a sample of 27. How do you evaluate stakeholders knowledge about PBF? the actual facility business plan? Does the business plan Please explain the strengths, weaknesses and suggestions contribute to the improvement of quantity and quality of for improvement. health services? 28. Have you shared your best practices and lessons on PBF 2. Who is involved in the development of the business plan at with others? the facility level? Is there adequate capacity to develop 29. What do you suggest to improve coordination and that? If not what do you recommend? experience sharing on PBF 3. How far facilities adhere to the implementation of their 30. How do you evaluate the role of PBF approach to improve business plan? the health status of your woreda? 4. How often do you make contractual agreements with the 31. What are the positive aspects and weaknesses of PBF facilities and health offices? What problems did you approach and its implementation? encounter? 32. How did you manage your essential commodity needs to 5. Do the health facilities and health offices (Zonal and achieve your PBF health outcomes? Your suggestions for Woreda health offices) provide accurate performance data improvement. for verification? What is the average percentage of 33. What are the key lessons from the PBF experiences to variation between declared and verified data? expand to other woredas and sustain it success?
34 MARCH 2019 © CORDAID annexes EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
6. How do you address missing data or absence of staff during verification? 7. What are the major bottlenecks to undertake verification? 8. Do you provide feedbacks and coaching on the verification results to improve reporting? 9. What are the main mechanisms to verify declared health services from health facilities and health offices? 10. Do you think the business plan, contractual agreement , verification and payment linked to results contribute to the increased quantity and quality of health services? What were the mechanisms? 11. What is your opinion to improve the verification process? 12. What are the strengths and weaknesses of the program in achieving its targets? 13. What is your opinion to make the program sustainable and scalable at regional and country levels?
Questionnaire for the Community Based Organization (CBO) 1. Why do you think that patient verification is important? 2. Who supports you to discharge your assignment in the community? 3. What is the longest distance you have travelled to trace the patient ? 4. How are you perceived by the household during your visit? Do the community members recognize your role? 5. In the verification process what things are working and what are not working? 6. What suggestions do you have to improve the verification process? 7. Did you encounter a case where you were not able to trace a patient for verification? If yes, what was your impression and feedback? 8. Have you done patient satisfaction survey? 9. Are the patients helpful in providing their feedbacks? 10. Are you welcomed by the household when you discuss sensitive health issues? 11. Are you comfortable in handling community quality score (quality of services, waiting time and cost ) , any issue (if any) 12. In the process of patient satisfaction assessment and what are not working? 13. What suggestions do you have to improve the patient assessment process? 14. Do you observe improvement in patient satisfaction because of PBF? 15. Why do you think it is important to implement PBF? 16. How is your work coordinated with HEWs and other community health workers? 17. Do you observe improvements of quality of health services through the implementation of PBF? In what aspects, please list them if possible 18. What suggestions do you have to improve the process of PBF?
MARCH 2019 © CORDAID 35 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA annexes
Focus Group Discussion Guide (Community level)
Estimated time ...... Minutes
Date of Interview:......
Name and Location of the community:......
......
Names and titles of FGD participants:
NAME TITLE SEX: M/F AGE GROUP
Time Start: ...... Time End: ......
Facilitator:......
Note-taker/s:......
Introduction: Discussion Questions: Good morning/afternoon! Thank you for sparing your time 1. Are you aware of the PBF project? If so, please tell us about today to speak with us. The team here is working to conduct the project. (Probe: a) external evaluation of the PBF project. The purpose of this 2. Do you think that this project is relevant? project evaluation is to provide an informed assessment of 3. What are the health service delivery improvements achievements during the last three years and provide recom- introduced by the project? mendations for sustainability and scalability of the program Probe: in other woreda, zones and regions. ▪▪ availability of drugs and supplies; Our team has had the opportunity to review project docu- ▪▪ laboratory services; ments to get a sense of the design and implementation of the ▪▪ cleanness of the compound; project. However, review of the documents alone is not ▪▪ caring and respectfulness of the health personnel; enough. Therefore, we would like to speak with you today to ▪▪ Any other hear about your experience, in your own words, in order to help us better understand how this project is working. 4. Did CBOs reached you and asked you on the service you have got and your level of satisfaction by the health centers and Yabelo hospital? Probe: What was your feedback? Did you inform areas for improvement? Are these issues you provided feedback improved over time? 5. Any suggestion on areas for improving the health service coverage and quality of services? On the PBF project?
36 MARCH 2019 © CORDAID annexes EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
ANNEX D: LIST OF PEOPLE CONSULTED
1. Key Informant Interview: FMOH, ORHB, ZHO, WHOs, and Health Facilities,
NO. NAME POSITION NAME OF INSTITUTION/ NAME OF ORGANIZATION/ WOREDA FACILITY
1. Dr. Feven Girma A/Head Partnership and Coordination FMOH Directorate
2. Mesfin Kebede Expert Partnership and coordination FMOH Directorate
3. Dr.Abdulkader Gelgelo Deputy Head ORHB ORHB
4. Derjie Abdissa Head, Quality Directorate ORHB ORHB
5. Meaza Hailu Expert ORHB ORHB
6. Fikremariam Gezahegn Program Manager Cordaid Ethiopia Cordaid Ethiopia
7. Nurky Ebrahim Team Leader Cordaid Yabelo Office Cordaid Yabelo Office
8. Gelgelo Verifier Cordaid Yabelo Office Cordaid Yabelo Office
9 Fenayu Dereje OPD focal person El Waye Adii-Galchet Health Center
10 Roba Kalichu Human Resource focal person El Waye Adii-Galchet Health Center
11 Roba Boru Accountant El Waye Adii-Galchet Health Center
12 Mahamed Aliyi. OPD El Waye Adii-Galchet Health Center
13 Barite Bagajace MCH Focal Person El Waye Adii-Galchet Health Center
15 Haleluya Bereket Accountant El Waye Adii-Galchet Health Center
16 Jemal Mohamed EPI focal Person El Waye Adii-Galchet Health Center
17 Mahamed Husen Emergency focal person El Waye Adii-Galchet Health Center
18 Molu Tahule HC Director El Waye Adii-Galchet Health Center
19 Eskindir Tahir HC Director Yabelo Rural Did-Yabelo HC
20 Sudita huka Sasubdu Garii Yabelo Rural Did-Yabelo HC
21 Sasi Tandi HEP Yabelo Rural Did-Yabelo HC
22 Demise Bekele OPD Yabelo Rural Did-Yabelo HC
23 Malich Surupra Guard Yabelo Rural Did-Yabelo HC
24 Mihiret Tefera Pharmacy Yabelo Rural Did-Yabelo HC
25 Rafo Husca Mid-Wife Yabelo Rural Did-Yabelo HC
26 Lemechu Usman HC Director Yabelo Rural Dikale HC
27 Adem Ahmed, HMIS Yabelo Rural Dikale HC
29 S/R Emebet Gobi MCH Yabelo Rural Dikale HC
30 Mohamed Suleman EPI Yabelo Rural Dikale HC
31 Alemayehu Demirew Lab.Tec Yabelo Rural Dikale HC
32 Harbora Bekele Head of Yabelo Rural Woreda Yabelo Rural Health Office Health Office
33 S/R Enanish Ada Acting Head, Yabelo Urban Yabelo Urban Health office
34 Desalegne Debebe Wash Focal Person Yabelo Urban Health office
35 Dermo Daleche TB/FP Yabelo Urban Office Health office
36 Sintayehu Hamid HIV Yabelo Urban Office Health office
MARCH 2019 © CORDAID 37 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA annexes
NO. NAME POSITION NAME OF INSTITUTION/ NAME OF ORGANIZATION/ WOREDA FACILITY
37 Dawit Getachew HIT Yabelo Urban Office Health office
38 Qabele Jotani MCH Yabelo Urban Office Health office
39 Yonas Teferi HR Yabelo Urban Office Health office
40 Kitissa Messele Director, PHCU Yabelo Urban Yabelo HC
41 Obortu Halake Yabelo Urban Yabelo HC
42 S/R Haymanot Aklilu Yabelo Urban Yabelo HC
43 S/R Konjit Telle FP Yabelo Urban Yabelo HC
44 S/R Elsabet Tafesse Yabelo Urban Yabelo HC
45 Molu Dima V/Head of Borena Zone Health Borena Zone Borena Zone Health Officet Department
46 Tegene Worku Health Facility Support Process Borena Zone Borena Zone Health Office Coordinator
47 Chala Befekadu Hospital Support Process Borena Zone Borena Zone Health Office Coordinator
48 Bimam Beaghe Head, X-ray Dept Borena Zone Yabelo Hospital
49 Sintayehu Abera Head,HR Borena Zone Yabelo Hospital
50 Bushra Abdella V/Head Medical Ward Borena Zone Yabelo Hospital
51 Osman Kedir v/head medical ward Borena Zone Yabelo Hospital
52 Shewaye Ambaw Head, GYNI Berana Zone Yabelo Hospital
53 Dida Bute Head, Pharm Borena Zone Yabelo Hospital
54 Abdula Qedan G/C, CEO Borena Zone Yabelo Hospital
55 Nagare Dejene Reform Officer Borena Zone Yabelo Hospital
56 Betrearow Sileru Head, Lab Borena Zone Yabelo Hospital
55 Sallad Aliyo Head Gomalo Woreda Health Office
56 Bekele Megersa MCH Coordinator Gomalo Woreda Health Office
57 Tdo Jelde CBHI Coordinator Gomalo Woreda Health Office
58 Abdurahman Tola Head Gomalo Surupha Health Center
59. Umer Menka member Gomalo Surupha Health Center
60 Beti Dalu Gomalo Surupha Health Center
61. Gezahegn Chofero Head Yabelo Rural Haro Wayu Health Center
62 Amarech Endale Clinical Nurse Yabelo Rural Haro Wayu Health Center
63. Tigist Zenebe Mid Wifery Yabelo Rural Haro Wayou Health Center
64. Alekere Hargessa Regulation head El Waye El Waye Woreda Health Office
65. Turu Muta Head El Waye El Waye Woreda Health Office
66. Gebasa Giro CDC Coordiantor El Waye El Waye Woreda Health Office
67. Nanye Beke Head El Waye El Waye Health Center
68 Daniel Hunkene Immunization head El Waye El Waye Health Center
69. Derebe Geresu Lab Technicain El Waye El Waye Health Center
70. Mohamed Rashedama MCH, Mid Wifery El Waye El Waye Health Center
38 MARCH 2019 © CORDAID annexes EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
2. Community Based Organization (CBO)
NO. NAME OF CBO PARTICIPANT WOREDA NAME OF HEALTH CENTER CBOS ATTACHED
1. Daboya Tene Gomalo Surupha
2. Kelea Gelgelo Gomalo Surupha
3. Hassen Gelo Gomalo Surupha
4. Galm Tache Yabelo Rural Did Yabelo
5. Adi Molu; Yabelo Rural Did Yabelo
6. Loko tunu Yabelo Rural Did Yabelo
7. Sake Sard Yabelo Rural Did Yabelo
8. Sake Dherge Yabelo Rural Did Yabelo
9. Dadi Dalacha Yabelo Rural Did Yabelo
10. Scure Gagna Yabelo Rural Did Yabelo
11. Ghakki Jarso Yabelo Rural Did Yabelo
12. Loko Bidu Yabelo Rural Did Yabelo
3. FGD
3.1 Names and titles of FGD participants: Adii Gulchat Health Center
NAME TITLE SEX: M/F AGE GROUP
Bakamo Guyo F 20
Elma Bagaja F 80
Waro Kitiso M 20
Jarso Boru M 20
Gamure Garbicha M 30
Abdiishaku Sord M 20
Hussen Jirmo M 25
Nura Dabaso M 50
Kiyyo Dida M 28
Torbe Kanchoro M 20
3.2 Names and titles of FGD participants: Did-Yabelo Health Center
NAME TITLE SEX: M/F AGE GROUP
Loko Galbech F 20
Jilo Okotu F 25
Lelo Haka F 30
Tiru Jarso F 30
Did Dima M 19
Galpalo Adi M 25
MARCH 2019 © CORDAID 39 EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA annexes
3.3 Names and titles of FGD particpants: El Waye Health Center
NAME TITLE SEX: M/F AGE GROUP
Abdulnaser Gelgelo M
Tiru Abegaze F
Teke Arero F
Elema Golo F
Kebede Guyo F
Gatela Dembela F
Adi Mamo F
Doko Kuchara F
Boko Chacho M
Kebede Boka F
Kemele Guba F
Bokayu Geelma F
Denga Godana M
3.4 Names and Titles of FGD Participants: Yabelo Hospital
NAME TITLE SEX: M/F AGE GROUP
Wako Wario M 70
Rahama Doyo F 20
Samson Tsegye M 34
Debretu Gemeda F 35
Abrahim Adem M 38
Nenko Dembela M 32
Denga Duba M 75
Gure Jaba M 55
Dalacka Ashane M 36
Loko Libe F 22
Bush Kote M 56
Bokayo Sora F 55
Ahade Boru Merchant F 55
40 MARCH 2019 © CORDAID annexes EXTERNAL EVALUATION OF PHASE 1 OF THE PBF SHOW CASE PROJECT IN BORENA
ANNEX E: REFERENCES
▪▪ Borena Zonal Health Department, 2018. Power Point Presentation of the findings of the project, PBF Workshop, March 28-30th, Debre Zeit ▪▪ Cordaid, PBF Portal Database for Phase I PBF showcase project in Borena Zone, Ethiopia. ▪▪ Cordaid, 2018. Power Point Presentation on the Findings of Internal review of the PBF phase I project. PBF Workshop, March 28-30th, 2018, Debre Zeit. ▪▪ Cordaid and Oromia Regional Health Bureau, 2016. Performance Based Financing Implementation Manual. ▪▪ Contractual Agreement between Purchaser and Providers-Sample ▪▪ Monthly Quantitative reports of the health facilities-Sample ▪▪ Quarterly Quality Assessment Report by Borena Zonal Health Office-Sample ▪▪ Quarterly Quality Assessment Report by the Woreda Health Offices-Sample ▪▪ Sieleunou, Isidore, 2014. Baseline study Report-Pilot Performance Based Financing Project in Borena Zone, Oromia Regional State. ▪▪ Verification Report by CBOs-Sample
MARCH 2019 © CORDAID 41 about cordaid contact
Cordaid works to end poverty Inge Barmentlo and exclusion. We do this in Program Manager the world’s most fragile and [email protected] conflict-affected areas as well as in the Netherlands. We Maarten Oranje engage local communities to RBF / Data Expert rebuild trust and resilience [email protected] and increase people’s self- reliance. Our professionals Akinyi Walender provide humanitarian Country Director Ethiopia assistance and create opportu- [email protected] nities to improve security, healthcare and education and Cordaid the Netherlands stimulate inclusive economic Lutherse Burgwal 10 growth. We are supported by 2512 CB The Hague more than 268,000 private +31(0)70-31 36 300 donors in the Netherlands www.Cordaid.org and by a worldwide partner network. CARE. ACT. SHARE. LIKE Cordaid.