Performance Management of Last Mile Vaccine Distribution

Nov. 2015

Brittany G. Johnson, Ekta Jhaveri, Wendy Prosser, Aida Coehlo, Prashant Yadav

ACKNOWLEDGMENTS

This work could not have been possible without the tireless work of the VillageReach team in country and Seattle. Particularly Ruth Bechtel and Tatenda Mutenga who provided regular follow-up and support.

Of course, the research would not have been possible without the distribution teams’ willingness to participate and be available for follow-up. In Maputo, Margarida Matsinhe. In Gaza Alberto Mabota. In Niassa Gregorio Júnior, and EPI teams in all 3 provinces.

Sarah Alphs Mwenda provided direction and momentum in the development of this study. Leila Yosef provided initial follow-up transcription and translation for the first six months of study implementation.

About VillageReach

VillageReach work with their partners to deliver vaccines to the last mile. Through this mission they research the dependencies within complex systems and identify the requirements for new solutions to thrive in sub-Saharan countries. VillageReach provides the data, tools, learning and support to empower key institutional stakeholders, including governments and global health partners, to implement change and take innovation through to scale and sustainability.

About the Healthcare Initiative at the William Davidson Institute

WDI’s Healthcare Initiative develops intellectual capital that helps increase access to essential medicines, vaccines and other health technologies in developing countries. We do this by helping create scalable and sustainable models for supply chains and delivery in healthcare. We help stimulate stakeholder discussions around new solutions to the many problems affecting global healthcare and explore new ideas through models, field experiments, and pilot implementations. Through this research, the initiative supports policymakers in philanthropic organizations, developing country governments, bilateral and multi-lateral agencies, pharmaceutical and vaccine manufacturers, and healthcare investment firms.

Contents Acronyms ...... 3 Executive Summary ...... 4 Introduction ...... 6 Study Background ...... 8 Consideration of Study Findings ...... 8 Experimental Design ...... 10 Analysis Methods ...... 16 Analysis of MIS Data...... 16 Root Cause Analysis ...... 22 Results ...... 30 Conclusion ...... 32 Annex ...... 34

Acronyms CMAM Central de Medicamentose Artigos Médicos (Central Medical Store) DPS Direcção Provincial de Saúde (Provincial Health Management) DLS Dedicated Logistics System EPI Expanded Program on Immunization LMIS Logistics Management Information System MISAU Ministério de Saúde (Ministry of Health) NGO Non-Governmental Organization PAV Programa Alagado de Vacinação (see EPI) SELV Sistema Electronica de Logistica de Vacinas (see LMIS) WDI William Davidson Institute VR VillageReach

Executive Summary Since 2011, VillageReach has implemented a Dedicated Logistics System (DLS), whereby the provincial Expanded Program on Immunization (EPI) delivers vaccines directly to health facilities on a monthly basis, skipping the district level and streamlining distribution. This program has successfully increased vaccine coverage in the provinces where it operates and with a higher cost-efficiency. However, to achieve these results, VillageReach has provided continual managerial support with a cost-share approach to government distribution teams, with the occasional additional financial support.

Between March 2014 and 2015 VillageReach in collaboration with the William Davidson Institute analyzed a strategy to shift accountability for vaccine distribution to the government. They implemented a control match study considering the impact of private sector management strategies. The research team selected three provinces, Gaza and Maputo were intervention sites and Niassa a control site. Intervention sites held monthly meetings with senior leadership to review and resolve distribution issues. These monthly meetings reviewed distribution issues charts which the distribution team tracked over the course of the month. EPI chiefs held discussions according to the structure of a resolution report to work through root causes of distribution issues and potential resolutions. Additionally, the province that showed the greatest delta in change over the baseline received a financial award for performance improvement.

All provinces showed improvement in vaccine delivery and availability over the course of the study. Gaza and Maputo showed marginal significance that was attributed to the tools introduced. Additionally, a similar intervention was leveraged by the VillageReach Mozambique office in conjunction with a new logistics information management system to review DLS data gathered on a monthly basis with all provinces, leading to additional noise in the data. Personnel changed in the control province over the course of the study, contaminating the control.

Based on baseline and endline study results, the monthly meetings had significance in ensuring that responsibilities of the distribution team were fulfilled. Additionally, these meetings provided timely feedback and support from supervisors. Based on the written monthly resolution reports, creative problem solving took place to resolve issues within the control of the distribution team. Nonetheless, distribution teams felt that without VillageReach asking for these reports on a monthly basis, it would be unlikely that they will continue to meet. This suggests that a required management directive is necessary in maintaining good practice.

Monthly resolution reports provided a continual resource for root cause identification, making it feasible to consider key impact areas for engagement among government and outside actors. For instance, through these reports the research team discovered that of the deliveries made over the course of the study, 20% more would have been possible had refrigerators been functional. This provides evidence for future work by government and other vaccine supply chain stakeholders to identify key strategic areas for interventions and resolve the most pressing bottleneck issues.

Introduction Vaccines save millions of lives each year and are among the most cost-effective health interventions available. While immunization coverage has dramatically increased over the last three decades, the coverage rates have plateaued at 80%. One in five children are still without access to vaccines, resulting in an estimated 1.5 million child deaths each year and tens of thousands of permanent disabilities from vaccine-preventable diseases such as diarrhea and pneumonia.

Improving access to lifesaving vaccines starts with a well-functioning supply chain which is essential for ensuring the availability of quality vaccines in the right quantities and at the required time to populations in need. VillageReach is a social enterprise that supports supply chains and last mile delivery across Africa. Since 2002, VillageReach has worked to build the capacity of provinces and districts in Mozambique to carry out health logistics activities to improve immunization coverage rates. Their work has highlighted a number of factors contributing to the current inefficiencies and ineffectiveness of the vaccine supply chain including

 Information deficits due to poor communication and reporting infrastructure  Deficient financial resources and unpredictable financial flows  Poor quality of infrastructure (e.g. electricity, roads, warehouse facilities)  Inadequate human resources in terms of number, capacity and performance due to lack of accountability structures

These gaps increase the costs of vaccine delivery and create large disparities in the quality of the immunization system across geographies.

In partnership with the Direcção Provincial de Saúde (DPS) in four provinces, VillageReach has been engaged in an approach to accelerate access to vaccines and improve supply chains through the deployment of a streamlined logistics system, the Dedicated Logistics System (DLS). This system has been designed to reduce the burden on frontline health workers by consolidating logistics tasks to a small team of two to four dedicated Field Coordinators for each province integrating supervision and data collection into a system design, and leveraging existing transport infrastructure.

Figure 1: Mozambique’s Public Health Supply Chain (left) and DLS (right)

At the provincial depot, the medical chief manages the provincial EPI chief. The EPI chief manages district EPI chiefs and the logistics coordinator at the province level. Each month, a VillageReach field officer works with government field coordinators (which was often an EPI chief, logistics coordinator or a district EPI chief) to make distributions directly to all health clinics.

VillageReach’s work to scale the DLS in four provinces (Maputo, Gaza, Niassa, Cabo Delgado) has yielded positive results. However, the current approach has led to questions regarding the sustainability of the system. VillageReach has had to provide ad hoc financing to ensure that deliveries happen as planned. High rates of staff turnover led to untrained staff in the Field Coordinator position, coupled with insufficient engagement from the provincial health authorities in monitoring the Field Coordinators’ work, put at risk the sustainability of improved performance of the vaccine supply chain. At the time of this report, it was unclear how government would take on full management and financial responsibility of the DLS. A performance management system could create better accountability and increased scrutiny within the DLS leading to more sustainable management approach of the system.

We implemented a control-matched study across three provinces in Mozambique receiving monthly support from VillageReach for the vaccine supply chain to determine the potential impact of a performance management framework implemented by the provincial teams on key delivery indicators. This study introduced performance management tools and incentives to improve provincial distribution team identification and resolution of distribution challenges and start a discussion on shifting the success to be sustainably managed by the government.

This study provided empirical evidence that distribution teams resolve a litany of problems by working directly with districts and health units. Over the course of this study, 1,470 visits to health units took place in Gaza and Maputo. 20% more could have taken place had refrigerators been working. A total of 287 additional visits took place during study implementation over the previous year of distribution.

Study Background NGO gap filling in areas such as fuel, per-diem, and maintenance is not a sustainable long-term solution. NGO technical assistance should primarily serve as a means to demonstrate to the government where the challenges are and how they could be addressed. For long-term sustainability, better accountability and performance metrics within the system will help in highlighting where the problems lie. Otherwise, the problems can remain oblivious to senior leadership in government.

The objective of this study was to introduce a performance management and accountability framework to determine the potential impact on vaccine delivery. The hypothesis was that monthly problem solving by the distribution team and involving senior leadership, paired with incentives for high performance, would resolve delivery bottlenecks and ultimately increase vaccine availability. The study was implemented in two intervention provinces and one control province to measure the impact of accountability and institutionalized root cause analysis on vaccine supply chain performance.

The research team measured change in performance using VillageReach Management Information System (MIS) and later the Sistema Electronica de Logistica de Vacinas (SELV). These information systems gave a complete picture of information pertaining to distribution cycles, facilities covered in any given month and vaccine stockouts.

Consideration of Study Findings This study was developed to create a vehicle for performance accountability that would put the annual distribution plans, actual reported performance, budgeted vs. actual amounts, and reasons for under achievement into one unified framework. This was intended to move forward a more systematic review of planning activities at a senior management level such as the EPI Director or Medical Chief, on a monthly basis. In the short term this could lead to optimal allocation of critical resources such as fuel and transport. In the long run it can highlight budgeting shortfalls or other bottlenecks in the budget allocation process. For this type of intervention to achieve impact, the performance accountability structure must be backed by senior leadership at the province.

Findings from this report can also provide pointers to Gavi in its supply chain strategy and create further impetus in the immunization system strengthening financing window to earmark funds for such initiatives in multiple countries. The study sheds light on the feasibility of sustaining the gains achieved by VillageReach, allowing for an easier handover to MOH for management of DLS. Gavi strategy requires data to inform key vaccine supply chain flows of immunization planning, global supply chains, global to country interface, in-country supply chains and vaccine delivery. Data is being gathered. The question that this study adds to is how this data can be used to positively impact delivery to meet coverage standards – particularly as these supply chains are subject to imminent increases in volume, doses, and cost.

Study results indicate which bottlenecks constrain delivery, problem resolution feasible to distribution teams, and issues that require a new strategy or central level involvement. Among these constraints, performance management tools provide a framework to think through issues. A major challenge to this

8 type of critical thought, was the demotivation of distribution teams when issues were not resolved – fuel availability was one of these challenges.

Experimental Design Intervention studies with a control group are considered to provide the most reliable evidence about the effect of an intervention. A key distinguishing characteristic of an intervention study is that the intervention being tested (in this case the distribution issues chart and resolution report) is allocated by the study designer to a group of two or more study areas. Key outcomes metrics are tracked prospectively to compare the intervention vs. the control areas. The vaccine supply chain in these four provinces in Mozambique is such that an intervention of the nature being tested in this study is most applicable at the Provincial level. While the most robust intervention study design is the randomized controlled trial (RCT) where the intervention or control is randomly allocated to the study area, the number of provinces in Mozambique and other logistical considerations prevented us from using a fully randomized control design. The research team tried to select provinces for the control and intervention arms of this study to ensure they are similar in as many respects as possible with the exception of the intervention under investigation. Admittedly, there are only 11 provinces and there are observed differences between them in vaccination rates, vaccine stockout frequency, and other variables of interest. The research team therefore conducted a pre-post analysis using difference-in-differences to further isolate the effect of this intervention.

The research team utilized a time-series design combined with a match control. The provinces included in this study were three: Gaza, Maputo, and Niassa. These were chosen due to the implementation of VillageReach’s MIS, to monitor performance, and received ongoing follow-up for vaccine supply chain improvement. The intervention to improve overall performance accountability and incentivize best performance took place in Gaza and Maputo - interventions described below. Niassa provided the control.

Table 1: Specification of intervention and control provinces

Gaza Maputo Niassa Study status Intervention Intervention Control Eligibility Easy access for Easy access for Limited access training and follow-up training and follow-up for follow-up Facilities 97 82 129 Districts 12 8 16 Length of 10 days 12 days 14 days distribution run Pilot: Total field 2 2 4 Gaza coordinators Maputo Total cars 2 2 2 MIS 2011 2011 2011 Control: SELV July 2014 July 2014 July 2014 Niassa

Initially the control and intervention areas were intended to be selected using matching criteria including population density, existing health infrastructure, vaccine coverage rates, vaccine stockout rates, number of years since the launch of the MIS program in the province, and other potentially confounding

10 variables. However, during the outset of the study, Niassa and Cabo Delgado were facing uncertain status in their participation in VillageReach distribution. The DPS leadership in Cabo Delgado was weak with consistent financial flow issues, negatively effecting the distribution system. Niassa had DPS commitment but less than ideal capacity of staff, both at the DPS and the VillageReach field officer. Despite this, Niassa was selected as the control. Gaza and Maputo were selected as intervention provinces based on the stability they offered to observe possible study results and the ability to follow- up with the implementation of the study.

Intervention 1a: Strengthening overall performance and accountability among EPI stakeholders Each month after the distribution cycle has been completed by EPI, the VillageReach Field Officer in the intervention province facilitated a meeting of all the EPI stakeholders which included provincial EPI Chief, Field Coordinators, and Logistics Technician to discuss the factors that led to differences in planned versus actual performance on three indicators:

1. Percent of health units that received a delivery within 33 days 2. Percent of health units that report a stockout at time of delivery 3. Percent of health units submitting a report

Field coordinators documented these factors using a distribution issues chart (Annex) over the course of their monthly deliveries. At monthly meetings, EPI Provincial Chiefs would lead discussions using a resolution report to process the root cause of the issues identified and think through possible resolutions.

Delivery within 33 days was a primary factor in determining health unit stockouts. Stockouts are attributable to determining the correct cycle and buffer stock and ensuring that replenishment happens on time to replenish the cycle stock before stockout. The following formula was used to determine quantity of vaccines to leave at health units.

Equation 1: Monthly vaccine stock-up-to policy at public health units in Mozambique

Number of doses was determined to satisfy required need until next distribution, which in the case of Mozambique was one month. A late delivery directly impacts the probability of a stockout. MIS data collects data on replenishment timing, not months of cycle and buffer stock. These were checked by distribution teams.

Months of cycle and buffer stock?

Probability of stockout

Replenishment timing discipline  1 Figure 2: Factors of commodity stockouts

The monthly meeting and the reporting of its outcomes was carried out through the use of two tools: a distribution issues chart and monthly resolution report (see Annex for the tools). The ultimate goal of this exercise was to have a consensus on the primary causes impeding planned distribution targets. The intent was to systematically capture these causes and develop a proactive approach to resolving the main bottlenecks.

Information examined in the monthly meetings included

 Reasons for deviating from projected number of facilities in annual distribution plan  Reasons behind vaccine stockouts  Reasons for facilities to not submit reports

This information was then discussed in the meeting using a categorization framework that was part of the resolution report. The outcome summary of each of these meeting was reported to the Medical Chief by the Provincial EPI Chief.

Intervention 1b: Rewarding best performance and sharing best practices Recognizing best practices, rewarding them, and sharing them was an important complement to the performance accountability approach in Intervention 1a which focuses more on systematically identifying the causes of problems.

There was no universally accepted definition of what would constitute best performance in the EPI distribution system. Also, the maximum achievable performance for a given province depended upon

1 In the field, unexpected trends in adherence, such as a surge of children arriving for vaccines, could impact probability of stockout – particularly if this unaccountable in previous calculations of target group and percent coverage.

12 their spatial-geographical constraints, road infrastructure, and complementary resources available from development partners such as EGPAF and VillageReach. In order for the best performance to reflect these differences, performance was measured as the improvement over a province specific baseline. Provinces that achieve the best performance received $4,000 to use for EPI computers. Best performance was monitored using MIS data of deliveries within 33 days, stockouts at time of visit, and health units submitting monthly reports. The following formula was used to determine change in performance.

Equation 2: Change in performance over baseline

퐸푛푑푙푖푛푒 (퐴푣푒푟푎푔푒 표푓 푝푒푟푓표푟푚푎푛푐푒 푂푐푡표푏푒푟 푡표 퐷푒푐푒푚푏푒푟 2014) − 퐵푎푠푒푙푖푛푒

퐵푎푠푒푙푖푛푒 (퐴푣푒푟푎푔푒 표푓 푝푒푟푓표푟푚푎푛푐푒 푂푐푡표푏푒푟 푡표 퐷푒푐푒푚푏푒푟 2013)

Each performance metric was weighted to determine which province achieved the greatest performance across all three metrics. The following weights were used:

Table 2: Weight of performance metrics to determine province with greatest improvement

Performance Metric Weight Health units visited every 32 days or less 60% Products stocked out across facilities 20% Health units submitting monthly reports 20%

Best practices from top performance was documented with the intention of creating a community of practice among EPI staff from different provinces to engage in learning from each other’s hands-on experiences. This would also feed into a central repository of best practices in the vaccine supply chain for all Gavi supported countries. This report elaborates the various contributing factors to performance improvements.

Data collection VillageReach routinely collects distribution performance data using MIS. This data was analyzed throughout the study and compared with data available from the previous year to provide a time series analysis of performance on the indicators analyzed. Additionally, as part of the study itself, monthly performance data outlined in Intervention 1a was collected from each distribution team as they recorded root cause in charts and potential actions in meeting reports.

Figure 3: Study and data collection timeline

To further inform study results, key informant interviews were conducted at baseline and endline with the Medical Chief, EPI Chief, Field Coordinators, District EPI Chiefs, and VillageReach Field Officer in each province. A midline study was conducted with Provincial EPI Chief, Field Coordinators and Officers six months into the study.

Integrity of the experimental design Two issues challenge the experimental design. Over the course of the intervention, VillageReach monitoring and evaluation officer started asking distribution teams questions regarding performance in a structure similar to the one put forward in this study and intended to be implemented among distribution teams, without minor VillageReach involvement. Additionally, the intervention provinces did not implement the tools fully as directed.

In July 2014, VillageReach deployed an upgraded LMIS tool (built on the OpenLMIS platform) that provides more visualization and easy access to data from the health facility level. Hand-in-hand, field officers started reviewing this MIS data with provinces as a regular management approach, yet contaminating results of this study. The intervention time between April and July, when the performance management intervention was the sole intervention, does not provide significant evidence to separate attribution of performance improvements due to this study versus VillageReach support. Additionally, this support as well as personnel change is credited for helping the control province identify and resolve a key issue behind delay in visits. Calculations are made to understand improvements in intervention and control provinces, but this background puts to question any results.

Implementation of the study in Gaza and Maputo was imperfect. Gaza failed to submit charts relating to missed visits and reports for the first six months of implementation. Maputo did not hold monthly meeting with the EPI manager, who likewise could not lead meetings and spearhead actions relating to

14 distribution improvements. Likewise, it is uncertain whether the full potential impact of this intervention was captured in this study.

Analysis Methods The research team focused on data from the management information system (MIS) managed by VillageReach to determine the impact the introduction of performance tools and incentives had on vaccine distribution and the statistical significance of this impact. The primary null hypothesis was that the implementation of a performance management and accountability intervention would have no impact on improving vaccine supply chain performance. Additionally, the research team collected the resolution reports and distribution issues charts collected monthly and analyzed these for an understanding of the MIS data and root cause for poor performance.

The research team did not succeed in rejecting the null; however, the root cause analysis suggests a range of activities available to the distribution team to improve vaccine distribution over the course of the study.

Analysis of MIS Data The following graphs represent the percent of total health units visited in intervention and control countries from a year preceding the intervention through the year following the intervention. The green dashed line marks the start of the study. These graphs include the metric of visits under 33 days which was investigated during the intervention as well as a measure for visits under 66 days to better illustrate the timing discipline of distribution teams.

Figure 4: Provincial performance data for health units visited

The other key measurement was the percent of stockouts at health units at time of delivery. The following graphs present province performance before and after the intervention. The green dashed line represents the start of the intervention.

Figure 5: Provincial performance data for health units stocked out

The top graph represents reported stockouts in health units. This is the measure the research team analyzed throughout and the measure used to compare province level performance. However, this is not the most accurate measure for stockouts as MIS data on stockouts is only available for health units that are visited. The second graph suggests what actual stockout levels might be. Health unit stocking policy is calculated based on the previous months demand and a two week safety stock. Districts carry two months of additional stock for the district which is intended for emergency replenishment. If a health unit is not visited within 2 months, it is reasonable to believe that a stockout occurs - represented in the bottom graph.2

We performed two types of time series analyses on the MIS data to determine the impact of the intervention on distribution performance: an analysis of data for the year preceding and proceeding from the intervention and analysis of Gaza and Maputo with Niassa.

2 It is also possible that health workers go and fetch vaccines to avoid stockouts – which would not results in the calculations presented. However, this task also puts strain on the system and impacts care provided at the health clinic. Thus, this figure still provides insight into disruption in public health services.

Pre post analysis The performance review intervention has not significantly improved replenishment timing discipline as measured by MIS data for the year preceding and proceeding from Intervention 1a in March. The table below records replenishments timing performance.

Table 3: Analysis of pre and post province performance of health units visited from MIS

Pre Post Average Effect Variance P value

Health units visited within 33 days Gaza 48% 60% + 12% 0.06 0.22 Maputo 57% 66% + 9% 0.08 0.35 Niassa 11% 38% + 27% 0.15 0.04 Health units visited within 66 days Gaza 68% 82% +14% 0.00 0.01 Maputo 80% 86% + 6% 0.02 0.35 Niassa 15% 56% +41% 0.15 0.00

Gaza and Niassa, intervention and control group respectively, both showed significant improvement in health units visited within 66 days. The ability to maintain replenishment timing discipline was largely dependent on the functioning of refrigerators. In Gaza and Maputo, refrigerators were the primary cause for missing a visit. Figure 6 following figure illustrates the breakdown of reasons given for health units not visited.

Figure 6: Causes for missed visits in Gaza and Maputo (percentages represent data from resolution reports)

A large segment of health units were not visited due to refrigerators not working. Had refrigerators been working, 20% more deliveries would have occurred over the course of this study. This intervention was

18 focused on accountability for distribution staff at province level to take action on problems within their control and functionality of refrigerators were considered outside this control. Operational interviews at midline and endline reveal that budget for refrigerator repair were handled by MOH and provinces separately. From midline assessment, there was at least one instance where distribution team reached out to provincial chief, who wrote a letter to MOH requesting assistance to replace various broken refrigerators.

In Niassa, the control province, no explicit measure was carried out defining reasons for not visiting health units – as this was a part of the intervention. For this reason, no quantifiable evidence explains the improvement to replenishment timing discipline. However, evidence was collected at the midline assessment that suggested the problem of broken refrigerators affected many health units. Furthermore, Niassa benefited from the SELV data review and personnel changes both in the DPS and the VillageReach field officer, mentioned previously. The distribution team identified these two areas as responsible for their identifying vehicle maintenance as a resolvable delivery bottleneck and thereafter improving performance.

Over the past year, health unit stockouts have significantly reduced. Stockouts were only recorded if a visit took place. For this reason, the research team also calculated a proxy measure where distribution occurring over 66 days from the previous distribution was considered a stockout. Because health facilities follow a 1 month stock replenishment policy and in principle hold an additional month of stock as buffer stock, missing a distribution run in a given month does not always result in a stock out (See Figure 2 earlier). However, missing two consecutive months of distribution to a health facility is very likely to result in a stock out (unless the facility had over ordered in the past, received stock from the district, or had gone to fetch stock from the district at their own expense and initiative).

Table 4: Analysis of pre and post province performance of health units experiencing one or more stockouts from MIS

Pre Post Average Effect Variance P value Health units with one or more stockout Gaza 24% 7% 17% 0.00 0.00 Maputo 40% 8% 32% 0.01 0.00 Niassa 15% 8% 7% 0.01 0.09 Health units visited with either a stockout or distribution exceeding 66 days Gaza 49% 25% 24% 0.00 0.00 Maputo 54% 22% 32% 0.04 0.00 Niassa 90% 46% 44% 0.15 0.00

Significant improvements were noticed in both Gaza and Maputo provinces. Niassa also showed improvement, but of no statistical significance over the previous year. This was not the case in considering visits taking more than 66 days, as Niassa greatly improved in its timing discipline. The intervention could explain the significance of these results. Endline interviews suggest that the tools did

19 increase distribution accountability by ensuring that teams make requests for vehicle, fuel and per diem on time every month.

However, these results may also be a general effect of the overall supply chain in vaccine distribution in Mozambique, since these improvements include the study control. Niassa, the study control, also shows improvement which is statistically significant to the 10% level. This is in part due to contamination. Niassa at the start of this study used a similar approach implemented by the VillageReach team in reviewing SELV data.

Difference- in-difference analysis Distribution and stock level performance in the intervention and control provinces before the start of the intervention (Table 3 and Table 4 column 1) make it evident that despite attempts to have a robust control and intervention group, non-trivial differences existed between the treatment and control groups prior to the start of this study (pre-intervention differences) . In an attempt to better isolate and understand the “true treatment effect” the research team also estimated the difference-in-difference (or "double difference") between the control and intervention groups.

For this analysis, the research team estimated the difference in the performance metrics in the intervention provinces before and after the intervention minus the difference in performance metrics in the control province before and after treatment.

Table 5: Analysis of control and intervention province performance of health units experiencing one or more stockouts from MIS

Control (Niassa) Intervention Average Effect Variance P value Health units visited within 33 days (Pre minus Post) Gaza 27% 12% - 15% 0.11 0.30 Maputo 27% 10% -17% 0.05 0.19 Weighted 27% 11% - 16% 0.08 0.25 Health units visited within 66 days (Pre minus Post) Gaza 41% 13% - 28% 0.02 0.05 Maputo 41% 5% - 36% 0.02 0.01 Weighted 41% 9% - 32% 0.02 0.03

As in the pre-post analysis, this analysis confirms that there is limited effect of the intervention on adherence to distribution timing (Table 5). In 66 day distribution windows, the research team observed that improvements in Niassa’s (control group) performance were in fact significantly greater than those in the intervention provinces.

Table 6 estimates this for stockouts and shows that there is a strong improvement in Maputo over the control group. Likewise, Gaza also showed a modest improvement. The average effect is significant allowing us to infer that the intervention did lead to a decrease in stockouts attributable to the intervention.

However, there were some other confounding factors occurring at the same time that may have contributed to this effect. For instance, stockouts were reported only when a visit took place. This may have led to results from data before the intervention (when visits were less frequent across provinces) showing better intervention figures. When visits were considered, as in the last three rows of the following table, a larger improvement in the control province was observed; however, these were not statistically significant.

Table 6: Analysis of control and intervention province performance of health units experiencing one or more stockouts from MIS

Control (Niassa) Intervention Average Effect Variance P value Health units with one or more stockout (Pre minus Post) Gaza 7% 16% 9% 0.02 0.10 Maputo 7% 32% 25% 0.04 0.00 Weighted 7% 23% 16% 0.03 0.05 Health units visited with either a stockout or distribution exceeding 66 days Gaza 44% 24% 20% 0.05 0.13 Maputo 44% 33% 11% 0.07 0.43 Weighted 44% 28% 16% 0.06 0.27

The magnitude of change in Niassa versus that in Gaza and Maputo must consider the imbalance in low performance in Niassa before the start of the intervention. As higher performing provinces, Gaza and Maputo required additional effort to identify and access performance improvements. Effort required to achieve improvements was not linear; as a low performer, Niassa had a low threshold for identifying improvement gains. For Niassa, gains included actionable items like changing the timing of when the distribution team takes a vehicle for maintenance. In the case of Maputo, these efficiency gains have potentially been realized leaving improvements that may be more resource or politically demanding to realize. The root cause analysis delves into the issues that were addressed in the intervention provinces.

Root Cause Analysis We conducted a more detailed analysis of distribution and stocking performance in Gaza and Maputo using the monthly distribution issue charts and resolution reports – some of this data was presented in the pie charts outlining MIS performance in delivery timing. Considered here are first distribution challenges and resolutions followed by stockout issues. Each of these sections considers the issues down to the district level. The final section outlines how these issues were managed and sometimes resolved by distribution teams. These results were verified using base-, mid-, and end-line surveys of key actors in the vaccine distribution network. Furthermore, Niassa completed these surveys to provide contrast of how distribution teams understand and manage these issues in a province not implementing the performance tools developed.

The purpose of this study was to increase the capacity and motivation of the EPI team to deliver vaccines in Mozambique through the implementation of a performance and accountability framework. The performance review tools developed helped the distribution team identify key problems, give them the confidence to state problems often considered out of their hands and motivation to reach out to authorities higher up in order to resolve these issues. Problem resolution was documented in the monthly reports which the study team reviewed. Actions taken by distribution teams to ensure health unit visits and prevent stockouts are included in text boxes at relevant sections to illustrate the processes undertaken and how issues were addressed.

Distribution within 33 days While the baseline survey identified lack of availability of vehicle and fuel as major challenges inhibiting routine distribution, the midline survey clearly showed that non-functioning refrigerators at health units was the major challenge. The midline assessment was conducted in October 2014 and data from resolution reports and distribution issue charts (March 2014- August 2014) were also analyzed to assess the distribution team’s performance and identify root causes hampering vaccine distribution. The study implementation continued until April 2015 and the distribution team has continued to send the reports and charts. The endline survey was conducted in September 2015.

Qualitative analysis of the monthly performance review data from March 2014 to April 2015 confirms the midline review findings that non-functioning refrigerators at health units was the major challenge impeding vaccine distribution. Similar to the midline findings, while the provinces knew that refrigerators were a recurring problem, the field team mentioned other challenges that directly affected their ability to conduct distribution such as on time availability of vehicle and fuel. These challenges, for the most, did not prevent the distribution team from visiting health units but did delay the start time of distribution and led to vaccine stockouts at some health units during the study implementation period. The table below summarizes month-by-month challenges faced by the distribution teams in Gaza and Maputo leading to missed visits to a health unit.

Table 7: Gaza and Maputo calendar of causes leading to missed visits from March 2014 to April 20153

ember

ust

ember

tember rua

il il

rch rch

uary

Ma Apr May Jun Jul Aug Sep Nov Dec Jan Feb Ma Apr Refrigerator Vehicle

Gaza Fuel Per diem

Refrigerator Vehicle Fuel Maputo Per diem

Refrigerators were a recurring issue during the entire study implementation period. Timely availability of vehicle and fuel were issues only in certain months, falling principally mid-year. The section below provides further insight into the challenges faced by the distribution teams that resulted in missed health unit visits and stockouts.

Challenges inhibiting routine distribution in Gaza The monthly resolution reports and distribution issues charts provided insight into the major reasons behind missed health unit visits by the distribution team within the 33 day time period. Distribution teams recorded 61% of missed monthly visits due to non-functioning refrigerators. In addition to this, inability to access roads especially during the wet season in Mozambique accounted for about 26% of the reasons for missed visits. Lack of staff at health units and health units being closed at time of visit accounted for 10% of the reasons for missed visits. The remaining 3% were attributed to other causes that did not present recurring issues and were therefore not investigated further.

The figure below provides a macro view of the situation and illustrates the districts with the highest number of missed visits and reasons for the same.

Figure 7: Reasons for missed visits in Gaza by district (Mar 2014-Apr 2015)

3 Distribution issues charts were submitted by both provinces from April 2014 onwards. Resolution reports provided data for March 2014. No meeting was conducted in either province in October 2014 due to a workshop.

Massingir 20 1 1 1 Chokwe 2 8 2 2 Refrigerator issue Manjacaze 13 1 Staff issue Chibuto 8 4 1 Road access issue Health unit closed Guija 2 5 2 Other Bilene 5 1 1 Xai-Xai 3 4

Health units within Massingir, Manjacaze and Chibuto districts had recurring refrigerator issues in one or two facilities. The other districts experienced a refrigerator issue which was often resolved by the following distribution cycle.

Challenges inhibiting routine distribution in Maputo Similar to the challenges faced in Gaza, non-functioning refrigerators were the leading cause behind distribution teams not visiting health units in Maputo. Non-functioning refrigerators accounted for approximately 83% of missed health unit visits. The figure below provides the reasons and number of missed visits by district.

Figure 8: Reasons for missed visits in Maputo by district (Mar 2014-Apr2015)

Manhica 11 2 1 Boane 12 1 Refrigerator issue Matituine 8 1 Staff issue Marracuene 1 3 Road access issue Health unit closed Moamba 2 Other Magude 2 Namahacha 2

Boane, Matituine and Manhiça districts included health units with recurring refrigerator issues. The other districts experienced a refrigerator issue which was often resolved by the following distribution cycle.

Inability to access roads during the wet season contributed to 13% of the reasons for missed visits. And a small percentage, 2% each, of the reasons for missed visits were attributed to health units being closed at time of visit and lack of staff. The most common reason for lack of staff at time of visits was due to a clinicians participation at various meetings and seminars and with no substitute staff at the health units to take care of business. This issue was additionally attributed to a shortage of staff at the health units.

Three health units in Maputo district were purposefully visited once every two months due to their distance: Ndelane, Santa Maria and Gueveza. Accordingly, distribution teams delivered a two month supply of vaccines to these units. Santa Maria and Ndelane were visited on the same distribution route. None of these health units had a stockout attributed to the distribution team visit plan.

Management of challenges inhibiting distribution A few key issues caused missed visits and other issues caused visit delays. Elaborated here are the issues contributing to missed visits: refrigerator failure, absent health unit staff, road access issues, and closed health units. The next sections elaborates issues of distribution delays.

Refrigerator issue Central level intervention was required to resolve refrigerator issues as budget for repair was handled by MOH and provinces separately. While the field coordinators were aware of their limited authority to influence the outcome, the resolution reports suggested that these performance tools gave them confidence to ask the right questions, correctly identify the problem and take necessary actions. The performance review tools made them more vocal about the issues. For example, from the midline assessment, there was at least one instance where Sensitizing authorities the distribution team reached out to provincial chief, who Problems with refrigerators prompted wrote a letter to MOH requesting assistance to replace the field coordinators to mention the same to the provincial chief, who wrote a various broken refrigerators. Most refrigerators were letter to MOH requesting assistance to obsolete and required urgent replacement. While the replace various non-functioning distribution team could not directly influence the refrigerators. Engaging the provincial replacement of refrigerators they could facilitate the visit of staff in distribution work has also created a technician to temporarily resolve maintenance issues. The awareness amongst them of challenges distribution team in Maputo included a technician during faced at the ground level by field staff. supervision visits who then supported local staff or residents in preventive maintenance.

Other contributing factors to non-functioning refrigerators included theft of solar panels and gas cylinders as well as late utility payments resulting in energy shut off. Distribution teams raised this purchase issue at meetings with district director of Manhiça. As a result, technicians from all health units were encouraged to create a system to pay the local utility on time and prevent energy shut offs affecting otherwise functioning refrigerators. Once this issue was resolve, affected refrigerators came back online. To resolve this issue, technicians at the health unit volunteered to monitor energy consumption and make required utility payments on-time. Stockouts at this facility in the year preceding the study were four times those recorded during the study.

While visits would not take place when refrigerators were not operational, vaccines were left at the district which was then responsible for coordinating a mobile brigade. The health unit staff was then responsible for coordinating their own transportation to retrieve a week’s supply in a cold box.

Staff issue Absence of nurses and technicians to receive vaccines at health units resulted in the visit not being counted, particularly in Gaza province. It is unclear why the staff was missing at time of the visit as the distribution teams informed health units of vaccine delivery prior to making a visit. Information from resolution reports revealed that Gaza technicians’ absence often occurred in the afternoon at health units without maternity wards. The issue was less common at health units with maternity wards and residence staff. After identifying this relationship, field coordinators scheduled visits during times they knew the Changes at health units technicians or nurses would be present. The distribution team made changes within the health units that resolved In Maputo’s Matituine district, absence of staff was missed deliveries. For example, the issue attributed to indiscipline on part of workers who would of lack of staff at certain health units in leave the health unit without knowledge of district officers. Maputo were attributed to the lack of Based on complaints of the field officer and field discipline of workers who left without coordinators, these workers were transferred. informing their supervisors. Based on complaints by the distribution team and Road access issue December to March was the wet season supervisory staff, these workers were in Mozambique. Heavy rains lead to road blockages and transferred. impeded distribution during these months. To account for DLS shows improvement starting in June this issue, the field distribution team deposited the stock of 2014 which resulted in 75 additional vaccines at district headquarters for health units they could children receiving vaccination. not visit. Then either someone from the district would drop off the vaccines or the health unit made arrangements to collect the vaccines. This was a policy established before the introduction of performance tools that made up this study. However, missed visits were still recorded in performance indicators.

Health unit closed Most health units close around 3:30 PM with the exception of some bigger facilities. If delivery was delayed, the health unit would likely be closed. If the staff lived close to the health unit and were advised of a visit beforehand, they met the distribution team and collected the vaccine at the time of delivery. However, health units were often short-staffed and there were instances when the staff would be on an official errand, like visiting the district facility. In these instances, the staff had no alternative but to close the health unit, despite the scheduled delivery.

However, not all health units were closed for legitimate reasons. For example, the issue of lack of staff at certain health units in Maputo were attributed to the lack of discipline of workers who left without informing their supervisors. When this was the case, distribution teams intervened. DLS shows improvement starting in June 2014. In regards to a particular health unit where the distribution team succeeded in transferring an ineffective staff member, eight additional visits to the unit were made over the preceding year. As a likely result, fewer stockouts were reported.

Other There were a handful of health units that did not receive deliveries. Reasons included political instability to missing keys. The annex elaborates reasons for missed visits by health unit.

Management of challenges delaying distribution Other issues did not impede a visit taking place, but a visit occurring after the established 33 day delivery window; principle issues related to vehicle and fuel availability.

Vehicle Baseline interviews identified vehicle availability and maintenance issues as the primary cause of delays in vaccine delivery. Distribution policy specified that a request for a vehicle be made at the province at least 10 days in advance and that the vehicle be taken for maintenance at the close of a distribution cycle (rather than the start). While this approach worked for most of the months, there were instances where vehicle maintenance caused delays in distribution. For example, in July 2014, Maputo distribution team identified the unavailability of transport as the cause of a two week vaccine stockout in several health units in the North Zone. In March 2015, Gaza distribution team reported stockout at two health units in Manjacaze district that they attributed to vehicle maintenance delays.

On further probing of why vehicles were available late in Maputo, distribution teams disclosed that on several occasions vehicles intended for vaccine distribution were used by the DPS for other activities. Also the DPS in Maputo was not following the period for vehicle maintenance in accordance with the distribution plan of the EPI vaccine program. Many times, the DPS did not have the financial ability to follow-up with all the maintenance required. While financial issues were beyond the control of the distribution team, they reached out to DPS and authorities higher-up to sensitize them on this issue so that they can receive vehicles on time in the future.

Fuel Similar to vehicle requests, request for fuel were made ten days in advance. Between Gaza and Maputo, Gaza experienced the most challenges with fuel availability. In May and July 2014, there was a shortage of fuel at DPS. The distribution team addressed this issue by reaching out to the district staff. In absence of assistance from the district, they reached out to Reaching out to partners for assistance the partners. In May, the cause of shortage was due to the The situation of fuel shortage at DPS in Gaza was resolved by the distribution supplier Petromoc not having sufficient stock of fuel and team reaching out to the district, Village Village Reach’s central office provided the distribution team Reach and EPI for assistance. In May and with 540 liters of fuel. In July, EPI had a stock of 360 liters of July 2014, the distribution team received fuel which was used at the beginning of distribution cycle. some stock of fuel through which they While fuel availability required central level resolution, the could start the distribution cycle. distribution team reached out to districts and partners for assistance.

In the endline survey, distribution teams mentioned the support provided by World Vision and Save the Children to provide vaccine transport to support mobile brigades in the districts they operate in all three provinces of the study: Gaza, Maputo, and Niassa.

The table below summarizes the major challenges that distribution teams faced over the last year, along with the scope of action, actions taken by the team and results.

Table 8: Summary of challenges and actions by distribution teams to deliver vaccines

Key Distribution team’s Steps taken Result challenge scope of action Refrigerator Report to district Reported to districts for replacement of Maputo province received 26 new Local technician part refrigerators, batteries and other parts refrigerators of distribution team Took technicians during visits to support Impact is second hand. Distribution Drop-off stock of vaccines local technicians and perform minor team did not have direct control at the district maintenance work over refrigerator and energy Advise on energy Requested an accountability system to purchases. purchase procedure purchase Credelac energy to prevent wastage from electricity outage Vehicle Request 10 days in The distribution team complied Endline impression was that these advance procedures requesting for vehicles 10 requests were made in a more Send vehicle for days in advance and sending for timely manner once monthly maintenance towards maintenance after distribution took resolution reports were required by close of distribution cycle place. VillageReach

Fuel Request 10 days in The distribution team complied with Gaza team reached out to partners advance procedures requesting for fuel 10 days in and district in situations of acute advance fuel shortages and resolved the issue, principally with the assistance of VillageReach Per diem Request 10 days in The distribution team complied with Endline impression was that these advance procedures requesting for per diem 10 requests were made in a more days in advance timely manner once monthly resolution reports were required by VillageReach

Stockouts There were two instances of stockouts in both Gaza and Maputo over the course of the intervention period. In Gaza, stockouts occurred in November 2014 and March 2015. The distribution team attributed this to late communication with the districts. In Maputo, stockouts occurred in July 2014 and were due to vehicles arriving late.4

BCG vaccines showed the highest instances of stockouts. Training health unit staff in quantification Once a vial of BCG vaccine was opened it needed to be The distribution team trained the health administered to children within a six hour time window. unit staff in determining vaccine need and following policies such as FIFO in Many times, vaccines could not be administered within this order to avoid vaccine expiration and/or window due to reasons such as children coming in later. stockouts. This was a practice in the Thus, the unused BCG vaccines should be discarded as per distribution teams starting before this MOH policy and global standards. Insufficient quantities study. delivered to account for this policy was the compounded by

4 One exception was a stockout in May 2014 at a health unit in Namahacha District which was attributed to a malfunctioning power station.

28 this wastage were primary contributing factors to BCG stockouts.

Challenges leading to stockouts in Gaza The two instances of stockouts occurred in November 2014 and March 2015. In both situations, it was delay in communication by the health unit staff to the districts that caused the stockout in which case health units did not report low stock but waited for the distribution team to arrive. In many instances, stockouts were averted when the health units at risk of Flexibility in distribution plan stockouts contacted the district which then sent over The fuel shortage in May 2014 led to a replenishment stock. During the endline survey, a field delay in start of distribution cycle. coordinator identified the issue relating to the arrival of Because of the delays, some health units more people than expected. In one instance, the clinic was in Manjacaze district had very low levels not open for previous visit which resulted in a stockout by of vaccine stock and they requested that the following months distribution. the distribution team visit them first in order to avoid stockouts. Challenges leading to stockouts in Maputo The two instances of stockouts occurred in May and July 2014. The cause of July stockouts was attributed late vehicle arrival, elaborated in the previous section of this report. One exception was Dibinduane health unit which had a stock in May 2015 due to malfunctioning of power substation in the area which caused a power failure.

Results Measured strictly in statistical terms the impact of the intervention piloted in this study had marginal impact on improving stock outs and distribution discipline. A study design with limited measurement power and poor ability to control other confounding Field Officer’s Observations interventions in the control group contributed to this. Some “There has been a lot of change in of the effects of this intervention will result from a change coordinators to improve. We control in management philosophy around problem resolution, the quality of data, we authenticate triage and escalation. These effects may take longer to the control and the performance show immediate performance gains. However, root-cause review – which is more frequent.” analysis and endline assessments suggest that the intervention enhanced the performance by the distribution Maputo Province teams and instilled a culture of problem resolution. Introduction of the resolution report helped the field team correctly identify the root causes of problems and formulate the necessary actions. Root cause analyses allow identification of simple solutions, like resolving payment timing issues. The field distribution team was aware of the problems before the study started but the performance tools gave them a framework to approach the problems, identify corrective actions and take actions within their control or report the issues to authorities higher up, where needed.

Gaza and Maputo showed similarities in their ability to take issues they determined in the baseline assessment to be outside their scope of impact and think through ways to address the issues. This was particularly apparent in Gaza’s efforts to address fuel availability and refrigerator functionality. The Maputo team Medical Chiefs’ Observations resolved various performance issues that resulted from health unit management, such as staff availability and on-time “They use a predefined agenda at payment of utility bills. The endline was used to determine meetings. We discuss some solutions whether these actions related to the performance tools. and register these in a document. For However, during the endline interviews, distribution teams did instance, when mobile brigades not recall taking many of these actions. This is not uncommon wasn’t carried out due to fuel, with root cause analyses which surface big and small issues coordinators requested district to alike. Small issues are generally easy to dismiss. reach out to NGOs, like World Vision.” Niassa also showed improvement in terms of distribution Gaza Province timing discipline. Based on mid- and endline interviews, reasons for this improvement had to do primarily with a more “We have the meeting notes from effective field officer and monthly review of performance data VillageReach and the reports help with VillageReach. These responses confirm the results of the evaluate change between past and other provinces—that data analysis can lead to improved present; Field Coordinators are performance. This cannot be separated with a change in obliged to do a report after management. distribution.” Maputo Province

Endline interviews suggested that monthly meetings were necessary to achieve gains in performance and increase in accountability led the distribution team to be more aware of their own performance in terms of fuel, vehicle, and per diem requests. One field officer suggested that over the last year field coordinators had taken on more of the planning and preparation for vaccine distribution. A large factor of this accountability was the request by VillageReach for the monthly reports and the fact that if they underperformed, VillageReach would bring this to the attention of the DPS to request new coordinators. An instance of this happened in Maputo, where the field officer noticed the field coordinator was not providing sufficient oversight of clinic record keeping and as a result of multiple reminders and reprisals, found a replacement. This same field officer cited that distribution improved as a result of the monthly meetings with field coordinators. While many of the root causes cited previously existed, through monthly meetings they started to be resolved. This discussion suggests that meetings provided an amount of accountability which VillageReach could act upon to determine coordinator performance.

The incentive of USD 4,000 to spend on computers was awarded to Gaza, the intervention province that showed the greatest improvement - 49% over the baseline. While the EPI Chief and field coordinator were happy to receive this prize, they did not suggest this was a motivator for their performance.

The reports themselves have not been used in any meaningful way outside the meetings, for instance to argue for greater resource allocation from senior political staff. When asked whether field coordinators could operate without VillageReach assistance, one field officer said, “technically yes, resource-wise, no.” In other words, her conclusion was that the largest constraint facing the distribution teams was resource related. For instance, the baseline study in Gaza and Maputo suggested that vehicle availability and per diem payments were major inhibitors of timely delivery and the cause of stockouts. Through the course of this study, the primary issue identified to facilities not visited was the functioning of refrigerators. Fuel and per diem availability were likely an issue. These were regularly addressed by VillageReach over the course of the study, and thus had minimal effect on performance measurements.

This study was unable to statistically establish that implementation of this performance management and accountability framework impacted vaccine supply chain performance. While performance improvements were seen, these cannot be attributed to this study using MIS data. The nature of the study was not powered to detect effect in the small sample size.

In conclusion, the impact of this study had marginal effect on the performance of distribution teams in Gaza and Maputo. A few confounding factors include the oversight VillageReach already provides as well as resource when key items like fuel are unavailable. Another is the constraint some of these factors impose on performance. The constraint of low budget for health delivery is likely greater than the constraint imposed by teams that are not engaged in grass-roots management and problem solving.

Conclusion While improvement in some of the performance indicators was not statistically significant, endline interview responses suggested that monthly meetings organized around resolution reports led to a culture of problem resolution. Providing a forum to plan the next month’s distribution cycle and discuss problems with PAV leadership will lead to performance improvements in the long term as this becomes a more routinely utilized management philosophy within the provincial vaccine supply chain organization. The performance tools added insight into data gathered by MIS. Resolution reports documented instances where meetings served to address issues encountered at province, district and health unit levels.

These performance tools are easy to reproduce and could help MOH sustain the performance achieved by VillageReach. In Niassa, we did see improvement without the structure of the resolution report but with the review of collected data with the VillageReach team and change in management. In all three provinces, the field officer played an important role in ensuring that documents were considered and meetings took place. Thus to replicate this study and achieve similar gains, one person in the distribution team would need to be accountable for the execution of regular meetings and documentation. Furthermore, the field officer was accountable to VillageReach. If this is to be implemented without NGO support, national EPI management would have to take interest and provide follow-up.

Table 9: Key considerations of study interventions

Performance Tools (Intervention 1a) Incentive (Intervention 1b) Role Distribution team defines and resolves Distribution team awarded for issues identified during delivery performance Impact to Distribution team resolves issues as they Award provided was less a motivator VillageReach arise than VillageReach oversight of VillageReach and other interested parties performance and payment of distribution have a root cause analysis of distribution services bottlenecks Resources One 2 hour meeting per month USD 4,000 dedicated to computers

Handoff of this study to government actors would require sharing best practices and performance improvements with national EPI management at MOH to gain their buy-in and identify a key staff member who could provide the necessary follow-up. Ideally, this follow-up would be incentivized by that person saving time through a reduction of other reports. In regards to recognition through salary- based incentives, study results would suggest that these are secondary to a structured management approach and performance review.

Given the results of the root cause analyses regarding the high impact of broken refrigerators to missed visits, one avenue to investigate is the feasibility and impact of having a dedicated team of technicians to respond to maintenance needs, possibly as a third party supplier. Other areas of investigation would need to be considered: the effectiveness and constraints of refrigerator technicians currently embedded

32 in the distribution team, such as is the case in Maputo; a cost benefit analysis between repairing a refrigerator and replacing it; the availability of funds for cold chain maintenance; improved preventive maintenance done on a regular basis through distributions; and the feasibility of the government to maintain repair services.

There are cost benefits of implementing an institutionalized root cause analysis. For instance, through the distribution issues chart, the research team concluded that 20% more vaccine deliveries would take place across Gaza and Maputo were refrigerators functioning. The analysis offered by the distribution issues chart and resolution report provides insight to the extent of supply chain problems to inform where, if any, interventions are needed. This process could save NGOs and government from instituting new, capital intensive systems that do not correctly address the problem.

Annex

Annex 1: Reasons for missed visits to health units in Gaza Figure 6 below provides a micro view of the situation. It illustrates the reasons and number of missed visits at individual health units. Of the 97 health units in Gaza province, a total of 27 health units (28%) were not visited at some point during the study implementation period.

Figure 9: Reasons for missed visits to health units in Gaza (Mar 2014- Apr 2015)

70 health units regularly visited Machamba 10 1 1 1 Macaringue 9 Alto Changane 7 Cucuine 5 Nalazi 2 2 1 Chimbembe 3 1 Cumba 4 Manhique 4 Chiaquelane 2 1 Chilaulene 1 2 Macasselane 2 1 Refrigerator issue Mamonho 3 Staff issue Betula 2 Road access issue Vila do Milenio 1 1 Health unit closed Changanine 2 Other Machua 1 1 Maqueze 2 Ndambine2000 2 Tlawene 1 1 Tuane 1 1 Zimilene 2 Cubo 1 Conhane 1 Massavasse 1 Mazivila 1 Muianga 1 Messano 1

As seen from Figure 6 above, at health units such as Machamba and Macaringue in Massingir district, non-functioning refrigerators have been the reason for missed visits for over half of the study implementation period. In Macaringue, refrigerators were not functioning in 9 of 13 months and in

Machamba the number was 10. Machamba health unit has not been visited during the entire study implementation period as in the other three months visits were missed due to road access issues, lack of staff at health unit and health unit being closed at time of visit. Alto Changane health unit in Chibuto district also shows a high number of missed visits due to non-functioning refrigerators.

Inability to access roads due to heavy rains prevented visits at several health units, in particular, at Cumba health unit in Chokwe district and Chimbembe health unit in Guija district. The ‘other’ category of reasons for missed visited includes the following over which the distribution team had no control:

 Nalazi and Chimbembe health units were not visited in April 2015 as there was situation of political instability in Guija district and possibility of armed men in these areas.  Conhane health unit could not be visited in February 2015 as the EPI room was locked and the keys were missing.  Chiaquelane health unit was without power at time of visit in February 2015.

Annex 2: Reasons for missed visits to health units in Maputo Figure 8 below provides a breakdown of the reasons for missed health unit visits by individual health units. Of the 82 health units in Maputo, a total of 15 health units were not visited at some point during the project implementation period.

Figure 10: Reasons for missed visits to health units in Maputo (Mar 2014-Apr 2015)

67 health units regularly visited Josina Machel 11 Mirona 10 1 Mabilibili 6 Machubo 1 3 Calanga 2 Chicutso 2 Refrigerator issue Kululua 2 Staff issue 3 De Fevereiro 1 Road access issue Agua de Maputo 1 Health unit closed Chanculo 1 Other Gueveza 1 Mahulane 1 Manhangane 1 Ndelane 1 Radio Marconi 1

Health units like Mabilibili in Matituine district, Mirona in Manhiça district and Josina Machel in Boane district had no refrigerators for almost the entire duration of the study implementation period. The situation at these three health units were unique and details are provided below:

 Josina Machel – There was a theft of solar panels at Josina Machel before start of study implementation. Many health units in Mozambique, especially in rural areas, operate off-grid and depend upon alternate sources of fuel supplies to support refrigeration, sterilization ad lighting. At Josina Machel, solar panels are used. And theft of solar panels is also common. In absence of a refrigerator, vaccines are dropped by the mobile brigades run by the districts.  Mabilibili – This health unit had a functioning refrigerator but it was transferred to Catuane health unit which had a larger population to vaccinate and a non-functioning refrigerator. Vaccines at Mabilibili were dropped off by the district’s mobile brigades and the health unit received a refrigerator in April 2015.  Mirona – This health unit was completely destroyed due to a storm/heavy winds approximately 3 years ago. At present, vaccines are dropped off by district’s mobile brigades.

Refrigerators break down either due to lack of electricity within the health unit due to theft of solar panels or theft/lack of gas cylinders and delay in energy stock purchases. For example, solar panels at Josina Machel health unit were stolen 3 years ago. Mahulane and Kulula health units also had solar panels stolen which resulted in non-functioning refrigerators. Similar to the solar panel theft, the gas cylinder at Chanculo health unit was stolen that resulted in a non-functioning refrigerator in April 2014. This situation prevailed for a long time and after insistence by the Village Reach country team, a new gas cylinder was allocated by the district director (which was for his personal use). Non-functioning refrigerators have also been attributed to a poor system of buying and paying for Credelac energy. Health units were purchasing energy from Credelac only when they would fully extinguish the existing stock. There was no system set-up to purchase a new stock of energy when the current stock hit a certain level in the health unit. For example, the refrigerator stopped working at 3 de Fevereiro health unit in Dec 2014 as there was no electricity due to lack of energy. Additionally, malfunctioning of transformer substations within the area the health unit is located has also caused refrigerators to break down; Like Manhagane health unit in April 2014 and Dibinduane health unit in May 2014. Stockout at Dibinduane health unit has been attributed to this reason.