Field Evaluation of Freeze- Preventive Vaccine Carriers in Nepal This report was written by PATH and supported in whole or part by a grant from the Bill & Melinda Gates Foundation. The views expressed herein are solely those of the authors and do not necessarily reflect the views of the foundation.

Contact information Pat Lennon Portfolio Leader of Supply Systems and Equipment Medical Devices and Health Technologies [email protected]

Mailing Address PO Box 900922 Seattle, WA 98109 USA Street Address 2201 Westlake Avenue Suite 200 Seattle, WA 98121 USA www.path.org

© 2020 PATH. All rights reserved. Photos in report are credited to PATH. June 2020 Contents

ABBREVIATIONS ...... V EXECUTIVE SUMMARY ...... VI 1 INTRODUCTION ...... 1 2 FREEZE-PREVENTIVE VACCINE CARRIERS ...... 1 2.1 Equipment description ...... 2 3 FIELD EVALUATION IN NEPAL ...... 3 3.1 Objectives ...... 4 3.2 Implementation partner ...... 4 3.3 Approvals ...... 4 4 METHODOLOGY ...... 4 4.1 Pilot design ...... 4 4.1.1 Sampling methodology ...... 4 4.1.2 Trainings ...... 6 4.1.3 Field implementation ...... 6 4.2 Vaccine transport system in Nepal ...... 7 4.3 Data collection ...... 7 4.3.1 Evaluation of performance ...... 7 4.3.2 Evaluation of acceptability ...... 8 4.3.3 Evaluation of health system impact ...... 8 5 RESULTS ...... 8 5.1 Performance of three freeze-preventive vaccine carriers ...... 8 5.1.1 Mean kinetic temperature ...... 12 5.1.2 Excursions ...... 14 5.1.3 Ambient temperature analysis ...... 15 5.1.4 Vaccine vial monitor life loss ...... 17 5.1.5 Cool-down rates and times ...... 18 5.1.6 Freezer performance...... 20 5.2 Acceptability ...... 22 5.2.1 Feedback on the freeze-preventive vaccine carrier ...... 22 5.3 Systems fit and costing analysis ...... 22 5.3.1 Systems fit ...... 22

iii 5.3.2 Costing data ...... 23 5.4 Electronic temperature monitors ...... 26 5.4.1 LogTag ...... 26 5.4.2 Parsyl ...... 26 6 DISCUSSION ...... 26 6.1 Context of use ...... 26 6.2 Carrier performance ...... 27 6.2.1 Mean kinetic temperature ...... 27 6.2.2 Excursions ...... 27 6.2.3 Vaccine vial monitor life loss ...... 27 6.2.4 Ambient temperature analysis ...... 27 6.2.5 Cool-down rates and times ...... 27 6.2.6 Freezer performance...... 28 6.3 Limitations and data gaps ...... 28 7 CONCLUSION AND RECOMMENDATIONS ...... 28 8 REFERENCES ...... 30

iv Abbreviations

AHW auxiliary health worker ANM auxiliary nurse midwife BCG bacillus Calmette–Guérin BPKIHS B.P. Koirala Institute of Health Sciences CCH cold chain handler CCP cold chain point DTP diphtheria-tetanus-pertussis DVS district vaccine store EPI Expanded Programme on Immunization FPVC freeze-preventive vaccine carrier HepB hepatitis B HP health post JE Japanese encephalitis MKT mean kinetic temperature MOH Ministry of Health MR measles-rubella NA not applicable OPV oral poliovirus vaccine PHC primary health care PCV pneumococcal conjugate vaccine PQS Performance, Quality and Safety (World Health Organization) PUF polyurethane foam SVC standard vaccine carrier TD tetanus-diphtheria UNICEF United Nations Children’s Fund VVM vaccine vial monitor WHO World Health Organization

v Executive summary

Preventing vaccine freezing is one of the biggest barriers in vaccine management. Toward addressing this challenge, B.P. Koirala Institute of Health Sciences in Dharan, Nepal, in collaboration with PATH, conducted a field evaluation of freeze-preventive vaccine carriers (FPVCs) in 24 health facilities located across one hilly district and one plains district of eastern Nepal. The primary study objectives were to evaluate the performance, acceptability, and fit of the FPVCs within the existing immunization system. The study was carried out in three phases. The AOV International FPVC was evaluated in phase 1 (May to August 2018) under simulated use. Phase 1a was conducted from May to July 2019, during which two additional carriers (Leff Trade and Blowkings) were evaluated, also under simulated use. During phase 2 (November to December 2019), all three carriers were studied under actual-use conditions. This report details the results of phase 2. The evaluation design had half the FPVCs contain conditioned ice packs and the other half unconditioned ice packs. The FPVCs were prepared by cold chain handlers and transported by health workers to outreach vaccination sessions carrying the vaccines. Quantitative and qualitative data were collected from health workers from all 24 health facilities, logbooks, and electronic temperature monitors in all three types of vaccine carriers and in some deep freezers/refrigerators. Study results indicate these long-range FPVCs successfully prevented freezing in outreach vaccination settings and during transport, with only one site registering temperature readings below 0°C. That site had very low ambient temperatures, below the minimum rated ambient temperature for freeze prevention specified by the World Health Organization’s Performance, Quality and Safety team. Most health posts had low ambient temperatures during this phase due to winter weather, and the vaccines remained in the optimal temperature zone of 0°C to 10°C for 75% of the time during the recorded sessions. The average cool-down times were less than 4 hours across all vaccine carrier brands and ice-pack states, and more than 90% of carriers that started out above 10°C cooled to below 10°C by the time the vaccination session ended. The average vaccine vial monitor life loss was just 0.25% per vaccination session. By preventing freezing and keeping vaccines at the optimal temperature, the vaccine carriers used in this study averted a potential loss in potency of freeze-sensitive vaccines. An average of $97 worth of vaccines were taken to each session in the standard vaccine carriers, of which $70 worth were freeze- sensitive vaccines. Thus, any one freezing incident during an outreach session could expose about $70 worth of vaccines to freezing temperatures and could potentially damage these vaccines. Freezing was a risk in standard vaccine carriers, occurring in 3% of monitored sessions during the phase 1 data collection, based on LogTag temperature recorder data. This potential loss of vaccine potency could have been prevented by using FPVCs. The benefit–cost calculations showed that an FPVC would be a good value for money when comparing its benefit of limiting freeze-sensitive vaccines from being exposed to freezing temperatures compared to its annualized purchase price. The study findings point to the importance of understanding the intended environment of use when evaluating new technologies. Global immunization supply chain entities should consider including recommendations for countries procuring new freezers to address potential consequences by procuring FPVCs or, where appropriate, consider use of cool water packs instead of ice packs. Although vaccine vial monitors provide health workers with a well-established protocol to know whether vaccines have been exposed to high temperatures, immunization programs will have to weigh the risks of vaccines freezing versus vaccines being above the recommended 0°C to 10°C temperature storage range potentially for several hours as the FPVC cools to below 10°C.

vi 1 Introduction

World Health Organization (WHO) guidelines recommend storage conditions for all routine immunization vaccines to ensure their potency is maintained through their life cycle. However, a poorly functioning cold chain may expose vaccines to both freezing and high temperatures. Vaccine vial monitors (VVMs) were introduced on all vaccines procured by the United Nations Children’s Fund (UNICEF) more than 20 years ago to provide health workers with a means of verifying vaccine quality following heat exposure over time. However, vaccine freezing in passive cooling devices remains one of the most pressing obstacles to effective vaccine management.1 Unlike heat exposure, where the effect on potency takes from 2 to 30 days at 37°C, freezing can happen in an instant for alum adjuvant, freeze-sensitive vaccines. Freezing can irreversibly damage vaccines adsorbed onto aluminum salt adjuvants, such as diphtheria, tetanus, pertussis, hepatitis B (HepB), and Haemophilus influenzae type b, reducing vaccine potency and compromising protective immunogenicity in recipients. Freezing is not easy to identify, and anecdotal reports indicate the shake test2–4 is not routinely implemented when freezing is suspected. Health workers and managers, therefore, are not aware of the frequency of freezing nor of its negative consequences on vaccine potency and on adverse events following vaccination.5 The importance of protecting vaccines from freezing has significant cost implications, as countries continue to introduce vaccines that are more expensive and sensitive to freezing, such as pentavalent (DTP-HepB-Hib), pneumococcal conjugate, human papillomavirus, influenza, and others. To prevent vaccine freezing, WHO recommends proper conditioning of frozen ice packs to greater than 0°C or use of cool water packs.a When using cool water packs, studies have shown temperature profiles that remain in the safe zone (greater than 0°C to below 20°C) for 6 to 7 hours.6 To conserve space in refrigerators and have a longer cold life, many contexts prefer to use conditioned ice packs instead. A number of studies conducted in low-, middle-, and high-income countries have reported the exposure of vaccines to freezing temperatures, during both the storage and transport segments of the vaccine cold chain.7 These studies demonstrate that the majority of vaccines are exposed to freezing temperatures during transport in standard cold boxes and vaccine carriers to lower levels of the health system, showing that vaccines placed with ice packs inside insulated standard vaccine carriers (SVCs) can freeze.6,7 Studies have also indicated that the WHO-recommended practice of conditioning ice packs is not routinely followed.5,8 Data from many countries show that vaccines continue to freeze at all levels of the health system.6 As noted in the WHO Vaccine Management Handbook on passive containers, “there is no single recipe for transporting vaccines safely in passive containers. Every method carries risks, and these risks have to be identified, understood, and managed in the specific operational context.”9

2 Freeze-preventive vaccine carriers

To address freezing during transport in standard cold boxes and vaccine carriers to lower levels of the health system, the WHO Performance, Quality and Safety (PQS) team, under the WHO Department of Essential Medicines and Health Products, developed new performance specifications for cold boxes and vaccine carriers that include freeze prevention.10 In 2015, UNICEF issued a request for information for freeze-preventive vaccine carriers (FPVCs). PATH refined a freeze-preventive liner concept using only foam and water sachets capable of freeze prevention a. Where confusion between fully frozen ice packs and conditioned ice packs may result, this report occasionally refers to fully frozen ice packs as stony hard.

1 in laboratory tests using –25°C ice packs at 10°C ambient temperature. With technical assistance from PATH and others, several manufacturers of PQS-prequalified vaccine carriers have adapted their designs to prevent vaccines from freezing in laboratory settings. In early 2016, PATH published the solution (including water or use of other thermal buffer materials) through Research Disclosure to ensure open access to the technology.11

2.1 Equipment description

The FPVC uses a liner containing foam and a thin layer of water as a phase-change material to separate the ice packs from the vaccines. This liner buffers the vaccines from direct exposure to frozen ice packs while the heat from the water sachets conditions/ warms the frozen water packs. In preliminary laboratory testing, these carriers met WHO performance specifications for freeze prevention by maintaining temperatures in the vaccine compartment at greater than 0°C (±0.5°C) and less than 10°C at ambient temperatures of 43°C and 15°C, while maintaining “cold life” requirements. The accepted temperature range for passively cooled devices is different from actively cooled refrigerators, for which the storage range is 2°C to 8°C. Cold life for FPVCs is 15 hours for short-range vaccine carriers and 30 hours for long-range vaccine carriers. In the laboratory, PATH compared the coldest locations in SVCs and FPVCs during the first few hours using unconditioned ice packs in both carriers (see Figure 1).

Figure 1. Comparing standard carriers with freeze-preventive carriers in the laboratory.

2 To prevent freezing from the rapid drop in temperature that occurs when a not fully conditioned ice pack is placed in an SVC, FPVCs are intended to take longer to cool the vaccine storage compartment to the acceptable temperature range. To address this need for a more gradual cool-down rate, WHO PQS added this thermal requirement to the new performance specification: all locations in the vaccine compartment are cooled to 10°C or cooler within 8 hours of placing ice packs in the carrier at 43°C ambient temperature.10 This requirement was established acknowledging that a relatively short period of heat exposure at the end of the cold chain would have limited overall effect on total vaccine life as validated by a VVM. Per WHO PQS specifications for freeze prevention,10 FPVCs must:

• Have a minimum vaccine storage capacity of 1 liter for long-range use and 0.5 liter for short-range use.

• Weigh no more than 8 kg when fully loaded with vaccines and ice packs.

• Allow ice packs that have expanded when frozen to still fit within the ice-pack compartment. The primary advantage of FPVCs is in preserving vaccine quality and preventing freezing even when fully frozen, unconditioned ice packs are used in the carrier. FPVCs are also expected to simplify logistics by reducing the amount of time required by logisticians and health workers to prepare vaccine carriers for use in outreach vaccination sessions. Additional advantages of these designs could include reduction of training burden when compared to training logistics staff to use carriers that use cool or conditioned water packs. The AOV International, Leff Trade, and Blowkings FPVCs (see Figure 2) used in this evaluation were prequalified by WHO PQS.

Figure 2. Freeze-preventive vaccine carriers used in the study.

AOV Leff Trade Blowkings

3 Field evaluation in Nepal Nepal was selected as the location of the field evaluation because of the commitment of the Ministry of Health (MOH) to preventing freezing during vaccine transport, as well as the variety of climatic and environmental conditions in the country. In phase 1 (May to August 2018), we evaluated the AOV FPVC in simulated use at 24 health facilities across one hilly district and one plains district of eastern Nepal (12 facilities per district). In phase 1a (May to July 2019), we evaluated Leff Trade and Blowkings FPVCs under the same conditions as phase 1. In phase 2 (November to December 2019), we evaluated the three FPVCs (AOV, Blowkings, and Leff Trade) at the same locations as phase 1 and phase 1a.

3 3.1 Objectives

The primary objectives of the evaluation were to provide information to the MOH on the cold chain system in eastern Nepal and assist them in determining the potential utility of FPVCs. Specific objectives were to understand, based on conditions of simulated and actual use, whether FPVCs: 1. Perform according to the WHO PQS specifications. 2. Are acceptable to end users. 3. Have any significant impacts on the health system.

3.2 Implementation partner

B.P. Koirala Institute of Health Sciences (BPKIHS) was chosen as the implementing study partner because of their technical reputation and experience in conducting field studies in collaboration with the MOH, as well as PATH’s past experience working with the institute.

3.3 Approvals

PATH obtained ethics approvals from the BPKIHS Institutional Review Committee and the Nepal Health Research Council for the overall study. The MOH Family Health Division granted approval to BPKIHS and PATH to conduct the field evaluation in Dhankuta and Sunsari Districts. Phase 1 and phase 1a results (simulated use of vaccine carriers) were shared with the MOH and the Nepal Immunization Advisory Committee to obtain approval to progress to phase 2 (actual use of vaccine carriers at the field level). Approval was granted in October 2019, and Phase 2 was implemented in the project intervention facilities/districts from November to December 2019.

4 Methodology

This section describes the design of phase 2, provides background on vaccine sessions in Nepal, and outlines the data collected and analyzed during the study.

4.1 Pilot design

Training for health workers was organized before the phase 2 implementation. Unlike phase 1 and phase 1a, in this phase actual routine immunization vaccines were loaded and transported in the FPVCs to outreach vaccination sessions. The testing and data collection occurred in two monthly rounds corresponding to the monthly immunization schedule used in the study regions. This report shares the findings from phase 2.

4.1.1 Sampling methodology

The phase 2 study was conducted in the same pilot intervention areas as phases 1 and 1a (refer to phase 1 and phase 1a data analysis reports). Two districts, Sunsari (plains region) and Dhankuta (hilly region), were identified in conjunction with national and district MOH authorities. Three blocks were selected in Sunsari District and two blocks were selected in Dhankuta District, with the aim to include six health posts under each primary health care (PHC) center at which to pilot test FPVCs in outreach

4 sessions. The second PHC center selected in Sunsari District failed to meet the criterion of having six health posts; therefore, an additional block was selected to maintain uniformity of the two districts. Criteria for site selection included:

• The plains and hilly regions of Sunsari and Dhankuta Districts, respectively.

• Six health posts under each PHC center.

• Use of an alternate delivery mechanism for vaccine transport to health posts. The selected health posts, their corresponding PHC centers, and their regions are shown in Figure 3.

Figure 3. Study locations and ice-pack conditions.

Nepal

Sunsari Dhankuta (plains region) (hilly region)

PHC center PHC center PHC center PHC center HP Itahari Harinagara Sitaganj Dandabazar Pakhribas

3 HPs 3 AHWs 3 HPs 3 HPs using fully 3 HPs using with fully 3 HPs using using fully 3 HPs using using fully 3 HPs using frozen conditioned frozen conditioned frozen conditioned frozen conditioned (hard) ice ice packs (hard) ice ice packs (hard) ice ice packs (hard) ice ice packs packs packs packs packs

Itahari Pakali Harinagara Sitaganj Rajarani Khuwafak Sanne Phalate

Hashpos Chandbel Rasi Amahibell Maha- Faxiv Chumbang Golikhark haaa a a bharat gg

Khanar Ekamba Madhya- Chimdi Danda- Bhedetar Pakhribas Muga horsahi bazar

Abbreviations: AHW, auxiliary health worker; HP, health post; PHC, primary health care.

LogTag® TRIX-8 temperature recorders (LogTag Recorders) were placed inside (internal) and outside (ambient) the carriers at all 24 sites. Trek 1.1 recorders (Parsyl, Inc.) were also placed inside the carriers. A LogTag or Parsyl device was placed in 17 out of 24 sites in freezers (all except Faxiv, Khuwafak, Golikharka, Madhya Rasi, Muga, Phalate, and Rajarani).

All 24 health posts in Dhankuta and Sunsari that participated in the phase 1 FPVC field evaluation were included in phase 2 data collection. Eight Blowkings (four stony hard and four conditioned), nine Leff Trade (five stony hard and four conditioned), and seven AOV FPVCs (three stony hard and four conditioned) were used by health workers from across the 24 health posts. Though initial sampling stipulated health facilities receive Blowkings FPVCs, eight received Leff Trade and eight received AOV.

5 Health workers at Chumbang health post decided to use the Leff Trade FPVC instead of an AOV because they felt it was easier to carry through the difficult hilly terrain. See Table 1.

Table 1. Phase 2 sampling matrix.

FPVC manufacturer Ice-pack state Sites

AOV Conditioned 4 sites total: Golikharka, Madhya Rasi, Muga, Rasi

AOV Stony hard 3 sites total: Amahibela, Chimdi, Pakhribas

Blowkings Conditioned 4 sites total: Ekamba, Chandbela, Pakali, Phalate

Blowkings Stony hard 4 sites total: Dandabazar, Rajarani, Sitaganj, Sanne

Leff Trade Conditioned 4 sites total: Bhedetar, Faxiv, Khuwafak, Harinagara

Leff Trade Stony hard 5 sites total: Chumbang*, Mahabharat, Khanar, Hasposha, Itahari

Abbreviation: FPVC, freeze-preventive vaccine carrier. *Health workers at Chumbang used a Leff Trade carrier instead of an AOV carrier because the Leff Trade was easier to carry.

4.1.2 Trainings

Before initiating phase 2, all staff involved in routine vaccination at the 24 health facilities were oriented to the objectives of the pilot and trained according to the protocol. This training included cold chain handlers (CCHs), auxiliary health workers (AHWs), and auxiliary nurse midwives (ANMs). Government immunization staff—including district immunization officers, block and district medical officers, and logisticians at the two districts included in the pilot—were also oriented in all aspects of the project implementation. The training was provided in the Nepali or Hindi language. The trainings were conducted at the block level with 108 participants attending across both districts: 29 participants attended the training organized in Itahari on November 7, 2019; 32 attended in Harinagara and Sitaganj on November 7, 2019; 19 in Pakhribas on November 6, 2019; and 28 in Dandabazar on November 10, 2019.

4.1.3 Field implementation

During phase 2, real vaccines were loaded by the AHW/ANM for each health post and transported in the FPVC. CCHs at half the facilities in both districts were instructed to condition the ice packs (leave the packs outside the freezer for 30 to 60 minutes [determined based on room temperature] or until the ice was slushy when shaken), and CCHs at the other half of the facilities were instructed to not condition the ice packs. The FPVCs were transported to the field containing the vaccines for actual use on routine vaccination session days to understand the performance of the carriers as per WHO PQS specifications. The AHWs/ANMs also ensured the placement of temperature monitoring devices inside and outside the FPVCs. The protocol specified that electronic temperature monitoring devices be placed in the freezers to capture the temperatures to which the ice packs were exposed. The health workers used the FPVCs as per MOH guidelines for vaccine storage and conducting outreach sessions (except for leaving some ice packs conditioned in the assigned sites).

6 4.2 Vaccine transport system in Nepal

Outreach vaccination sessions in Nepal are conducted once a month for a continuous period of 3 to 7 days, depending on the number of session sites to which the respective health post caters. Distribution of vaccines from the regional medical store to the health post also occurs monthly. Figure 4 depicts the flow of vaccines from the regional medical store to the outreach sessions.

Figure 4. Vaccine delivery system in Nepal.

Abbreviations: AHW, auxiliary health worker; ANM, auxiliary nurse midwife; CCP, cold chain point; DVS, district vaccine store; HP, health post.

4.3 Data collection

Both quantitative and qualitative methods were used to collect data during phase 2, as described below.

4.3.1 Evaluation of performance

The efficiency of the FPVC depends on its ability to prevent vaccine freezing and maintain internal temperatures within an acceptable range: 0°C to 10°C. The key performance parameters measured include mean internal temperature and mean kinetic temperature (MKT), cool-down time and rate, temperature excursions, the effect of ambient temperature on internal temperature, and percentage VVM life loss. In addition, the mean and range of temperatures for some of the freezers were measured to understand the environment in which the ice packs were being frozen.

7 4.3.1.1 Temperature WHO PQS-prequalified LogTags were used to collect temperature data every 10 minutes from the FPVCs. Both internal vaccine carrier temperatures and ambient temperatures were collected to determine conformity with PQS specifications under varying ambient temperature conditions. To capture the temperatures of the ice packs, LogTag and Parsyl devices were also placed in the deep freezers or refrigerators used to freeze the ice packs. In addition to recording temperatures with the LogTag, secondary temperature data and data on humidity were collected with the Parsyl Trek 1.1, which is commercially available for the food industry but had not been used in vaccine transport, nor is it PQS prequalified.

4.3.2 Evaluation of acceptability

The acceptability of FPVCs was assessed based on qualitative feedback from health workers (AHWs, ANMs), and CCHs. Qualitative feedback was collected among all health workers from the 24 health facilities in the study using an in-depth interview guide. The discussions focused on how well the FPVCs performed vaccine delivery and transportation during outreach.

4.3.3 Evaluation of health system impact

Both qualitative and quantitative data on the health system were collected from health workers in all 24 facilities. Following an in-depth interview guide, qualitative feedback was gathered from ANMs, AHWs, and CCHs at the end of the pilot testing about the use of FPVCs during outreach sessions, including storage capacity, freeze prevention, size and weight, staffing, and training-related aspects. Similar feedback was collected from medical officers and district immunization officers. Quantitative data on the health system and cost impact were collected by the block monitors using a baseline data collection tool during phase 1a . Data were collected from health workers, CCHs, and medical officers. Separate tools were developed for each health facility level and outreach session. The key indicators collected included information on staff involved in vaccination, mode of travel, and related costs when traveling to vaccination sessions. Data from a pilot logbook also provided information on the quantities of vaccines taken to each session.

5 Results

This section presents the quantitative and qualitative results from the study. It covers the context of use, the thermal performance data from the carriers and freezers, the acceptability feedback from end users, the systems fit and costing analysis, and results from using electronic temperature loggers.

5.1 Performance of three freeze-preventive vaccine carriers

Temperature data were collected for 24 FPVCs in five blocks (24 health posts). The dips in the graph represent the start of the vaccine compartment cooling down, after the ice packs were placed in the carriers. The time of these dips was used in calculating the session start time, and the end time noted in the session monitoring log was used to calculate the session end time. The cool-down patterns show the effect of the FPVC’s barrier mechanism intended to slow down the rapid drop in temperature experienced by the SVC as soon as ice packs are introduced. It is at this point that freezing most often occurs (see Figure 5).

8 Figure 5. Example temperatures during 3 normal days of outreach in a row.

Abbreviations: FPVC, freeze-preventive vaccine carrier; SVC, standard vaccine carrier.

Figure 6 shows a typical graph of when ice packs were placed in the carriers the night before the vaccination session, which occurred in 9 of the 12 sites in Dhankuta and 1 of the 12 sites in Sunsari. Travel between the cold chain points and the health posts in hilly Dhankuta was challenging. The temperatures remained low for most of the outreach because the ice packs were not removed from the carriers between sessions.

Figure 6. Comparison of cool-down times for a standard and a freeze-preventive vaccine carrier during outreach session 1, day 1.

Abbreviations: FPVC, freeze-preventive vaccine carrier; SVC, standard vaccine carrier.

9 Table 2 shows the sample size (number of data points collected) for the phase 2 temperature analysis. Data were collected across five health posts and taken at 10-minute intervals for a minimum of 70 hours (Harinagara, 425 data points) and a maximum of 181 hours (Dandabazar, 1,088 data points).

Table 2. Phase 2 temperature data collected for freeze-preventive vaccine carriers.Data taken at 10-minute intervals, so each sample accounts for 10 minutes of time during an outreach period.

District Block Number of freeze-preventive Sample size for temperature vaccine carriers tested analysis

Sunsari (plains) Itahari 6 1,014

Harinagara 3 425

Sitaganj 3 687

Dhankuta (hilly) Dandabazar 6 1,088

Pakhribas 6 542

Total sample size 24 3,756 (626 hours)

A total of 153 sessions were planned for phase 2, but only 141 were conducted (see Table 3 and Table 4). In Itahari, one session in November and three sessions in December were not organized because there were too few beneficiaries at the outreach sites. Similarly, in Dandabazar five sessions in November and two in December were not held, and one session in Pakhribas was not held, all due to few target beneficiaries.

Table 3. Number of sessions planned and held in the health facilities in phase 2 in the project intervention areas.

District Block November 2019 December 2019

Sessions Sessions Sessions Sessions Sessions Sessions planned held not held planned held not held

Sunsari Itahari 23 22 1 23 20 3 (plains) Harinagara NA NA NA 15 15 0

Sitaganj 14 14 0 14 14 0

Dhankuta Dandabazar 20 15 5 20 18 2 (hilly) Pakhribas NA NA NA 24 23 1

Abbreviation: NA, not applicable.

Because they used the Leff Trade vaccine carrier during an earlier phase, health workers at Chumbang health post preferred to use the Leff Trade carrier instead of the AOV carrier allotted during phase 2. Data were analyzed for the number of sessions being organized by carrier type (see Table 4).

10 Table 4. Number of sessions planned and held in the health facilities in phase 2 by freeze-preventive carrier type.

FPVC type November 2019 December 2019 Total

Sessions Sessions Sessions Sessions Sessions Sessions planned held not held planned held not held

AOV 9 9 0 31 31 0 40

Leff Trade 27 22 5 31 27 4 49

Blowkings 21 20 1 34 32 2 52

Total 57 51 6 96 90 6 141

Abbreviation: FPVC, freeze-preventive vaccine carrier.

The tables below show the sample size and number of carriers tested, by manufacturer. Temperature data were collected by a LogTag device, and each sample represents a temperature measurement taken at a 10-minute interval. The LogTags continuously collected temperature readings at 10-minute intervals, but for data analysis purposes, temperature readings were taken during recorded sessions only. The temperature data also rely on the logbook session start times and health post return times, as recorded in the logbooks by health workers.

Table 5. Phase 2 temperature data collected for the Leff Trade freeze-preventive vaccine carrier.

District Block Number of Leff Trade carriers tested Sample size for temperature analysis

Sunsari Itahari 3 474 (plains) Harinagara 1 99

Dhankuta Dandabazar 4 790 (hilly) Pakhribas 1 103

Total sample size 1,466 (244 hours)

Table 6. Phase 2 temperature data collected for the Blowkings freeze-preventive vaccine carrier.

District Block Number of Blowkings carriers tested Sample size for temperature analysis

Sunsari Itahari 3 540 (plains) Sitaganj 1 249

Dhankuta Dandabazar 2 298 (hilly) Pakhribas 2 195

Total sample size 1,282 (213 hours)

11 Table 7. Phase 2 temperature data collected for the AOV freeze-preventive vaccine carrier.

District Blocks Number of AOV carriers tested Sample size for temperature analysis

Sunsari Itahari 0 0 (plains) Sitaganj 4 764

Dhankuta Dandabazar 0 0 (hilly) Pakhribas 3 244

Total sample size 1,008 (169 hours)

5.1.1 Mean kinetic temperature

Vaccines experienced a range of temperatures inside the carriers over time. Calculating MKT provides a single temperature value that would affect the potency of vaccine in the same way as if it were left at that temperature for the same amount of time (as the range of temperatures). Although taking a simple average of temperature values often provides a similar result to MKT (see Table 8), MKT is a more rigorous metric for accounting for the effect of higher temperatures on the thermal degradation of vaccines in carriers. MKT was influenced by the length of time the ice packs were inside the carriers. The longer the ice packs remained in the carrier after cool-down, the lower the MKT. In hilly Dhankuta District, 9 of 12 health posts placed their ice packs in the carriers the day before some or all of the outreach sessions, so by the time the session began, the carriers were already cool. Table 8 shows the mean temperature and the MKT for each block. The MKT is slightly higher than the mean temperature, reflecting the effect of higher temperatures on raising the MKT. The blocks in Dhankuta had the lowest mean temperatures, while the highest MKTs were found in Itahari and Sitaganj.

Table 8. Phase 2 comparison of arithmetic mean internal temperature and mean kinetic temperature by block in freeze-preventive carriers.

District Block Arithmetic mean (°C) Mean kinetic temperature value (°C)

Sunsari Itahari 9.6 10.4

Sunsari Harinagara 8.2 8.6

Sunsari Sitaganj 9.8 10.4

Dhankuta Dandabazar 4.6 4.8

Dhankuta Pakhribas 3.5 3.6

Table 9 compares the mean internal temperature and MKT for the stony hard ice packs to conditioned ice packs across all blocks and carrier types. There was no significant difference in MKT between the sites that were assigned stony hard ice packs and the sites that were assigned conditioned ice packs.

12 Table 9. Mean internal temperature and mean kinetic temperature for fully frozen and conditioned ice packs.

Assigned ice-pack state Arithmetic mean (°C) Mean kinetic temperature value (°C)

Stony cold 6.7 7.2

Conditioned 6.6 7.0

The following three figures show the mean internal temperature and MKT at each health post, grouped by carrier type. Four health facilities were assigned stony hard ice packs (see Table 1 for the sampling matrix), and the remaining four were assigned conditioned ice packs. Not having a freezer on-site likely prevented some sites from using stony hard ice packs, and warmer than expected freezer temperatures may have preconditioned some of the ice packs. However, the data were analyzed for each vaccine carrier type assuming that the category assigned to each site was indicative of the true state of the ice packs. For blocks with stony hard ice packs, the highest mean temperature was recorded in Dandabazar (6.8°C), followed by Sitaganj (6.1°C), as shown in Figure 7. The highest mean internal temperatures for the conditioned ice packs were recorded in Chandbela and Ekamba (12.3°C and 9.9°C, respectively). All facilities maintained an MKT between 0°C and 10°C, except for Chandbela (13.6°C) and Ekamba (10.7°C).

Figure 7. Mean internal temperature and mean kinetic temperature for Blowkings freeze-preventive carriers.

13.6

10.7

7.1 7.2 5.9 12.3 9.9 3.0 6.8 1.9 1.6 5.6 6.1 2.9 1.8 1.6

Dandabazar Phalate Rajarani (Stony Sanne (Stony Chandbela Ekamba Pakali Sitaganj (Stony (Stony Hard) (Conditioned) Hard) Hard) (Conditioned) (Conditioned) (Conditioned) Hard) Hilly Plains Mean Internal Temp Mean Kinetic Temp

Chumbang health post was assigned an AOV, but the health care workers used a Leff Trade carrier instead, with stony hard ice packs. For sites with stony hard ice packs, the highest mean internal temperature was recorded in Hashposha (11.7°C), followed by Khanar (9.2°C). The highest mean internal temperatures for FPVCs with conditioned ice packs were recorded in Harinagara and Bhedetar (11.8°C and 8.1°C, respectively). All facilities maintained an MKT between 0°C and 10°C, except for Harinagara (12.8°C), Hashposha (12.6°C), and Khanar (10.4°C).

13 Figure 8. Mean internal temperature and mean kinetic temperature for Leff Trade freeze-preventive carriers.

12.8 12.6

10.4 9.4 8.2

4.5 11.8 11.7 3.8 4.0 3.2 9.2 8.1 8.8

3.8 4.4 4.0 2.9

Bhedetar Chumbang Faxiv Khuwafak Mahabharat Harinagara Hasposha Ithari (Stony Khanar (Stony (Conditioned) (Stony Hard) (Conditioned) (Conditioned) (Stony Hard) (Conditioned) (Stony Hard) Hard) Hard) Hilly Plains

Mean Internal Temp Mean Kinetic Temp

Because the Chumbang site used a Leff Trade carrier, only three sites with stony hard ice packs were evaluated with the AOV carriers. For blocks with stony hard ice packs, the highest mean temperature was recorded in Amahibela (13.8°C), followed by Chimdi (9.6°C). The highest mean internal temperatures for FPVCs with conditioned ice packs were recorded in Rasi and Madhyarashi (6.9°C and 5.9°C, respectively). All facilities maintained an MKT between 0°C and 10°C, except for Amahibela (14.2°C).

Figure 9. Mean internal temperature and mean kinetic temperature for AOV freeze-preventive carriers.

14.2

9.8

7.1 6.0 13.8 4.8 4.3 3.8 9.6 6.9 5.9 4.8 4.1 3.8

Golikhark Muga (Conditioned) Pakhribas (Stony Amahibela (Stony Chimdi (Stony Hard) Madhyarashi Rasi (Conditioned) (Conditioned) Hard) Hard) (Conditioned) Hilly Plains Mean Internal Temp Mean Kinetic Temp

5.1.2 Excursions

A total of 3,756 readings were considered for the FPVCs, each representing a temperature measurement taken every 10 minutes while the vaccines were inside the carriers. Because vaccine carriers should maintain temperatures between 0°C and 10°C, internal temperature readings above or below this range were deemed excursions. Nine temperature readings were below 0°C and 925 were above 10°C (see Table 10). A total of 75.13% of readings were within 0°C to 10°C.

14 Table 10. Distribution of temperature readings by temperature category for freeze-preventive carriers.

Internal temperature Number of readings Percentage

< 0°C 9 0.24%

0°C to 10°C 2,822 75.13%

> 10°C 925 24.63%

Total readings 3,756

5.1.2.1 Excursions below 0°C Of 21 temperature readings, nine sub-zero readings (one freezing event) were recorded for the FPVC at Pakhribas health post. This represents 10% of the readings taken across all the outreach sessions at that health post, and 43% of the readings taken during that specific vaccination session, which lasted only 210 minutes (3.5 hours). The freezing event (below 0°C) was noted in the hilly region in the AOV vaccine carrier with stony hard ice packs, and occurred between 10:00 a.m. and 2:00 p.m., during the vaccination session. 5.1.2.2 Excursions above 10°C A total of 925 temperature excursions exceeding 10°C were recorded (Table 11). This represented 34.03% of the readings for the AOV, 22.58% for the Leff Trade, and 19.58% for the Blowkings FPVC. Of the 925 readings above 10°C, 104 were in the hilly region and 821 were noted in the plains areas. A total of 547 temperature readings were of carriers with stony hard ice packs, and 378 were of carriers with conditioned ice packs.

Table 11. Distribution of freeze-preventive vaccine carrier temperature excursions above 10°C, by carrier type.

Carrier type Number of readings above 10°C/total readings Percentage

Leff Trade 331/1,466 22.58%

Blowkings 251/1,282 19.58%

AOV 343/1,008 34.03%

Total readings 925/3,756 24.63%

5.1.3 Ambient temperature analysis

Per WHO PQS specifications, the equipment must maintain an appropriate internal temperature at ambient temperatures up to 43°C and down to 10°C. Figure 10 plots the mean ambient temperature for each health post against the MKT for the carriers at that health post, taken as an average over all three sessions. It shows that the FPVCs in warmer health posts tended to have higher internal temperatures compared to those in cooler settings.

15 Figure 10. Mean internal temperature versus mean ambient temperature by health post.

The project was conducted in winter (November 2019 to December 2019). Figure 11 compares ambient and internal temperature data for the 24 health facilities. The highest average ambient temperature was found in Hasposha, at 24.6°C, and the lowest was found in Rajarani, at 12.8°C.

Figure 11. Internal temperature versus ambient temperature by health post.

Note: Ambient temperature data for the Muga health facility could not be downloaded via LogTag due to a device challenge.

16 5.1.4 Vaccine vial monitor life loss

Although the primary purpose of this study was to assess the ability of the FPVC to prevent freezing, it is also important to consider its ability to keep vaccines between 0°C and 10°C to avoid compromising potency due to heat exposure. VVMs are included on Expanded Programme on Immunization vaccine vials to provide a visual indicator of the long-term effect of heat on the vaccines. The rate of color change in a VVM, which serves as a proxy for the potency degradation of the vaccine, can be described through equations that depend on the category of VVM.12 These equations are able to estimate the percentage of vaccine life that has been lost by accounting for the amount of time the vaccines have been exposed to different temperatures. For this analysis, this will be referred to as VVM life loss. Figure 12 is a graph plotted using the VVM life loss equations and shows how many days a VVM can remain at a constant temperature before it loses 100% of its life. The most heat-sensitive vaccines, represented by VVM2, can remain at 20°C for 18 days before losing 100% life, while VVM7 vaccines can withstand 78 days and VVM 14 vaccines can withstand 156 days at 20°C. For comparison, the highest MKT for any FPVC in any block was 13.8°C.

Figure 12. Days until 100% vaccine vial monitor life loss at constant temperature.

200

180 VVM2 160 VVM7 VVM14 140

120

100

80

60 Days until 100% life loss100%lifeDaysuntil 40

20

0 0 5 10 15 20 25 30 Constant temperature (°C)

Abbreviation: VVM, vaccine vial monitor.

To analyze VVM life loss, the vaccination session times were taken from the logbook, and the internal temperature of the vaccine carrier during each session window was used to calculate the percentage VVM life loss for that session. The values for each of these sessions were aggregated in the tables below. The first table looks at what typical VVM life loss was like during this phase of the study, across all carriers and health posts, while also identifying the sites with the most and least VVM life loss. The typical VVM life loss was just 0.25%, with the maximum remaining under 1%.

17 Table 12. Overall vaccine vial monitor 2 life loss statistics for phase 2.

Vaccine vial monitor 2 life loss per session Location

Average 0.25% NA

Maximum 0.95% Muga

Minimum 0.04% Mahabarat

Abbreviation: NA, not applicable.

The next analysis looked at the average VVM life loss per session for each of the five blocks (see Table 13). Results were consistent across the blocks. Dandabazar had the least percentage VVM life loss per session at an average of 0.18%, but the most VVM life loss per session was only 0.30%, in Pakhribas.

Table 13. Vaccine vial monitor 2 life loss by block.

Block Average vaccine vial monitor 2 life loss per session

Dandabazar 0.18%

Harinagara 0.29%

Itahari 0.25%

Pakhribas 0.30%

Sitaganj 0.28%

When we compared the assigned ice-pack state, there was also little difference between the sites assigned stony hard ice packs (0.22%) and the sites assigned conditioned ice packs (0.29%). Finally, we compared the average VVM life loss per session for each brand of carrier. The averages were quite close, with the least VVM life loss in the Blowkings carriers at 0.21% and the most in the AOV carriers at 0.31%, which is a negligible difference. The average VVM life loss for Leff Trade was 0.24%.

5.1.5 Cool-down rates and times

Table 14 shows the average cool-down times and the average cool-down rates for the carriers in each block across all carrier brands and ice-pack conditioning states. A cool-down time was only calculated if the carrier temperature started above 10°C and cooled down to below 10°C. A cool-down rate was calculated if the carrier started above 10°C and then decreased in temperature, regardless of whether it fell to less than below 10°C. The majority of the cool-downs took place in Itahari and Sitaganj because those sites replaced their ice packs daily and held two rounds of outreach sessions during the study period. In Itahari, Pakhribas, and Sitaganj, the cool-down rates were greater than 4°C/hour, which is the minimum rate needed to cool a vaccine carrier from 43°C to 10°C in less than 8 hours, per PQS requirements. However, slower cool-down rates can still result in complete cool-downs if the starting temperature is below 43°C. The shortest average cool-down time was in Dandabazar, at 1.15 hours,

18 while the longest was in Harinagara, at 3.12 hours. The fastest average cool-down rate was 4.75°C/hour in Pakhribas and the slowest was 3.23°C/hour in Dandabazar. Nearly all cool-downs were successful, reaching to less than 10°C, but Sitaganj had the slowest percentage of successful cool-downs at 81%.

Table 14. Average cool-down rates and times by block for all carriers.

Block Number of cool- Average cool-down Average cool-down Percentage cooled downs time rate down

Dandabazar 12 1.15 hour 3.23°C/hour 100%

Harinagara 7 3.12 hour 3.32°C/hour 100%

Itahari 40 1.91 hour 4.25°C/hour 95%

Pakhribas 6 1.39 hour 4.75°C/hour 100%

Sitaganj 27 1.92 hour 4.02°C/hour 81%

Table 15 shows the average cool-down times and rates for stony hard versus conditioned ice packs, again across all three carrier brands. It is important to note that this reflects the assigned ice-pack state and that sites may have had to use conditioned ice packs if they did not have a freezer on-site. The sites assigned stony hard ice packs had faster cool-down times and rates on average compared to the sites with conditioned ice packs, although a slightly higher percentage failed to reach 10°C compared to the conditioned carriers.

Table 15. Average cool-down times and rates by conditioning state for all carriers.

Conditioning Number of cool- Average cool- Average cool- Percentage state downs down time down rate cooled down

Stony hard 61 1.76 hour 4.26°C/hour 90%

Conditioned 31 2.07 hour 3.52°C/hour 97%

Each carrier brand was assigned to eight health posts, with half of those health posts using conditioned ice packs and half of them using stony hard ice packs. Acknowledging that there are variations between health posts that could have influenced the results, such as freezer quality and ambient temperatures, Table 16 shows the average cool-down times and rates for each carrier brand. Blowkings had the fastest cool-down time and rate at 1.49 hour and 5.23°C /hour, while AOV had the longest cool-down time and the slowest cool-down rate, at 2.88 hour and 2.05°C/hour.

19 Table 16. Average cool-down rates and times by carrier brand.

Carrier brand Number of cool- Cool-down time Cool-down rate Percentage cooled downs down

AOV 22 2.88 hour 2.05°C/hour 77%

Blowkings 31 1.49 hour 5.23°C/hour 97%

Leff Trade 39 1.71 hour 4.13°C/hour 97%

5.1.6 Freezer performance

Freezer data were collected from 17 health posts. To better understand freezer performance, following the beginning of phase 2 (from November 20 to December 26, 2019), LogTags were placed in 17 freezers for 2 months of data collection (62,906 readings taken at 10-minute intervals). Figure 13 looks at the continuous readings taken within the 24-hour period leading up to each vaccination session as noted in the logbook. For each 24-hour period, the mean, maximum, and minimum freezer temperatures were calculated. The figure below shows the average mean, the average minimum, and the average maximum freezer temperatures across all vaccination sessions for each health post. Taking the average helped smooth out outliers that might have occurred if the LogTags were removed from the freezer, for example. The mean freezer temperatures 24 hours prior to the sessions ranged from –22.4°C in Khanar to 1.6°C in Chumbang. The lowest average minimum freezer temperature was measured in Khanar at –24.4°C, while the highest average minimum was observed in Chumbang at –3°C. The highest average maximum freezer temperature was measured in Pakhribas at –10.9°C, while the lowest average maximum freezer temperature was found in Khanar at –14.7°C.

Figure 13. Mean freezer temperature by health post.

10.9 6.6 7.4

0.8 -0.7 -1.0 0.1 1.6 -3.1 -2.7 -1.0 -2.0 -2.4 -5.2 -5.9 -7.8 -5.2 -8.5 -9.8 -3.8 -3.0 -12.4 -9.4 -10.9 -11.1 -10.5 -14.7 -10.3 -10.6 -9.3 -10.6 -15.4 -14.4 -13.9 -13.8 -15.4 -18.0 -15.2 -19.0 -16.6 -16.9 -22.4 -20.5 -19.5 -19.3 -22.9 -24.4 -24.2

Avg mean freezer temp Avg min freezer temp Avg max freezer temp

Table 17 looks at freezer performance throughout the entire duration of the study. Because the LogTags were placed at different times, some health posts recorded more readings than others. The data have been trimmed to remove ambient temperature readings at the beginning and the end of the recording

20 period, as well as temperature spikes that would have been highly unlikely to occur if the LogTags had remained in the freezer. This was done to ensure greater accuracy of the average and maximum readings; however, there is a risk that some legitimate readings may have been trimmed, or that some incorrect readings may have remained. The average freezer temperatures ranged from below –20°C in Khanar, Sitaganj, and Itahari, to above 0°C in Pakhribas (5.1°C), though most had average temperatures between –15°C and 0°C. The lowest minimum temperature was in Itahari at –28.8°C, while the highest minimum temperature was in Chumbang at –7.7°C. The lowest maximum temperature was in Chandbela at –5°C while the highest maximum temperature (assuming the LogTag was present in the freezer while the reading was taken) was 19.8°C in Mahabarat.

Table 17. Mean temperature and range of freezer temperatures from November 20 to December 26, 2019.

Health post Total readings Average temperature (°C) Maximum (°C) Minimum (°C)

Amahibela 191 –6.6 2.4 –10.2

Bhedetar 6,586 –1.9 12.6 –22.5

Chandbela 715 –16.4 –5.0 –23.5

Chimdi 5,760 –12.8 18.2 –23.0

Chumbang 1,707 –3.3 8.9 –7.7

Dandabazar 6,178 –4.4 14.5 –9.5

Ekamba 5,786 –12.1 1.4 –20.8

Harinagara 1,291 –8.7 6.2 –18.4

Hasposha 5,462 –10.5 19.4 –22.5

Itahari 1,858 –21.2 –1.0 –28.8

Khanar 5,892 –22.4 6.9 –32.4

Mahabharat 6,162 –3.0 19.8 –27.4

Pakali 5,466 –10.8 –1.1 –17.4

Pakhribas 1,991 5.1 15.5 –24.2

Rasi 1,292 –16.6 –1.9 –24.9

Sanne 818 –10.1 0.4 –16.8

Sitaganj 5,751 –21.9 –2.1 –26.5

21 5.1.6.1 Device malfunctions None of the facilities covered during the study reported any major FPVC malfunctions. Reports of minor damage included the shoulder patch on the strap and torn foam discs at two facilities.

5.2 Acceptability

5.2.1 Feedback on the freeze-preventive vaccine carrier

Joint interviews of ANMs and AHWs were conducted at all 24 health posts to obtain feedback on FPVC acceptability and use. All health workers appreciated the ice-pack partitions, which prevented direct contact between ice packs and vaccine vials. They felt this increased protection of the vaccines from potential freezing and reduced wastage due to damaged vial labels from water accumulation. Health workers from all health facilities noted that FPVC internal temperatures were being maintained (based on the VVM status). They did not observe water accumulation at the bottom of the FPVCs except at one health post, where minimal droplets were observed during winter months. Health workers who used the stony hard ice packs said these packs saved time (compared to the ice packs in the SVCs), as they did not need to be conditioned. The weight and size of the FPVC were the two most pressing issues echoed by almost all the health workers and health officials. In the plains region, the larger size of the FPVC made it harder for health workers to place it on their bikes or scooters. They also noted that the larger-sized ice packs (0.6 liters versus 0.4 liters in short-range vaccine carriers) might contribute to the heavier weight of the FPVC. Additional FPVC design issues noted by a few health workers were difficulty in opening and closing the lid, the way the foam discs tore easily—five of the seven health facilities using the AOV said the quality of the foam discs needs to be improved. Some health workers noted that having insufficient transport space in their bikes and scooters to carry the FPVCs to the outreach sites resulted in an operational challenge.

5.3 Systems fit and costing analysis

5.3.1 Systems fit

All health workers agreed that the FPVC benefits enhanced safety, reduced vaccine wastage, and prevented vaccine freezing. Health workers acknowledged the larger size and heavier weight of the FPVCs in comparison to SVCs; however, many did not find this to be a significant issue. Three out of seven facilities that used the AOV and one of nine facilities that used the Leff Trade vaccine carrier mentioned that weight was a significant issue, and one facility that used the AOV mentioned size as a major issue. None of the facilities that used the Blowkings FPVC mentioned weight or size as a major issue. All the facilities that mentioned weight as a significant issue were located in hilly areas. Some of the health workers who mentioned size to be a minor issue said their chief concern was that the FPVCs would not fit on their scooters or bikes the way the SVCs do. District-level health officials and block monitors stated that the size and weight of FPVCs might be something health workers could adapt to. ANMs were familiar with the smaller, lighter SVCs traditionally used and in general did not like the larger, heavier FPVCs, although some mentioned that with time they might adjust if FPVCs are implemented and scaled up by the national MOH. Most ANMs, particularly in the facilities that used the stony hard ice packs, pointed out that the packs often expanded on freezing and were hard to insert into and remove from the ice-pack compartment of the FPVC. Further, the larger 0.6-liter ice packs used by long-range vaccine carriers such as the FPVCs

22 were more difficult to fit in the freezer compartments of domestic refrigerators because of limited capacity. It also took longer to freeze these ice packs, compared to the 0.4-liter ice packs used in the short-range vaccine carriers, which was even more challenging in facilities that experienced frequent power cuts. Ice packs for the standard and Blowkings vaccine carriers were the same size (0.4-liter), so they could be used interchangeably, as observed at one health post, and were easier to freeze in the freezer compartment in the refrigerator. Additional feedback from health workers included:

• One health worker suggested two versions of the FPVC should be developed, one that is the existing/standard size for regular sessions and another that is smaller in size/capacity, especially for catchment areas that are sparsely populated and thereby have fewer beneficiaries. A smaller version would make it easier for the health workers to carry the vaccine carriers to these areas, most of which are situated in very remote locations.

• Health workers were also requested to provide backpacks when backpacks were not provided by the manufacturers along with the vaccine carriers. Backpacks were provided only with AOV and Leff Trade vaccine carriers. A few said the backpacks need more pockets, and one said the straps should be padded for ease of carrying.

5.3.2 Costing data

Using data from the study logbook, we obtained information from 54 outreach sessions conducted during the study. On average, during each outreach session health post staff targeted to vaccinate 15 children aged under 1 year (range 1 to 72 children) and an average of 4 pregnant women (range 0 to 19 women). Table 18 shows the vaccines carried to these sessions, the price per dose, volume per dose, the number of doses per vial, and which of these vaccines are freeze sensitive. An average of 20 vials were carried to each session (range 2 to 79), and the majority of these vials were pneumococcal conjugate vaccine, which is in a 2-dose vial. These vaccines were transported in the SVCs. The average value of vaccines (bacillus Calmette–Guérin for tuberculosis, oral polio, diphtheria-tetanus-pertussis, pneumococcal conjugate, measles-rubella, Japanese encephalitis, and tetanus-diphtheria) carried to each session was $96.61 (range $7.39 to $370). Of these vaccines, 72% of the value was for freeze-sensitive vaccine, so the average value of freeze-sensitive vaccine per session were approximately $70 (range $7.39 to $257). Thus, any one freezing incident during an outreach session could potentially damage about $70 worth of vaccine. On average during an outreach session, 0.38 liters of vaccine and diluent were carried in the SVCs (maximum 1.6 liters). The vaccine storage volume of each FPVC was 1.5 liters—adequate space to carry the vaccines taken to most of the outreach sessions included in this study.

23 Table 18. Types, average quantities, values, and volumes of vaccines taken to outreach sessions.

BCG OPV DTP* PCV* MR JE TD*

Price per dose $0.105 $0.13 $0.69 $3.05 $0.656 $0.41 $0.129

Volume per dose 1.5044 0.8787 2.109 4.8 5.2514 2.9 2.122

Doses per vial 20 20 10 2 10 5 10

Freeze sensitive No No Yes Yes No No Yes

Quantity of vials carried per outreach session

Number of vials for each vaccine Total quantity

Average 2 2 2 9 2 2 1 20

Minimum 0 0 0 1 0 0 1 2

Maximum 11 7 10 30 8 9 4 79

Value of vaccines carried per outreach session

Value for each vaccine Total value: Total value: all vaccines freeze-sensitive vaccines

Average $4.20 $5.20 $13.80 $54.90 $13.12 $4.10 $1.29 $96.61 $69.99

Minimum $0.00 $0.00 $0.00 $6.10 $0.00 $0.00 $1.29 $7.39 $7.39

Maximum $23.10 $18.20 $69.00 $183.00 $52.48 $18.45 $5.16 $369.39 $257.16

Volume of vaccines carried per outreach session

Volume for each vaccine in cm3 Total volume Total volume in cm3 in liters

Average 60.2 35.1 42.2 86.4 105.0 29.0 21.2 379.2 0.379

Minimum 0.0 0.0 0.0 9.6 0.0 0.0 21.2 30.8 0.031

Maximum 331.0 123.0 210.9 288.0 420.1 130.5 84.9 1,588.4 1.588 *Freeze-sensitive vaccines. Abbreviations: BCG, bacillus Calmette–Guérin; DTP, diphtheria-tetanus-pertussis; JE, Japanese encephalitis; MR, measles-rubella; OPV, oral poliovirus; PCV, pneumococcal conjugate vaccine; TD, tetanus-diphtheria.

An FPVC costs $45; when this purchase price is divided over 5 or 10 years, the annualized price is $9 and $4.50, respectively. Freezing was a risk in SVCs, occurring in approximately 3% of sessions, based on LogTag monitoring data (145 out of 5,312 temperature readings were below 0°C in SVCs). We computed the benefit–cost ratios using the data in Table 18 on the minimum, average, and maximum value of freeze-sensitive vaccines taken to each session, taking into account the annualized price of the FPVC (as the cost) and the value of freeze-sensitive vaccines that could potentially be prevented from exposure to freezing temperatures if FPVCs were used (as the benefit). A benefit–cost ratio greater than 1 shows that the benefits outweigh the costs. The larger the benefit–cost ratio, the better the value for money.

Table 19 shows that all the benefit–cost ratios are greater than 1, except when we use the minimum value ($7.39) of freeze-sensitive vaccines taken to a session and the purchase price of the FPVC is annualized over a 5-year period. However, even when we use the minimum value of freeze-sensitive vaccines taken

24 to a session but annualize the FPVC purchase price over a 10-year period, the benefit of the value of vaccines prevented from exposure to freezing outweighs the annualized purchase price of the FPVC. As the value of freeze-sensitive vaccines taken to each outreach session increases, the benefit–cost ratio increases. Annualizing the price of the FPVC over a longer period further improves the benefit–cost ratio.

Table 19. Benefit–cost ratio per health facility per year, taking into account the price of the freeze-preventive vaccine carrier and the value of vaccines prevented from exposure to freezing.

Value Value Value Value in calculation or calculation method

Cost

Price of an FPVC $45.00 C

Annualized price of an FPVC over 5 years $9.00 C1 = C / 5

Annualized price of an FPVC over 10 years $4.50 C2 = C / 10

Assumption (based on study data)

Percentage of outreach sessions where vaccines 2.73% P1 are exposed to freezing temperatures

Benefit

Minimum Average Maximum

Value of freeze-sensitive vaccines taken to an $7.39 $69.99 $257.16 V outreach session

Number of outreach sessions held per month 3 4 4 S

Annual value of freeze-sensitive vaccines $7.26 $91.70 $336.94 B = P1 x V x S x 12 prevented from exposure to freezing during outreach sessions (taking into account the probability of vaccines being exposed to freezing temperatures in standard vaccine carriers at study sites)

Benefit–cost ratios

Benefit–cost ratio (assuming 5-year useful life for 0.81 10.19 37.44 B / C1 the FPVC)

Benefit–cost ratio (assuming 10-year useful life 1.61 20.38 74.88 B / C2 for the FPVC) Abbreviation: FPVC, freeze-preventive vaccine carrier.

25 5.4 Electronic temperature monitors

5.4.1 LogTag

LogTag electronic temperature monitors were used to collect temperature data every 10 minutes during the study. These devices are PQS prequalified and are a well-accepted global standard for reliable data collection in refrigerators. They were the source of all temperature data reported in this study. Data were downloaded monthly by block monitors for analysis. Only one LogTag had errors and did not record temperature (ambient temperature).

5.4.2 Parsyl

A new electronic temperature monitoring device, the Parsyl Trek 1.1, was also included in this study, in order to obtain preliminary feedback on its ability to transmit data on temperature and humidity and to understand its potential value in the ability to rapidly upload data. The Trek is not PQS prequalified, so the data collected were not used for study analysis, other than the humidity and light data, which did not appear to have an effect on temperatures. The Trek 1.1 device includes a light sensor, helpful to evaluate the inside of a vaccine carrier (or a freezer door opening) so that field practices can be identified and corrected if necessary. Analysis is also being done by Parsyl. Feedback was obtained from CCHs and health workers on the Trek device. The devices were sometimes replaced two or three times for the same health facility due to functional issues. There were challenges in updating devices and the Parsyl mobile app with software upgrades and during the final data download and transfer process, which impacted four devices. Data could not be retrieved from four devices. Challenges in transfer of data from the Parsyl device at the field level were most likely due to mobile app issues or phone connectivity. In addition, it is likely that exposure to water negatively impacted the functioning of both types of monitoring devices. During phase 1a, health workers commented that the SVCs and FPVCs were collecting water, so part way through the study they placed the Parsyl and LogTag devices in Ziploc bags to prevent water damage, and the same practice was followed during the phase 2.

6 Discussion Study findings pointed to several factors that affected the acceptability, systems fit, and variations in cool- down times of the PFVCs.

6.1 Context of use The study included districts in hilly and plains settings, with the intention to evaluate acceptability and systems fit in varying environments of use, including remote areas. However, in most instances the logbook data indicated that outreach vaccination sessions lasted for less than 8 hours, even in the hilly areas. And while it was thought that long-range vaccine carriers would be required for use in hilly regions, short-range carriers, as are used in the study districts, are better suited for outreach sessions of less than 8 hours in duration, regardless of the terrain.

26 6.2 Carrier performance

6.2.1 Mean kinetic temperature

All the health posts had MKTs below 15°C, with averages ranging from 1.6°C to 14.8°C. The hilly sites tended to have lower MKTs than the plains sites. The assigned ice-pack state did not seem to affect MKT during this phase.

6.2.2 Excursions

The FPVCs consistently kept the vaccines between 0°C and 10°C, with 75% of temperature readings in this zone overall. The Blowkings carriers had the fewest readings above 10°C (20% of readings), while AOV had the most, at 34% of readings. The LogTags recorded nine readings of less than 0°C in Pakhribas in the AOV FPVC. All temperatures remained above –0.5°C. While this is still not ideal, it is better than prolonged exposure to much lower temperatures, such as –7.5°C seen in a standard carrier in an earlier phase of this study. There were no standard carriers in this phase of the study, so there is unfortunately no way to know how a standard carrier would have performed at such low ambient temperatures. Pakhribas had very low ambient temperatures, below the PQS minimum for freeze prevention. The mean ambient temperature for the 24 hours prior to the start of the session was 11.5°C. PQS testing for carriers only requires them to show freeze prevention at a minimum rated ambient temperature of 15°C. Data from the rest of the health posts indicate that the carriers were successful in preventing freezing at temperatures above and slightly below this threshold.

6.2.3 Vaccine vial monitor life loss

In part due to low initial temperatures in the vaccine carriers, the percentage VVM life loss was very low across all health posts and carrier brands. The session with the greatest VVM life loss occurred in Muga, at 0.95%, primarily because the vaccines remained in the carrier for multiple days. The average VVM life loss per session including all health posts, conditioning states, and manufacturers was just 0.25%. With the one exception of the Pakhribas health post, the FPVCs were successful in both preventing freezing and in maintaining vaccine potency. FPVCs are well suited for scenarios like these winter vaccination sessions, where low ambient temperatures pose a greater risk of freezing the vaccines.

6.2.4 Ambient temperature analysis

Plotting the average ambient temperature against the mean internal temperature for each health post showed some correlation between the temperature outside the carrier and the temperature inside the carrier. This could explain why internal temperatures were much lower in this phase of the study, since winter weather resulted in low ambient temperatures.

6.2.5 Cool-down rates and times

More than 90% of the time that ice packs and vaccines were inside a carrier, the carrier cooled to below 10°C within 8 hours. Average cool-down times were less than 2 hours for all blocks except Harinagara. Both the Leff Trade and Blowkings FPVCs cooled to below 10°C in 97% of all sessions, while the percentage was lower for the AOV carriers, at 77%.

27 6.2.6 Freezer performance

Freezers were more consistent at supplying sub-zero temperatures compared to phase 1 and phase 1a, although only three health posts had average freezer temperatures below –20°C. Freezers may have performed better due to lower ambient temperatures, which would keep the freezer cooler even in the event of a door opening or power loss. It may also be that electricity is more consistent in the winter.

6.3 Limitations and data gaps Data gaps occurred for several reasons, including LogTag malfunction from excessive exposure to water, battery issues, computer differences in session timing (which required careful recalculation), and language barriers. Additional limitations included:

• The terminology and recording of the timing for the outreach sessions was not consistent, which resulted in the need to make assumptions and to remove some data.

• The exact timing of ice-pack removal from freezers and placement in carriers and the actual timing of carrier use (with vaccines in the carriers) is unclear due to inexact time recording and intentional limiting of logging requirements to limit the data collection burden for health workers.

7 Conclusion and recommendations

Preventing vaccine freezing in passive cooling devices remains one of the most pressing obstacles to effective vaccine management. It is challenging, operationally, to prevent freezing. Health workers worldwide have only the shake test as a tool to verify whether vaccine freezing has occurred—and this is a test that anecdotal reports indicate is not often done. Study results validate the long-range FPVC as a viable means to prevent freezing in outreach vaccination settings. The FPVCs used in this Nepal study prevented any incidents of freezing, and, therefore, the potential loss in efficacy of freeze-sensitive vaccines. Based on the catchment areas in this study, where an average of 15 children were vaccinated during an outreach session, the use of the FPVC could result in averted costs, on average, of $70 in vaccine value per outreach session where freezing could have occurred. The costing analysis shows the FPVC to be a good value for money. Cool-down times for the FPVCs to reach 10°C were more variable than laboratory testing predicted, which appears to have been primarily driven by a larger than expected variation in freezer performance. More research is needed to understand to what extent the variations are due to the quality of the freezers, availability of consistent power, or other factors. While VVMs provide health workers with a well- established protocol to know whether vaccines have been exposed to high temperatures, immunization programs will have to weigh the risks of vaccine freezing versus vaccines being in a vaccine carrier for potentially several hours as it cools to below 10°C. The freezer temperature data showed highly variable freezer performance. The data also point to a potential increase in risk of freezing temperatures in SVCs in places where freezer performance is optimal. In global settings where new, highly efficient water pack freezers are being introduced, there may be an inadvertent additional risk of vaccine freezing. Global immunization supply chain entities such as Gavi, the Vaccine Alliance, could include recommendations for countries procuring new freezers to

28 anticipate potential consequences and also procure FPVCs; or where appropriate, consider use of cool water packs instead of ice packs. This study reinforced how critical it is to verify the context of use prior to introducing any new technology. The FPVCs included in this study met their performance objectives; however, health workers were accustomed to a smaller short-range SVC, which may have influenced their perceptions regarding the larger long-range FPVCs. That is, the scenarios of use in eastern Nepal, where outreach vaccination sessions lasted less than 8 hours, may not have been the ideal context of use. There is a need for both short-range and long-range FPVCs to address the varying contexts of use globally.

29 8 References

1. Hanson CM, George AM, Sawadogo A, Schreiber B. Is freezing in the vaccine cold chain an ongoing issue? A literature review. Vaccine. 2017;35(17):2127–2133. doi: 10.1016/j.vaccine.2016.09.070. 2. Techathawat S, Varinsathien P, Rasdjarmrearnsook A, Tharmaphornpilas P. Exposure to heat and freezing in the vaccine cold chain in Thailand. Vaccine. 2007;25(7):1328–1333. https://doi.org/10.1016/j.vaccine.2006.09.092. 3. Edstam JS, Dulmaa N, Tsendjav O, Dambasuren B, Densmaa B. Exposure of hepatitis B vaccine to freezing temperatures during transport to rural health centers in Mongolia. Preventive Medicine. 2004;39(2):384–388. http://dx.doi.org/10.1016/j. ypmed.2004.01.029. 4. Xue H, Yang B, Kristensen D, Chen D. A freeze-stable formulation for DTwP and DTaP vaccines. Human Vaccines & Immunotherapeutics. 2014;10(12):3607–3610. https://doi.org/10.4161/21645515.2014.980195. 5. World Health Organization (WHO). Preventing Freeze Damage to Vaccines. Aide Memoire for Prevention of Freeze Damage to Vaccines. WHO/IVB/07.09. Geneva: WHO; 2007. http://apps.who.int/iris/bitstream/handle/10665/69673/WHO_IVB_07.09_eng.pdf;jsessionid=50F8811 8AB62C98E39D6B87473744AB2?sequence=1. 6. Kartoglu U, Ganivet S, Guichard S, et al. Use of cool water packs to prevent freezing during vaccine transportation at the country level. PDA Journal of Pharmaceutical Science and Technology. 2009;63(1):11–26. https://www.ncbi.nlm.nih.gov/pubmed/19455939. 7. WHO. Temperature Sensitivity of Vaccines. WHO/IVB/06.10. Geneva: WHO; 2006. http://apps.who.int/iris/bitstream/10665/69387/1/WHO_IVB_06.10_eng.pdf. 8. Matthias D, Robertson J, Garrison M, et al. Freezing temperatures in the vaccine cold chain: a systematic literature review. Vaccine. 2007;25:3980–3986. http://epela.net/epela_web/document_lib/Freezing_temperatures_in_the_vaccine_cold%20_chain.pdf. 9. WHO. How to use passive containers and coolant-packs for vaccine transport and outreach operations. In: WHO Vaccine Management Handbook, Module VMH-E7-02.1. WHO/IVB/15.03. Geneva; WHO: 2015. https://apps.who.int/iris/handle/10665/183584. 10. WHO website. PQS Catalogue page. http://apps.who.int/immunization_standards/vaccine_quality/pqs_catalogue/catdocumentation.aspx?i d_cat=18. Accessed June 16, 2020. 11. PATH. Freeze-Prevention Design for Vaccine Carriers, Cold Boxes and Other Passive Cold Chain Equipment to Prevent Freezing of Vaccines, Drugs or Pharmaceuticals in the Cold Chain. Alexandria, VA: Research Disclosure; 2016. https://www.technet-21.org/en/library/main/4833-research- disclosure-freeze-prevention-design-for-vaccine-carriers-cold-boxes-and-other-passive-cold-chain- equipment-to-prevent-freezing-of-vaccines-drugs-or-pharmaceuticals-in-the-cold-chain. 12. Goldwood G, Diesburg S. The effect of cool water pack preparation on vaccine vial temperatures in refrigerators. Vaccine. 2018;36(1):128–133. doi: 10.1016/j.vaccine.2017.11.024.

30