Final Performance Evaluation of Usaid/Madagascar Integrated Social

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR INTEGRATED SOCIAL

MARKETING PROGRAM

March 2017 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Rachel Jean-Baptiste, MPH, PHD, Oxford Epidemiology Services LLC; Juan Manual Urrutia Valenzuela, MBA, Oxford Epidemiology Services LLC; Iain McLellan, MA; Heritiana Andrianaivo, Oxford Epidemiology Services LLC; Ramy Razafindralambo, MSc; Jean Clement Andriamanampisoa, MSc.

Cover Photo: Top Réseau promotion materials. Photo credit: Iain McLellan

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR INTEGRATED

SOCIAL MARKETING PROGRAM

March 2017

USAID Contract No. AID-OAA-C-14-00067; Evaluation Assignment Number: 300

DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

This document is available online in English and French. Online documents can be located in the GH Pro website at http://ghpro.dexisonline.com/reports-publications. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from:

Global Health Program Cycle Improvement Project 1331 Pennsylvania Avenue NW, Suite 300

Washington, DC 20006 Phone: (202) 625-9444

Fax: (202) 517-9181

http://ghpro.dexisonline.com/reports-publications

This document was submitted by GH Pro to the United States Agency for International Development in Madagascar (USAID/Madagascar) under USAID Contract No. AID-OAA-C-14 00067.

ACKNOWLEDGMENTS

This evaluation would not have been possible without the support, cooperation, and sharing of information and experiences, perceptions, and viewpoints of different stakeholders, providing vital material for this report’s findings and conclusions. The team wishes to acknowledge a debt of gratitude to all those, including Top Réseau facilities, Points d’Approvisionnement Relay Communautaire (PARCs), Points d’Approvisionnement (PAs), community health workers, districts, and national leaders, US Government implementing partners, and other partners of the Government of Madagascar, who gave generously of their time, and shared their thoughts, at times extensively and with great depth. Special thanks are due to the leadership and staff of the Integrated Social Marketing (ISM) consortium including PSI and partners for their continuous support, flexibility, and practical help. The data collectors who supported us in the field were indispensable and helped shaped our interpretation of the information we received. The USAID staff who support and oversee ISM deserve special mention for their time and sharing of their insights in the role of ISM with the evaluation team. In addition, we would like to thank USAID/Madagascar staff, including Azzah Al-Rashid, Jocelyne Andriamiadana, Melinda Manning, and Sara Miner for setting a solid direction for the evaluation, and remaining engaged throughout this whole learning activity. Special thanks goes to Vololontsoa Raharimalala for her superb work in coordinating this activity from start to finish. We appreciate her responsiveness. And last, but certainly not least, the evaluation team would like to sincerely thank Melinda Pavin for rapid yet focused technical review, Crystal Thompson for her indefatigable administrative support throughout the implementation of this evaluation, and Laurie Chamberlain, for her patience during the editing and final production.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / v

CONTENTS ACKNOWLEDGMENTS ...... V ACRONYMS ...... VIII EXECUTIVE SUMMARY...... XI I. INTRODUCTION ...... 1 Evaluation Purpose ...... 1 Evaluation Questions ...... 1 II. PROJECT BACKGROUND ...... 3 III. EVALUATION METHODS AND LIMITATIONS ...... 5 Data Analysis Plan ...... 7 Limitations of the evaluation ...... 9 IV. FINDINGS ...... 10 Question 1A: Is the ISM Program likely to achieve its objectives as outlined in the intermediate results and results framework? ...... 11 Question 1B: What were the most and least successful activities implemented by the Program? Why? ...... 13 Question 2: How were the three Program intermediate results integrated (from a management perspective and at the activity level)? What were the benefits or disadvantages to this integration? ...... 15 Question 3: How was the target population (women of reproductive age, children under five, youth 15-24 years old, and those living in rural and underserved areas) functionally defined by the ISM Program? Did Program activities reach this population? Why or why not? ...... 17 Question 4: How was Program learning (including operational research and, particularly, results of innovative activities) documented, disseminated, and applied to improve ISM Program activities and interventions by other health sector partners? ...... 20 Question 5: To what extent are ISM activities socially and economically sustainable? How could the Program design be adapted to improve sustainability? ...... 21 Other relevant findings ...... 27 V. CONCLUSIONS ...... 28 VI. RECOMMENDATIONS ...... 30 ANNEX I. SCOPE OF WORK ...... 32 ANNEX II. EVALUATION METHODS AND LIMITATIONS ...... 60 ANNEX III. PERSONS INTERVIEWED ...... 136 ANNEX IV. DATA COLLECTION INSTRUMENTS ...... 150 ANNEX V. SOURCES OF INFORMATION ...... 264 ANNEX VI. DISCLOSURE OF ANY CONFLICTS OF INTEREST ...... 266

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / vi

FIGURES Figure 1. Deep Dive Districts ...... 10 Figure 2. PSI TRAC FP Modern Method Contraception Prevalence %MWRA ...... 14 Figure 3. Proportion of rural WRA using FP at Top Réseau clinics ...... 19 Figure 4. Studies & Researches hold by ISM (evolution) ...... 20 Figure 5. Sustainability of Efforts: IR 1 ...... 24 Figure 6. Sustainability of Efforts: IR 2 ...... 25

TABLES Table 1. Data Collection Breakdown ...... 10 Table 2. Summary of ISM Program Achievements up to FY 2016 ...... 12 Table 3. Comparison of Community, Pharmacy and Top Réseau Distribution of Pilplan and Confiance (ISM-Distribution Activity Level Indicator FY 2013 – FY 2016, Excel) ...... 15 Table 4. Evaluation of Sustainability Dimensions by IR ...... 23 Table 5. Product Sustainability ...... 26

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / vii

ACRONYMS

ACT Artemisinin Based Combination Therapy AOR Agreement Officer’s Representative BCC Behavior Change Communications CBD Community-Based Distribution CHV Community Health Volunteer CPF Conseilleur de Planning Familiale CQI Continuous Quality Improvement CROM Regional Body of Doctors (Conseil Regional d’Ordre des Medecins) CU5 Children Under Five CYP Couple Years of Protection DAM Medical Drugs Agency (Direction de l’Agence de Médicaments) DHIS District Health Information System DQA Data Quality Assurance DTK Diarrhea Treatment Kits EMMS Environmental Mitigation and Monitoring Statement EOP End-of-Program ETL Education through Listening FGD Focus Group Discussion FISA Fianakaviana Sambatra FP Family Planning FY Fiscal Year GFATM Global Fund to Fight AIDS, Tuberculosis and Malaria GOM Government of Madagascar HF Healthy Family HNI Human Network International HPN Health, Population, and Nutrition IEC Information, Education, Communication IMCI Integrated Management of Childhood Illnesses IPC Interpersonal Communication IPs Implementing Partners

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / viii

IPTp Intermittent Preventive Treatment – Pregnancy IR Intermediate Result ISM Integrated Social Marketing IUD Intra-Uterine Device KII Key informant interview LFP Learning for Performance LLIN Long-Lasting Insecticide Nets LMIS Logistics Management Information System LTM Long Lasting Method M&E Monitoring and Evaluation MCH Maternal and Child Health MEAH Ministère de l'Eau, de l'Assainissement, et de l'Hygiène (Ministry of WASH) MIS Malaria Indicator Survey MIS Monitoring and Information Systems MOH Ministry of Health MVU Mobile Video Unit NGO Non-governmental Organization NMCP National Malaria Control Program OPQ Optimizing Performance and Quality PA Supply Point (Point d’Approvisionnement) PARC Point d’Approvisionnement Relay Communautaire PE Peer Educator PHC Primary Health Care PMI Presidential Malaria Initiative PSI Population Services International Q Quarter QA Quality Assurance QC Quality Control RDT Rapid Diagnostic Test RH Reproductive Health SAF Sampan’ Asa Fampandrosoana/Fiangonan’iI Jesosy Kristy eto Madagasikara SALFA Sampan’ Asa Loterana momban’ny FAhasalamana SBCC Social and behavior change communication SMS Short Message System STI Sexually Transmitted Infection

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / ix

TA Technical Assistance TB Tuberculosis TOT Training of Trainers TR Top Réseau TraC Tracking Results Continuously UN United Nations USAID US Agency for International Development WASH Water, Sanitation and Hygiene WRA Women of Reproductive Age WASH Water, Sanitation and Hygiene

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / x

EXECUTIVE SUMMARY

BACKGROUND The Integrated Social Marketing (ISM) Program is a five-year Cooperative Agreement (Number AID-687-A-13-00001) for $36,823,053, funded by the United States Agency for International Development (USAID)/Madagascar from January 1, 2013 through December 31, 2017. The project is implemented by Population Services International (PSI), as prime partner, with IntraHealth, Banyan Global, Human Network International (HNI) as international partners, and SAF and SALFA as partners local to Madagascar. The ISM Program operates in 20 of 22 regions of Madagascar. The main objective is to use an integrated social marketing approach to increase the use of lifesaving health products and services, particularly in the areas of family planning (FP)/reproductive health (RH), maternal and child health (MCH), and malaria. Three primary intermediate results (IRs) are expected as outcomes of the ISM Program: IR 1: Increased adoption and maintenance of health behaviors, particularly in the areas of FP, water, sanitation and hygiene (WASH) practices, diarrhea, pneumonia, malaria prevention and treatment, nutrition, RH, and others. IR 2: Improved quality of selected health services in the private sector, focuses on working with a network of 250+ private franchised Top Réseau clinics to deliver integrated health services, specifically FP, integrated management of childhood illnesses, youth services, and malaria. IR 3: Increased availability of lifesaving health products and services such as contraceptives, condoms, diarrhea treatment kits (DTK), and long lasting insecticide treated nets (LLINs).

EVALUATION PURPOSE This final performance evaluation served two purposes: 1) to learn to what extent the project’s objectives and goals – at all result levels – have been achieved; and 2) to inform the design of follow-up projects. This evaluation will assist the Mission in reaching decisions related to: 1) the effectiveness of current approaches to improve health behavior, improve quality of private sector health services, and increase availability of health products and services; 2) the type of mechanisms the Mission should use in any future assistance to the health sector for social marketing and related interventions, and 3) the nature and scope of possible future interventions in the sector, based on lessons learned from the current project.

EVALUATION QUESTIONS Specific questions that will guide this Performance Evaluation are stated in the Evaluation Scope of Work and were discussed in detail with USAID/Madagascar, and are understood to be the following:

A. Achieving objectives: Is the ISM Program likely to achieve its objectives as outlined in the intermediate results and results framework? What were the most and least successful activities implemented by the Program? Why?

B. Integration of Project Components: How were the three Program intermediate results integrated (from a management perspective and at the activity level)? What were the benefits or disadvantages to this integration?

C. Definition and reach of “underserved” and “rural”: How was “underserved” and “rural” defined operationally for the target populations (women of reproductive age,

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / xi

children under five, youth 15-24 years old, and those living in rural and underserved areas) by the ISM Program? Did Program activities reach this population? Why or why not?

D. Program learning and dissemination: How was Program learning (including operational research and, particularly, results of innovative activities) documented, disseminated, and applied to improve ISM Program activities and interventions by other health sector partners?

E. Sustainability: To what extent are ISM activities socially and economically sustainable? How could the Program design be adapted to improve sustainability?

EVALUATION METHODS Evaluation Design: Cross-sectional design with mixed methods for data collection

Data Collection Methods: Methods used include: document reviews, focus group discussions, key informant interviews, client exit interviews from Top Réseau, household surveys, phone interviews with Top Réseau staff, and observation visits to Top Réseau, warehouses, radio stations, retailers, pharmacies, PARC, Supply Points (PAs or Points d’Approvisionnement), and Community Health Volunteer (CHV) locations where socially marketed products are kept.

Ethical considerations and assurances: The team obtained verbal informed consent from every participant, and protected their identity in reporting the findings. The team obtained special additional written permission from participants whose pictures or video were taken. Limitations Limited time constraints and safety concerns led to the selection of only districts and health facilities that are geographically accessible.

EVALUATION FINDINGS In total, the team collected data from 618 individuals. Web-based and phone surveys reached 166 of the 180 Top Réseau clinics with functional phone numbers provided by PSI. The team conducted 180 household surveys in the six districts visited, 32 client exit interviews from seven Top Réseau Clinics, and interacted with 68 beneficiaries through focus group discussions in the three regions and six districts where data was collected. In addition, the team interviewed 11 Government of Madagascar (GOM) officials, three USAID officials, 17 PSI/ISM staff (group interview), two USAID IPs, four PARCs, 11 PAs, 50 CHVs (in focus group discussions), 26 interpersonal communication (IPC) agents (most of whom were Conseil Planning Familial (CPFs) or family planning counselors, three radio station managers, and five wholesalers. The team conducted 27 retail checks. Informed consent was obtained for all participants. The team also reviewed 36 documents.

Question 1: Achieving objectives IR1: Increased adoption and maintenance of health behaviors. Measurable progress has been made in terms of the adoption and maintenance of health behaviors, and the ISM Program is on track to achieve many of its targets. Appropriate treatment of diarrhea among children under five rose from 3.6% in the baseline to 8.1% in Fiscal Year (FY) 2014 and it is likely that the project will achieve its target of 12% by end-of-project (EOP). However, there are two targets that ISM is unlikely to achieve given current status and rate of implementation. While the

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / xii

contraceptive prevalence rate of 29.2% in the baseline has risen to a current rate of 36.28%, it is unlikely to achieve its EOP target of 44% as stated in its most recent performance monitoring plan (PMP) given its current FP sales pattern. Second, only 220 urban and 15 rural Top Réseau report providing ISM services in 2016, substantially less than the target of 233 urban and 41 rural. Most successful for IR 1: Radio has been the most effective channel in reaching the ISM target populations; 52.7% of parents of under five children, said they had been exposed to messages on pneumonia prevention and treatment on the radio (2014 Pneumonia TRaC). Some 45.5% said they had heard radio spots on Sur’Eau (2014 Diarrhea TRaC). Least successful for IR 1: Mobile Video Unit (MVU), through the primary midmedia channel to target rural areas that were not covered by radio signals, did not reach its targets nor did it have impressive impact. Three vehicles conducted 440 shows in 2015. In FY 2016 the three MVUs met only 47.7% of its goal of reaching 79,200 people.

IR2: Improved quality of selected health services in the private sector. ISM contributed to the development of well trained staff and expanded services offered specifically for Family Planning and Maternal and Child Health. Annual ISM supervision reviews found that 92.3% TR Family Planning providers scored at least 80% on FP quality standards in 2015 compared to 58.2% in baseline. One hundred and twenty Youth Peer Educators (YPEs), who organize events and visits schools and 240 FP Counselors, who mostly do house to house visits, promote use of TR clinics especially to the underserved, poor including youths. 69,065 clients who visited Top Réseau clinics seeking FP services used a voucher given to them by YPEs and FP Counsellors that offered a discount. This allowed ISM to reach 104% of its FY 2015 target. Youths making up 16.7% of total TR FP clients (24,000/143,000) in FY2016. 24% of TR clients were youths (exit survey). Most successful for IR 2: 230 out of the 251 were found to be providing the three services introduced by ISM (FP, MCH and Malaria). There was evidence of Top Réseau providing FP services to the underserved as nearly half of FP clients came with voucher in FY 2016. Least successful for IR 2: Inadequate marketing and promotion results in a low recognition of the TR brand and poor understanding of concept. 54% of TR clinics provide 90% of the total number of Family Planning consultations. This is an indication that the full potential of the network is not used.

IR3: Increased availability of lifesaving health products and services. A combination of community, commercial and pharmaceutical distribution, combined with Top Réseau outlets, has made socially marketed commodities affordable and widely available. There have been some stock outs but usually as a result of national level supply and not failures of the supply chain. Brands like Sur’Eau, Pilplan and Protector Plus have a high degree of brand loyalty. The injectables Confiance and Implanon sales have grown steadily since being introduced.

Most successful: The community distribution of commodities through CHVs reaches rural and poor populations at prices that are affordable to them. The small margin the CHVs get from their sales contributes greatly to their motivation and sense of contributing a valuable service to their community. Least successful: Some challenges with the motivation of the managers of PARC supply points. But those interviewed also felt a sense of contributing to society through their social marketing work, as well as making a modest profit.

Question 2: Integration of Project Components The team found the most concrete evidence of integration of the three IRs within Top Réseau clinics: IR 1 – SBCC was represented through mass media, and IPC including CPFs promoting

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / xiii

the use of Top Réseau clinics (IR 2). Evidence suggested this worked, as 66,000+ users of TR for FP services came in with a voucher only obtainable through interaction with a CPF. Socially marketed products (IR 3) were made available at Top Réseau clinics (IR 2) and sold to clients. The team found that such linkages were beneficial. The linkage between generic SBCC and specific brands, products and points of purchase is less evident, mainly because of the poor performance of mid media efforts. The team could not find evidence of concerted efforts to document and monitor linkages and their effect. It is, therefore, impossible to establish if such efforts could have had disadvantages in program implementation. Additionally, the multiplicity of programs conducting SBCC campaigns both through mass media and IPC, makes it difficult to differentiate any individual program’s contribution to impact on the community distribution.

Question 3: Definition and reach of “underserved” and “rural” ISM slightly varied its definition of rural and underserved depending on the project component. IR1:: Operational definition for the target population for SBCC was not explicit. Discussions with PSI revealed that rural and underserved were operationally defined as those not reached by mass media. Mobile Video Units (MVU) was the SBCC vehicle employed for those not reached by mass media. However, MVUs did not meet its targets Nevertheless, the SBCC through radio did reach large proportion of the population. Indeed, 43.7% of rural women aged 15-49 listen to radio at least once a week and more than 68% in urban areas (MDG survey, 2013), and more than half of the participants in the household survey stated radio as a trusted source of health information. IR2: The ISM program Mobile Video Unit road show in Toliara. Photo credit: Iain supports Top Réseau clinics in both urban McLellan and rural areas to provide services to underserved and rural populations. The ISM program set up the e-voucher system at the Top Réseau clinics to subsidize the cost of consultation and allow low income population to access the Top Réseau services, and this has facilitated service delivery to a large number of people who might otherwise not have accessed care. More than half (67%) of the rural Top Réseau clients belong to the poorest wealth quintile classes (Top Réseau Client Exit survey, PSI 2015). However, reach is low - only 6.4% of clients in all Top Réseau clinics are rural, and 14% in urban Top Réseau are from the lowest wealth quintile. IR3: By working together with other USAID projects, specifically Mahefa and Mikolo, the community distribution strategy has reached the rural population, and most product sales are at that level. However, the extent to which this same group was able to reach the urban poor was unclear. The commercial and pharmaceutical channel only serve the segment of population that has capacity and willingness to pay.

Question 4: Program learning and dissemination ISM conducted 42 studies including 29 on results of innovative activities. However, they have disseminated only six. The team found evidence of use of research to implement ISM, particularly around decision making on products, but also key processes like distribution. The

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / xiv

team also found that most ISM research activities are designed and executed by PSI in-house. ISM research staff said they make minimal use of freelance researchers and outside research firms due to concerns about quality. Other USAID implementing partners and the GOM expressed dissatisfaction regarding their level of engagement in the research process, as well as having minimal access to research results. Partners would like to benefit more from ISM research for their decision-making.

Question 5: Sustainability Analysis of the system for Increase adoption and maintenance of health behaviors (IR 1) revealed that this is largely driven by PSI. PSI staff take the lead in ensuring that messages that promote healthy behaviors are developed, scripted, and aired. The system to improve quality of health services in the private sector (IR 2) in use by PSI was largely driven locally. The team found that almost all inputs, including the providers and private clinic space as well as the majority of necessary equipment were all local. Processes, including patient flow, data management, stock management, among others, were all local. PSI contributed to recruitment of new clients, including youths, through IPCs and the voucher system. Outputs, specifically provision of health services including family planning services, were all local. The system for community distribution (IR 3) was largely dependent on PSI and other partners for key inputs. Specifically commodities, as well as staff (PARCs, PAs) were dependent on PSI as they only exist because of the subsidy created from the distribution chain. Community Health Volunteers are not dependent on PSI but are largely supported by other USAID partners. Nevertheless, if another partner was mandated to provide similar support, the existing structures could still deliver products to clients in rural areas. In the system for commercial distribution, the team found that except for commodity subsidy, other inputs, including working capital, transport, space, and staff were all local. Processes were largely local, except for transportation from PSI to the warehouses. Outputs, specifically sales, were all local. And for each project component, ownership, advocacy, policy environment, organizational infrastructure and financial viability all need additional focus to ensure long term social and economic sustainability.

EVALUATION CONCLUSIONS Q1 Achieving objectives: Of seven key results ISM was to accomplish, they are on track to accomplish all except CYP rate of 44% with one year left to implement when the current rate is 36.2%. ISM is also not likely to have 273 Top Réseau, including 40 rural Top Réseau offering integrated services in at least three health areas (FP/RH, IMCI/nutrition, and malaria). Available evidence suggests there are currently 235 Top Réseau providing any health service, including 15 rural Top Réseau clinics.

Q2 Integration of Project Components: Linkage between IR1, IR2 and IR 3 is clear and evident at the Top Réseau clinics. This is demonstrated by the use of 66,887 vouchers at TR clinics that were handed out by IPC SBCC agents to underserved women in need of FP. The linkage between social marketing distribution and the clinics resulted in a contribution of the franchise to CYP. The team concluded that integration as seen at the Top Réseau facilitated access to health services delivery and commodities, though Top Réseau clinics did not reach a significant number of people. Linkage between IR1 and IR 3 was limited to product promotion. There is evidence of a consistent effort on the part of ISM to use mass media communications to promote products and brands. No evidence of concrete efforts to promote and measure the impact of integration overall.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / xv

Q3 Definition and reach of “underserved” and “rural”: While mass media reached some rural and underserved populations, a substantial proportion of such populations were not reached. There are few Top Réseau in urban poor and rural areas, and there is evidence that the Top Réseau franchise is underperforming – 4% of Top Réseau clinics provide nine percent (9%) of services. This may be contributing to weak coverage among the urban poor and rural populations. Community distribution succeeded in reaching target populations in the rural areas by working with other USAID partners. Commercial social marketing mainly services women of reproductive age and youth in urban areas who can afford to buy products at commercial prices, and some products, particularly for children under five, are not available in the commercial sector. Distribution is not reaching the urban poor and underserved, except for those who visit Top Réseau clinics with a voucher.

Q4 Program learning and dissemination: The ISM program produced extensive research, including 42 studies to date and several other studies planned over the next year. Dissemination of results has been limited, and the use of the research by partners is also limited.

Q5 Sustainability: IR2 and IR3 were built on highly subsidized local systems and are more likely to remain at the end of ISM. Ownership, advocacy, policy environment, organizational infrastructure and financial viability all need additional focus to ensure sustainability. Limited efforts to rely on local systems for the management and supervision of outputs of IR1 including communication products and research. ISM has not created or implemented a sustainability schedule. At district level in particular, there is minimal collaboration between ISM and MoH.

EVALUATION RECOMMENDATIONS THROUGH END OF PROJECT ● Develop a closeout plan that takes into consideration sustainability and asset transfer

● Improve performance monitoring of Top Réseau by first cleaning the Top Réseau data and making it more user friendly.

● Develop an advocacy strategy that involves District Medical Inspectors and Ordre de Medecin to help strengthen the Top Réseau network.

● Consider piloting a model of service delivery where the services go to the urban poor and the rural communities, thereby increasing coverage. Leverage mobile video outreach to include provision of the three key services delivered through ISM (FP, MCH, Malaria) as well as related product sales.

● Create a research dissemination schedule together with USAID to identify which of the 42 or more research reports should be disseminated, and also ensure that all research reports are available online.

● Develop a sustainability schedule based on their Cooperative Agreement that details efforts they will make to further strengthen the likelihood of sustainability of the three IRs including all products.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / xvi

I. INTRODUCTION

EVALUATION PURPOSE USAID requested an evaluation of the Integrated Social Marketing program to learn the extent ISM has accomplished its objectives, and how best to plan for a follow-on activity. Specifically, data collected from this evaluation will assist the Mission in reaching decisions related to: 1) the effectiveness of current approaches to improve health behavior, improve quality of private sector health services, and increase availability of health products and services; 2) the type of mechanisms the Mission should use in any future assistance to the health sector for social marketing and related interventions; and 3) the nature and scope of possible future interventions in the sector, based on lessons learned from the current project.

1) Achieving objectives: A. Is the ISM Program likely to achieve its objectives as outlined in the intermediate results (IRs) and results framework? B. What were the most and least successful activities implemented by the Program? Why? Areas to consider:

o Social and behavior change communication (SBCC) compared to other components

o Understand target setting (could reach more?)

2) Integration of Program Components: A. How were the three Program IRs integrated (from a management perspective and at the activity level)? B. What were the benefits or disadvantages to this integration? Areas to consider:

o Linkages made between components and impact on end-users o Organization of linkages (planning and implementation)

o Components critical mass vs. better separated out o Balanced or too focused on one component

3) Definition and reach of “underserved” and “rural”: A. How was “underserved” and “rural” defined operationally for the target populations (women of reproductive age, children under five, youth 15-24 years old, and those living in rural and underserved areas) by the ISM Program?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 1

B. Did Program activities reach this population? Why or why not? Areas to consider: o Market-driven (availability and affordability)

o Socio-economic needs-driven (availability)

o Wealth quintiles

o Rural vs. urban underserved reached

4) Program learning and dissemination: How was Program learning (including operational research and, particularly, results of innovative activities) documented, disseminated, and applied to improve ISM Program activities and interventions by other health sector partners? Areas to consider:

o Research dissemination to partners o Utility of research to partners

5) Sustainability: A. To what extent are ISM activities socially and economically sustainable? B. How could the Program design be adapted to improve sustainability? Areas to consider:

o Multiple dimensions of sustainability, including social and economic dimensions

o Inclusion of sustainability in close-out plan

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 2

II. PROJECT BACKGROUND

The Integrated Social Marketing Program is a five-year Cooperative Agreement (Number AID 687-A-13-00001) funded by the United States Agency for International Development (USAID)/Madagascar. The project is implemented by Population Services International, as prime partner, with IntraHealth, Banyan Global, Human Network International (HNI) as international partners, and SAF (Sampan’ Asa Fampandrosoana/Fiangonan’iI Jesosy Kristy eto Madagasikara) and SALFA (Sampan’ Asa Loterana momban’ny FAhasalamana) as partners local to Madagascar. The award is for a total of $36,823,053, running from January 1, 2013 through December 31, 2017. The goal of the program is to improve the health of the Malagasy people – especially women of reproductive age (WRA), children under five (CU5), and youth 15-24 years old living in rural and underserved areas through an increasingly sustainable social marketing program that delivers essential health products and services. The ISM Team will apply its expertise in social marketing, social franchising and SBCC to bring more users into the Malagasy health market. The ISM Program operates in 20 of 22 regions of Madagascar (see map). The main strategic objective is to use an integrated social marketing approach to increase the use of lifesaving health products and services, particularly in the areas of family planning (FP)/reproductive health (RH), maternal and child health (MCH), and malaria. PSI also works in partnership with USAID’s integrated health programs, MIKOLO and MAHEFA (and later CCH), to expand community distribution of products and services. By the end of this program, the Malagasy people will see improvements in their health status with regard to FP, RH, MCH, and malaria. Three primary Intermediate Results (IR) are expected as outcomes of the ISM Program:

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 3

IR1: Increased adoption and maintenance of health behaviors. The “Healthy Family” behavior change communication (BCC) campaign focuses on increased knowledge and adoption of preventative behaviors, and utilization of commodities related to: family planning; water, sanitation and hygiene (WASH) practices; diarrhea, pneumonia and malaria prevention and treatment; nutrition; reproductive health, and others. Radio, TV, mobile video units (MVU), innovative interpersonal communication techniques, and a variety of additional information, education and communication (IEC) materials and activities all combine to positively influence health behavior. In partnership with MIKOLO and MAHEFA, community health volunteers (CHV) are trained and equipped to provide education and distribute critically important health products within isolated rural areas.

IR2: Improved quality of selected health services in the private sector. PSI’s network of nearly 250 private, franchised “Top Réseau” health clinics deliver a variety of health care services primarily in the areas of FP/RH, integrated management of childhood illnesses (IMCI), youth services, and malaria. PSI and its partners IntraHealth, Banyan Global, SAF, and SALFA focus on expanding access to quality health care services through training, quality assurance, capacity-building, supervision, promotional support, access to financing, and more. Rural and urban Top Réseau clinics are present in 74 of the 114 districts across Madagascar.

IR3: Increased availability of lifesaving health products and services. PSI is expanding access to affordable health products such as contraceptives, condoms, diarrhea treatment kits (DTK), drinking water treatments, pneumonia and malaria medicines, and long-lasting insecticide-treated nets (LLINs). PSI distributes these social marketing commodities through a network of nearly 1,200 commercial, pharmaceutical, and community-based outlets. Within the ISM Team, HNI provides mobile technology support to make e-voucher and mobile money payment initiatives easier and more accessible to consumers and retailers. The findings of this evaluation are expected to contribute to USAID decision-making on the level and type of support to further reach the rural and underserved populations with BCC, high quality health services, and necessary commodities. Specifically, the evaluation provides an opportunity for USAID to identify gaps – including strengths and weaknesses – and gather evidence upon which a post-ISM strategy can be based.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 4

III. EVALUATION METHODS AND LIMITATIONS

Overall Evaluation Design: A cross-sectional design with mixed methods for data collection defined the design of the evaluation (methods used to collect this data are summarized in Table 1). A system-strengthening approach to guide data collection, analysis, and reporting on Evaluation Questions 1, 2, and 3, a descriptive approach to respond to Question 4, and Oxford Epi’s multi-dimensional Sustainability Framework to answer question 5. These are fully described in the Evaluation Work Plan (Annex 2). Type of Evaluation: This was a final performance evaluation.

Summary of Stakeholder Engagement: Data were collected from multiple stakeholders including: ISM staff and sub-partners Banyan and IntraHealth; various members of USAID’s health team, other USAID partners intricately involved in the implementation of the ISM on the ground (that implement Mahefa and Mikolo projects); the Government of Madagascar, specifically Ministry of Health, specifically the Office of the Director General, and Office of the Director of Family Health, Salama, and Ministry of WASH (MEAH or Ministère de l'Eau, de l'Assainissement, et de l'Hygiène); district medical officers; and Top Réseau staff, Point d’approvisionnement relay communautaire (PARCs), Point d’Approvisionnement (PA or Supply Point), CHVs, radio station managers, commercial wholesalers, retailers, including supermarkets, kiosks, boutiques and pharmacies, and beneficiaries. Sampling strategy: A detailed sampling strategy is included in the Evaluation Work Plan (Annex 2). In brief, using software developed by Oxford Epi, the team attempted to reach all 251 Top Réseau currently on PSI’s registry for participation in a phone interview. In addition, the team conducted “deep dive” into two districts each (one urban, one rural) in Diana (Diego 1 and Ambilobe), Toliara (Toliara 1 and Toliara 2), and Itasy (Miarinarivo and Avironimamo). Regional and district selection criteria are described in the Evaluation Work Plan (Annex 2). Within each district, the team collected survey data from 30 households using systematic random sampling during which every third house was selected. USAID Point de Approvisionnement (Supply Point) supplying provided a list of national-level Community Health Volunteers with socially marketed stakeholders, and in every district the team products in Toliara. Photo credit: Iain McLellan interviewed either the District Medical Officer or his/her designee(s). Sampling of other stakeholders for key informant interviews (KIIs) and focus group discussions (FGDs) were done by convenience.

Data Collection Methods: The team used multiple data collection methods to address each evaluation question. Summarized in Table 1 below, these include document reviews, focus group

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 5

discussions, key informant interviews, client exit interviews from Top Réseau, household surveys, phone interviews with Top Réseau staff, and visits to Top Réseau clinics, warehouses, radio stations, retailers, pharmacies, PARC, PA, and CHV distribution locations where socially marketed products are kept. Where data from one set of discussions seem to indicate a finding, team members used subsequent data collection encounters to further validate, invalidate, or broaden/deepen understanding of the finding by discovering new perspectives. Team members debriefed each other regularly by phone and email and compared notes and findings. A full description of data collection methods can be found in the Evaluation Work Plan (Annex 2).

Table 1. Data Collection Methods Employed for Each Evaluation Question

Data Collection Evaluation Questions Method 2: 3: Coverage 1: ISM 4: 5: Integration of Achievements Research Sustainability of IRs Underserved Document Review X X X X X KII: Government of X X X X Madagascar (GOM) KII: USAID X X X X X KII: PSI X X X X X KII: Top Réseau X X X X KII: PARC X X X KII: PA X X X KII: Radio Station X X X X KII: Interpersonal X X X Communication (IPC) FGD: CHV X X X FGD: IPC X X X FGD: Youth – Males X X X FGD: Youth – X X X Females FGD: Women 15-49 X X X (Parents CU5) FGD: Men 15-49 X X X (Parents CU5) Client Exit Interview X X X Household Survey X X X Top Réseau Phone X X X X Interview

Ethical considerations and assurances. The team obtained verbal informed consent from every participant, and protected their identity in reporting the findings. The team obtained special additional written permission from participants whose pictures or video were taken.

Deviations and adjustments. Procedures used to ensure that the data are of highest achievable quality include the following: The team developed the tools together, conducted

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 6

training (on Top Réseau phone interviews and household surveys), and almost always collected the qualitative data in pairs. In addition, the team made use of local dialect translators to further ensure smooth communication. All quantitative data (surveys) were entered into an easy-to-use web database, and analyses conducted in real time. Periodic reviews of the analysis by team members led to implementation adjustments and improvement in the data collection process.

DATA ANALYSIS PLAN For Question 1: ISM Achievements, we documented targets vs. achievements to date, triangulated by findings from KIIs and household surveys. The team also analyzed reach through SBCC-based data from the household survey, triangulated with data from PSI’s Tracking Results Continuously (TRaC) 2015 for Family Planning and others done in 2014 on diarrhea prevention and treatment and pneumonia; efficacy of Top Réseau for reaching the underserved and rural by reviewing data from PSI Top Réseau exit client interview, the evaluation team’s own client exit interviews, as well as utilization of each Top Réseau. This allowed the team to document the proportion of Top Réseau providing services to the target population. This analysis was further disaggregated by urban/rural. For the commercial sector, the team analyzed processes and efficiencies that facilitated or hinder target achievements by reviewing results from stakeholder KIIs, household surveys, review of reports, documents, and other data from PSI. In addition, the team conducted a critical analysis of the status, and activities that were the most and least successful of each project IR using data from KIIs, reports, and raw data (provided in Excel) about the performance of Top Réseau from PSI.

For Question 2: Integration, the following were analyzed:

Linkages between components. The team analyzed the quantitative and qualitative evidence from multiple sources demonstrating the extent to which activities from one IR were operationally planned and implemented together with at least one other IR. We also reviewed any evidence for training and capacity building of staff that focused on integration of the IRs. Where integration did happen, the team reviewed the impact on beneficiaries.

Balance between the components. During initial briefing meetings with USAID, the team was told of USAID’s interests in the balance of funds and activities of each of the components, and how they compare. This was a sub-question for Evaluation Question 2. The team attempted to reviewed collected information to identify the extent to which there was special focus on one component compared to the others, or if all components were treated equality in terms of time, staff, research, and budget. Despite repeated requests, we did not receive a budget breakdown by IR from PSI, but instead reviewed the information available in their work plan as illustrative.

For Question 3: Coverage of Target Populations, the team documented PSI’s explanation of how they operationally defined “underserved” and “rural.” The team then compared Top Réseau selected by PSI because they are located in a rural area with the most up to date list (2016) of rural communes obtained from the Madagascar Ministry of Interior to identify, documented any discrepancies, and eliminated any Top Réseau located in an area considered “urban” or “peri-urban” by the Ministry of Interior from further analysis as part of the group of “rural” Top Réseau. The team used results of the diminished number of rural Top Réseau as the actual current number of rural Top Réseau, and analyzed data on their utilization and reach. Adding to this were additional analyses of perceptions of Top Réseau providers (from phone

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 7

interviews) and clients of Top Réseau (client exit interviews) of how far they live to the Top Réseau. Analysis for this question continued with reviewing the types of patients reached by social marketing products. For this, we analyzed the performance of the community distribution arm separately from the Commercial Sector arm of Social Marketing. Qualitative results from FGDs with beneficiaries, CHVs, and KIIs with PAs and PARCs were analyzed for agreement or diversity with regards to product reach in the rural areas. Because the team did find some CHVs working in urban areas in Diana Region, we collected and analyzed data on product reach of urban underserved from them as well, and used that to triangulate with data from urban household surveys. The urban CHVs promoted and distributed social marketing products and they were separate from peer educators who only do promotion but no distribution. To better understand and document the reach of SBCC, analysis of proportion of beneficiaries who have heard of health messages from the various media mix was analyzed from the evaluation team’s household survey by urban/rural, and triangulated with data from beneficiary FGDs, as well as 2015 Family Planning and 2014 Diarrhea and Pneumonia TRaC surveys.

For question 4: Program Learning. The team used documents provided by PSI and by USAID, as well as KIIs with both groups to analyze the research infrastructure that PSI has operationalized for ISM, and the types of research that PSI has conducted so far over the life of the project. We further analyzed the extent to which this research has been disseminated by requesting this information from PSI, and triangulating it with results of KIIs with other partners, including USAID implementing partners (IPs) and the GOM who could have reasonably made use of the data from PSI research. Per guidance from USAID, analysis on this area focused mostly on the dissemination process employed by PSI, whether or not certain types of research are disseminated, and documenting PSI’s reasons or challenges to doing so. While we intended to document the perceived utility of the currently disseminated research, the data to do this was largely unavailable.

For Question 5: Sustainability, results were analyzed in two ways. First, understanding the economic fragility of the environment within which the ISM is implemented, USAID/Madagascar requested that the evaluation team review the extent to which systems that currently implement SBCC, private service delivery, and socially marketed products through community distribution as well as commercial distribution were sustainable (Figure 1). To do this, we relied on a common and well accepted definition of a system as a series of inputs, processes, and outputs, and a system was more likely to be sustainable if its inputs, processes and outputs had minimal external dependencies. We further analyzed findings using the Oxford Epi Sustainability Framework previously described in the Evaluation Protocol (Annex 2). Responses to a set of questions with a scale from 1 to 10 collected during KIIs with stakeholders, as well as phone interviews with Top Réseau clinics were combined and averaged to produce a score for each of the dimensions of sustainability, including shared understanding and ownership, policy environment, advocacy capacity, staffing number and capacity, physical infrastructure (space, technology, etc.), organizational infrastructure (management, Quality Assurance (QA)/Continuous Quality Improvement (CQI) processes, etc.), public image/confidence, in addition to financial viability. These averages were color coded (red = 0 – 4.9; yellow = 5.0 – 7.9; green = 8.0 to 10.0) to indicate level of sustainability by dimension. This analysis was further refined by reviewing results

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 8

from KIIs with stakeholders, phone surveys, and beneficiary household surveys. In a number of instances, this triangulation led to modifications to the colors (from one color to another), or the addition of a color mix. Thus, an aspect with an average score that made it red could also have some yellow to depict that some work was underway. Each domain was then used as a building block to sustainability, and represented by colors – red (significant support needed), yellow (critical support needed), and green (minimal additional support needed). Final results were summarized using the “building blocks” to build “sustainable houses” thus, providing a visual depiction of current status of sustainability.

LIMITATIONS OF THE EVALUATION The team notes that while it had access to raw data from PSI that added to analysis of IR2 (Top Réseau), it did not have similar data for IRs 1 and 3. Further, the team was able to collect quantitative data from the majority of those involved in the implementation (and at the same time beneficiaries) of IR2 (Top Réseau) using Oxford Epi’s real time data system, since names and phone numbers were readily available, but it was not able to do the same for the other IRs given restrictions on time, money, and demographic differences. For example, phone interviews would not have worked well in the assessment of community distribution, since a significant number of those involved may not have regular access to working phones. In addition, limited resources, time constraints and safety concerns led to the selection only of districts and health facilities that are geographically accessible. This may limit the team’s capacity to understand the extent to which ISM activities reached rural and underserved populations throughout the 20 of 22 regions of Madagascar, where it operates. However, several efforts have been made to correct for this shortcoming, ensuring geographic spread of the selected regions (South west, Central, North), and intentional urban/rural stratification of selected districts. In addition, the online surveys that reached the majority of Top Réseau providers transcend boundaries due to geographical accessibility, as does raw data and reports from PSI on their activities in the whole country. These provided the team with significant additional insights, and further increased our confidence in our ability to identify necessary recommendations.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 9

IV. FINDINGS

In total, data was collected from 618 people (see table below for breakdown). Web-based and phone surveys reached 166 of the 180 Top Réseau clinics with functional phone numbers provided by PSI. The team conducted 180 household surveys in the six districts visited, 32 client exit interviews from seven Top Réseau clinics, and interacted with 68 beneficiaries through focus group discussions in the three regions and six districts where data was collected. In addition, the team interviewed 11 GOM officials, three USAID officials, 17 PSI/ISM staff (group interview), two USAID IPs, four PARCs, 11 PAs, 50 CHVs (in FGDs), 26 IPCs (most of whom were Conseilleurs de Planning Familial (CPFs), three radio station managers and five wholesalers. The team also conducted 27 retail checks. Informed consent was obtained for all participants. The team also reviewed 64 Figure 1. Deep Dive Districts documents (36 documents plus 28 sub-contracts, Annex 4).

Table 1. Data Collection Breakdown

Source Number Top Réseau clinicians- phone 166 Household interview 180 Client exit interview 32 Top Réseau clinic visits 7 Beneficiaries (FGD) 68 Central GOM 11 District GOM 6 USAID/Madagascar Officials 3 PSI/ISM staff 17 USAID IPs 2 PARCs 4 PAs 11 CHVs 50 IPCs 26 Radio Station Managers 3 Wholesalers 5 Retail Checks 27 TOTAL 618

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 10

QUESTION 1A: IS THE ISM PROGRAM LIKELY TO ACHIEVE ITS OBJECTIVES AS OUTLINED IN THE INTERMEDIATE RESULTS AND RESULTS FRAMEWORK? IR1: Increase adoption of maintenance of health behaviors The team found that ISM was likely to achieve many of its targets. Table 2 summarizes baseline (as documented in the Cooperative Agreement) vs. achievements vs. targets for the key indicators listed in the Cooperative Agreement. Already measurable progress has been made in terms of the adoption and maintenance of health behaviors. Increases in levels of knowledge on the importance of hygiene, clean water, latrines, hand washing, diarrhea treatment are evident (TRaC, evaluation FGDs). Data from TRaC and Evaluation Team’s FGD and household surveys demonstrated that messages about malaria prevention were well understood (nets, rapid treatment). Appropriate treatment of diarrhea among children under five rose from 3.6% in the baseline to 8.1% in Fiscal Year (FY)14 and it is likely that the project will achieve its target of 12% by end-of-program (EOP). Similar steady progress has been made in under-five children sleeping under an LLIN. Preliminary data from the Malaria Indicator Survey (MIS 2016) presented to the team by PSI indicated that the EOP target of 80% has already been met, though data from the evaluation survey done in the three zones visited by the evaluation team indicate that only 64.3% of CU5 slept under an LLIN the previous night. Final data from MIS 2016 now indicate 73.4% of children under five years old slept under a bednet the previous night. Differences in these two findings may be due to differences in sampling and scope. The MIS had a national representative sample, while the evaluation survey was limited to three regions, one of which (Itasy) did not receive bednets from PSI in the most recent distribution campaign. Greater acceptance of FP among women of all ages, including youths, was reflected in FGDs conducted by the evaluation team. “Life is expensive and revenue is low and I already have lots of children,” was one comment from a woman of reproductive age who used Confiance injectables. Though not all men have been convinced, progress was made in getting men to accept family planning. Eighty percentof women said they were able to convince their partner to allow them to use oral contraceptives (2015 FP TRaC PSI, report amended 2016). Objectives were also achieved in terms of family health. Levels of knowledge on the importance of hygiene, clean water, latrines, hand washing, diarrhea treatment was evident (TRaC, FGDs). Malaria prevention was well understood (nets, rapid treatment).

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 11

Table 2. Summary of ISM Program Achievements up to FY 2016 (Source: MIS Program data summarized in FY 2016 Q3 Report and Annexes)

Indicator Baseline Achievements to Targets Likely by Date EOP

Contraceptive Prevalence 29.2% 36.28% 40.2% N Rate (FP TRaC 2015)

Couple Years of Protection 561.510 907.232 831.600 Y (CYP) per Year

U5 Slept under 76.5% 73.4% 64.3% 80.0% Y LLIN (Eval HH survey)

Diarrhea treated 3.6% 8.1% 12% Y appropriately (TRaC FY 14)

Sur’Eau (liters of treated 2,522,270 L 2,801,450L 5,925156 L Y water)

#TR providers providing 3 213 Urban 220 Urban 233 Urban N services 0 Rural 15 Rural 40 Rural

% CHV reporting 80% 71% 90% Y no stock outs

Radio, was the preferred channels of communication on family planning and family health. This was reflected in the 12 FGDs held with ISM target populations, the household study conducted by the evaluation team and triangulated with the ISM TRaC studies. For example, the Family Health drama and local radio were cited as important communication channels by about half of respondents in the household study. When asked specifically on which channel they had heard messages regarding MCH, 41% of respondents cited radio compared to 8.4% for television and 5.6% for CHVs. Another measure of the reach of radio was the 58.3% of urban and 42.8% of rural respondents who said they heard a radio spot promoting Sur’Eau (Diarrhea FY15 TRaC). CHVs were not a channel of communication in this project beyond product promotion. However, while the contraceptive prevalence rate of 29.2% in the baseline has risen to a current rate of 36.28%, it is unlikely to achieve its original EOP target of CYP of 40.2% given its current family planning sales pattern. The evaluation team also found it unlikely that ISM would accomplish its target of Top Réseau providers offering integrated services in at least three health areas – FP/RH, IMCI/nutrition, and malaria. The target as per the Cooperative Agreement was that by the EOP, ISM would have 40 rural and 233 urban Top Réseau clinics providing these services. To date, ISM has contracts with 251 clinics – 36 rural and 215 urban. The ISM Data provided to the evaluation team revealed that of the 251 clinics in the Top Réseau franchise, only 235 were providing any of services concerned (FP, MCH, and Malaria). This includes 17 of the 36 “rural” sites that are part of ISM through partners SAF and SALFA. When the evaluation team compared the listed communes of the 17 rural clinics, we found that only 15 could be considered “rural” per guidelines from the Madagascar Ministry of Interior (Liste et Classement des Communes, Madagascar Ministry of Interior, 2016). Thus, the data suggests that 220 urban and 15 rural Top Réseau clinics are providing these integrated services. This finding was discussed extensively with PSI and while they agree that data may not be available for each facility, they suggest that as long as facilities are under contract that they should be counted as an achievement for that indicator.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 12

QUESTION 1B: WHAT WERE THE MOST AND LEAST SUCCESSFUL ACTIVITIES IMPLEMENTED BY THE PROGRAM? WHY? IR1: Increased adoption and maintenance of health behaviors Most: Radio has been the most effective channel in reaching the ISM target populations parents of U5 children, according to the 2014 TRaCs on pneumonia and diarrhea. Fifty-three percent said they had been exposed to messages on pneumonia prevention and treatment on the radio. About 46% said they had heard radio spots on Sur’Eau. By contrast 13.2% saw TV spots, 20.1% saw or got materials, and 0.5% attended a group activity or talk by a CHV (2014 Diarrhea TRaC). Most significantly, the two studies found that those exposed to radio messages were on average nine times more likely to treat water, have confidence in Sur’Eau, and believe in pneumonia treatment of a CU5 than non-exposed parents (2014 Pneumonia and Diarrhea TRaCs). Radio reach is more limited in rural areas but is still the most cost-effective means for reaching target populations. A total of 48,689 radio spots were broadcast on 80 radio stations in 2015. Fifty-four percent of household study respondents said they had heard the Family Health radio drama.

Least: ISM aired 8,954 television spots in 2015 that reached mostly urban, upper wealth quintiles who own TVs. Eleven percent of parents said they saw spots on pneumonia and 17.2% saw spots on Sur’Eau (2014 TRaCs on Pneumonia and Diarrhea). In the evaluation team’s household survey, less than eight percent said they had seen TV programming on LLIN and maternal and child health. Mobile Video Unit shows was the primary mid media channel used to reach rural areas that were not covered by radio signals. Three vehicles conducted 440 shows in 2015. The MVUs are relatively expensive considering their very limited contact with target populations. In FY 2016, the three MVUs met only 47.7% of their goal of reaching 79,200 people and at that rate will be hard-pressed to reach the EOP goal of 111,078. Messages delivered “seems not to be retained or misunderstood,” the TRaC FP reported. Other mid media, such as point of purchase promotional materials and posters, were also limited. Plans have been made for increasing CHV visibility with signage but MOH approval was only received in August 2016.

IR2: Improve quality of health services in the private sector Most: Top Réseau, the network of 251 franchised private sector clinics, expanded into rural areas with the addition of 40 clinics (PSI reports that 36 are currently operating, though ISM service delivery report only includes data for 15) that joined the network. ISM contributed to the development of well trained staff and expanded services offered specifically for FP and MCH. Annual ISM supervision reviews found that 92.3% TR Family Planning providers scored at least 80% on FP quality standards in 2015 compared to 58.2% in baseline. One hundred and twenty (120) Youth Peer Educators, who organize events and visits schools and 240 FP Counselors, who mostly do house to house visits, promote use of Top Réseau clinics especially to the underserved poor, including youths. In FY 2015, as many as 66,887 FP clients came with a voucher, evidence of Top Réseau providing FP services to the underserved (ISM FY16 Q3 report Annex 3, MIS Program data). These vouchers provide a discount without which it is unlikely that these clients would have been able to afford FP services. This allowed ISM to reach 104% of its FY 2015 target. Youth Peer Educators, who also escorted youths to Top Réseau clinics, contributed to youths making up 47.4% of total Top Réseau FP clients (68,804/143,032) in FY 2016 (ISM Données par Centres Top Réseau FY13 and FY16 Excel Report). Twenty-four

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 13

percent of TR clients were found to be youths in the exit study (Madagascar (2015): Top Réseau Clients Exit Survey, Baseline 2015). The quality of the services of the Top Réseau clinics and the affordable socially marketed commodities were appreciated by the ISM target populations who used them, according to the two exit studies, household surveys, and the FGDs. Top Réseau clinics are providing 19.21% for the total Couple Years of Protection provided by the ISM program, (174,247 CYP out of 907,222). In the ISM FY 2017 budget, IR2 has been allocated $1,400,000, which is 29.84% of the total budget of $4,661,000 for all three IRs. In terms of FP commodities mix, the proportion of user dependent products, Confiance and Piplan, sold through Top Réseau clinics is a relatively small 1.8% when compared to community and pharmacy outlets, while provider dependent products, like IUDs are all delivered through Top Réseau clinics.

Least: Regular supervision is part and parcel of the ISM implementation model. However, several findings make us question the regularity of supervision. First, PSI did not have an up-to date list of contact information for Top Réseau clinics. Second, when Top Réseau clinics were asked in phone interviews how often they are supervised, we received a variety of answers; and finally, Top Réseau clinicians were unable to explain the metrics by which the quality of their service delivery is assessed. Fifty-four percent of Top Réseau clinics provide 90% of the total number of all consultations, and 50% provide 90% of FP consultations (ISM Données par Centres Top Réseau FY 2013 – FY 2016, Excel). This is an indication that the full potential of the network is not being used. Inadequate marketing and promotion results in a low recognition of the Top Réseau brand and poor understanding of concept.

IR3: Increase availability of health products and services A combination of community, commercial, and pharmaceutical distribution, combined with Top Réseau outlets, has made socially marketed commodities affordable and widely available. There have been some stock-outs but usually as a result of national level supply and not failures of the supply chain. All FP methods have been, in general, increasing in the market (Figure 2). Confiance and Implanon sales have grown steadily since being introduced. There has been more limited demand for ViaSur. Though ViaSur sales have grown from 34,144 in the baseline to 99,952 to date, they are well below the EOP target of 1,122,932. Yes condoms have been introduced in just a few regions. Pneumox has been difficult to sell and has not yet been made available to CHVs as they are waiting to be trained in its use. Some of the mature brands like Sur’Eau, Pilplan and Protector Plus have been socially marketed in Madagascar for over 25 years and are very well known and have a high degree of brand loyalty. According to the Family Planning TRaC 2015 and the Figure 2. PSI TRAC FP Modern Method Contraception FGDs, the socially marketed Prevalence % Married Women of Reproductive Age products are affordable. There was also consensus among beneficiaries, CHVs, and pharmacies interviewed in FGDs and KIIs that

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 14

small increases in prices are possible (e.g increasing price of nets from the current 3,000 Ariary up to 5,000 Ariary, as per one participant in an FGD with mothers of children under five. Most participants in that same FGD were fine with a 500 Ariary increase on nets. These numbers should be taken cautiously as this was not intent to pay study). But large increases or ending subsidies would eliminate ability of many users to pay, especially the youth and rural populations.

Most and Least Successful Activities: Products and Services

Most: The community distribution of commodities through CHVs reaches rural and poor populations at prices that are affordable to them. The small margin the CHVs get from their sales contributes greatly to their motivation and sense of contributing a valuable service to their community. The CHVs said in one of the six FGDs that they have “seen big changes” in the acceptance of family planning and have a constant demand for the products they sell. “Family Planning is no longer a taboo” (CHV FGD participant). The revised delivery system – that now includes delivery of products to PARCs, and facilitates, through a voucher system that pays for transport, Points d’approvisionnement (PA) to pick up products from PARCs, and CHVs to pick up products from PAs – has strengthened the system and reduced stock-outs. Sixty-one percent of Pilplan obtained through community distribution compared to 39.2% through pharmacy distribution. (See Table 3 below).

Least: Though PSI asserts that PARC turnover is less than 10%, the evaluation did find that there were some challenges with the motivation of PARCs, particularly those in remote areas, as PAs sometimes opt to go to another PARC and without PAs, PARCs do not sell their products. However, those interviewed also appreciate their contribution to society through their social marketing work while making a modest profit. There was some slippage of Pilplan and community Sur’Eau that was found for sale in two commercial markets. Table 3. Comparison of Community, Pharmacy and Top Réseau Distribution of Pilplan and Confiance (ISM-Distribution Activity Level Indicator FY 2013 – FY 2016, Excel)

FY13 FY14 FY15 z FY16

Pilplan Community 1,222939 1,712,114 2,157,552 1,654,997

Pilplan Pharmacies 807,726 1,123,318 1,419,689 949,711

Pilplan Top Réseau 7,567 18,091 18,679 21,770

Confiance Community 826,471 1,355,153 1,721,973 1,625,685

Confiance Pharmacies 444,959 689,047 458,823 349,212

Confiance Top Réseau 42,232 57,408 63,397 63,994

IUD Top Réseau 13,584 29,287 28,838 28,838

QUESTION 2: HOW WERE THE THREE PROGRAM INTERMEDIATE RESULTS INTEGRATED (FROM A MANAGEMENT PERSPECTIVE AND AT THE ACTIVITY LEVEL)? WHAT WERE THE BENEFITS OR DISADVANTAGES TO THIS INTEGRATION? 2A. How were the three Program intermediate results integrated (from a management perspective and at the activity level)?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 15

ISM’s perception of integration was found to be different than that of USAID. To ISM, integration referred to integrated health services, namely family planning/reproductive health, maternal child health, and malaria prevention. Project management was organized to address and reinforce linkages among health areas. However, USAID’s perception of integration, as documented in ISM’s Cooperative Agreement and per the evaluation questions, was to identify linkages between the three IRs. PSI admitted that they did not consciously plan to integrate the components, and did not organize internal managerial functions to accommodate this. The Mission was also interested in knowing whether or not more resources went to one IR vs the others. One way to assess for this was to analyze the extent of the distribution of human and financial resources allocated to each program’s components. The evaluation team requested a budgetary breakout of expenditures by IR but never obtained it; PSI’s response was that their financial reporting system does not use Intermediate Results. Of all documents submitted to the evaluation team, only the FY 2017 work plan includes information on how budget allocations addressed IRs for budgeting and planning purposes. As a result, the team was not able to conduct any in-depth analysis on how the three IRs were integrated from a management perspective. On an activity level, ISM’s integrating factor to produce sustained health behavior changes was the execution of an overarching SBCC campaign that would induce consumers to use enhanced quality services at the Top Réseau franchise clinics, and/or to obtain lifesaving commodities in the commercial sector (pharmacies, depots, grocery stores) in the community through the Community Health Volunteers. Conceptually this SBCC campaign was implemented through a mix of three different communication channels, mass media (national TV and radio, local or community radio), mid media (primarily using Video Mobile Units), and Inter Personal Communications (Youth Peer Educators, Family Planning Counsellors and to some extent the CHVs supported by program subcontractors and other non-governmental organizations (NGOs)). These messages were to motivate beneficiaries to use the services of Top Réseau clinics, and to buy the products. The team found the most concrete evidence of integration of the three IRs within Top Réseau clinics: IR1 – SBCC was represented through mass media, IPC, including CPFs promoting the use of Top Réseau clinics. Evidence suggested this worked, as 66,000+ users of Top Réseau for FP services came in with a voucher only obtainable through interaction with a CPF. Within a Top Réseau, health services (IR2) were provided by doctors who are well trained, and socially marketed products (IR3) were made available and sold to clients.

As for the integration of IR1 and 3, there is evidence of a consistent effort to use mass communications to promote products and brands. The linkage between generic SBCC and specific brands, products and points of purchase is less Focus group discussions with Top Réseau youth volunteers. Photo credit: Iain McLellan evident, mainly because of the poor performance of mid media efforts. The team could not find evidence of concerted efforts to

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 16

document and monitor linkages and their effect. It is, therefore, impossible to establish if such efforts could have had disadvantages in program implementation. Additionally, the multiplicity of programs conducting SBCC campaigns, both through mass media and IPC, makes it difficult to differentiate any individual program’s contribution to impact on the community distribution.

2B. What were the benefits or disadvantages to this integration? The team found that such linkages were beneficial. For example, vouchers represent the only approach by ISM to target and reach the urban poor. Thus, it is likely that many of those who received FP services using a voucher at a Top Réseau might not have otherwise received it. The linkage between social marketing distribution and the clinics resulted in a contribution of the franchise to CYP. Specifically, the Top Réseau clinics accounted for 19.21%, of the total CYP 2016 according to ISM or 174,247 CYP. However, as noted earlier, this integration did not benefit as many people as it could have, in part due to underperformance by 46% of the clinics contributing only to 10% of the number of consultations (ISM Données par Centres Top Réseau FY 2013 – FY 2016, Excel).

QUESTION 3: HOW WAS THE TARGET POPULATION (WOMEN OF REPRODUCTIVE AGE, CHILDREN UNDER FIVE, YOUTH 15-24 YEARS OLD, AND THOSE LIVING IN RURAL AND UNDERSERVED AREAS) FUNCTIONALLY DEFINED BY THE ISM PROGRAM? DID PROGRAM ACTIVITIES REACH THIS POPULATION? WHY OR WHY NOT? IR1: Increased adoption and maintenance of health behaviors The definition of rural and underserved targets for mass and mid-media coverage was not explicit in the program. When asked, PSI said they defined “underserved” as those not reached by radio. Their strategy here was to use mobile video units to circulate mainly in remote rural areas, but also for really poor urban areas where the population is likely to not have access to radio. However, this did not yield the anticipated results. Data from this evaluation, as well as previous research done by PSI, show that the MVU is the least effective communication channel in terms of cost and numbers reached. Almost all beneficiaries who participated in rural and urban FGDs reported never seeing a MVU. Recollection of specific messages by those who had seen them was sparse. In fact, the scope of the MVU is rather limited in rural areas with only three units reaching a total of 143,000 people during the life of the project to date (ISM FY 2015 Annual Report). PSI reports difficulties in accessing rural areas and the insecurity of certain urban areas as major constraints. The MVUs did some events in urban as well as rural areas. However, PSI reported low attendance to these events. They would gather approximately 1,000 people on a given evening. When the cost and time of travelling are calculated against the numbers reached, MVUs turn out to be limited in reach and relatively costly.

Program Activity Reach FGDs and KIIs revealed that local radios and IPCs are most effective in reaching target populations. This data was triangulated by the TRaC survey results in 2014, which showed that 42.8% of people in rural areas heard radio spots on the Sur'Eau product and 50.0% on pneumonia (Diarrhea TRaC, 2014). Indeed, radio is the most common media channel used by households in comparison to TV and other media – 43.7% of rural women aged 15-49 listen to radio at least once a week and more than 68% in urban areas (MDG survey, 2013). Greater involvement of local radio stations in adapting messages to the local context, culture, and

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 17

dialects could potentially reach a wider audience and have greater impact on behavior changes, according to KIIs with radio stations and district-level ISM partners.

IR2: Improved quality of selected health services in the private sector

The ISM program relies on the official definition of rural communes to recruit rural Top Réseau clinics and target poor populations in rural areas. This criterion is justified by the high proportion of poor within the rural population, as 77% of rural populations are considered to be affected by poverty (MDG survey, 2013). In collaboration with SALFA and SAF FJKM, the program also identified 36 clinics to join the network and expand health services for the poor. From data provided by PSI, only 17 of the 36 had provided any services in FY 2016. And of the 17, two clinics were found to be actually located in communes officially classified as urban, namely Mananara Avaratra and Maroantsetra (Ministry of Interior Affairs, 2015).

As many as two-thirds, or 67%, of the rural Top Réseau clients belong to the poorest wealth quintile classes (Top Réseau Client Exit survey, PSI 2015), confirming that the clinics managed to reach a fraction of the poor in rural areas. However, the program data show that only 6.4% of clients in all Top Réseau clinics are rural. They are mainly composed of WRA and CU5. In addition, only 7.8% of the WRA using the FP services of the Top Réseau clinics come from rural areas (PSI Top Réseau service statistics Excel spreadsheet). The low number of rural clinics largely explains the low coverage of rural populations. And although these rural Top Réseau clinics are located closer to the populations living in poor areas, their impact on coverage is hampered due to accessibility difficulties. Client Exit Interviews conducted by the evaluation team revealed a high proportion of clients (62.5%) said that clinics are far from their homes.

Program Activity Reach The ISM program has increased the number of Top Réseau clinics in both urban and rural areas to increase the number of underserved and rural populations amongst their clients. The ISM program has set up an e-voucher system at the Top Réseau clinics that subsidizes the cost of consultation and allows low income populations to access the Top Réseau services. Youth aged 15-24 are defined by the ISM as part of these vulnerable people. The team’s analysis of the proportion of youth among clients of the Top Réseau clinics users of family planning services indicates that the voucher system is effective in reaching these vulnerable people. Indeed, this proportion has risen from 35.6% in 2013 to 48.1% in FY 2016 (ISM Excel spreadsheet). Thanks to the voucher system (based on socio-economic and demographic criteria defining the poorest), IPC agents succeeded in increasing the number of youth served by the Top Réseau clinics. In addition to youth, Top Réseau clinics also have to target other underserved people in urban and rural areas. The following paragraphs analyze separately the coverage of target populations by the Top Réseau clinics in urban and then in rural settings.

Urban Top Réseau Clinics The ISM program has set the goal of extending TR quality services offered at affordable prices in order to reach the urban poor. However, it is not clear whether the recruitment procedure for Top Réseau implemented by ISM has adequately considered the location of medical practitioners in poor neighborhoods as a key selection criterion. In addition, the profile of urban Top Réseau clients shows that 65% of the clients belong to the richest categories whereas only 14% of the clients are in the poorest category (Top Réseau Clients Exit Survey, PSI 2015).

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 18

According to the evaluation team’s analysis, this disproportionate percentage of Top Réseau clients from the highest wealth quintiles is explained in part by the consultation fees established by the Top Réseau clinics. These fees were deemed to be too expensive according to 32.3% of the clients (Evaluation client exit survey, 2016). Consultation prices vary between US$ 1-5 but can reach US$ 16 at a Top Réseau clinic in Diego I. Compared to the weekly per capita income estimated at US$ 26.47 in Madagascar (http://www.tradingeconomics.com/madagascar/gdp-per capita-ppp), as of 2017 these average consultation prices are relatively high as they represent 5 24% of the weekly expenses of a person living in an urban area.

The voucher system was put in place by the ISM program to alleviate these costs for some clients and proved to be effective. However, due to the limited resources of the ISM program to finance this system, the scope of vouchers for the urban poor remains limited. Not all the urban Top Réseau clinics had extensive coverage of underserved populations. This was due to their customer base already being low in terms of numbers of clients. Indeed, 90% of the total clients of all Top Réseau clinics come from 54% of the operating clinics, most of them in urban areas according to evaluation team calculations. Thus, half of the Top Réseau clinics have limited client base and, as a result, cannot cover significant numbers of vulnerable urban population.

Rural Top Réseau Clinics IR3: Increased availability of lifesaving health products and services The ISM program has developed its social marketing activities to improve the availability of products at the level of commercial and pharmaceutical distributors in urban / peri-urban areas and to improve community-based distribution (CBD) through supply points serving rural areas.

Community Distribution The ISM program has established supply points (PAs) and community relay supply points (PARCs) in the intervention zones of the USAID health projects, namely Mahefa, Mikolo, and Santénet2, which target rural areas and underserved populations. The CHVs work closely with these projects and source from the PAs to sell the products to the populations in their villages. With a few exceptions (CHVs work in the city of Diégo I), supply points and CHVs are essentially serving the rural population, especially women WRA and CU5. On the other hand, youth are not the priority Figure 3. Proportion of rural WRA using targets of CHVs that have not been specifically FP at Top Réseau clinics trained to educate them on reproductive and youth health. The focus group results revealed that despite the end of the Santénet 2 and Mahefa projects, most of the CHVs continue to provide their services and products to women and CU5 thanks to the IPC techniques already well proven and the availability of some products. The community distribution approach is particularly effective in reaching the rural poor because most products are available at affordable prices for rural people. Nevertheless, IMCI products like Via'Sur, Pneumostop, Pneumox, Artemisinin Based Combination Therapy (ACT), and Sur'Eau have seen stock-outs in two regions visited.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 19

Commercial and pharmaceutical distribution. The program's approach to targeting underserved populations through the commercial and pharmaceutical channel involves selling products in urban areas for a segment of the population that has some capacity and willingness to pay. Participants in the KIIs and focus groups reported that youth using Protector Plus condoms, WRA women, and CU5 parents are the main customers of commercial resellers and pharmacies. However, products for CU5 such as Hydrazinc, Pneumostop, Pneumox, and Sur'Eau are not available in pharmacies in the regions visited during the evaluation. In addition, Pilplan and Confiance, the highest selling products available in pharmacies and retailers are sold at prices considered as high by 34% (Pilplan) and by 40% (Confiance) of respondents (Evaluation household survey) and are not intended for the urban poor. The market segmentation strategy adopted by the ISM program explains the price differentiation applied in the commercial channel, thus leaving out a fraction of the underserved urban population that does not have the capacity to pay.

QUESTION 4: HOW WAS PROGRAM LEARNING (INCLUDING OPERATIONAL RESEARCH AND, PARTICULARLY, RESULTS OF INNOVATIVE ACTIVITIES) DOCUMENTED, DISSEMINATED, AND APPLIED TO IMPROVE ISM PROGRAM ACTIVITIES AND INTERVENTIONS BY OTHER HEALTH SECTOR PARTNERS? Documentation of Research in ISM program is effective and used to improve the program. An ISM Research Strategic framework exists. This document delineates steps for conducting research studies: study design, training, field work, analysis, report preparation and dissemination. To date, ISM conducted 42 research studies, including 29 on results of innovative activities (eq. 69%). Many of these studies were completed in FY 2015 and FY 2016.

Most ISM research activities were designed and executed by PSI in-house. ISM has a staff of six people with skills in qualitative and qualitative methodology. In KIIs ISM research staff said they make minimal use of freelance researchers and outside research firms in order to guarantee quality. All research is documented either as a paper or PowerPoint presentation. Research results are used for project improvement, including annual work plan development Figure 4. Studies & Research conducted by ISM and monthly planning meetings with partners (SAF, SALFA, etc.). ISM also uses results to launch new products, and to modify products and key processes such as distribution, as well as establish pricing structures. SBCC materials and media outputs such as spot ads were all developed based on research including pre- and post-testing. Limited sharing with others partners. Some dissatisfaction was expressed by ISM partners, including those in both governmental and non-governmental organizations, regarding the level of sharing of research results and data. There were indications that sharing was limited to discussion during a meeting or a PowerPoint presentation to invited participants but not distributed systematically to all partners. It is also likely that persons who attended such sharing

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 20

events did not systematically share the data with colleagues, thus limiting exposure to PSI research results within their organization. The net result is limited numbers of those intervening in the health sector having access to ISM data. Out of the 42 studies, only six have been disseminated as of FY 2016.

Partners do not report using ISM research for their decision-making. The type of data collected by many ISM studies could very well be used by other IPs and stakeholders to inform their work plans, implementation, message development, SBCC activities, and mass media use. Yet KIIs with IPs and the GOM revealed concern that ISM data was not systematically diffused despite the fact that they have limited access to other Madagascar-specific data. They viewed this as a real missed opportunity to improve their work.

Government, both at national and in districts, reported limited sharing of research or programmatic data by the ISM program. MOH officials in particular interviewed in KIIs suggested that a more systematic diffusion of ISM data would help them with strategic planning and decision-making. Interviews with the Director General of the MOH revealed not only dissatisfaction, but unmet need and missed opportunities, as illustrated by this quote: "I've never received results of research" from PSI. I have not seen studies of BCC done by PSI, yet research results are very important to the MOH. We don't have enough research that is Madagascar-based. We could revise our strategy based on research" (DG, MOH). According to the MOH, not only was research data not shared satisfactorily, neither was programmatic data: "They [PSI] do not even share their workplan for the district with us. When we ask them for data, they say that they must first get permission from their headquarters" (District Medical Officer).

Partners are not satisfied with the level of engagement in the research process. Partners also mentioned the desire to participate more fully in the research design process, including the selection of research questions. TRaC tools are standard but for other types of research, sharing the process of developing instruments would contribute to strengthening partners’ capabilities for the development for future research.

QUESTION 5: TO WHAT EXTENT ARE ISM ACTIVITIES SOCIALLY AND ECONOMICALLY SUSTAINABLE? HOW COULD THE PROGRAM DESIGN BE ADAPTED TO IMPROVE SUSTAINABILITY? Sustainability of Systems Analysis of the system for Increase adoption and maintenance of health behaviors (IR1) revealed that this is largely driven by PSI. PSI staff takes the lead in ensuring that messages that promote healthy behaviors are developed, scripted, and aired. Interviews with PSI staff suggested that almost all inputs, including behavioral studies, were done in-house by PSI. Processes were mostly PSI’s, with some involvement of local organizations. Specifically, PSI developed the SBCC strategies, including behavior goals, media mix plan, training of IPCs, and the development of materials including posters and spot ads. Data from PSI suggests the involvement of contractors to organize promotional events, produce TV and radio spot ads, and produce “Healthy Family” campaign materials under close supervision and management of PSI.

For SBCC, PSI was largely responsible for exploitation of outputs though an extensive network of radio stations aired the ISM spots and programs. PSI distributed print materials, and all IPCs (except CHVs) were trained by PSI. CHVs were trained by other USAID partners with material from PSI. At the national level, ISM provided useful technical support to the GOM in SBCC

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 21

strategic planning and the development of homogenized messages and materials. KIIs with the GOM revealed that ISM was particularly active in its collaboration with the MEAH and provided a consultant to develop its SBCC strategic plan. A high level official within the MEAH expressed appreciation for ISM support in producing and paying for the diffusion of spot ads on radio and television that promote hand washing, construction of latrines as well as the promotion of Sur’Eau. However, PSI is viewed as the leader of the activity, and the MEAH expressed feelings of dependency on PSI for communications. “We can’t do research, produce or disseminate messages so we depend on PSI.” (KII, MEAH, GOM)

The MEAH also expressed that they would welcome training in SBCC strategic planning at both the national and regional level to reduce dependency on PSI. KIIs with USAID and PSI revealed that ISM also collaborated with the MOH’s National Malaria Program and the SBCC sub committee made up of international organizations and NGOs. ISM conducted a valuable behavior study on obstacles to LLIN use for Roll Back Malaria, and took the lead in organizing mass distribution of LLINs. However, KIIs with district leaders revealed no evidence of local leaders and institutions working with ISM to increase adoption of healthy behaviors.

The system to improve quality of health services in the private sector (IR2) in use by PSI was largely driven locally. The team found that almost all inputs, including the providers and private clinic space as well as the majority of necessary equipment, were all local. PSI’s input included training providers in integrated delivery of FP, STI, youth services, cervical cancer, IMCI, etc.), provision of additional furniture/materials (including LLINs), and ensuring access to commercial FP products to Top Réseau clinics. Processes, including patient flow, data management, stock management, among others, were all local. PSI contributed to recruitment of new clients, including youths, through IPCs and the voucher system. Outputs, specifically provision of health services, including FP services, were all local. However, the Ministry of Health at national and district levels expressed dissatisfaction with their level of engagement, particularly around sharing data for reporting and decision making. In particular, interviews with members of the DMO office in two towns expressed a lack of recognition of their office by PSI, and were in turn not as familiar with the activities of the ISM project.

ISM used two systems to increase availability of health products and services (IR3) – community distribution, and commercial distribution. In the system for community distribution, a key input – specifically, commodities – was all PSI. Other important inputs, including staff (PARCs, PAs) were indirectly PSI as they only exist because of the subsidy created from the distribution chain. Community health volunteers are not dependent on PSI but are largely supported by other USAID partners. While supported by national policy, it was unclear to the evaluation team the extent to which existing CHVs would continue without external support, or what the capacity of the country was to recruit new ones as this was out of the scope of the evaluation. Processes, in particular, transportation and advertising, were mostly PSI. However, PSI’s introduction/addition of key stopping points along the way (PARC to PA to CHV) is well established, and could be adopted/adapted by others willing to subsidize transportation of PAs to PARCs as is done now by PSI. The output – or specifically, sales – were all local.

In the system for commercial distribution, the team found that except for commodity subsidy, other inputs, including working capital, transport, space, and staff, were all local. Without the subsidy, however, there would not be a commercial distribution of most socially

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 22

marketed products. Processes were largely local, except for transportation from PSI to the warehouses. The commercial sector alone moves the product from wholesale to retail outlets, and conducts their own stock management and restocking activities. PSI also helps the commercial sector in marketing the products through various channels. Outputs – specifically, sales – were all local.

Sustainability Dimensions

Table 4. Evaluation of Sustainability Dimensions by IR*

IR IR1 IR2 IR3 IR3 SBCC Top Réseau Social Marketing Community Distribution (qual) Ownership 6.6 8.0 Policy 6.1 3.0 Advocacy 4.3 7.0 6.7 Staffing 7.3 9.0 6.0 Physical 6.3 6.8 5.7 Org 7.6 7.0 7.0 Financial 4.2 8.5 1.0 Public Trust *Red = 0 – 4.9 (not yet sustainable); Yellow = 5.0 – 7.9 (needs more work, but on its way); Green = 8.0 to 10.0 (sustainable) Table 4 summarizes findings for each IR based on data from interviews. For SBCC, this was largely qualitative, but for the other areas people were asked to rate various questions and the averages are summarized. Domains were further substantiated by document review, and used to build sustainable “houses.” KIIs with GOM at national and district levels, as well as with radio station managers, revealed that for SBCC, local ownership of messaging and processes was relatively low. While aspects of the policy environment are improving (e.g. malaria communication plan developed with significant input from PSI, KII USAID, PSI), there is still significant work to be done to line up Madagascar’s communication policies with the health needs of Malagasy for FP, IMCI, and other areas. The team found minimal local capacity for advocacy for SBCC; staffing involved in ISM’s SBCC efforts were mostly PSI’s. Though physical infrastructure was present (e.g., local radio and TV stations) and functional, organizational infrastructure to enable SBCC to be locally derived and implemented was severely challenged. KIIs with GOM officials revealed that the MOH is in the process of reorganizing the way it handles its health communication needs. The team was not able to review the extent to which there are budget lines within the MOH budget for SBCC activities, but was told by GOM officials that additional financial support is still needed for SBCC as well as most aspects of the MOH, and that systems became even more fragile due to the recent political crisis. Nevertheless, interviews with would-be beneficiaries revealed a great deal of confidence placed in health messages heard through the radio and, to a lesser extent, TV. Additionally, participants in FGDs held in five of six districts were able to correctly explain ways to prevent diarrhea and malaria, an indication that SBCC messages were learned, though not necessarily through the ISM efforts, as there are other actors involved. KIIs with GOM officials

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 23

suggests minimal, if any, financial viability of these efforts. “If PSI doesn’t do it, it is a big problem.” (KII, MEAH, GOM). Results are summarized in Figure 6.

For efforts to improve quality of selected health services in the private sector (IR2) through Top Réseau, the team found that the Top Réseau network does not exist as an independent body, and the glue that ties all clinics together is PSI through the ISM project. PSI also owns the Top Réseau brand. As such, the Top Réseau network itself is not sustainable without PSI. Nevertheless, we analyzed each dimension within the framework as if Top Réseau was an entity, and summarize results in Figure 6. While ownership of clinics was high, ownership of the idea of private facilities providing subsidized care to the poor was not. While service provision would continue without PSI, prices would likely increase. “At these prices it would not be profitable given the cost of the [family planning] products plus time to insert. But since we buy cheap from PSI and they also give us needed materials like gloves, it’s ok. If PSI is not here, we will pass these costs on to the client.” (Top Réseau Medical Coordinator)

Another district official noted that there were likely differences in types of ownership among the various types of clinics recruited to become members of the Top Réseau, depending on their initial (pre-Top Réseau) orientation. “Members of Top Réseau who are part of another network that already have the same mission of reaching the poor and vulnerable, that’s OK. For example, Dispensaire Catholic has a special mission to reach the poor, they are interested in reaching the poor. But for private offices, those that are owned privately/personally, it is more difficult. They are only interested in gaining as much money as possible as quickly as possible. Top Réseau model is not sustainable there…” (KII District Official, GOM) The policy environment for private service provision is becoming increasingly favorable. Nationally, critical policies for Universal Health Coverage (Strategie Nationale sur la Couverture Sante Universelle Madagascar, 2015), as well as privatization of health services (Contractuelle policy, unseen by the evaluation team) are all supportive of including private sector providers. KIIs with GOM at national and district levels all reveal that they consider the role of private sector health providers is important for health service provision, particularly in urban areas where a segment of the population completely rejects free services. There appear to be severe limitations on branding that makes it difficult for Top Réseau to maximize its potential for attracting new clients. According to PSI, the Top Réseau brand was revitalized in FY 2015 with new signs and fresh paint. However, the size of the Top Réseau sign is dictated by ministerial decrees, and the medical code of ethics also Figure 5. Sustainability of Efforts: IR 1 prohibits the promotion of franchise brands through mass media. In that same year, the National Doctors Association also formally asked PSI/Madagascar to stop its broadcasts promoting Top Réseau through radio and TV spots.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 24

Nevertheless, the evaluation team observed that signs for other non-public health entities, including Marie Stopes, were significantly larger than the ISM Top Réseau. We also noted through FGDs that beneficiaries were more likely to know about Marie Stopes or FISA (Fianakaviana Sambatra) than about Top Réseau. KIIs with district officials revealed an awareness of the restrictions due to branding on Top Réseau’s ability to attract new clients, and expressed a willingness to work strategically with PSI to overcome some of these barriers. One district official suggested that “Yes, they can advertise if they are providing a new service, but must have the OK of the District”. (KII District Official, GOM)

Unfortunately, these same district officials also expressed dissatisfaction with their current relationship with PSI, minimizing their chances of collaborating to improve this important aspect of ISM. The Top Réseau network itself does not have any advocacy capacity. As indicated earlier, Top Réseau staff are all local and independent of PSI for salary. Our findings indicate that the only “staff” that influence the work on Top Réseau paid by PSI are the CPFs. Top Réseau Figure 6. Sustainability of Efforts: IR 2 themselves, as well as key stakeholders, expressed confidence that clinics had adequate number of staff with capacity to deliver quality services. Similarly, the basic physical infrastructure clinics would remain post-PSI or ISM. However, many of the clinics receive additional furniture and some technological support from the ISM project. Additionally, they receive access to subsidized FP commodities without which they may provide less FP services. The team found that organizational infrastructure is still relatively week for Top Réseau. In the clinics visited by the team, we found weak documentation processes with indicator measurements that sometimes do not align with GOM requirements, weak communication between clinics and the MOH, and no internal systems that reward nor sanction. There are also relatively weak administrative processes that may be leading to inefficiencies. While a full evaluation of such was outside the scope of this work, ISM partners did express sentiments that the clinic doctors still had minimal skills to help them run their clinics as a profitable business. “It is difficult to promote an effective management culture and financial goals when clinics depend on PSI’s equipment and product donations.” (KII, ISM Partner) As individual clinics, they would continue to offer an integrated package of services inclusive of family planning. However, the low purchasing power of the majority of Malagasy currently, coupled with the lack of buy-in to the idea of serving the poor and underserved by most private clinics that do not already have this as a mission endangers financial viability of this effort. ISM intended to help Top Réseau providers to expand their businesses by obtaining loans. Successful business expansion could potentially allow for eventual replacement of PSI/ISM subsidies to patient care and socially marketed products. To date only 11 of the 251 TRs with whom PSI has contracts has obtained loans. (KII PSI and other ISM partners)

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 25

Results from Top Réseau client exit interviews, as well as household surveys, indicate a high level of trust and public confidence in Top Réseau. However, the evaluation team was not able to validate the quality of services provided, and when asked, Top Réseau clinicians (phone interviews, KIIs) were not able to share the metrics on which they are evaluated by PSI for the quality of services provided, and did not demonstrate any evidence of active engagement in continuously trying to improve such metrics.

Table 5. Product Sustainability Source: (PSI Madagascar, FP COGs Analysis 11.29.2016)

COST RECOVERY PRODUCT BRAND % Consumer Price to Cost of Goods Male Condom YES 213% EC Norlevo 136% CHX Arofoitra 94% CONDOM Protector + 48% IUD Copper T 19% Injectable Confiance 10% Oral Contraceptive Pilplan 8% Female Condom Feeling 6% Cycle Beads Rojo 3% Implant Implanon 2% Implant Jadelle 2% Implant Zarin 1% Injectable Sayana Press 0%

For efforts to increase availability of lifesaving health products and services (IR2), KIIs with PSI revealed that all of the current brands promoted by the ISM are owned and registered to PSI. Thus, local ownership from that perspective is non-existent. The team did find local ownership of the process for distribution in the commercial sector – as long as these commodities are available at subsidized prices, social marketing in the commercial sector will move products to clients by itself. In community distribution, PSI delivers products to PARCs and subsidizes the transportation of PAs to pick up products from PARCs. From PAs to CHVs to clients, the system works independently of PSI, as long as products are available.

There are no policies to support or promote social marketing in the commercial sector. However, the new MOH CHVs (currently being updated) does encourage participation of CHVs in community distribution of socially marketed products, and stakeholders at all levels within the GOM were incredibly supportive of community distribution. KIIs with persons involved along the supply chain of both commercial distribution and community distribution revealed that there is no formal organizational body that conducts advocacy on their behalf to ensure availability of socially marketed products. An unfortunate though exemplary demonstration of this is the absence of anyone who can negotiate with the GOM to deter their plans to remove Confiance from the market despite its solid market penetration and consumer loyalty to the brand, among

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 26

beneficiaries (KIIs with PSI and discussions during debrief). None of the staff involved in either commercial or community distribution are paid by PSI outside of margins made on sales. However, PSI reports a high attrition rate among its staff that work on social marketing, an indication of the extensive work done in addition to moving products. With regards to physical infrastructure, KIIs with both commercial and community distribution revealed a desire for more space. The evaluation team did note instances where commodities were not adequately protected against possible theft at the PARC and PA levels (cabinets, or storage rooms without locks). Organizational capacity was weak for both commercial and community distribution. Within the commercial sector, the team did not find any stated Quality Assessment/Quality Control (QC) practices, and PSI staff at national and regional levels expressed concern about the lack of capacity in supply chain management as well as marketing/detailing of the products. The team found that financial viability is possible with both commercial and community distribution as long as the margins remain at least where they are. If margins decrease, data from FGDs and KIIs along the supply chain of both systems revealed that in both sectors, most would stop selling the products.

OTHER RELEVANT FINDINGS Overall Management: The evaluation team noted some differences in understanding of the project between USAID and PSI. For example, in attempts to collect information from PSI about the integration of all three IRs (Evaluation Question 2), PSI informed the team it thinks of integration in terms of health services delivery (i.e., FP with MCH and/or IMCI). Their daily management of the project follows this type of integration, and KIIs with PSI revealed that they rarely do they consciously think and plan integration in terms of the three IRs; feedback from the debriefing meeting with them reiterated this. Despite numerous requests, PSI did not provide the team with a global breakdown of costs associated with each IR, nor any concrete information about barriers or enablers to integrating the three IRs, and the team is unable to provide any insight into what it takes to manage the project in an integrated – as intended – vs as separate IRs. Another big difference noted by the team was that while USAID views the growth of ISM (in terms of additional activities) as making the project more “unwieldy,” PSI perceived their ability to add new activities as “freedom to innovate.” They have used this freedom to introduce a number of innovations, including current discussions on using drones to bring commodities to hard-to-reach parts of Madagascar. Finally, ISM is on its second Chief of Party, and its fourth Agreement Officer's Representative (AOR). These important changes in personnel have potentially introduced discontinuity in communication between the project and its management at USAID, which may also have contributed to the project seeming more unwieldy to USAID.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 27

V. CONCLUSIONS

Question 1 – Achievements: The ISM program led by PSI provided essential support to the Malagasy people during a critical period of political and economic instability. Of seven key results ISM was to accomplish, they are on track to accomplish most except LLIN target (currently at 73.4%, compared to 80% target), and CYP rate of 40.2% with one year left to implement when the current rate is 36.2%. ISM is also not likely to have 273 Top Réseau, including 40 rural Top Réseau offering integrated services in at least three health areas (FP/RH, IMCI/nutrition, and malaria). Available evidence suggests there are currently 235 Top Réseau reporting the provision of any health service, including 15 rural Top Réseau clinics. Debrief discussions and document exchanges with PSI about this particular indicator suggest that this may be a reporting issue. The team also concludes there may be efficiencies to be gained with Top Réseau clinics, as currently half of them are providing more than 90% of the services reported.

The ISM has collaborated with Mikolo and Mahefa in other USAID-community-based funded activities, to provide them with technical support in SBCC and ISM product-specific messaging, which they then pass on to their CHVs. There have been positive changes in behavior and attitudes towards health, including growing acceptance of FP, but it is unclear what has been the extent of the influence of ISM SBCC, in part due to the involvement of other SBCC players (Mikolo, Mahefa, WASH, MCH, and others). Though ISM intended to have a multichannel SBCC strategy (Mass media, mid-media and interpersonal communication) but it has been primarily a single channel one with only the mass media, and in particular local radio in the rural areas having a significant impact. The mid media – in the form of the Mobile Video Units – has proven to be a weak channel. The ISM program, with its 1,000 employees and large field offices, was somewhat unwieldy and the focus on both the promotion of its socially marketed products as well as integrated private health service delivery and SBCC overall, in coordination with multiple partners, made SBCC very complex and coordination challenging at all levels. Progress was made with ISM technical assistance to the development of strategic SBCC plans in the areas of WASH and malaria in particular. ISM also provided leadership in ensuring that messages and strategies were focused on inspiring specific changes in behavior. ISM did contribute to the development of skills of small organizations it contracted to promote socially marketed products and generic behaviors. In fact, 28 contracts were given to groups to organize events, produce radio and TV spots, and other activities. ISM was less successful in meeting its mandate of preparing local partners to sustain SBCC interventions into the future as ISM was central in developing, supervising, and controlling the work of the contractors. ISM is only now trying to identify communication agencies to take on more responsibility for developing and managing the SBCC outputs. Radio coverage was limited but still the most effective mass media according to target populations – with potential for reaching 50% to 75% of the population. CHVs were motivated by sales of ISM products, which keep commodities available and sustains IPC. The team found that there is a need for better coordination of SBCC strategic planning and message and materials development.

Question 2 – Integration: Linkages between IR1 and IR2 and IR3 are clear and evident at the Top Réseau clinics. The linkage the IPC (Youth Peer Educators and Family Planning Counselors)

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 28

component of SBCC and services provision is demonstrated by the use of 64,000 vouchers handed out by them for FP consultations. The linkage between social marketing distribution and the clinics resulted in a contribution of the franchise to CYP. The team concluded that integration as seen at the Top Réseau facilitated access to health services delivery and commodities. However Top Réseau did not reach a significant number of people. Linkage between IR1 and IR3 was limited to product promotion. There is evidence of a consistent effort on the part of ISM to use mass media communications to promote products and brands. The linkage between generic SBCC and health behaviors promoted at points of sale of commodities was less evident, mainly because of the poor performance of mid media efforts. The evaluation team did not find evidence of concrete efforts to promote and measure the impact of integration overall.

Question 3 – Coverage: The SBCC media mix selected has limited success in reaching rural and urban underserved populations. Not all of the recruited Top Réseau are performing at their potential, resulting in weak coverage among the urban poor and rural populations. While ISM has 251 Top Réseau clinics under contract, they are only reaching 0.3% of rural WRA in Madagascar. Community distribution succeeded in reaching target populations in the rural areas. Commercial social marketing mainly services WRA and youth in urban areas who can afford to buy at commercial prices. Distribution is not reaching the urban poor and underserved, except for those who visit Top Réseau clinics with a voucher.

Question 4 – Research: The ISM program produced extensive research, including 42 studies to date and several other studies planned over the next year. ISM relies on research for a number of key project activities, including product development and message pre-testing. However, dissemination of results has been limited, and the use of the research by partners is also limited.

Question 5 – Sustainability: From a systems perspective, IR2, and IR3 were built on highly subsidized local systems, and as such, are more likely to remain at the end of ISM. However, various dimensions of sustainability – including, to some extent, ownership, advocacy, policy environment, organizational infrastructure, and financial viability – all need additional focus to ensure sustainability. The team found limited efforts to rely on local systems for the management and supervision of outputs of IR1, including communication products and research, and although ISM contracted local organizations it maintained tight management control of them. The team also found that ISM has not created – let alone implemented – a sustainability schedule. At district level, in particular, there is minimal collaboration between ISM and MOH and dissatisfaction was expressed in KIIs, particularly around data sharing.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 29

VI. RECOMMENDATIONS

THROUGH END-OF-PROJECT ● Overall, PSI needs to develop a close-out plan that takes into consideration sustainability and asset transfer, in collaboration with the GOM. The sooner plans are drawn for sustainability and asset transfer the better.

● Based on the findings for Question 1, improved performance monitoring of Top Réseau is needed. The first task would be cleaning the Top Réseau data and making it more user-friendly. This includes, among other actions, organizing the data by individual clinics instead of by clinicians so that it is easy and quick to tally the work of sites. ISM should also consider designing a strategy for non-performing Top Réseau providers. They will also need to recruit additional Top Réseau in the rural areas in order to meet the target of 40 Top Réseau clinics providing the three services in rural areas, though perhaps it would be wiser to focus on improving performance of poorly performing sites. This is a discussion that USAID may choose to have with PSI to decide a way forward. With regards to CYP, ISM should continue to promote and increase sales of long-term contraceptive products such as IUDs and implants since these have more impact on CYP than short-term methods.

● Based on findings for Question 3, in addition to the recommendations previously stated for Top Réseau, ISM should consider piloting a model of service delivery where the services go to the urban poor and the rural communities, thereby increasing coverage. Already Top Réseau clinicians participate in outreach activities when invited, but they do not provide care at such events. ISM should consider supporting Top Réseau clinicians, particularly those with very low utilization rates, to provide services at such events, or to organize new mobile outreach events. The ISM team should think through different strategies for urban poor vs. rural populations, including temporary use of existing buildings (churches, schools) for such events to the extent that such activities are in compliance with national norms and regulations. If mobile video outreach continues, Top Réseau clinicians should leverage their use to include provision of the three key services delivered through ISM (FP, MCH, malaria), as well as related product sales.

● Based on findings for Question 4, ISM should create a research dissemination schedule together with USAID to identify which of the 42 or more research reports should be disseminated, when, and how. ISM should also ensure that all research reports are available online. Finally, ISM should ensure that there is a clean, searchable database of all project indicators available. PSI has been using the District Health Information System (DHIS) 2 to accomplish a similar goal. However, since they did not share it with the team, and the data that they did share with the team required substantial additional formatting in order for it to be analyzed, improvements can be made in this area.

● Based on findings for Question 5, ISM should develop a sustainability schedule based on their Cooperative Agreement, which details efforts they will make to further strengthen the likelihood of sustainability of the three IRs, including all products. As part of this activity, they should increase engagement of the district leadership in the areas where they work, and should further engage local research organizations that could potentially

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 30

play a stronger role in helping to answer key programmatic questions. ISM should also increase effectiveness of its financial and marketing training to Top Réseau staff, and increase the number of Top Réseau clinics accessing outside funding to grow their clinics. ISM should further strengthen their relationship with the GOM at all levels especially around data with the Ministry of Health. Specifically, this includes extensive sharing of research results, as well as work plans and programmatic results within each district where they work. Lastly, ISM and USAID should further engage in discussions and resolve the issues around the sale of Confiance in Madagascar. USAID has made substantial investments over two decades in this brand, and it is well accepted by the population. Additionally, USAID needs to clearly establish who owns the branding of all of the ISM products, most of which have been developed with USAID funds.

● Our results indicate there may be room for charging additional fees for some of the socially marketed products. We therefore recommend a thorough, third party review of PSI Willingness to Pay studies with special focus on products that are near cost recovery to see if increasing prices could help move the products closer towards sustainability. Additional recommendations in this area include nurturing public private relationships with relevant pharmaceutical companies, as well as companies that deal in mineral, petroleum, chocolate, and other relevant industries to help with cost-share of subsidies for socially marketed products. We further suggest that USAID provides technical assistance to strengthen the policy environment around social marketing, and further focus on strengthening marketing, financial accountability skills among those selling social marketing products.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 31

ANNEX I. SCOPE OF WORK

Assignment #: 300 [assigned by GH Pro]

Global Health Program Cycle Improvement Project (GH Pro)

Contract No. AID-OAA-C-14-00067

EVALUATION OR ANALYTIC ACTIVITY STATEMENT OF WORK (SOW) Date of Submission: 9-8-16 Last update: 10-5-16

I. TITLE: PERFORMANCE EVALUATION OF USAID/MADAGASCAR INTEGRATED SOCIAL MARKETING PROGRAM II. REQUESTER / CLIENT  USAID/Washington Office/Division:

 USAID Country or Regional Mission Mission/Division: Madagascar / HPN

III. FUNDING ACCOUNT SOURCE(S): (CLICK ON BOX(ES) TO INDICATE SOURCE OF PAYMENT FOR THIS ASSIGNMENT)  3.1.1 HIV  3.1.4 PIOET  3.1.7 FP/RH  3.1.2 TB  3.1.5 Other public health  3.1.8 WSSH threats  3.1.3 Malaria  3.1.9 Nutrition  3.1.6 MCH  3.2.0 Other (specify):

IV. COST ESTIMATE: (NOTE: GH PRO WILL PROVIDE A COST ESTIMATE BASED ON THIS SOW) V. PERFORMANCE PERIOD Expected Start Date (on or about): October 2016 Anticipated End Date (on or about): February 23, 2017

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 32

VI. LOCATION(S) OF ASSIGNMENT: (INDICATE WHERE WORK WILL BE PERFORMED) Madagascar

VII. TYPE OF ANALYTIC ACTIVITY (CHECK THE BOX TO INDICATE THE TYPE OF ANALYTIC ACTIVITY) EVALUATION:

 Performance Evaluation (Check timing of data collection)

 Midterm  Endline  Other (specify): Performance evaluations focus on descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making. Performance evaluations often incorporate before-after comparisons, but generally lack a rigorously defined counterfactual.

 Impact Evaluation (Check timing(s) of data collection)  Baseline  Midterm  Endline  Other (specify): Impact evaluations measure the change in a development outcome that is attributable to a defined intervention; impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. Impact evaluations in which comparisons are made between beneficiaries that are randomly assigned to either a treatment or a control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured.

PEPFAR EVALUATIONS (PEPFAR Evaluation Standards of Practice 2014) Note: If PEPFAR funded, check the box for type of evaluation

 Process Evaluation (Check timing of data collection)

 Midterm  Endline  Other (specify):

Process Evaluation focuses on program or intervention implementation, including, but not limited to access to services, whether services reach the intended population, how services are delivered, client satisfaction and perceptions about needs and services, management practices. In addition, a process evaluation might provide an understanding of cultural, socio-political, legal, and economic context that affect implementation of the program or intervention. For example: Are activities delivered as intended,

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 33

and are the right participants being reached? (PEPFAR Evaluation Standards of Practice 2014)

 Outcome Evaluation Outcome Evaluation determines if and by how much, intervention activities or services achieved their intended outcomes. It focuses on outputs and outcomes (including unintended effects) to judge program effectiveness, but may also assess program process to understand how outcomes are produced. It is possible to use statistical techniques in some instances when control or comparison groups are not available (e.g., for the evaluation of a national program). Example of question asked: To what extent are desired changes occurring due to the program, and who is benefiting? (PEPFAR Evaluation Standards of Practice 2014)

 Impact Evaluation (Check timing(s) of data collection)  Baseline  Midterm  Endline  Other (specify): Impact evaluations measure the change in an outcome that is attributable to a defined intervention by comparing actual impact to what would have happened in the absence of the intervention (the counterfactual scenario). IEs are based on models of cause and effect and require a rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. There are a range of accepted approaches to applying a counterfactual analysis, though IEs in which comparisons are made between beneficiaries that are randomly assigned to either an intervention or a control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured to demonstrate impact.

 Economic Evaluation (PEPFAR) Economic Evaluations identifies, measures, values and compares the costs and outcomes of alternative interventions. Economic evaluation is a systematic and transparent framework for assessing efficiency focusing on the economic costs and outcomes of alternative programs or interventions. This framework is based on a comparative analysis of both the costs (resources consumed) and outcomes (health, clinical, economic) of programs or interventions. Main types of economic evaluation are cost- minimization analysis (CMA), cost-effectiveness analysis (CEA), cost-benefit analysis (CBA) and cost-utility analysis (CUA). Example of question asked: What is the cost-effectiveness of this intervention in improving patient outcomes as compared to other treatment models?

VIII. BACKGROUND If an evaluation, Project/Program being evaluated:

Activity/Project Name Integrated Social Marketing (ISM) Program

Implementer Population Services International/Madagascar

Cooperative Agreement # AID-687-A-13-00001

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 34

Total Estimated Ceiling of $36,823,053 the Evaluated Project (TEC)

Life of Project December 2012 – December 2017

Active Geographic Regions 20 of 22 regions of Madagascar

Development Objective Sustainable health impacts accelerated for the Malagasy people

USAID Office USAID/Madagascar Health, Population and Nutrition (HPN) Office

Activity/Project AOR Jocelyne Andriamiadana

Background of project/program/intervention:

A. Health context in Madagascar Reducing maternal, infant and child mortality in Madagascar is a critical priority for the international community and the Government of Madagascar (GoM). Although the country has made significant progress to reduce child deaths, maternal mortality has stagnated over the past twenty plus years; at present, 10 women die each day as a result of birth-related complications. In addition, each day, 100 children die from preventable causes, including malaria, which is the third leading cause of death for children.

The 2009 coup d'état plunged the country further into crisis, stalling development and further deteriorating the health system. With financial and political restrictions placed on the GoM during this period, USAID/ Madagascar shifted to a humanitarian support strategy and invested nearly $250 million in innovative community health services and systems: scaling-up access to diagnosis and treatment for simple pneumonia, diarrhea, and malaria as well as condoms, and oral and injectable contraceptives. USAID/Madagascar equipped and trained an extended cadre of more than 17,000 community health volunteers (CHV) in 20 of 22 regions covering about 1,200 mostly rural communes to expand basic, life-saving services. Today, this system, which USAID now directly supports in 15 regions, provides health services to 9.5 million people or about 64 percent of Madagascar’s rural population.

Following successful elections in December 2013, multi and bi-lateral organizations normalized relations; the USG lifted restrictions in May 2014. The GoM initiated the development of a health sector development strategy, the Plan de Développement du Secteur Sante (PDSS), in January 2014. The plan outlines a five-year strategy to improve health services and outcomes and was launched in mid-2015. Madagascar also launched an action plan in response to the African Union’s Campaign for the Accelerated Reduction of Maternal, Newborn and Child Mortality in Africa (CARMMA). The ambitious plan aims to reduce, by 2019, the maternal mortality ratio from 478 to 300 deaths per 100,000 live births and the neonatal mortality rate from 26 to 17 deaths per 1,000 live births. Furthermore, in June 2014, the GoM committed to redouble its efforts at the Acting on the Call: Ending Preventable Child and Maternal Deaths meeting, which mobilized governments and their partners from 24 priority countries to address

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 35

maternal and child mortality.

B. Overview of ISM Program In December 2012, during the period of bilateral restrictions, PSI/Madagascar (PSI) was awarded the Cooperative Agreement Number AID-687-A-13-00001 for the Integrated Social Marketing (ISM) Program. The award is for a total of $36,823,053, running from January 1, 2013 through December 31, 2017. The goal of the program is to improve the health of the Malagasy people -- especially women of reproductive age, children under five, youth 15-24 years old, and those living in rural and underserved areas. The ISM Program operates in 20 of 22 regions of Madagascar (see attached map). The main strategic objective is to use an integrated social marketing approach to increase the use of lifesaving health products and services, particularly in the areas of family planning/reproductive health, maternal and child health, and malaria.

PSI and its partners IntraHealth, Banyan Global, Human Network International (HNI), SAF and SALFA apply their combined expertise in social marketing, health clinic social franchising, and behavior change communication to bring more users into the Malagasy health market. PSI also works in partnership with USAID’s integrated health programs, MIKOLO and MAHEFA (and later CCH), to expand community distribution of products and services. Three primary intermediate results (IRs) are expected as outcomes of the ISM Program:

IR1: Increased adoption and maintenance of health behaviors. The ‘Healthy Family’ behavior change communication (BCC) campaign focuses on increased knowledge and adoption of preventative behaviors, and utilization of commodities related to: family planning (FP); water, sanitation and hygiene (WASH) practices; diarrhea, pneumonia and malaria prevention and treatment; nutrition; reproductive health (RH), and others. Radio, TV, mobile video units (MVU), innovative interpersonal communication techniques, and a variety of additional information, education and communication (IEC) materials and activities all combine to positively influence health behavior. In partnership with MIKOLO and MAHEFA, community health workers (CHW) are trained and equipped to provide education and distribute critically important health products within isolated rural areas.

IR2: Improved quality of selected health services in the private sector. PSI’s network of nearly 250 private, franchised ‘Top Réseau’ health clinics deliver a variety of health care services primarily in the areas of FP/RH, integrated management of childhood illnesses (IMCI), youth services, and malaria. PSI and its partners IntraHealth, Banyan Global, SAF, and SALFA focus on expanding access to quality health care services through training, quality assurance, capacity-building, supervision, promotional support, access to financing, and more. Rural and urban Top Réseau clinics are present in 74 of the 114 districts across Madagascar.

IR3: Increased availability of lifesaving health products and services. PSI is expanding access to affordable health products such as contraceptives, condoms, diarrhea treatment kits

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 36

(DTK), drinking water treatments, pneumonia and malaria medicines, and long-lasting insecticide-treated nets (LLINs). PSI distributes these social marketing commodities through a network of nearly 1,200 commercial, pharmaceutical, and community-based outlets. Within the ISM Team, HNI provides mobile technology support to make e-voucher and mobile money payment initiatives easier and more accessible to consumers and retailers.

The ISM cooperative agreement was modified in September 2014 to reflect the lifting of bilateral restrictions on the GoM, but the program description was not significantly modified as a result.

C. Summary of the Project M&E Plan The ISM results framework and performance monitoring plan (with objectives and achievements) is included with the Background Documents for the Team’s review.

Strategic or Results Framework for the project/program/intervention (paste framework below)

The USAID/Madagascar HPN Office Results Framework presents the development hypothesis. The ISM Program contributes under IRs 1 and 2, and to lesser extent, 3 and 4.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 37

What is the geographic coverage and/or the target groups for the project or program that is the subject of analysis?

20 of 22 regions of Madagascar: Diana, Sava, Itasy, Analamanga, Vakinankaratra, Sofia, Boeny, Melaky, Alaotra-Mangoro, Atsinanana, Analanjirofo, Amoron'i Mania, Haute Matsiatra, Vatovavy- Fitovinany, Atsimo-Atsinanana, Ihorombe, Menabe, Atsimo-Andrefana, Androy, Anosy

IX. SCOPE OF WORK A. Purpose: Why is this evaluation or analysis being conducted (purpose of analytic activity)? Provide the specific reason for this activity, linking it to future decisions to be made by USAID leadership, partner governments, and/or other key stakeholders. This evaluation is meant to serve a dual purpose: (1) to learn to what extent the project’s objectives and goals—at all result levels—have been achieved; and (2) to inform the design of a new social marketing project.

B. Audience: Who is the intended audience for this analysis? Who will use the results? If listing multiple audiences, indicate which are most important. The audience of the evaluation report will be:

● USAID/Madagascar Mission, specifically the HPN team, the Global Health Bureau, and the implementing partner ● PSI and its partners ● An Executive Summary will be provided to the MOH.

C. Applications and use: How will the findings be used? What future decisions will be made based on these findings? This evaluation will assist the Mission in reaching decisions related to: (1) the effectiveness of the current approach to improving health behavior, improving the quality of private sector health services, and increasing the availability of health products and services; (2) the type of mechanism(s) the Mission should use in any future assistance to the health sector for social marketing and related interventions; and (3) the nature and scope of possible future interventions in the sector, based on lessons learned from the current project.

PSI and its partners will learn about their strengths and weaknesses to adjust their close-out and sustainability strategy accordingly.

The MOH will learn more about USAID’s support to the dissemination of key BCC messages and how effective various methods have been; about the transfer of learning from private providers to public providers; and about USAID’s contribution to CHVs ability to deliver health commodities to remote populations.

D. Evaluation/Analytic Questions & Matrix:

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 38

a) Questions should be: a) aligned with the evaluation/analytic purpose and the expected use of findings; b) clearly defined to produce needed evidence and results; and c) answerable given the time and budget constraints. Include any disaggregation (e.g., sex, geographic locale, age, etc.), they must be incorporated into the evaluation/analytic questions. USAID policy suggests 3 to 5 evaluation/analytic questions. b) List the recommended methods that will be used to collect data to be used to answer each question. c) State the application or use of the data elements towards answering the evaluation questions; for example, i) ratings of quality of services, ii) magnitude of a problem, iii) number of events/occurrences, iv) gender differentiation, v) etc.

Suggested methods for Sampling Frame answering this question Who is the best source for What data sources and data this information? What is Evaluation Question collection and analysis methods the sampling criteria? will be used to produce the evidence for answering this question?

1 Is the ISM Program likely to Program reports, KII and FDG, achieve its objectives as program data, etc. outlined in the intermediate results and results framework? What were the most and least successful activities implemented by the Program? Why?

2 How were the three Program Management interviews, intermediate results review of program reports, integrated (from a discussions with beneficiaries management perspective and (TR providers, CHVs, etc.) at the activity level)? What were the benefits or disadvantages to this integration?

3 For the ISM target population Program data and reports (ie: (women of reproductive age, exit surveys); discussions with children under five, youth 15- beneficiaries (TR providers, 24 years), how was CHVs, clients, etc.) "underserved" and "rural" operationally defined? Did Program activities reach this population, why or why not? 4 How was Program learning Other partners, USAID, (including operational management, program reports

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 39

research and, particularly, results of innovative activities) documented, disseminated, and applied to improve ISM Program activities and interventions by other health sector partners?

5 To what extent are ISM activities socially and economically sustainable? How could the Program design be adapted to improve sustainability?

E. Methods: Check and describe the recommended methods for this analytic activity. Selection of methods should be aligned with the evaluation/analytic questions and fit within the time and resources allotted for this analytic activity. Also, include the sample or sampling frame in the description of each method selected. General Comments related to Methods: Based on the information provided within this Statement of Work, the Evaluation Team may wish to propose an alternate approach, factoring in the information that will be available to them.

The Evaluation Team should consider a range of possible methods and approaches for collecting and analyzing the information, which are required to carry out the evaluation. Data collection methodologies will be discussed with and concurred by USAID/Madagascar at the beginning of the evaluation work; this shall include the proposed relevant summary tables, graphs, and annexes. The team will use participatory methods and activities that will enhance collaboration and dialogue among counterparts, particularly partners. For instance, the Evaluation Team should consider meeting and getting feedback from counterparts, beneficiaries, implementing entities, and stakeholders (regional and bilateral), such as Top Réseau (TR) health providers, supply point operators, the audience of various BCC interventions, PSI staff (central and regional), sub-grantee staff, the MOH, and other USAID implementing partners.

 Document and Data Review (list of documents and data recommended for review) This desk review will be used to provide background information on the project/program, and will also provide data for analysis for this evaluation. USAID/Madagascar will provide the Evaluation Team with key documents describing the objectives set and results reported by the ISM Program, including:

1. Cooperative Agreement # AID-687-A-13-00001 with PSI.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 40

2. Current ISM Program results framework and PMP. 3. Annual work plans for ISM Program implementation. 4. Annual reports through FY 2015 and quarterly reports for FY 2016. 5. Available reports or studies produced by ISM Program. 6. Project descriptions of related USAID-funded community health projects.

Other reports and surveys of interest might include:

● PSI’s Tracking Results Continuously (TRaC) surveys ● Exit surveys of TR clients ● MDG survey (2012-2013) ● Top Réseau (TR) service quality assessments ● Madagascar (South) 2012 MICS ● Madagascar MIS 2013 ● Madagascar MIS 2011 ● Madagascar DHS 2008-9

 Secondary analysis of existing data (This is a re-analysis of existing data, beyond a review of data reports. List the data source and recommended analyses)

Data Source (existing Description of data Recommended analysis dataset)

ISM monitoring data ISM has routine monitoring data Trend analysis from beginning of that is uses to track progress project. and report to USAID.

PSI sales data Data of sales from PSI supported sites

PSI franchise service data Service data from PSI franchises

TRAC survey data

 Key Informant Interviews (list categories of key informants, and purpose of inquiry)

Key informant interviews with ISM project staff (PSI, IntraHealth, Banyan Global, HNI, SAF and SALFA), TR providers, peer educators, CHVs, national and regional government officials, other project partners and distributers (MIKOLO, MAHEFA and CCH), USAID HPN staff, and other program stakeholders. A list of key informants will be developed in consultation with USAID during the Team Planning Meeting (TPM).

 Focus Group Discussions (list categories of groups, and purpose of inquiry)

Focus group discussions (FGD) with program beneficiaries (services and commodities) to gain their perspective regarding use of products and services supported under ISM. Focus groups may include clients of ISM supported services, CHVs, peer educator and TR providers. Focus

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 41

group participants should be representative of both men and women. Men and women will participate in separate FGDs to adjust for the potential power differential between men and women, and to assure women’s voice is heard equally to men. The final list of groups will be determined in consultation with USAID during the Team Planning Meeting.

 Group Interviews (list categories of groups, and purpose of inquiry)

Optional: Key informants can be grouped and interviewed together, as long as the respondents feel free to express their opinions openly.

 Client/Participant Satisfaction or Exit Interviews (list who is to be interviewed, and purpose of inquiry)

Optional: ISM has conducted exit interviews. These data/report will be reviewed to determine if additional exit interviews with clients of franchised services supported by ISM are needed.

X. HUMAN SUBJECT PROTECTION The Analytic Team must develop protocols to insure privacy and confidentiality prior to any data collection. Primary data collection must include a consent process that contains the purpose of the evaluation, the risk and benefits to the respondents and community, the right to refuse to answer any question, and the right to refuse participation in the evaluation at any time without consequences. Only adults can consent as part of this evaluation. Minors cannot be respondents to any interview or survey, and cannot participate in a focus group discussion without going through an IRB. The only time minors can be observed as part of this evaluation is as part of a large community-wide public event, when they are part of family and community in the public setting. During the process of this evaluation, if data are abstracted from existing documents that include unique identifiers, data can only be abstracted without this identifying information. An Informed Consent statement included in all data collection interactions must contain:

● Introduction of facilitator/note-taker ● Purpose of the evaluation/assessment ● Purpose of interview/discussion/survey ● Statement that all information provided is confidential and information provided will not be connected to the individual ● Right to refuse to answer questions or participate in interview/discussion/survey ● Request consent prior to initiating data collection (i.e., interview/discussion/survey)

XI. ANALYTIC PLAN Describe how the quantitative and qualitative data will be analyzed. Include method or type of analyses, statistical tests, and what data it to be triangulated (if appropriate). For example, a thematic analysis of qualitative interview data, or a descriptive analysis of quantitative survey data.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 42

All analyses will be geared to answer the evaluation questions. Additionally, the evaluation will review both qualitative and quantitative data related to the project/program’s achievements against its objectives and/or targets. Quantitative data will be analyzed primarily using descriptive statistics. Data will be stratified by demographic characteristics, such as sex, age, and location, whenever feasible. Other statistical test of association (i.e., odds ratio) and correlations will be run as appropriate.

Thematic review of qualitative data will be performed, connecting the data to the evaluation questions, seeking relationships, context, interpretation, nuances and homogeneity and outliers to better explain what is happening and the perception of those involved. Qualitative data will be used to substantiate quantitative findings, provide more insights than quantitative data can provide, and answer questions where other data do not exist.

Use of multiple methods that are quantitative and qualitative, as well as existing data (e.g., project/program performance indicator data, MICS, MIS, DHI and HMIS data, etc.) will allow the Team to triangulate findings to produce more robust evaluation results. The Evaluation Report will describe analytic methods and statistical tests employed in this evaluation.

XII. ACTIVITIES List the expected activities, such as Team Planning Meeting (TPM), briefings, verification workshop with IPs and stakeholders, etc. Activities and Deliverables may overlap. Give as much detail as possible.

Background reading – Several documents are available for review for this analytic activity. These include ISM proposal, annual work plans, M&E plans, quarterly progress reports, and routine reports of project performance indicator data, as well as survey data reports (i.e., DHS and MICS). This desk review will provide background information for the Evaluation Team, and will also be used as data input and evidence for the evaluation.

Team Planning Meeting (TPM) – A four-day team planning meeting (TPM) will be held at the initiation of this assignment and before the data collection begins. The TPM will:

● Review and clarify any questions on the evaluation SOW ● Clarify team members’ roles and responsibilities ● Establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion ● Review and finalize evaluation questions ● Review and finalize the assignment timeline ● Develop data collection methods, instruments, tools and guidelines ● Review and clarify any logistical and administrative procedures for the assignment ● Develop a data collection plan ● Draft the evaluation work plan for USAID’s approval ● Develop a preliminary draft outline of the team’s report ● Assign drafting/writing responsibilities for the final report

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 43

Briefings – Throughout the evaluation the Team Lead will provide briefings to USAID. The In- Brief and Debrief are likely to include the all Evaluation Team experts, but will be determined in consultation with the Mission. These briefings are:

● Evaluation launch, a call/meeting among the USAID, GH Pro and the Team Lead to initiate the evaluation activity and review expectations. USAID will review the purpose, expectations, and agenda of the assignment. GH Pro will introduce the Team Lead, and review the initial schedule and review other management issues. ● In-brief with USAID, as part of the TPM. At the beginning of the TPM, the Evaluation Team will meet with USAID to discuss expectations, review evaluation questions, and intended plans. The Team will also raise questions that they may have about the project/program and SOW resulting from their background document review. The time and place for this in-brief will be determined between the Team Lead and USAID prior to the TPM. ● Workplan and methodology review briefing. At the end of the TPM, the Evaluation Team will meet with USAID to present an outline of the methods/protocols, timeline and data collection tools. Also, the format and content of the Evaluation report(s) will be discussed. ● Evaluation Workplan and Protocol will be submitted to USAID/Madagascar at the close of the TPM that includes: o Evaluation matrix, including evaluation methods o Criteria for purposive sampling for health facility site selection o Evaluation questions o Data collection plan o Evaluation workplan for USAID’s approval o Data collection methods, with instruments, tools and guidelines, including consent statements o Assignment timeline o Timeline for field work and deliverables ● In-brief with project to review the evaluation plans and timeline, and for the project to give an overview of the project to the Evaluation Team. ● The Team Lead (TL) will brief the USAID weekly to discuss progress on the evaluation. They will also provide a midterm briefing to the Activity Manager, HPN team, and Program Office on the status of the evaluation, including potential challenges and emerging opportunities. As preliminary findings arise, the TL will share these during the routine briefing, and/or in an email. ● A final exit debrief between the Evaluation Team and USAID will be held at the end of the evaluation to present preliminary findings to USAID. During this meeting a summary of the data will be presented, along with high level findings and draft recommendations. For the debrief, the Evaluation Team will prepare a PowerPoint Presentation of the key findings, issues, and recommendations. The evaluation team shall incorporate comments received from USAID during the debrief in the evaluation report. (Note: preliminary findings are not final and as more data sources are developed and analyzed these finding may change.) ● Stakeholders’ debrief/workshop will be held with the project staff and other stakeholders identified by USAID. This will occur following the final exit debrief with the Mission, and will not include any information that may be procurement deemed

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 44

sensitive or not suitable by USAID. ● Final Presentation to be done virtually, utilizing web conferencing software (e.g., Adobe Connect or GoTo Meeting) to discuss the summary of findings and recommendations to USAID/Madagascar. This presentation will be scheduled as agreed upon during the TPM, including who will attend, date & time, length of presentation, etc.

Fieldwork, Site Visits and Data Collection – The evaluation team will conduct site visits to for data collection. Selection of sites to be visited will be finalized during TPM in consultation with USAID. The evaluation team will outline and schedule key meetings and site visits prior to departing to the field.

Evaluation/Analytic Report – The Evaluation/Analytic Team under the leadership of the Team Lead will develop a report with findings and recommendations (see Analytic Report below). Report writing and submission will include the following steps:

1. Team Lead will submit draft evaluation report to GH Pro for review and formatting 2. GH Pro will submit the draft report to USAID 3. USAID will review the draft report in a timely manner, and send their comments and edits back to GH Pro 4. GH Pro will share USAID’s comments and edits with the Team Lead, who will then do final edits, as needed, and resubmit to GH Pro 5. GH Pro will review and reformat the final Evaluation/Analytic Report, as needed, and resubmit to USAID for approval. 6. Once Evaluation Report is approved, GH Pro will re-format it for 508 compliance and post it to the DEC. The Evaluation Report excludes any procurement-sensitive and other sensitive but unclassified (SBU) information. This information will be submitted in a memo to USAID separate from the Evaluation Report.

Data Submission – All quantitative data will be submitted to GH Pro in a machine-readable format (CSV or XML). The datasets created as part of this evaluation must be accompanied by a data dictionary that includes a codebook and any other information needed for others to use these data. It is essential that the datasets are stripped of all identifying information, as the data will be public once posted on USAID Development Data Library (DDL).

Where feasible, qualitative data that do not contain identifying information should also be submitted to GH Pro.

XIII. DELIVERABLES AND PRODUCTS Select all deliverables and products required on this analytic activity. For those not listed, add rows as needed or enter them under “Other” in the table below. Provide timelines and deliverable deadlines for each.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 45

Deliverable / Product Timelines & Deadlines (estimated)

 Launch briefing October 10, 2016

 In-brief with USAID October 17, 2016

 Workplan and methodology review briefing October 24, 2016

 Workplan with timeline October 24, 2016

 Analytic protocol with data collection tools October 24, 2016

 In-brief with target project / program October 25, 2016

 Routine briefings Weekly

 Out-brief with USAID with Power Point November 18, 2016 presentation

 Findings review workshop with November 21, 2016 stakeholders with Power Point presentation

 Draft report Submit to GH Pro: December 12, 2016 GH Pro submits to USAID: December 16, 2016

 Final report Submit to GH Pro: January 12, 2017 GH Pro submits to USAID: January 19, 2017

 Raw data (cleaned datasets in CSV or XML January 12, 2017 with code sheet or data dictionary)

 Report Posted to the DEC February 23, 2017

 Other (specify):

Estimated USAID review time Average number of business days USAID will need to review deliverables requiring USAID review and/or approval? 10 Business days

XIV. TEAM COMPOSITION, SKILLS AND LEVEL OF EFFORT (LOE) Evaluation/Analytic team: When planning this analytic activity, consider:

● Key staff should have methodological and/or technical expertise, regional or country experience, language skills, team lead experience and management skills, etc. ● Team leaders for evaluations/analytics must be an external expert with appropriate skills and experience. ● Additional team members can include research assistants, enumerators, translators, logisticians, etc. ● Teams should include a collective mix of appropriate methodological and subject matter expertise.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 46

● Evaluations require an Evaluation Specialist, who should have evaluation methodological expertise needed for this activity. Similarly, other analytic activities should have a specialist with methodological expertise. ● Note that all team members will be required to provide a signed statement attesting that they have no conflict of interest, or describing the conflict of interest if applicable.

Team Qualifications: Please list technical areas of expertise required for this activities

● List desired qualifications for the team as a whole ● List the key staff needed for this analytic activity and their roles. ● Sample position descriptions are posted on USAID/GH Pro webpage ● Edit as needed GH Pro provided position descriptions Overall Team requirements: The team shall, at a minimum, meet the following requirements:

1. Include at least four members. 2. An evaluation team leader with at least 10 years of experience in evaluation design, methods, management, and implementation; a post graduate degree in public health, international affairs, economics, or related fields; extensive experience in community health, social marketing, or SBCC; familiarity with USAID programs, policies and objectives; and excellent spoken and written skills in French and English. 3. Specialists in the following technical areas, at a minimum: SBCC, social marketing, and private sector health service delivery. Additional technical expertise in community distribution and social franchising strongly desired. 4. An appropriate mix of speakers of English, French and Malagasy, accounting for tasks related to document review, field visits and key stakeholder interviews, and report writing. 5. Demonstrate an understanding of the community health context in Madagascar. 6. Demonstrate familiarity with USAID’s Evaluation Policy and guidance included in the USAID Automated Directive System (ADS) in Chapter 200.

The recruitment of local Malagasy evaluators is highly encouraged.

Team Lead: This person will be selected from among the key staff, and will meet the requirements of both this and the other position. The team lead should have significant experience conducting project evaluations. Roles & Responsibilities: The team leader will be responsible for (1) providing team leadership; (2) managing the team’s activities, (3) ensuring that all deliverables are met in a timely manner, (4) serving as a liaison between the USAID and the evaluation/analytic team, and (5) leading briefings and presentations. Qualifications:

● Minimum of 10 years of experience in public health, which included experience in implementation of health activities in developing countries ● Post graduate degree in public health, international affairs, economics, or related fields;

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 47

● Experience in evaluation design, methods, management and implementation ● Demonstrated experience leading health sector project/program evaluation/analytics, utilizing both quantitative and qualitative s methods ● Extensive experience in community health, social marketing, or SBCC is highly desirable ● Excellent skills in planning, facilitation, and consensus building ● Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders ● Excellent skills in project management ● Excellent organizational skills and ability to keep to a timeline ● Good writing skills, with extensive report writing experience ● Experience working in the region, and experience in Madagascar is desirable ● Proficient in English and French (spoken and written) ● Familiarity with USAID projects ● Familiarity with USAID policies and practices − Evaluation policy − Results frameworks − Performance monitoring plans Key Staff 1 Title: Evaluation Specialist Roles & Responsibilities: Serve as a member of the evaluation team, providing quality assurance on evaluation issues, including methods, development of data collection instruments, protocols for data collection, data management and data analysis. S/He will oversee the training of all engaged in data collection, insuring highest level of reliability and validity of data being collected. S/He is the lead analyst, responsible for all data analysis, and will coordinate the analysis of all data, assuring all quantitative and qualitative data analyses are done to meet the needs for this evaluation. S/He will participate in all aspects of the evaluation, from planning, data collection, data analysis to report writing. Qualifications:

● At least 10 years of experience in USAID M&E procedures and implementation ● At least 5 years managing M&E, including evaluations ● Experience in design and implementation of evaluations ● Strong knowledge, skills, and experience in qualitative and quantitative evaluation tools ● Experience implementing and coordinating other to implements surveys, key informant interviews, focus groups, observations and other evaluation methods that assure reliability and validity of the data. ● Experience in data management ● Able to analyze quantitative, which will be primarily descriptive statistics ● Able to analyze qualitative data ● Experience using analytic software ● Demonstrated experience using qualitative evaluation methodologies, and triangulating with quantitative data ● Able to review, interpret and reanalyze as needed existing data pertinent to the evaluation ● Strong data interpretation and presentation skills ● An advanced degree in public health, evaluation or research or related field ● Proficient in English and French

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 48

● Good writing skills, including extensive report writing experience ● Familiarity with USAID health programs/projects, primary health care or health systems strengthening preferred ● Familiarity with USAID and PEPFAR M&E policies and practices − Evaluation policies − Results frameworks − Performance monitoring plans

Key Staff 2 Title: Evaluation Social Marketing and SBCC Specialist Roles & Responsibilities: Serve as a member of the evaluation team, providing technical expertise to evaluate social marketing and SBCC activities. S/he will provide technical expertise on social marketing and social behavior change communications, including demand creation for health products and services, and adopting healthy behaviors. S/He will participate in all aspects of the evaluation, including planning, data collection, data analysis and report writing. Qualifications:

● At least 8 years of experience working with social marketing and SBCC programs in developing country settings ● Experience should include mass media, community-based interventions, and interpersonal communications (IPC) ● Experience working with formal and non-formal private sector networks, for commodity sales and distribution, franchise service networks ● Experience in social marketing and demand generation for FP/RH, MCH and malaria commodities, ● Experience and knowledgeable on evaluation methodologies related to social marketing and SBCC ● Proficient in written and spoken English and French ● Good writing skills, with experience producing evaluation and/or technical reports ● Experience working in the region, and experience in Madagascar is desirable

Key Staff 1 Title: Evaluation Private Sector Community Health Specialist Roles & Responsibilities: Serve as a member of the evaluation team, providing expertise in community health, specifically focused on private sector health service delivery, including community distribution and social franchising. S/He will participate in planning and briefing meetings, development of data collection methods and tools, data collection, data analysis, development of evaluation presentations, and writing of the Evaluation Report. Qualifications:

● At least 8 years’ experience working on community health activities, including experience in the private sector and social franchising; USAID project implementation experience preferred ● Strong background in strengthening health services at the community level

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 49

● Demonstrated understanding of community engagement for services and commodities that include demand creation and prevention ● Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders ● Experience conducting evaluations and/or related research, including development of data collection tools ● Experience conducting qualitative data collection and analysis, such as key informant interviews, focus groups and/or observations ● Proficient in English and French. ● Good writing skills, specifically technical and evaluation report writing experience ● Experience in conducting USAID evaluations of health programs/activities

Other Staff Titles with Roles & Responsibilities (include number of individuals needed):

Local Evaluation Logistics /Program Assistant will support the Evaluation Team with all logistics and administration to allow them to carry out this evaluation. The Logistics/Program Assistant will have a good command of French, Malagasy and English. S/He will have knowledge of key actors in the health sector and their locations including MOH, donors and other stakeholders. To support the Team, s/he will be able to efficiently liaise with hotel staff, arrange in-country transportation (ground and air), arrange meeting and workspace as needed, and insure business center support, e.g. copying, internet, and printing. S/he will work under the guidance of the Team Leader to make preparations, arrange meetings and appointments. S/he will conduct programmatic administrative and support tasks as assigned and ensure the processes moves forward smoothly. S/He may also be asked to assist in translation of data collection tools and transcripts, if needed.

Local Evaluators (3 local consultants) to assist the Evaluation Team with data collection, analysis and data interpretation. They will have basic familiarity with health topics, as well as experience conducting surveys interviews and focus group discussion, both facilitating and note taking. Furthermore, they will assist in translation of data collection tools and transcripts, as needed. The Local Evaluators will have a good command of French, Malagasy and English. They will also assist the Team and the Logistics Coordinator, as needed. They will report to the Team Lead.

Will USAID participate as an active team member or designate other key stakeholders to as an active team member? This will require full time commitment during the evaluation or analytic activity.  Yes – If yes, specify who:

 Significant Involvement anticipated – If yes, specify who: USAID/Madagascar staff may observe during data collection  No

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 50

Staffing Level of Effort (LOE) Matrix: This LOE Matrix will help you estimate the LOE needed to implement this analytic activity. If you are unsure, GH Pro can assist you to complete this table.

a) For each column, replace the label "Position Title" with the actual position title of staff needed for this analytic activity. b) Immediately below each staff title enter the anticipated number of people for each titled position. c) Enter Row labels for each activity, task and deliverable needed to implement this analytic activity. d) Then enter the LOE (estimated number of days) for each activity/task/deliverable corresponding to each titled position. e) At the bottom of the table total the LOE days for each consultant title in the ‘Sub-Total’ cell, then multiply the subtotals in each column by the number of individuals that will hold this title.

Level of Effort in days for each Evaluation/Analytic Team member

Evaluation/Analytic Team

Private Team Logistics / Activity / Deliverable SBCC Sector Local Lead / Eval Admin Spec Comm’ty Evaluator Spec Coord Services

Number of persons → 1 1 1 1 .3

1 Launch Briefing 0.5

2 HTSOS Training (international 1 1 1 consultants)

3 Desk review 5 5 5 2

4 Preparation for Team convening in- 2 country

5 Travel to country 2 2 1

6 In-brief with Mission 0.5 0.5 0.5 0.5 0.5 with prep

7 Team Planning 4 4 4 4 4 Meeting

8 Workplan and methodology review 0.5 0.5 0.5 0.5 0.5 briefing with USAID

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 51

9 In-brief with project 0.5 0.5 0.5 0.5 0.5 with prep

10 Data Collection DQA Workshop (protocol orientation 2 2 2 2 for all involved in data collection)

11 Prep / Logistics for 0.5 0.5 0.5 2 0.5 Site Visits

12 Data collection / Site Visits (including 16 16 16 16 16 travel to sites)

13 Data analysis (in- 6 6 6 2 6 county + remote)

14 Debrief with Mission 1 1 1 1 1 with prep

15 Stakeholder debrief 1 1 1 1 1 workshop with prep

16 Depart country 2 2 1

17 Draft report(s) 6 5 3 1 1

18 GH Pro Report QC

Review & Formatting

19 Submission of draft

report(s) to Mission

20 Final Presentation (virtual) with USAID 1 1 1 with prep

21 USAID Report

Review

22 Revise report(s) per 3 2 2 USAID comments

23 Finalize and submit

report to USAID

24 USAID approves

report

25 Final copy editing and

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 52

formatting

26 508 Compliance

editing

27 Upload Eval Report(s) to the DEC

Sub-Total LOE 53 50 46 31 35

Total LOE 53 50 46 31 105

If overseas, is a 6-day workweek permitted  Yes  No

Travel anticipated: List international and local travel anticipated by what team members.

The Evaluation will cover urban and rural sites selected from 3-4 regions:

● one in/around Antananarivo ● one in the south or east (Mikolo region); or one in the north or west (Mahefa region) ● one NOT covered by a USAID bilateral (see map).

Almost all regions are accessible within one day drive or via air. Travel within regions can be challenging, but is feasible during the proposed timeframe (Oct-Nov).

XV. LOGISTICS Visa Requirements List any specific Visa requirements or considerations for entry to countries that will be visited by consultant(s):

90 day visa can be issued at the airport upon entry

List recommended/required type of Visa for entry into counties where consultant(s) will work

Name of Country Type of Visa  Tourist  Business  No preference

 Tourist  Business  No preference

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 53

Clearances & Other Requirements Note: Most Evaluation/Analytic Teams arrange their own work space, often in their hotels. However, if Facility Access is preferred GH Pro can request it.

GH Pro does not provide Security Clearances, but can request Facility Access. Please note that Facility Access (FA) requests processed by USAID/GH (Washington, DC) can take 4-6 months to be granted. If you are in a Mission and the RSO can grant a temporary FA, this can expedite the process. If FA is granted through Washington, DC, the consultant must pick up his/her FA badge in person in Washington, DC, regardless of where the consultant resides or will work.

If Electronic Country Clearance (eCC) is required, the consultant is also required to complete the High Threat Security Overseas Seminar (HTSOS). HTSOS is an interactive e-Learning (online) course designed to provide participants with threat and situational awareness training against criminal and terrorist attacks while working in high threat regions. There is a small fee required to register for this course. [Note: The course is not required for employees who have taken FACT training within the past five years or have taken HTSOS within the same calendar year.] If eCC is required, and the consultant is expected to work in country more than 45 consecutive days, the consultant must complete the one week Foreign Affairs Counter Threat (FACT) course offered by FSI in West Virginia. This course provides participants with the knowledge and skills to better prepare themselves for living and working in critical and high threat overseas environments. Registration for this course is complicated by high demand (must register approximately 3-4 months in advance). Additionally, there will be the cost for one week’s lodging and M&IE to take this course.

Check all that the consultant will need to perform this assignment, including USAID Facility Access, GH Pro workspace and travel (other than to and from post).  USAID Facility Access (FA) Specify who will require Facility Access:

 Electronic County Clearance (ECC) (International travelers only)  High Threat Security Overseas Seminar (HTSOS) (required with ECC)

 Foreign Affairs Counter Threat (FACT) (for consultants working on country more than 45 consecutive days)  GH Pro workspace Specify who will require workspace at GH Pro:

 Travel -other than posting (specify):

 Other (specify):

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 54

XVI. GH PRO ROLES AND RESPONSIBILITIES GH Pro will coordinate and manage the evaluation/analytic team and provide quality assurance oversight, including:

● Review SOW and recommend revisions as needed ● Provide technical assistance on methodology, as needed ● Develop budget for analytic activity ● Recruit and hire the evaluation/analytic team, with USAID POC approval ● Arrange international travel and lodging for international consultants ● Request for country clearance and/or facility access (if needed) ● Review methods, workplan, analytic instruments, reports and other deliverables as part of the quality assurance oversight ● Report production - If the report is public, then coordination of draft and finalization steps, editing/formatting, 508ing required in addition to and submission to the DEC and posting on GH Pro website. If the report is internal, then copy editing/formatting for internal distribution.

XVII. USAID ROLES AND RESPONSIBILITIES Below is the standard list of USAID’s roles and responsibilities. Add other roles and responsibilities as appropriate.

USAID Roles and Responsibilities USAID will provide overall technical leadership and direction for the analytic team throughout the assignment and will provide assistance with the following tasks:

Before Field Work

● SOW. o Develop SOW. o Peer Review SOW o Respond to queries about the SOW and/or the assignment at large. ● Consultant Conflict of Interest (COI). To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CV’s for proposed consultants and provide additional information regarding potential COI with the project contractors evaluated/assessed and information regarding their affiliates. ● Documents. Identify and prioritize background materials for the consultants and provide them to GH Pro, preferably in electronic form, at least one week prior to the inception of the assignment. ● Local Consultants. Assist with identification of potential local consultants, including contact information. ● Site Visit Preparations. Provide a list of site visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line items costs. ● Lodgings and Travel. Provide guidance on recommended secure hotels and methods of in-country travel (i.e., car rental companies and other means of transportation). During Field Work

● Mission Point of Contact. Throughout the in-country work, ensure constant availability of the Point of Contact person and provide technical leadership and direction for the team’s work.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 55

● Meeting Space. Provide guidance on the team’s selection of a meeting space for interviews and/or focus group discussions (i.e. USAID space if available, or other known office/hotel meeting space). ● Meeting Arrangements. Assist the team in arranging and coordinating meetings with stakeholders. ● Facilitate Contact with Implementing Partners. Introduce the analytic team to implementing partners and other stakeholders, and where applicable and appropriate prepare and send out an introduction letter for team’s arrival and/or anticipated meetings. After Field Work

● Timely Reviews. Provide timely review of draft/final reports and approval of deliverables.

XVIII. ANALYTIC REPORT Provide any desired guidance or specifications for Final Report. (See How-To Note: Preparing Evaluation Reports)

The Evaluation/Analytic Final Report must follow USAID’s Criteria to Ensure the Quality of the Evaluation Report (found in Appendix I of the USAID Evaluation Policy).

a. The report should be approximately 30 pages (excluding executive summary, table of contents, acronym list and annexes). b. The structure of the report should follow the Evaluation Report template, including branding found here or here. c. Draft reports must be provided electronically, in English, to GH Pro who will then submit it to USAID. d. For additional Guidance, please see the Evaluation Reports to the How-To Note on preparing Evaluation Draft Reports found here. Reporting Guidelines: The draft report should be a comprehensive analytical evidence-based evaluation/analytic report. It should detail and describe results, effects, constraints, and lessons learned, and provide recommendations and identify key questions for future consideration. The report shall follow USAID branding procedures. The report will be edited/formatted and made 508 compliant as required by USAID for public reports and will be posted to the USAID/DEC. The findings from the evaluation/analytic will be presented in a draft report at a full briefing with USAID and at a follow-up meeting with key stakeholders. The report should use the following format:

● Executive Summary: concisely state the most salient findings, conclusions, and recommendations (3-5 pages) ● Table of Contents ● Acronyms ● Evaluation/Analytic Purpose and Evaluation/Analytic Questions (1-2 pages) ● Project [or Program] Background (1-3 pages) ● Evaluation/Analytic Methods and Limitations (1-3 pages) ● Findings (organized by Evaluation/Analytic Questions) ● Conclusions ● Recommendations ● Annexes

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 56

- Annex I: Evaluation/Analytic Statement of Work - Annex II: Evaluation/Analytic Methods and Limitations - Annex III: Data Collection Instruments - Annex IV: Sources of Information o List of Persons Interviews o Bibliography of Documents Reviewed o Databases o [etc] - Annex V: Disclosure of Any Conflicts of Interest - Annex VI: Statement of Differences (if applicable)

The evaluation methodology and report will be compliant with the USAID Evaluation Policy and Checklist for Assessing USAID Evaluation Reports

------

The Evaluation Report should exclude any potentially procurement-sensitive information. As needed, any procurement sensitive information or other sensitive but unclassified (SBU) information will be submitted in a memo to USIAD separate from the Evaluation Report. ------

All data instruments, data sets (if appropriate), presentations, meeting notes and report for this evaluation/analysis will be submitted electronically to the GH Pro Program Manager. All datasets developed as part of this evaluation will be submitted to GH Pro in an unlocked machine-readable format (CSV or XML). The datasets must not include any identifying or confidential information. The datasets must also be accompanied by a data dictionary that includes a codebook and any other information needed for others to use these data. Qualitative data included in this submission should not contain identifying or confidential information. Category of respondent is acceptable, but names, addresses and other confidential information that can easily lead to identifying the respondent should not be included in any quantitative or qualitative data submitted.

XIX. USAID CONTACTS Primary Alternate Alternate Contact Contact 1 Contact 2

Name: Melinda Manning Hary (Vololontsoa) Azzah Al-Rashid Sara A. Miner Raharimalala

Title: Project Monitoring Operations Health Promotion & Evaluation Specialist Advisor Assistant

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 57

USAID Program Office, Office of Health, Office of Health, Office/Missi USAID/Madagascar Population and Population and on Nutrition (HPN), Nutrition (HPN), USAID/Madagasca USAID r /Madagascar

Email: mmanning@usaid vraharimalala@usai aal- [email protected] .gov d.gov [email protected]

Telephone: 033 44 326 76 +(261) 33 44 327 +(261) 33 44 326 54 86

Cell Phone: +(261) 34 07 428 +(261) 34 07 428 22 06

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 58

Annex 1: ISM Map

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 59

ANNEX II. EVALUATION METHODS AND LIMITATIONS

1. INTRODUCTION

The Integrated Social Marketing (ISM) Program is a five-year Cooperative Agreement (Number AID-687-A-13-00001) funded by the United States Agency for International Development (USAID)/Madagascar. The project is implemented by Population Services International (PSI), as prime partner, with IntraHealth, Banyan Global, Human Network International (HNI) as international partners, and SAF and SALFA as partners local to Madagascar. The award is for a total of $36,823,053, running from January 1, 2013 through December 31, 2017. The goal of the program is to improve the health of the Malagasy people – especially women of reproductive age (WRA), children under five (CU5), youth 15-24 years old living in rural and underserved areas through an increasingly sustainable social marketing program that delivers essential health products and services. The ISM Team will apply its expertise in social marketing, social franchising and behavior change to bring more users into the Malagasy health market. The ISM Program operates in 20 of 22 regions of Madagascar (see attached map). The main strategic objective is to use an integrated social marketing approach to increase the use of lifesaving health products and services, particularly in the areas of family planning (FP)/reproductive health (RH), maternal and child health (MCH), and malaria. PSI also works in partnership with USAID’s integrated health programs, MIKOLO and MAHEFA (and later CCH), to expand community distribution of products and services. By the end of this program, the Malagasy people will see improvements in their health status with regard to FP, RH, MCH, and malaria. Three primary intermediate results (IRs) are expected as outcomes of the ISM Program:

IR1: Increased adoption and maintenance of health behaviors. The “Healthy Family” behavior change communication (BCC) campaign focuses on increased knowledge and adoption of preventative behaviors, and utilization of commodities related to: family planning (FP); water, sanitation and hygiene (WASH) practices; diarrhea, pneumonia and malaria prevention and treatment; nutrition; reproductive health (RH), and others. Radio, TV, mobile video units (MVU), innovative interpersonal communication techniques, and a variety of additional information, education and communication (IEC) materials and activities all combine to positively influence health behavior. In partnership with MIKOLO and MAHEFA, community health workers (CHW) are trained and equipped to provide education and distribute critically important health products within isolated rural areas.

IR2: Improved quality of selected health services in the private sector. PSI’s network of nearly 250 private, franchised Top Réseau health clinics deliver a variety of health care services primarily in the areas of FP/RH, integrated management of childhood illnesses (IMCI), youth services, and malaria. PSI and its partners IntraHealth, Banyan Global, SAF, and SALFA focus on expanding access to quality health care services through training, quality assurance, capacity-

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 60 building, supervision, promotional support, access to financing, and more. Rural and urban Top Réseau clinics are present in 74 of the 114 districts across Madagascar.

IR3: Increased availability of lifesaving health products and services. PSI is expanding access to affordable health products such as contraceptives, condoms, diarrhea treatment kits (DTK), drinking water treatments, pneumonia and malaria medicines, and long-lasting insecticide-treated nets (LLINs). PSI distributes these social marketing commodities through a network of nearly 1,200 commercial, pharmaceutical, and community-based outlets. Within the ISM Team, HNI provides mobile technology support to make e-voucher and mobile money payment initiatives easier and more accessible to consumers and retailers. The findings of this evaluation are expected to contribute to USAID decisions on the level and type of support to further reach the rural and underserved populations with behavior change communication, high quality health services, and necessary commodities. Specifically, the evaluation provides an opportunity for USAID to identify gaps, including strengths and weaknesses, and gather evidence upon which a post-ISM strategy can be based.

2. CONTEXT – ISM II PROJECT OVERVIEW

When awarded the project in December 2012, the vision of the ISM Team was to create a more efficient and integrated health market that reaches those Malagasy families most in need of health products and services in new ways. They expected that after five years, they will have succeeded in building a strengthened private sector health system capable of reaching more people and serving as a catalyst to build public sector health capacity when the Government of Madagascar (GOM) became able to fully re-engage. The main strategy implemented by the ISM team is to grow the total market for products and services by building on successes and lessons learned using three essential approaches:

1. Strengthen behavior change communication by linking it more directly to health-seeking behaviors. Towards this end, the ISM team applied a theoretical framework called PERForM, implemented through a marketing planning process called DELTA. The DELTA process developed archetypes or audience profiles that ensures that messages are well positioned and relevant to rural audiences, including youth, couples, men and women within the realm of influence of the program. Using this interpersonal communication approach, clients are directly referred to relevant, high quality services nearby where more information is provided and where products and service delivery happen at the same time. This approach is intended to lead to IR 1: Increased adoption and maintenance of health behaviors.

2. Build on the successful strategy of integrating more high-quality services within the Top Reseau franchise. PSI/Madagascar evolved the Top Réseau franchise from a youth-focused HIV and reproductive health (RH franchise to one that offers a broader integrated package of family health services, including provision of a broad range of long-acting and permanent methods. With funding from this project, PSI’s strategy was to further strengthen the quality of service delivery in a sustainable manner, as well as business development and management capacity of existing Top

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 61

Réseau clinics. In addition, PSI intended to add 40 new Top Réseau providers in rural areas, and 20 in urban areas over the five-year project implementation period. This approach is intended to lead to IR 2: Improved quality of selected health services in the private sector.

3. Improve reach and efficiency in rural areas. Rural areas of Madagascar are largely underserved, with significant room for improving reach of socially marketed health products and services. The ISM Team intended to extend rural reach across all three intermediate results. To more efficiently reach rural populations, the ISM Team integrated community radio, local events and interpersonal communications using well trained IPC agents and refer them to Top Réseau newly integrated SAF and SALFA clinics. They also intended to expand community distribution supply points to ensure consistent resupply of CHWs through the MAHEFA and Santénet2 programs. These community supply points were already a proven successful innovation launched under the predecessor social marketing program to correct inefficiencies that led to stock- outs, and the ISM team intended to further scale these supply points. This approach was largely intended to lead to IR 3: Increased availability of lifesaving health products and services. The cooperative agreement called for ISM to work closely with the various private sector clinics and other USAID partners that focus on community systems to increase access and use of health services and commodities among the rural poor by leveraging private sector providers, since financial and political restrictions were put on the GOM by the US Government because of the coup d’état in 2009. In September 2014, the ISM cooperative agreement was modified to reflect the lifting of bilateral restrictions on the GOM, but the program description was not modified as a result.

Evaluation Team A team of six experts, three international and three Malagasy professionals will conduct the Final Performance Evaluation of the ISM Program as follows:

Dr. Rachel Jean-Baptiste: Team Leader will be responsible for 1) providing team leadership, 2) managing the team’s activities, 3) ensuring that all deliverables are met on time, 4) serve as liaison between USAID and the evaluation/analytic team, and 5) leading briefings and presentations. She will also serve as the evaluation specialist on the team, providing the overall quality assurance on evaluation issues, including methods, development of data collection instruments, protocols for data collection, data management and data analysis. She will oversee the training of all engaged in data collection, insuring highest level of reliability and validity of data being collected. She is the lead analyst, responsible for all data analysis, assuring that all qualitative and quantitative data are analyzed accordingly to meet the needs of this evaluation.

Mr. Iain McLellan: As Evaluation Social Marketing and SBCC Specialist, Mr. McLellan will serve as a member of the evaluation team, providing technical expertise to evaluate social marketing and SBCC activities. He will provide technical expertise on social marketing and social behavior change communications, including demand creating for health products and services, and

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 62 adopting healthy behaviors. He will participate in all aspects of the evaluation, including planning, data collection, data analysis and report writing.

Mr. Juan Manuel Urrutia: As the Evaluation Private Sector Community Health Specialist, Mr. Urrutia will provide expertise in community health, specifically focused on private sector health service delivery, including community distribution and social franchising. He will participate in planning and briefing meetings, development of data collection methods and tools, data collection, data analysis, development of evaluation presentations, and writing of the Evaluation Report.

Ms. Ramy Razafindralambo, Mr. Heritiana Andrianaivo, and one more local evaluator: As local evaluators, they will assist the Evaluation Team with data collection, analysis and data interpretation. They will have basic familiarity with health topics, as well as experience conducting surveys, interviews and focus group discussions, both facilitating and note taking. They will assist in translation of data collection tools and transcripts. All six (6) team members will conduct Key Informant Interviews with stakeholders ISM, USAID, IPs, and the Government of Madagascar, and will be involved in data collection in three (3) regions and six (6) districts.

Research Assistants: The team will include 6 to 10 part-time local research assistants who will help with the collection and data entry of quantitative data, as well as with note-taking and translation, as needed.

3. PERFORMANCE EVALUATION PURPOSE

Evaluation Purpose This evaluation will serve two purposes: 1) to learn to what extent the projects’ objectives and goals – at all result levels – have been achieved; and 2) to inform the design of the new social marketing project. The audience of this evaluation is USAID/Madagascar Mission, specifically the HPN team, the Global Health Bureau; the implementing partner PSI and its partners; and the Madagascar Ministry of Health.

This evaluation will assist the Mission in reaching decisions related to: 1) the effectiveness of current approaches to improve health behavior, improve quality of private sector health services, and increase availability of health products and services; 2) the type of mechanisms the Mission should use in any future assistance to the health sector for social marketing and related interventions, and 3) the nature and scope of possible future interventions in the sector, based on lessons learned from the current project.

Evaluation Questions Specific questions that will guide this Performance Evaluation are stated in the Evaluation Scope of Work and were discussed in detail with USAID/Madagascar, and are understood to be the following:

1) Achieving objectives:

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 63

A. Is the ISM Program likely to achieve its objectives as outlined in the intermediate results and results framework? B. What were the most and least successful activities implemented by the Program? Why? Areas to consider:

o SBCC compared to other components o Understand target setting (could reach more?) o Balance between resource allocation and expected results 2) Integration of Project Components: A. How were the three Program intermediate results integrated (from a management perspective and at the activity level)? B. What were the benefits or disadvantages to this integration? Areas to consider:

o Linkages made between components and impact on end users o Organization of linkages (planning and implementation) o Components critical mass vs. better separated out o Balanced or too focused on one component 3) Definition and reach of “underserved” and “rural”: A. How was “underserved” and “rural” defined operationally for the target populations (women of reproductive age, children under five, youth 15-24 years old, and those living in rural and underserved areas) by the ISM Program?

B. Did Program activities reach this population? Why or why not? Areas to consider:

o Market driven (availability and affordability) o Socio-economic needs driven (availability) o Wealth quintiles o Rural vs. urban underserved reached 4) Program learning and dissemination: How was Program learning (including operational research and, particularly, results of innovative activities) documented, disseminated, and applied to improve ISM Program activities and interventions by other health sector partners? Areas to consider:

o Research dissemination to partners o Utility of research to partners 5) Sustainability: A. To what extent are ISM activities socially and economically sustainable? B. How could the Program design be adapted to improve sustainability? Areas to consider:

o Multiple dimensions of sustainability, including social and economical dimensions o Inclusion of sustainability in close out plan

3. METHODOLOGY

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 64

Guiding Principles This Final Performance Evaluation is taking place one year before the ISM project ends in December 2017. As such, it will focus on the goals, objectives, and subsequent implementation of the overall project over the past four years. The intent is not to document statistical impact of the project, since we do not have baseline with which to compare. Instead, our overarching guiding principle will be to focus on analyzing the efforts made by ISM, the experience and engagement of stakeholders, specifically its partners, Top Réseau, distributors and retailers, as well as its providers of interpersonal communication, and to provide a critical analysis of efficiencies in what was done and what needs to be done to sustain and build on gains. We understand our task to be largely analytical in nature. As such, we propose two overarching principles to guide this evaluation: the systems strengthening for access and reach principle, and the sustainability principle. The first, systems strengthening for access and reach, will analyze how ISM operated with social marketing structures, the private services delivery (Top Réseau), all supported by the SBCC to expand access and quality of services to underserved and rural populations using an adaptation of the BRAC Graduation Model1 and endorsed by the World Bank2, CGAP, and others3 as a gold standard in evaluating programmatic ability to reach the poorest. We will evaluate the extent of access and reach of underserved and rural populations along the continuum of the Graduation Model, specifically to include 1) targeting (defining the target population for Top Réseau, Social Marketing, Community Distribution, and SBCC), 2) consumption support (demand creation (SBCC), demand-based subsidies (vouchers to target population), supply-based subsidies (social marketing margins, fee subsidies at Top Réseau), etc.), 3) skills transfer (behavior change (SBCC), management capacity (Top Réseau), training to SM agents (PA, wholesalers, others), 4) savings (health behavior for consumers (CYP, DALY), cost recovery and cost efficiency) and 5) asset transfer (basic equipment, technology, and brand value) if any. For the purposes of this evaluation, we will consider facilitation by the ISM Project

1 http://borgenproject.org/brac-graduation-model-success-in-fight-against-poverty/ 2 http://www.worldbank.org/en/results/2013/04/04/graduation-program-creating-pathways-out-of-extreme-poverty- into-sustainable-livelihoods 3 https://www.cgap.org/sites/default/files/CGAP-Focus-Note-Reaching-the-Poorest-Lessons-from-the-Graduation- Model-Mar-2011.pdf

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 65 that resulted in a Top Réseau obtaining a loan or other external investments as ‘asset transfer’ (see Figure 1). This principle will guide our work to answer questions 1, 2, and 3. The second principle, sustainability, will analyze the extent to which ISM gains will continue after project closeout, with particular focus on access to private health services and lifesaving commodities by the rural and underserved. This includes establishment of Top Réseau and social marketed products as brands of confidence for the provision of quality health services, and their ability to continue to reach the most underserved and rural populations. Building on Oxford Epi’s Sustainability Framework used in a number of other USAID evaluations, the team will assess the following dimensions: ownership, policy environment, advocacy capacity, staffing, physical infrastructure, organizational infrastructure, and financial viability. This principle will guide our work to answer Evaluation Question 5. Analysis of Evaluation Question 4 will be guided by the concept of continuous learning, and also creating value through research and dissemination.

Evaluation Design The design of this evaluation is cross-sectional with regional and district stratification. The Team’s approach to data collection is inclusive and comprehensive, with multiple methods of data collection proposed. Data collection will include the following:

Review of Project documents, including ISM quarterly and annual reports, work plans, Performance Management Plans, among others. ISM reports and planning documents have been provided. As additional relevant documents are identified, the evaluation team will request them and they will be reviewed.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 66

Review of these documents will allow the evaluation team to obtain a clear picture of what ISM was intended to do, what it has actually accomplished, and some indication into strengths and weaknesses of the project.

Building from this understanding, the Evaluation team will then conduct key informant interviews with ISM leadership and staff to document and analyze the ISM project. Specifically, the interviews will allow ISM staff to articulate the following for Evaluation Question 1:

1) What services does ISM provide, to whom, and how? 2) Target setting, re-evaluation, and reaching targets

3) How is SBCC implemented? Types of Human Resources ISM needs, and partners they work with to provide SBCC, challenges and opportunities

4) Type of information they find critical to their daily operations and mechanisms in place to obtain it on a regular basis

5) Internal governance/accountability processes that allow them to succeed (or hinders success) This will allow the evaluation team to be clear on the structures and processes that enable or hinder success of the ISM Project.

For Evaluation Question 2, the following will be evaluated:

Linkages between components – For this we will look at whether or not activities are planned and implemented together, and if this happens on a routine basis or special occasions; training and capacity building of staff that facilitates/hinders integration; where integration happened; impact of integration on beneficiaries;

Balance between the components – For this we will look at the extent to which there was special focus on one component compared to the others, or if all components were treated equality in terms of time, staff, research, and funding. We will also capture challenges and opportunities to deeper integration between components.

Evaluation Question 3 will be evaluated in the following manner:

We will document the definition of “underserved and rural” used by PSI, as well as the understanding of what drove this definition (market-driven (affordability) vs. population poverty level as defined by wealth quintiles). We will then document findings from the field that supports or negates this definition, and that sheds light on the implementation process. These results will be further triangulated by results from household surveys where we will document if Top Réseau are located in urban, peri-urban, and rural areas, and if users of Top Réseau live in urban, peri- urban, or rural areas; and 2) if socially marketed products are reaching urban, peri-urban, or rural areas, and 3) if SBCC done through the ISM project are reaching urban, peri-urban, or rural areas.

For Evaluation Question 4, the evaluation will be two-pronged:

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 67

First, we will evaluate the types of research that PSI conducts in Madagascar, and extent to which this research is disseminated. Per guidance from USAID, analysis on this area will focus mostly on the dissemination process employed by PSI, whether or not certain types of research are not disseminated, and documenting PSI’s reasons or challenges to doing so.

Secondly, to the extent possible, we will document the perceived utility of the currently disseminated research, as well as the research done by PSI that is not disseminated, based on our discussions with partners.

To answer Evaluation Question 5 data will be collected based on a sustainability framework with the following key dimensions:

A. SHARED UNDERSTANDING of the goals and objectives of the ISM Program to include sustainability of activities, services and gains after its completion among Top Réseau clients

B. OWNERSHIP AND RESPONSIBILITY of the Top Réseau for achieving sustainable goals

C. POLICY ENVIRONMENT that supports sustainability of ISM activities and gains, or governmental responsibility to promote access and use of health services among rural and underserved populations;

D. ADVOCACY CAPACITY (including capacity of Top Réseau for strategic planning and decision making, access to decision makers, etc.)

E. STAFFING (number and type required to deliver the data services; capacity building plans; etc.)

F. ACCESS TO TECHNOLOGY NECESSARY TO THE AIM (includes meds, supplies, as well as IT for data management) G. FINANCIAL VIABILITY (outside of support from ISM or other donor funding) H. FINANCIAL MANAGEMENT SYSTEMS (ability to fundraise, account for and manage funds on their own)

I. RECOGNITION AND SUPPORT of staff for work well done

J. INFRASTRUCTURE (Organizational, Physical, Informational)

K. PUBLIC IMAGE/TRUST (both from the private sector’s perspective, and that of beneficiaries) This Oxford Epi Sustainability Framework builds on previous work done by USAID, the World Bank, and others in the area of governance. Oxford Epi has successfully used data from this framework to other USAID evaluations. Given these evaluation questions, and the Team’s understanding of the needs of USAID/Madagascar Mission, this framework will facilitate the collection of the right data.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 68

Overview of Data Collection

QUESTION SOURCE METHOD DATA COLLECTION TOOLS

Q1 PSI Document review, including Document review documents related to cost capture guide Is the ISM Program likely Partners recovery to achieve its objectives as outlined in the Beneficiaries Program data analysis intermediate results and Communicators results framework? (IR1)

Providers (IR 2)

What were the most Marketers (IR 3) KII questionnaire and least successful activities implemented by FGD guides the Program? KIIs

FGDs

Why?

2 How were the three Program Data Document review Document review Program intermediate capture guide results integrated (from ISM Team KIIs, survey a management USAID IPs perspective and at the activity level)? Beneficiaries FGDs, exit survey KII and FGD guides Communicators FGD (IR1) Questionnaires What were the benefits or disadvantages to this Providers (IR2) integration? Marketers (IR3)

Q3 PSI KIIs KII guides

For the ISM target Program data Document review populations (women of capture guide reproductive age, Document review Questionnaires children under five, Beneficiaries HH survey youth 15-24 years old) how was “underserved” TR providers Phone survey and “rural” operationally FGD guides CHWs FGD defined?

Did Program activities reach this population?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 69

Why or why not?

Q4 ISM partners KIIs KII guide

How was Program Other USAID funded learning (including IPs operational research and particularly results of GOM innovative activities) Other GOM documented, partners disseminated

Q5 ISM partners KIIs KII guide

To what extent are ISM USAID FGDs (with beneficiaries) FGD guide activities socially and economically sustainable? GOM HH survey (with beneficiaries) HH survey

How could the Program Other GOM Survey (TR) TR survvey design be adapted to partners improve sustainability? Top Réseau providers

Medical Council

Beneficiaries

Data Collection Plan Data collection tools, including Key Informant Interviews, FGD guides, questionnaires, and checklists designed for this evaluation will examine and verify the completeness and effectiveness of certain project activities. These instruments include:

• Document review checklist: A checklist of key evaluation elements to check for in document reviews

• Key Informant Questionnaires (customized by audience)

o ISM o GoM o Relevant USAID IPs

o Top Réseau o Social Marketers o Media partners o Other donors, including Global Fund and UN • Focus Group Discussions

o Beneficiaries

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 70

o CHWs o SBCC IEC and IPC around Top Réseau • Exit interviews

o Users of Top Réseau • Web and phone-based Survey

o All Top Réseau • Household Survey

o Beneficiaries Key Information Interviews (KII) using structured interview guides: These will be tailored to the group of individuals being interviewed to elicit information to: a) validate and, where possible, verify project approaches, interventions, and achievements and their current technical and strategic appropriateness; b) secure opinions and perceptions of project implementation effectiveness and issues, and identifying gaps in project activities; c) obtain first-hand reports on training received, data and management systems changed, and overall capacity building d) determine how stakeholders and beneficiaries interact with the project, regarding issues of integration, market penetration, ownership, partnership, and collaboration; and, e) determine how ISM has enabled change in the area of access to quality health services by the underserved and rural populations.

Focus Group Discussions (FDGs) also using structured guides will focus on aspects of integration of the three components among beneficiaries; delivery and reach of socially marketed products among CHWs using the existing system, and the challenges and opportunities for SBCC in discussions with IPC workers. We will ask questions about perceptions of the various groups with regards to current experience, as well as ways things could be improved. This information may be helpful for the follow-on project.

Household Surveys will allow for documentation for the presence or absence of all three project components. It will also allow for an opportunity to quantify project outcomes (e.g., diarrhea among CU5) and compare it to exposure/use of the three project components. Research assistants will capture the data using paper-based tools, and if possible, will immediately enter the data into Oxford Epi’s real time data analysis software so that key insights can be gleamed by evaluators immediately and further inform other qualitative data collection activities.

Phone and we-based survey. Research assistants will contact all 257 Top Réseau and request an interview. This interview will explore issues around target definition, coverage of target group, access and utilization of health services, availability and delivery of commodities, and sustainability of the Top Réseau system.

Data collection instruments are included in the Annex.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 71

Sampling Strategy Selection Criteria for regions and districts: We selected three regions and six districts (one urban, one rural in each region) using the following criteria:

Regions Districts Exclude Antananarivo and Tamatave (per USAID) 2 in each selected region Accessible within maximum one day of travel (air or 1 urban- capital road) Have both urban and rural presence 1 rural – lowest population density MAHEFA operational zone + ISM Team Presence of at least one Top Réseau MIKOLO operational zone + ISM Team Presence of PAs Zone without either MAHEFA or MIKOLO + ISM Team Existence of Warehouse Media presence Contains the most Top Réseau clinics Is secure to travel and work Geographic distribution

This criterion led to the selection of Diana Region, with districts Antsiranana 1 and Antsiranara 2 in the north; Itasy Region, with districts Miarinarivo and Soavinandriana; and Atsimo Andrefana Region, with districts Tulear 1 and Tulear 2. Selection criteria for Top Réseau for online surveys: All Top Reseau clinics included

Selection criteria for data collected through Key Informant Interviews: To conduct the qualitative interviews of key informants, USAID has provided a list of 12 key informants, and the evaluation team will start with them. If, during the course of our interaction, we learn of an additional person we should interview and there is time, their name will be added to the list. Within the regions, types of persons to be interviewed includes: warehouse director, assistant distribution, wholesalers, NGO partner offices, radio station managers. Within districts, persons involved in key informant interviews will include the District Health Officer, Top Réseau clinic staff, Points d’approvisionnement, retailers (observations only), IPC, CHWs, clinic users, and beneficiaries outside of the clinic. Sampling of clinic users, beneficiaries, and CHWs and IPCs will be by convenience.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 72

This sampling strategy is expected to result in the following sample:

Location Source N Tool By Whom

Antananarivo ISM Team 5 KII questionnaire Evaluation Team

GOM 3

Other IPs 3

Other GOM Ptnrs 2

Conseil des Medicins 1

Media partners 3

National Top Réseau 257 Telephone/Web Survey Local Research Assistants

Region (3) Warehouse Director 3 KII Evaluation teams

Assistant Distributor 3

Wholesalers 3

NGO partners 3

Radio station 3 managers

District (6) District Health Officer 3 KII

Top Réseau 6 KII

Points d’appro 12 Checklist

Retailer observations 30 Observations

IPC 6 FGD

CHWs (6-10) 6 FGD

Beneficiaries 24 FGD

Clinic users 30 Exit interviews Local research assistants Beneficiaries 180 Limited HH survey

Data Analysis The data will be analyzed on the basis of the five major questions of the evaluation. Using the frameworks for sustainability and systems strengthening within the private sector previously described above, the analysis of each question will be summarized by key themes or

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 73 domains within the framework that applies to them. Both quantitative and qualitative methods will be used, and wherever possible, the data will be disaggregated by gender. Quantitative data will be captured and analyzed in real time using DatAdventure, a software developed by Oxford Epi for such work. Real time analyses to be produced will include graphs and pie charts of questions. If desired, access to the real-time data will be given to the evaluation COR at USAID, and she will be able to review results in real time. The raw data will be exportable into Excel tables. The Team will use DatAdventure to produce graphs of descriptive statistics. The data will be exported into Excel, and imported into SPSS for further analyses, including calculations of statistical differences where necessary by region, age and gender. Qualitative data will be analyzed using a Qualitative Data Analysis Matrix. The team will analyze the emergence of opinions, perceptions, and issues. The data will be synthesized to determine recurrent themes and issues. Where appropriate these data will be presented in tables. Quantitative and qualitative data will be analyzed on the basis of the five major questions of the Performance Evaluation. This analysis will be further enhanced by examining the data within the context of the domains within the four evaluation questions.

Overview of Data Analysis

Evaluation Question Analytic Key Themes Data Analysis Framework Method

1. Descriptive -age Quantitative characteristics of descriptive data: respondents in -gender evaluation data -mean age -geographical spread collection -% female -type of stakeholder -% per regions (North, South-West, Central)

-#Stakeholder:

2. Program Private Sector ● Processes and efficiencies Quantitative and Achievements Systems Strengthening that will lead to targets Qualitative to reach the poor being achieved or not Status, strengths, weaknesses of each component

3. Integration Private Sector ● End user behavior change Qualitative and Systems Strengthening and use of services and Quantitative to reach the poor products when 2 vs. 3 components integrated Description of the ● Positive behavior trends integration ● Multiplicative effects vs. diminishing returns Effect of the

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 74

Evaluation Question Analytic Key Themes Data Analysis Framework Method

● Balance of focus on integration on target components (money and population time) ● Regional variability any are found 4. Coverage of Private Sector ● Market definition Qualitative and Underserved and Systems Strengthening ● Market growth, share, Quantitative Rural Population to reach the poor penetration ● Utilization of health Socio-economic status services of beneficiaries (age, ● Positive behavior trends gender, economy) ● Regional variability Description of location of Top Réseau sites

Analysis of similarities and deviations by regions

5. Research Research and ● Research initiation Qualitative and Dissemination dissemination process ● Research implementation Quantitative ● Data use for program learning Summary of number ● Results dissemination of research conducted vs disseminated

6. Sustainability Sustainability ● GOM and Private Sector Qualitative and Framework preparedness Quantitative ● Institutional continuity of product and service delivery Preparedness (policy, based on demand ownership, advocacy, ● Community acceptability HR, technology, and confidence in Top finance, infrastructure, Réseau, Social Marketed confidence/trust) commodities ● Regional variability

The final report will list each Evaluation Question followed by Findings, Analysis, Conclusions, and Recommendations section. The end summary will focus on priority issues for USAID/Madagascar to address and major lessons learned based on the answers provided in examining the four Evaluation Questions. This approach should aid the Team in finding gaps in the current activities and processes. Specifically, the Team will:

1) Review ISM reported achievements against the PMP and work plan.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 75

2) Summarize commonalities related to the topics covered in the KIIs. Main topics will cover the project implementation process as depicted by the frameworks used; including: ownership; practicality; effectiveness; gaps; suggestions for improvements; and lessons learned.

3) Develop qualitative analysis tables highlighting results of discussions on key themes.

4) Develop quantitative analysis tables highlighting results from surveys. 5) Develop summary descriptive statistics of all participants.

6) Develop recommendations based on findings for each of the four questions. Ethical Considerations The evaluation team will implement a policy of informed consent for all key informant interviews and focus group discussions (see Annex 3 for an example) and all interviews will be done on a voluntary basis. Interviewees will be given the option to opt-out of particular questions or the whole interview, if at any time they believe a response would contain sensitive information. Survey takers will also be provided a similar option for informed consent and participation. The information provided as part of these interviews and discussions will not be linked to any specific person in the Final Report and all information provided will be kept confidential and used for planning purposes only. Only general identifying information (organization, geographical unit, gender, and age if reported voluntarily) will be utilized. Any information that could be directly linked to an individual will not be used. Only members of the Evaluation Team will have access to the transcripts and raw data. The Final Report will be a synthesis of the Team’s analysis drawn from interviews from numerous respondents. Any included quotes to highlight particular issues will not include names. We will not collect data from minors, or directly from patient records. We will not collect any data with personally identifiable information throughout this process.

Limitations of the Evaluation Selection only of districts and health facilities that are geographically accessible is a limitation to understanding the extent to which ISM activities reached rural and underserved populations throughout the country. This is due to limited resources and short time frame, and as such, the data collected will not be generalizable to Madagascar. However, several efforts have been made to correct for this shortcoming, ensuring geographic spread of the selected regions (Southwest, Central, North), and intentional urban/rural stratification of selected districts. In addition, the online surveys transcend boundaries due to geographical accessibility, and may provide additional insights, though only for activities that involve the Top Reseaus.

4. PREPARATIONS FOR FIELD WORK

The evaluation will be carried out by the Evaluation Team in cooperation with USAID/Madagascar and the ISM team. To ensure quality of data collection, the evaluation team leader will establish clear guidelines for data collection, specifically for how to conduct interviews. The SBCC expert will brief team members on how to best collect information

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 76 related to SBCC, while the Private sector expert will do likewise for data collection on franchise and market penetration. They will each lead data analysis in their respective areas. The Team is supported by a logistics specialist who has been working with the team from day 1 to set up appointments, and facilitate travel outside of Antananarivo. Given the few days available for data collection, the team is likely to split into three groups to more efficiently manage time and collect maximum data. The team may also employ part time research assistants who will help with conducting phone interviews for the online data collection. Accordingly, the Final Performance Evaluation will include the following steps:

1) Finalization of data collection tools

2) Formation of three data collection teams who will conduct visits to one region, 2 districts each

3) Review and conduct training on tools 4) Conduct data collection with quality control checks interspersed 5) Data entry and analysis 6) Report writing

5. TIMELINE AND DELIVERABLES

Date Tasks and Deliverables 10/11/2016 Launch briefing call with USAID/Madagascar 10/19/2016 In-brief with USAID/Madagascar 10/19-10/23/2016 Preparation of Evaluation Protocol, Analytic Plan, Timeline, elaboration of data collection tools and USAID Methodology review briefing material 10/24/2016 Workplan and methodology review briefing with USAID Submission of Evaluation Protocol and Timeline 10/28/2016 Inbrief with target project (ISM Team) 10/25-10/29/2016 Data collection in Tana, including with ISM Team Recruitment of research assistants for phone/web survey 10/30 Travel to Regions 10/31 – 11/9/2016 Data collection in regions and districts 11/10/2016 Travel to Antananarivo 11/11-11/17/2016 Analyze results, prepare outbrief presentation 11/18/2016 Informal outbrief USAID; Outbrief with PSI/ISM team

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 77

Date Tasks and Deliverables 11/21/2016 Formal outbrief with USAID 11/22-11/23/2016 Travel out of Madagascar 11/30-12/11/2016 Continue analysis and report writing 12/12/2016 Draft report to GH Pro; GH Pro submits to USAID 12/16/2016 1/12/2017 Final report to GH Pro; GH Pro submits to USAID 1/17/2016 1/12/2017 Raw data (cleaned datasets in CSV or XML with code sheet or data dictionary) 2/23/2017 Report posted on DEC * This timeline does not include the weekly updates which will be provided by the Team to USAID. These weekly updates will also provide the Team the opportunity to request any reasonable assistance from USAID, if needed.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 78

ANNEX 1 – DATA COLLECTION TOOLS

Key Informant Interviews with ISM Staff and Partners

Question/Information Required

00 Date of interview: ______/______/______

[ ] PSI

[ ] PSI partner ______

[ ] USAID

[ ] Other USAID partner

[ ] GoM

[ ] Other GoM partner

Which ISM component are you responsible for?

[ ] SBCC [ ] Private service delivery [ ] Social Marketing [ ] Not an ISM implementer

Thank you for making the time to talk with me today.

The USAID/Madagascar has asked GH Pro to conduct an evaluation of the performance of the ISM project implemented by PSI. This evaluation is meant to serve a dual purpose: (1) to learn to what extent the project’s objectives and goals—at all result levels—have been achieved; and (2) to inform the design of a new social marketing project.

You were suggested as a key person to inform this activity and we greatly appreciate your perspective, experiences and views.

Our questions are organized to obtain a good overview of how the ISM project functioned operationally, and to answer the evaluation questions. As such, we will be asking you questions about 1) project achievements vs. targets to date; 2)

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 79

Integration of program results and the benefits/disadvantages of such integration; 3) the ISM project’s definition/operationalization of “underserved” and “rural” as it relates to the project’s target population; 4) documentation, dissemination, and application of program learning; and 5) social and economic sustainability of ISM activities.

If there are staff members who are particularly suited for specific questions, we would appreciate the opportunity to include them in the interview (as part of the group) or to talk with them separately.

All of the answers you provide will be summarized and included in our report. We will also be conducting interviews and focus group discussions with other stakeholders, and will be triangulating all findings. The final report will be shared with you through USAID.

Before we begin, I want to let you know that any information or examples we gather during this interview process will not be attributed to any specific person or institution, unless you tell us that you would be willing to have your responses to be either quoted by you in the report, or otherwise attributed to you. You are also free to not respond to any of our questions or stop the interview at any time.

Our interview will take about one hour.

Before we go further, do you have any questions about this interview?

[ ] Consent provided ______[Interviewer/Recorder initials]

Do you have any questions before we begin?

May we continue with the interview?

[ ] Yes [ ] No

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 80

Name of Respondent______

Age ______Gender [ ] Male [ ] Female

Job Title______

Email: ______@______.______

Phone number: ______

Name of Respondent______

Age ______Gender [ ] Male [ ] Female

Job Title______

Email: ______@______.______

Phone number: ______

Name of Respondent______

Age ______Gender [ ] Male [ ] Female

Job Title______

Email: ______@______.______

Phone number: ______

Name of Respondent______

Age ______Gender [ ] Male [ ] Female

Job Title______

Email: ______@______.______

Phone number: ______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 81

Name of Respondent______

Age ______Gender [ ] Male [ ] Female

Job Title______

Email: ______@______.______

Phone number: ______

Name of Respondent______

Age ______Gender [ ] Male [ ] Female

Job Title______

Email: ______@______.______

Phone number: ______

Let’s begin by talking about the ISM Project.

01 What is the objective of the ISM project? (probe – what services does ISM offering/supporting, to whom, and how? How do you work with Top Reseau? CHWs? Supply chain and distribution actors?)

What activities does the ISM implement? [ PSI ONLY]

Can you talk about resource allocation – do you feel it is balanced well across all of these activities? [ PSI ONLY]

-how has this balance or imbalance affected your results? [ PSI ONLY]

How do the ISM partners interact to deliver on the objectives of the project? What are strengths

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 82

and weaknesses of this partnership? [PSI ONLY]

Of the three components, SBCC, social marketing and private services, what was/is the most successful? Why?

What was the least successful? Why?

[IF SBCC not mentioned]: Can you tell me a bit about your work with SBCC? [ PSI ONLY]

-are you reaching your targets?

-What has worked well?

-What has not worked well? Why?

What are some of the challenges ISM has experienced?

Will these challenges keep ISM from reaching its targets by the end of 2017?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 83

02 Can you tell us a bit about the method you have used to set your targets? (baseline data used? 1 Sources? Etc.) [ PSI AND ISM PARTNERS ONLY]

At which point do you involve the Mission? [ PSI ONLY]

How likely are you to achieve your targets?

Why?

Why not?

03 Let’s talk about integration within the ISM project. In implementing ISM, can you tell us how you 2 linked SBCC with service delivery with commodities?

(from planning to implementation; improvement processes; review of results)

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 84

Examples?

What does it take to create these linkages? Maintain it?

Can you give examples of where linking these three components has led to synergies?

Are there reasons you think it would be better to focus on only one or two? (if so, which one, or two?)

Overall, what proportion of resources went to each of the three components:

In terms of money:

______SBCC ______Private healthcare ______commodities

In terms of time:

______SBCC ______Private healthcare ______commodities

04 For the ISM target population (women of reproductive age, children under five, youth 15-24 years), 3 how was "underserved" and "rural" operationally defined?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 85

(probe: market-driven vs. wealth quintiles (lower wealth))

(probe: explore definition of urban, peri-urban, rural)

What data sources did you use, if any?

Did Program activities reach these populations? (document proof)

Why or why not?

Are you likely to reach your targets for [each target group] by December 2017?

How do you know?

Why?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 86

Why not?

07 We understand ISM has done a significant amount of research. What have they researched? Can 4 you please describe your research process, from initiation to end?

(probe: who initiates, who conducts, who analyses, who presents, who benefits? Involvement of USAID?)

08 Can you tell us about your use of the research data?

09 What do you do to disseminate your research findings? 4

Is this applied to every research project? If not, which ones does it apply to? Are there any exceptions?

10 Outside of the ISM project, is the ISM data being used? for what? By whom? 4

What has been ISM’s role in promoting this use?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 87

11 On a scale of 1-10, with 1 being not at all, to 10 being complete, how prepared do you 5 think the Government and the Private sector are to continue to increase commercially available health products and services without ISM or USAID support?

Family planning

Pilplan ______

Confiance ______

Yes youth condoms ______

Protector Plus ______

Feeling condom ______

Sayana Press ______

Norlevo ______

Implanon ______

Jadelle ______

IUDs (Copper T) ______

IUDs (Zarin) ______

Chlorhexidine ______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 88

Rojo-branded cycle beads ______

Malaria

LLINs ______

ACTs ______

RDTs ______

Child Survival

ViaSur (or equivalent) ______

HydraZinc (DTK) ______

Sur Eau Pilina Tablets and liquid ______

Pneumox ______

Zaza Tamady-branded micronutrient powder (MNP) ______

12 Are all of these products legally registered in Madagascar?

13 Who owns the brands/trademarks (ownership)?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 89

14 On a scale of 1-10, how much capacity would you say the distribution partners of the ISM project has to advocate for social marketing commodities?

______

Please explain

15 On a scale of 1-10, do distribution partners of the ISM project have the physical infrastructure for socially marketed products?

Space ______

Equipment ______

Transport ______

Access to relevant information ______

16 On a scale of 1 to 10, please rate the extent to which you believe that distribution partners of the ISM project have adequate staff to implement the ISM activities

______

Please explain

17 On a scale of 1-10, do distribution partners of the ISM project have the organizational infrastructure for socially marketed products:

Clear internal roles and responsibilities ______

capacity for marketing ______

Internal QA/CQI processes that are operational ______

Supply Chain Management ______

Financial management capacity ______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 90

Overall, on a scale of 1 to 10, how strong would you say the ISM’s distributors’ organizational capacity for social marketing? ______

18 On a scale of 1-10, how financially viable

On a scale of 1 – 10, what is the level of trust that the distributors/retailers have from their consumers?

______

19 5

On a scale of 1-10, with 1 being not at all, to 10 being complete, do you think that the Top Reseaus, overall:

Share the same understanding with the ISM project regarding their role in promoting health equity by providing high quality care to the poor as well as other clientele?

______

Take full ownership and responsibility for ensuring financial viability of the e-voucher (or other subsidy) system?

______

Has a policy environment supportive of their role as a provider of equitable services to the poor as well as other clientele?

______

Has capacity to advocate for additional resources to support/maintain its role as an equitable and high quality private service provider?

______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 91

--in particular, what is the advocacy capacity within the Top Reseaus to solicit additional support or to create public private partnerships that may benefit their catchment area?

Has sufficient staff numbers and types and capacity of staff to ensure that it can successfully serve as a private provider of high quality services?

Numbers: ______

Types: ______

Capacity: ______

Are staffing levels at the Top Reseaus in general appropriate to manage the delivery of the needed high quality primary healthcare services in the communities they serve?

Yes/No,

______

why/why not?

what changes to staffing are needed/have been completed recently to support service delivery?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 92

What are the plans and processes for continuous recruitment, education, orientation, retention, and evaluation?

Has sufficient access to necessary technology and supplies to facilitate its role as a high quality primary healthcare provider for their catchment area?

______

What technology and supplies do they need that they do not currently have access to?

- What is the access to computer, electricity, hardware/software, internet? And Who pays?

-what is their access to medicines, and who pays?

-what is their access to laboratory tests, and who pays?

Do the Top Reseaus have adequate financial resources outside of the ISM project for staffing, Internet, training, other resources to provide high quality primary healthcare services to its catchment area?

______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 93

Has capacity to attract additional paying clients if/when necessary?

______

Can you please elaborate what is their marketing strategy/strategies for obtaining new and retaining old clients?

Have mechanisms to recognize and reward good performance within their team?

______

Has access to medicines and supplies required to provide high quality primary healthcare to the underserved and rural populations?

Has

______

On a scale of 1 to 10, to what extent do Top Reseaus have the organizational infrastructure to continue to implement ISM activities in the absence of the ISM program?

documented mechanisms for continual learning ______documented process for staff recruitment ______and retention ______,

Well defined staff roles and responsibilities ______capacity for marketing ______

Referral and counter referral with larger (public or private) facilities to transfer more complicated cases ______

Internal QA/CQI processes that are operational ______

Routine data collection, analysis and reporting ______

Financial management capacity ______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 94

On a scale of 1 (barely) to 10 (significant) Please rate the extent to which the Top Reseaus enjoy positive media coverage at the community, district and provincial levels

______

On a scale of 1(negative) to 10 (complete positive), please rate the perception of communities of Top Reseaus

______

Does the public understand the concept of Top Reseau?

Is the public supportive of Top Reseau?

In your opinion, who does the public rely on as a credible source of expertise and credible health information?

[ ] Top Reseau

[ ] CHWs

[ ] Peer models

[ ] Radio

[ ] TV

[ ] Newspaper

[ ] Friends

[ ] Family

[ ] Other

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 95

What is the level of confidence in the services that Top Reseaus have from their catchment area?

______

Why do you think this perception is accurate/not accurate?

Urban:

Rural:

20 Overall, on a scale of 1 to 10, how likely are the Top Reseaus financially capable of continuing to provide services at lower or no cost to the poor without support from the ISM project or USAID?

______

Please detail what is funded in the ISM project for Top Reseaus:

______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 96

Of the 257 Top Reseau, what proportion are likely to continue without additional support from the ISM team?

What distinguishes these ones from the other Top Reseaus?

Where are they located?

21 Do you have a close out plan?

If yes, may we have a copy?

22 How will you/have you factor(ed) in sustainability in your close out plan?

23 How would you describe ISM’s added value in Madagascar to: 4

Private clinic scene:

Supply chain of essential commodities:

Community health workers:

Women of reproductive age?

Youth:

-males

-females

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 97

Children under 5 and their mothers?

Ministry of Health:

Ministry of Gender:

Other USAID partners:

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 98

24 Now that you have lived through the implementation of ISM, what would you have done differently gen if you could?

25 How could the program design be adapted to improve sustainability?

26 In an ideal world, what do you think should be the main focus of a follow on project? gen

Would you include all of these components together again?

If you had to remove at least one, which one would that be?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 99

Would you add others?

27 Do you have any other thoughts you would like to share?

Thank you very much! If you think of anything else, please do not hesitate to get in touch with us.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 100

ISM Final Performance Evaluation Document Review Guide

Reference Type Question 1: Question 2: Question 3: Question 4: Question 5: Comments Achievements Integration of Definition and Research Sustainability Components Coverage of Dissemination Rural and Underserved

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 101

ISM Final Performance Evaluation Exit Interview with Top Reseau Clinic Users

Question/Information Required

00 Date of interview: ______/______/______

Thank you for making the time to talk with me today.

You were suggested as a key person to inform this activity and we greatly appreciate your perspective, experiences and views.

Our interview will take about one hour.

Before we go further, do you have any questions about this interview?

[ ] Consent provided ______[Interviewer/Recorder initials]

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 102

Do you have any questions before we begin?

May we continue with the interview?

[ ] Yes [ ] No

Male [ ] [ ] Female

Marital Statut Célibataire______Mariée ______Autre______

Age Intervalle 15-24______25-35______35-49______

Distance de la Clinique Très Près____ _Assez Près______Loin______Très Loin_____

02 En général, combien de personnes habitent votre foyer ?

[___|___]

03 Combien d’enfants avez-vous ?

[___|___]

04 Quel est votre niveau d’études ?

[ ] non alphabétisé

[ ] Primaire

[ ] Secondaire

[ ] Certificat

[ ] Bac/Universitaire

[ ] Maitrise

A5 Quelle est votre occupation ?

[ ] Je n’ai pas d’emploi

[ ] paysan

[ ] pécheur

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 103

[ ] petit commerçant

[…] domestique

[…] épicier

[ ] Taxi

[ ] Professionnel

[ ] Police, sécurité

[ ] Tailleur

[ ] Coiffeur

[ ] Artist

[ ] Other______

06 Est-ce que votre époux(se) a un travail ?

[ ] Oui

[ ] Non

07 Est-ce votre première visite à la clinique ?

[ ] Oui

[ ] Non

07A Pourquoi préferez vous cette clinique au Centre de Santé ?

Meilleure qualité

Plus Proche

Connais le docteur

Vu ou entendu la publicité

Autre raison, expliquéz

08 Venez-vous souvent a la Clinique?

[ ] Plus d’une fois par mois

[ ] 1 fois par mois

[ ] 1 fois par trimestre

[ ] 1 fois par semestre

[ ] 1 fois par an

[ ] Moins d’une fois par an

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 104

09 Quel est le propos de votre visite

[ ] Enfant malade Age de l’enfant [ ] 0-30 jours ; [ ] 31 j – 1 ans ; [ ] 1 – 5 ans

Sexe de l’enfant [ ] male ; [ ] femelle

[ ] Vous êtes souffrante

[ ] Enceinte

[ ] Contrôle

[ ] Planning familial

[ ] Autre

10 Combien de temps vous avez attendu dans la sale d’attente?

[ ] 0 a 15 minutes

[ ] 15 a 30 minutes

[ ] 30 – 60 minutes

[ ] 1 heure a 2 heures

[ ] Autre ______

11 Quelle est votre durée du trajet pour venir a la clinique

[ ] 0 a 15 minutes

[ ] 15 a 30 minutes

[ ] 30 – 60 minutes

[ ] 1 heur a 2 heurs

[ ] Autre ______

12 On vous a posé des questions pour faire une fiche médicale?

[ ] Oui

[ ] Non

13 Est ce qu’on vous a expliqué en détail votre problème de santé ?

[ ] Oui

[ ] Non

14 Avez-vous reçu un bon conseil ?

[ ] Oui

[ ] Non

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 105

15 Sur une échèle de 1 (très mauvais expérience) a 10 (très bonne expérience), comment trouvez- vous la qualité de soin que vous avez reçu aujourd’hui ?

______

16 Comment avez-vous appris l’existence de cette Clinique ?

[ ] Un ami

[ ] J’ai vu sur la route

[ ] Radio

[ ] TV

[ ] Agents de Santé de la communauté

[ ] Autre agent communautaire)

[ ] Une autre clinique

17 Vous venez ici parce que vous connaissez le docteur ?

[ ] Oui

[ ] Non

18 Est-ce que vous avez entendu une publicité de la Clinique a la radio ?

[ ] Oui

[ ] Non

19 Est-ce qu’un agent de santé vous a conseillé de venir à la Clinique ?

[ ] Oui

[ ] Non

20 Est-ce que quelqu’un de proche, ami ou membre de la famille vous a conseillé de venir à la clinique ?

[ ] Oui

[ ] Non

21 Recommanderiez-vous la clinique a un de vos proches ?

[ ] Oui

[ ] Non

22 Achetez-vous des produits à la Clinique ?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 106

[ ] Oui

[ ] Non

23 SI Oui, qu’est-ce que vous achetez ?

______

24 Comment trouvez-vous le prix de la consultation

[..] Payé avec un voucher

[ ] Cher

[ ] Acceptable

[ ] Bon Marché

25 Reviendrez-vous à la Clinique si besoin ?

[ ] Oui

[ ] Non

Household Survey of ISM target populations:

• Female Youth 15-24

• Male Youth 15-24

• Women of Reproductive age 25-49

• Parents of children under 5

Date of interview: ______/______/______

Name of interviewer______

Profile of interviewee

__Female Youth 15-24

__Male Youth 15-24

__Women of Reproductive age 25-49

__Mothers children under 5

__Fathers of children under 5

Thank you for making the time to talk with me today.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 107

Our interview will take about one hour. Before we go further, do you have any questions about this interview?

[ ] Consent provided ______[Interviewer/Recorder initials]

Do you have any questions before we begin?

May we continue with the interview?

[ ] Yes [ ] No

01 Male [ ] [ ] Femelle

Marital Statu Célibataire______Mariée______Autre______

Age Interval 15-24______25-35______35-49______

Distance de la Clinique Trés Prés_____Assez Prés______Loin______Très Loin_____

02 En general, il y a combien de de personnes qui vivent dans votre maison?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 108

[___|___]

03 Combien d’ enfants avez-vous ?

[___|___]

04 Quel est votre niveau d’éducation ?

[ ] non alphabétisé

[ ] Primaire

[ ] Secondaire

[ ] Certificat

[ ] Bac/Universitaire

[ ] Maitrise

A5 Quelle est votre occupation ?

[ ] Je n’ai pas d’emploi

[ ] paysan

[ ] pécheur

[ ] petit commerçant

[…] domestique

[…] épicier

[ ] Taxi

[ ] Professionnel

[ ] Police, sécurité

[ ] Tailleur

[ ] Coiffeur

[ ] Artist

[ ] Other______

06 Est-ce que votre époux(se) travaille ?

[ ] Oui

[ ] Non

Let’s begin by talking about the ISM Project.

1) For each of the following promotions indicate

● Yes I have heard of it

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 109

● I heard about it on the radio ● I heard about it on television ● I heard about it through a Mobile Health Unit ● I heard about it through a Top Reseau clinic ● I heard about it through a community health worker

● Healthy Family Campaign promoting good health habits to mothers of children under 5 ● Yes I have you heard of it [ ] I heard about it on the radio

[ ] I heard about it on television

[ ] I heard about it through a Mobile Health Unit

[ ] I heard about it through a Top Reseau clinic

[ ] I heard about it through a community health worker

● Promotion of Long-Lasting Impregnated Nets (LLIN) [ ] I heard about it on the radio

[ ] I heard about it on television

[ ] I heard about it through a Mobile Health Unit

[ ] I heard about it through a Top Reseau clinic

[ ] I heard about it through a community health worker

● Malaria treatment (ACTs, RDTs, ITP) [ ] I heard about it on the radio

[ ] I heard about it on television

[ ] I heard about it through a Mobile Health Unit

[ ] I heard about it through a Top Reseau clinic

[ ] I heard about it through a community health worker

● Water hygiene products Sur’Eau solution and tablets [ ] I heard about it on the radio

[ ] I heard about it on television

[ ] I heard about it through a Mobile Health Unit

[ ] I heard about it through a Top Reseau clinic

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 110

[ ] I heard about it through a community health worker

● Diarrhea treatment products like Hydro-Zinc and the Diarrhea Treatment Kit [ ] I heard about it on the radio

[ ] I heard about it on television

[ ] I heard about it through a Mobile Health Unit

[ ] I heard about it through a Top Reseau clinic

[ ] I heard about it through a community health worker

● Promotions of short term Family Planning commodities like pills, condoms [ ] I heard about it on the radio

[ ] I heard about it on television

[ ] I heard about it through a Mobile Health Unit

[ ] I heard about it through a Top Reseau clinic

[ ] I heard about it through a community health worker

● Promotions long-lasting Family Planning commodities like injectables, implants [ ] I heard about it on the radio

[ ] I heard about it on television

[ ] I heard about it through a Mobile Health Unit

[ ] I heard about it through a Top Reseau clinic

[ ] I heard about it through a community health worker

● Promotion of Top Reseau clinics [ ] I heard about it on the radio

[ ] I heard about it on television

[ ] I heard about it through a Mobile Health Unit

[ ] I heard about it through a Top Reseau clinic

[ ] I heard about it through a community health worker

2. On a scale of 1 (not good) to 10 (very good) how would you rate the quality of the following communication channels for reaching people like you. Leave blank if unfamiliar

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 111

with channel.

__National radio spot advertisements

__National radio drama on Family Health

__Local radio

__Mobile Video Van

__Top Reseau clinic counseling and outreach

__Interpersonal communication by Community Health Workers or others

3. On a scale of one to ten how would you rate the quality of the following social marketing services and products:

__Long-Lasting Impregnated Nets (LLIN)

__Sur’Eau solution and tablets

__Diarrhea treatment products Hydro-Zinc and the Diarrhea Treatment Kit

__Short-term Family Planning Commodities like PIPLAN, PROTECTOR PLUS

__Long-term Family Planning Commodities like CONFIANCE, Implants and IUDs

__Emergency contraceptive Unipil

__Pneumonia treatment Pneumox and/or pneumostop

__Top Reseau clinics

4. On a scale of one to ten how would you rate access to the following social marketing services and products:

__Long-Lasting Impregnated Nets (LLIN)

__ Sur’Eau solution and tablets

__Diarrhea treatment products Hydro-Zinc and the Diarrhea Treatment Kit

__Short-term Family Planning Commodities like Birth Control Bills and condoms

__Long-term Family Planning Commodities like Injectables, Implants and IUDs

__Emergency contraceptive Unipil

__Pneumonia treatment “pneumax

__Top Reseau clinics

5. On a scale of one to ten how would you rate the affordability of the following social marketing services and products:

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 112

__Long-Lasting Impregnated Nets (LLIN)

__Sur’Eau solution and tablets

__Diarrhea treatment products Hydro-Zinc and the Diarrhea Treatment Kit

__Short-term Family Planning Commodities like Birth Control Bills and condoms

__Long-term Family Planning Commodities like Injectables, Implants and IUDs

__Emergency contraceptive Unipil

__Pneumonia treatment “pneumax

__Top Reseau clinics

6. What are the biggest challenges in accessing social marketing services and commodities. Check one of more.

__Never heard of them

__Not easy to access

__Not affordable

__Not meeting needs

__Stocks run out often

7. Please indicate on a scale of one to ten to what degree you agree with the following statements. One meaning that you don’t agree at all and 10 meaning that you agree very much. Mark zero if statement is not applicable to you.

__I have never heard of Top Reseau

__I have heard of Top Reseau clinics but never used its services

__I have never used the clinics because they are too far from where I live

__I have never used the clinics because they are too expensive

__I have used the services and find them to meet my needs

__I have used the services and find them to be affordable

__I have used the Family Planning services

__I have used the Family Health services for children under 5

__I would recommend Top Reseau clinics to others

Thank you very much for your participation.

8. when was the last time you used the Top Reseau Clinic ? 1, 2, 3

[ ] less than 2 weeks ago

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 113

[ ] 2 weeks to 1 month agao

[ ] 1 to 3 months ago

[ ] 3 to 6 months ago

[ ] More than 6 months ago

[ ] I have never visited a Top Reseau clinic

[..] I don’t remember

Why did you go to the Top Reseau clinic ?

[ ] Sick Child Age of child [ ] 0-30 days ; [ ] 31 days – 1 yr ; [ ] 1 – 5 yr

Sex of child [ ] male ; [ ] female

[ ] I was sick

[ ] Pregnancy

[ ] Contrôle

[ ] Family planning

[ ] Other

On a scale of 1 (very bad experience) to 10 (very good experience), how did you find the quality of service that you received during your last visit?

______

Comment-avez-vous appris de l’existence de cette clinique ?

[ ] Un ami

[ ] J’ai vu sur la route

[ ] Radio

[ ] TV

[ ] CHWs

[ ] Agence communautaire

[ ] Peer Educator (model mother/father, youth group, etc)

[ ] Une autre clinic

Autres

Achetez vous des produits a la Clinique?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 114

[ ] Oui

[ ] Non

Si Oui, qu’est ce que vous achetez ? 4

______

Trouvez vous le prix de la consultation

[ ] J’ai payé avec un voucher

Cher

[ ] Acceptable

[ ] Bon Marché

Reviendrez-vous a la Clinique si besoin?

[ ] Oui

[ ] Non

Do you currently use any method of contraception?

[ ] Yes

[ ] No

If yes, which type?

______

How many children 0 to 5 years old live in your house?

______

Of the children 0 to 5 who live in your house, how many had diarrhea in the past two weeks?

______

Do you currently have a bednet (LLIN)?

[ ] Yes

[ ] No

IF Yes, Did you sleep under it last night?

[ ] Yes

[ ] No

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 115

Did any of the children 0 to 5 years old who live in this house sleep under the bednet last night?

[ ] Yes

[ ] No

Thank you very much! If you think of anything else, please do not hesitate to get in touch with us.

KII Social Marketing Personnel and sales force Location: Date:

Key Information to be completed by the interviewer In your opinion: is this an urban, peri-urban, or rural site? General appearance of the store or point d’approvisionnement Cleanliness Signage and IEC materials display Product display

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 116

KEY INFORMANT INTERVIEWS SM PERSONNEL AND SALES FORCE

Question/Information Required

00 Date of interview: ______/______/______

Thank you for making the time to talk with me today.

You were suggested as a key person to inform this activity and we greatly appreciate your perspective, experiences and views.

Our interview will take about one hour.

Before we go further, do you have any questions about this interview?

[ ] Consent provided ______[Interviewer/Recorder initials]

Do you have any questions before we begin?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 117

May we continue with the interview?

[ ] Yes [ ] No

Age ______Gender [ ] Male [ ] Female

01 Describe your role as a PSI salesperson

02 Please list the products you sell and the prices you sell them at 1

03 What are your monthly sales target and do you frequently meet your targets, exceed or fall short 1 on your targets?

04 Please list the people you sell to 3

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 118

05 How many clients do you visit every day and how often do you visit them? 1

06 Please describe your interaction with a new client (he she comes to you, you visit the household to 1 recruit, you meet them in promotion events)

07 What are the most common questions clients ask you? 2

08 List starting with the one that gets you more clients the advertising and promotions that you use 2 (Radio, Point of Sale materials, PSI Mobile Video Units, person to person education, events and/or campaigns)

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 119

09 Please list the training you have received for PSI 2

10 In your opinion, how do your clients learn and decide about buying your products 2

11 What do you think of the products prices 2

[..] Expensive

[..] Acceptable

[..] Inexpensive

12 Please list the problems you may have encountered with stock 3

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 120

13 Please list, if any, other products, different from PSI´s that you sell

14 Please list the products that you receive from the government and that you give, for free, to the 5 clients

15 In your opinion, social marketing products that you sell are profitable or not too much? 3

16 How do social marketing products volumes compare to other similar products, for example 5 essential household goods (soap) or over the counter medicines

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 121

17 Where is the closest Top Réseau clinic, do you use it, do you send clients there? 3

18 How often do you attend meetings with personnel from the closest Top Réseau clinic? 2

19 Please list the services that the closest Top Réseau clinic offers 2

(Inteviewer can offer options if respondent has trouble)

20 Are your clients 3

urban,

rural,

very poor,

poor,

just fine,

rich

21 If prices were increased by 30%, how would this affect your sales? 3,5

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 122

22 If there were no social marketing products available, do you think that you would still be selling the 5 products at commercial prices?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 123

TOP RÉSEAU Sites visit guide Location:

Date:

Basic Information to be completed by the interviewer The day that the key person interview is to take place, it is recommended that you arrive ten to fifteen minutes ahead of time and just sit in the wait area and observe the flow, it will help you direct the conversation to obtain as much information about the quality assurance culture and client satisfaction. In your opinion: is this an urban, peri-urban, or rural site? General appearance of the clinic Cleanliness Signage and IEC materials display How many people were in the waiting room? To the best of your knowledge how would you describe the patients in the waiting room in terms of their Socio-Economic Status? How many staff work in the clinic? Write down any other details of interest that relate to your perception of quality upon arrival to the clinic.

Throughout the whole process think: Would I bring my husband/wife/children to this clinic?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 124

KEY INFORMANT INTERVIEWS WITH TOP RÉSEAU CLINIC´S STAFF

Question/Information Required

00 Date of interview: ______/______/______

Thank you for making the time to talk with me today.

We are visiting six Top Réseau Clinics and yours was selected.

Our interview will take about one hour.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 125

Before we go further, do you have any questions about this interview?

[ ] Consent provided ______[Interviewer/Recorder initials]

Do you have any questions before we begin?

May we continue with the interview?

[ ] Yes [ ] No

Age ______Gender [ ] Male [ ] Female

01 Describe when and how this clinic became a Top Réseau Member 1

02 What are the main reasons to become a Top Réseau Clinic and what are the advantages (more 1 clients, improved quality, less expensive commodities, expanded services)?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 126

03 List the new services offered, under Top Réseau, to your clients, their response, and the effect of 1 adding these services to the clinic´s number of clients

04 Describe your clients’ profile (age, gender, economic status, urban or rural)? 3

05 How is price of each service determined? 1

06 Describe what kind of technical support you have received and by whom (medical, finance and 2 administrative, counseling, supply management)?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 127

07 Describe the flow for a patient the first time he or she comes to your clinic, starting with the first 1 contact and until he or she sees the doctor, please give us detail on how and who creates the medical record (histoire clinic)

08 Describe the same flow for a repeat patient 1

09 Please describe the medical record keeping system, 1

10 How would you describe the counselling provided to the patient?

(who does it, when is it done)

11 Describe the basic elements of the Top Réseau minimum standards, 1

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 128

12 To your understanding how important is the application of the minimum medical standards 1

13 In what manner and to what degree, belonging to Top Réseau Plus has helped the clinic in 2 improving the quality of its services?

14 Please describe the Information, Education and Communications IEC materials used by the clinic, 2 who provides them, with what frequency (Brochures, Posters, Danglers, Signs, Other?

15 Describe the kind and frequency of detailing visits from PIS supervisors and detailers 2

16 Please list the other organizations that visit the clinic for supervision or promotion 2

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 129

17 List the promotional events, like Women’s day or malaria day in which the clinic has participated 2 and the role it played

18 To your understanding what are the sources where your first time clients get to know about the 2 clinic

19 Why do you think the clients come to the clinic (Distance, radio advertising, friend or family 2 recommended, prices, quality, doctor reputation, referral by a CSW or Community Agent)?

20 Please list the products that you sell and those that you give away for free 2

21 Please list the products that you buy from PSI 5

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 130

22 Please list the other public or private medical services in your area 3

23 Finally: if PSI was not supporting the clinic would it be able to continue providing the same services 5 at the same prices or what would you need to do and would you continue providing the same services?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 131

FGD Guide for Caregivers of Children under 5

Question/Information Required

00 Date of interview: ______/______/______

Name of FGD moderator______

Name of note taker______

Name of observer______

Thank you for making the time to talk with me today. The USAID/Madagascar has asked GH Pro to conduct an evaluation of the performance of the ISM project implemented by PSI. This evaluation is meant to serve a dual purpose: (1) to learn to what extent the project’s objectives and goals—at all result levels—have been achieved; and (2) to inform the design of a new social marketing project. We are meeting with four groups of young women including yours.

Our interview will take about one hour.

Before we go further, do you have any questions about this interview?

[ ] Consent provided ______[Interviewer/Recorder initials]

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 132

Do you have any questions before we begin?

May we continue with the interview?

[ ] Yes [ ] No

FGD session participants profile

Ages______

# Males ______

# Females______

Let’s begin by talking about the ISM Project.

01 Evaluation Question 1

1a Can you tell us what you know about the Healthy Family Campaign?

Probes: how did you hear about them? Have you interacted with peer educators (model mothers and model fathers)?

1b. Can you tell us what you know about products that can prevent or treat diarrhea among your children who are younger than 5 years old?

Probe: Sur Eau, Diarrhea Treatment Kit; how did you learn about them? Do you use them?

Probe: Piplan, Confiance, Proector, etc

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 133

1c. Can you tell us about your experience with bednets?

Probe: where did you get yours from? Did you pay (if so, how much)? How did you know to get one?

1d. Can you tell us about your experience the last time your child was sick with malaria?

Probe: did you get him/her tested? Did you give him/her treatment? Was it ACTs? Where did you get it from/go for care? After how many days?

02 The last time your child (under 5) was sick, what did you do?

(probe- seek medical care, obtain medicines, go to Top Reseau, talk with CHWs?)

03 2a What do you know about family planning? 1

Probe: short vs long term methods; how did you learn about them?

2b Which of these products have you purchased? How did you learn to purchase them? Where did you get them from?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 134

2c What do you know about the Top Reseau clinics?

Probe: used? Affordable? Quality? How did you hear about them?

04 Evaluation question 3 4

What would be most helpful to you to help you protect the health of your child?

05 Evaluation question 4

The information you have received via (radio, IPC, TV), was it easy for you to understand/relate to?

06 Evaluation question 5 4

5a Tell us about the cost of healthcare. How do you deal with paying when your child is sick? Has payment kept you from using healthcare in the past?

What about medicines you or your child may need. How do you deal with paying for them? Has payment kept you from buying medicines for your child in the past?

Thank you very much! If you think of anything else, please do not hesitate to get in touch with us.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 135

ANNEX III. PERSONS INTERVIEWED

PERSONS INTERVIEWED

ISM EVALUATION CONTACT Age Gender Title Email Phone number LIST Dr Phillipe Botralahy M Regional Dir- [email protected] +261 32 07 452 36 Antsiranana, PSI Dr. Didier M Responsable du Top 032 07 452 13 Reseau, PSI Antsiranana Dr. Marc Richard 42 M Espace Medicale - Top Reseau Diego 1 Chantal Coutiti F Grossite - PARC, 320227018 Diego 1, Jupiter Epicerie Fidy Lalao Andriamihaingo 47 M Superviseur [email protected] 032 07 155 22 Distribution PSI - Toliary I Rakotozafy Ravalohery 43 M Chauffeur de livraison [email protected] 032 07 455 26 (CL) PSI - Toliary I Rantosoa Raheva Christian 38 M Superviseur de Point [email protected] 032 07 455 54 de Distribution (SPD) PSI - Toliary I Razanalison Faly 48 M Coordinateur [email protected] 032 07 452 84 Régional de Franchise

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 136

ISM EVALUATION CONTACT Age Gender Title Email Phone number LIST Sociale PSI - Toliary I Rakotomalala Roger 42 M Superviseur de [email protected] 032 07 454 71 Communication PSI - Toliary I Andriamitantsoa Mamy Holiarimanga 45 F Coordinateur [email protected] 034 60 349 03 Régional USAID Mikolo Toliary I Rasolonirina Eymard 39 M Responsable en [email protected] 034 06 247 04 Mobilisation Communautaire (BR) USAID Mikolo - Toliary I Solondrainy Tsiarivelo Julie 34 F Superviseur [email protected] 034 70 488 51 Technicien d’Appui USAID Mikolo - Toliary I Nomenjanahary Marovavy 19 F Pair Educateur Jeune N/A 032 47 258 91 PSI - Toliary I Rasoanomenjanahary Hemamie 22 F Pair Educateur Jeune N/A : 034 63 904 51 Tendrisoa Lydia PSI - Toliary I Celestin Robena 21 M Pair Educateur Jeune [email protected] 034 27 200 98 PS I- Toliary I Razafison Tojoniriko Marcelin 22 M Pair Educateur Jeune N/A 032 43 028 88 Durando PSI - Toliary I

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 137

ISM EVALUATION CONTACT Age Gender Title Email Phone number LIST Andrianarivony Mialisoa Angela 20 F Pair Educateur Jeune N/A 034 92 866 60 PSI - Toliary I Rafanomezantsoa Sitraka 22 M Pair Educateur Jeune N/A 033 11 271 15 PSI - Toliary I Rasoanantenaina Bonette Clara 28 F Conseillère en N/A 033 05 259 17 Planification familiale (CPF) PSI - Toliary I Emile Prisca 25 F Conseillère en N/A 032 92 297 32 Planification familiale PSI - Toliary I Hirdjee Zeeshane 22 M PDG Tanora ZZ- [email protected] 032 11 410 41/ 034 11 410 41 Super grossiste Toliary I Razafinjatovo W. 59 M Gestionnaire Top [email protected] 032 02 837 86 Reseau Betesda Toliary I Raherinainasoa Elisah Micheline 30 F Sage femme Top N/A 032 60 820 45 Reseau Betesda Toliary I Ravelonirina Marthe Elodie 24 F Responsable [email protected] 033 43 790 26 Commerciale Radio Television Siteny (RTS) Toliary I Razafinjatovo Elisabeth Nirina 57 F Medecin Top Reseau [email protected] 034 84 342 30 Toliary I

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 138

ISM EVALUATION CONTACT Age Gender Title Email Phone number LIST Abdouulhhoussen M. 61 M Commercant - Super N/A 034 11 030 88 Grossiste/Parc Toliary II Rakotondramanana Fabien 40 M Adjoint technique / [email protected] 032 44 347 59 Service de Sante de District Toliary II Hotahiene Raphael 46 M Medecin Inspecteur [email protected] 032 05 616 48 Tulear II Manampiheriniony Rich 44 F Adjoint technique / [email protected] 033 12 717 51 Service de Sante de District Toliary I Razafindraibe Emile 52 M Medecin Chef Salfa [email protected] 034 67 982 86 Rasoarinindriana Lalao Voangy 54 F Medecin Salfa [email protected] 034 60 129 34 Raboanarison Rado M Receptioniste [email protected] 034 84 084 48

Simone Arizemo 43 F AC Mere Toliary II Daniel Francois Tsimavay 36 M AC Enfant Toliary II 032 61 119 62 Pascal Sebastien 54 M AC Enfant Toliary II 032 42 040 44 Ratovonirina Jolien 53 M AC Enfant Toliary II 032 40 580 82 Tsivehanarisoa 54 F AC Mere Toliary II Aliriny 45 F AC Enfant Toliary II Famoreta Auguste 50 M AC Mere Toliary II 032 28 133 42 Euphrasie Nomeny 33 F PA Ankililoake Toliary 032 59 644 34

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 139

ISM EVALUATION CONTACT Age Gender Title Email Phone number LIST II Raharinantenaina Juliolila F. A. 42 F PA Tsianisiha Toliary 033 28 133 42 II Theordin 23 M FGD Male youth - 033 45 513 12 Toliary I Manjaka 20 M FGD Male youth - Toliary I Alfreudial 24 M FGD Male youth - 032 84 944 17 Toliary I Harson 21 M FGD Male youth - Toliary I Stanislas 23 M FGD Male youth - 032 97 210 12 Toliary I Boumedien 25 M FGD Male youth - 03497 646 95 / 032 53 405 06 Toliary I Votso Draz Floratin 25 M FGD Male youth - 033 17 296 49 Toliary I Andranto Jean Thierry 22 M FGD Male youth - 032 63 636 00 Toliary I Josuvah Sarela Paul Prudhomme 21 M FGD Male youth - Toliary I Alfredelin 16 M FGD Male youth - 034 63 675 55 Toliary I Jenny Arof 18 F FGD female youth - 034 59 713 27 Toliary I

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 140

ISM EVALUATION CONTACT Age Gender Title Email Phone number LIST Santatra 16 F FGD female youth - 032 88 232 99 Toliary I Haingo Nirina 17 F FGD female youth - 034 68 287 10 Toliary I Rahenintsoa Evah 16 F FGD female youth - 034 87 536 62 Toliary I Faniloniaina Gaetta 18 F FGD female youth - 034 19 656 22 Toliary I Rosia 18 F FGD female youth - Toliary I Tacia 15 F FGD female youth - Toliary I Faustinah 18 F FGD female youth - Toliary I Nona Bertha Fandraina Sandersen 36 M CC Vorehe Toliary II 034 40 00194 Tenoavintsoa Fandahara 21 M CC Vorehe Toliary II Heliarisoa Rosina 20 F JPE Vorehe Toliary II Fillette 26 F JPE Vorehe Toliary II Rasoanantenaina Sanderlesse 15 F JPE Vorehe Toliary II Vaviniaina Brigettine 15 F JPE Vorehe Toliary II Fomenjanahary Marcellia 15 F JPE Vorehe Toliary II Central Rekiky 22 F JPE Vorehe Toliary II Hariniaina Marie Elisa 20 F FGD female youth - Vorehe -Toliary II

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 141

ISM EVALUATION CONTACT Age Gender Title Email Phone number LIST Filasoa Honoriny 16 F FGD female youth - Vorehe -Toliary II Raherinandraina Genevieve F FGD female youth - Vorehe -Toliary II Claire 23 F FGD female youth - Vorehe -Toliary II Soary 15 F FGD female youth - Vorehe -Toliary II Perline Faravavy Justine 25 F FGD female youth - Vorehe -Toliary II Zafy Pikhorin 18 M FGD male youth- Vorehe- Toliary II Ruberthon 17 M FGD male youth- Vorehe- Toliary II Retiaro 18 M FGD male youth- Vorehe- Toliary II Babay 20 M FGD male youth- Vorehe- Toliary II Sanderlinot 20 M FGD male youth- Vorehe- Toliary II Fernand 18 M FGD male youth- Vorehe- Toliary II Mbohiteny 17 M FGD male youth- Vorehe- Toliary II Viola 21 M FGD male youth- Vorehe- Toliary II

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 142

ISM EVALUATION CONTACT Age Gender Title Email Phone number LIST Fabrice 17 M FGD male youth- Vorehe- Toliary II Fandrana Sanderlin 44 M Motivateur TB - 034 97 535 44 Accountant SALFA Rafindranely Simonette 24 F Sage femme Top [email protected] 034 98 209 99 Reseau Salfa - Vorehe II Perline 22 F FGD Female parents- 034 21 901 30 Vorehe II- Toliary II Felicie 42 F FGD Female parents- Vorehe II- Toliary II Olga 28 F FGD Female parents- Vorehe II- Toliary II Renamo 35 F FGD Female parents- Vorehe II- Toliary II Clodia 27 F FGD Female parents- Vorehe II- Toliary II Rene 44 F FGD Female parents- Vorehe II- Toliary II Randria Bekongo 32 M FGD Male parents- Vorehe- Toliary II Solotiana 32 M FGD Male parents- Vorehe- Toliary II Retata 28 M FGD Male parents- Vorehe- Toliary II Rabonavison Tsitindry 30 M FGD Male parents-

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 143

ISM EVALUATION CONTACT Age Gender Title Email Phone number LIST Vorehe- Toliary II Ramanantsoa Heritiana Sandersen 31 M FGD Male parents- Vorehe- Toliary II Randrianiavison 30 M FGD Male parents- Vorehe- Toliary II Fnadahara 40 M FGD Male parents- Vorehe- Toliary II Sandra 38 F PA Belalanda Toliary 032 61 719 03/ 034 29 466 82 II Jean Francois 48 M CHW Enfant Toliary 032 92 141 27 II Hermine Janette 44 F CHW Mere Toliary II 033 41 258 60 Hubert Rozea 49 M CHW Mere Toliary II 032 56 650 95 Madalena 54 F CHW Toliary II 034 28 775 91 Manesa Marcel 61 M CHW Enfant Toliary 033 17 751 85 II Kaila Ramualdo 39 M Pharmacie Sanfily - 020 94 427 75 Toliary I Eugene Lahy 48 M Pharmacie Tsodrano - 032 58 792 73 Betania Rajaonarivony Miora Henintsoa F Formateur Promoteur 032 07 455 83 en Sante Razanadrasoa Christiana 15 F School student/youth Tiamjara Lydie 15 F School student/youth Ravola Josline 15 F School student/youth

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 144

ISM EVALUATION CONTACT Age Gender Title Email Phone number LIST Vololonirina Nazia 15 F School student/youth Salima Mohamed Said 15 F School student/youth Tiana Hellene 15 F School student/youth Nathalia 15 F School student/youth Christine 17 F School student/youth Temina Fredine F School student/youth Isra Helene F School student/youth Velo Charlette F School student/youth Rosoanirina Daniella F School student/youth Be Elia Catherina F School student/youth Zaimak F School student/youth Ravococonirinis Philibertine F School student/youth Ramariamanama Dally F School student/youth Amzidy Djamaldine M School student/youth Jaomisy Bemiteralandriel M School student/youth Bajio M School student/youth Armad Jaosany M School student/youth Noelison M School student/youth Abdo M School student/youth Amad Soileimana M School student/youth Fahardine M School student/youth

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 145

ISM EVALUATION CONTACT Age Gender Title Email Phone number LIST Joasany Angeraldo M School student/youth Henry Jaosany-Laurant M School student/youth Velozara Machelin M School student/youth Mariamo Caroline 35 F CHW 032 652 7931 Soavola Marie Prisca 25 F CHW 034 79 84554 Soa Marigaiva Clementine 22 F CHW [email protected] 032 25 13009

Volasua Nadege Paul 20 F CHW 032 7 645 668 R. Honrve Marthanas 23 F CHW [email protected] 032 56 96575

Romaric A 24 F CHW [email protected] 032 81 858 08

Ahmada Toilibo 39 M CHW 032 63 67843 Veronique 31 F CHW 032 22 295 62 Athena 29 F CHW 032 59 21113 Nadia 36 F CHW 032 51 08866 Elcime Soamany 38 F CHW [email protected] 032 0201367

Rose Christine Maria 27 F CHW [email protected] 032 76 75407 Sylvain M CHW Felix M CHW Jaolety M CHW Amaid Avilignitry M CHW

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 146

ISM EVALUATION CONTACT Age Gender Title Email Phone number LIST Miarana Richard M CHW Ampizara M CHW Retsizafe M CHW Tolizara Jaomamy M CHW Raymond Jean M CHW Abdoul Anzizi M CHW Mr Yvon Ratefison 42 M Supervisor de Distr, [email protected] 032 07 45461; PSI, Diego 1 Dr. Joamdalana Mboty Tina Fabricia 36 F FISA Director 032 40 530 56 Ramiarisoa Suzette 41 F CHW 032 5391859, Rasoarizamamy Victoire 52 F CHW 032 59 08847, Mana 52 F CHW 032 27 199 26, Dr Leticia Lydia Yasmine 39 F District Medical 032 40 71812, Officer Dr Lucien 44 M Technical Assistant 032 07 67959, Distr Med Officer Ramarozatovo Andry 40 M Comercial Distributer [email protected] 034 05 376 90, PANDA Dr. Mamy Rachelle Gustavie Rosalie 32 F Top Reseau [email protected] 032 44 06605, Coordinator, Ambilobe Dr. Behajaina Benarson Jeanne 57 F Assistant District Med [email protected] 034 37 93264, Officer Finoana Samizamdry 54 F DMOffice, MCH 034 11 30369; 032 55 77814,

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 147

ISM EVALUATION CONTACT Age Gender Title Email Phone number LIST director Rakotomalala Bruno 54 M DMOffice, HIV/STI [email protected] 032 62 17329; 034 766 00113,

Ravaooarisoa Estrine 53 F Mother No email No telephone Be Arthur Romain 36 M Mother No email 032 28 36806 Nadine Vianne 42 F Mother No email No telephone Vognymary Antanimiavotro 42 F Mother No email No telephone Edwige Mantaly 30 F Mother No email No telephone Raissa Mantaly 21 F Mother No email No telephone Noeline 25 F Mother No email No telephone Edwige 30 F Mother No email No telephone Dalissia 16 F mother No email No telephone Esmeralda 16 F mother No email No telephone Estelle 30 F mother No email No telephone Anjara 24 F mother No email No telephone Angele 45 F mother No email No telephone Dr Ramihantaniarivo Herlyne F Director General [email protected] 034 05 517 91 or 034 36 606 MOH 72 Dr Ravoniarisoa Marie Georgette F Directeur de la Sante [email protected] 032 04 013 46 Familiale MOH, Institut d'Hygiene Sociale

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 148

ISM EVALUATION CONTACT Age Gender Title Email Phone number LIST Randrema Harimanga Or Hamelo M Secretaire General or [email protected] 22 469 24/5/6 Solthi Joli Directeur General, MEAH Ambohijatovo Daniele Nyirandutiye F HPN Director, [email protected] 034 07 422 92 USAID John Yanulis M COP, MIKOLO [email protected] 032 69 550 29 Tantely Rajaobelina M Supply Chain [email protected] 034 07 428 23 Specialist, USAID Joss Razafindrakoto M Infectious Diseases [email protected] 034 07 428 24 Spec, USAID Chuanpit Chua-Oon COP, MAHEFA [email protected] 034 49 790 01 166 Top Reseau Clinics 180 Household Survey Participants

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 149

ANNEX IV. DATA COLLECTION INSTRUMENTS

The following data collection instruments were reviewed and approved by USAID prior to data collection:

1. Key Informant Interviews with PSI ISM Staff

2. Households with children under five, youth age 15 to 24 and women age 15 to 49 3. Telephone interview with Top Réseau members 4. Exit interview with Top Réseau clinic users 5. FGD with female parents age 15-49 of children under five 6. FGD with male parents age 15-49 of children under five 7. Focus Group Discussion Guide for Male Youth 15 to 24 8. Focus Group Discussion Guide for Community Health Workers 9. Focus Group Discussion Guide for Female Youth 15 to 24 10. Key Informant Interviews with distributers 11. KII social marketing personnel and sales force 12. KII with local radio station managers 13. KII with Top Réseau clinic staff 14. Top Réseau Site Visit Guide

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 150

1. KEY INFORMANT INTERVIEWS WITH PSI ISM STAFF

Question/Information Required

00 Date of interview: ______/______/______

[ ] PSI

[ ] PSI partner ______

[ ] USAID

[ ] Other USAID partner

[ ] GoM

[ ] Other GoM partner

Which ISM component are you responsible for?

[ ] SBCC [ ] Private service delivery [ ] Social Marketing [ ] Not an ISM implementer

Thank you for making the time to talk with me today.

USAID/Madagascar has asked GH Pro to conduct an evaluation of the performance of the ISM project implemented by PSI. This evaluation is meant to serve a dual purpose: (1) to learn to what extent the project’s objectives and goals—at all result levels—have been achieved; and (2) to inform the design of a new social marketing project.

You were suggested as a key person to inform this activity and we greatly appreciate your perspective, experiences and views.

integration of program results and the benefits/disadvantages of such integration; 3) the ISM project’s definition/operationalization of “underserved” and “rural” as it relates to the project’s target population; 4) documentation, dissemination, and application of program learning; and 5) social and economic sustainability of ISM activities.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 151

Before we begin, I want to let you know that any information or examples we gather during this interview process will not be attributed to any specific person or institution, unless you tell us that you would be willing to have your responses either quoted by you in the report, or otherwise attributed to you. You are also free to not respond to any of our questions or stop the interview at any time.

Our interview will take about one hour.

Before we go further, do you have any questions about this interview?

[ ] Consent provided ______[Interviewer/Recorder initials]

Do you have any questions before we begin?

May we continue with the interview?

[ ] Yes [ ] No

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 152

Let’s begin by talking about the ISM Project.

01 What is the objective of the ISM project? (probe – what services does ISM offer/support, to whom, and how? How do you work with Top Réseau? CHWs? Supply chain and distribution actors?)

What activities does ISM implement? [PSI ONLY]

Can you talk about resource allocation – do you feel it is balanced well across all of these activities? [PSI ONLY]

How has this balance or imbalance affected your results? [PSI ONLY]

How do ISM partners interact to deliver on the objectives of the project? What are the strengths and weaknesses of this partnership? [PSI ONLY]

Of the three components (SBCC, social marketing and private services), what was/is the most successful? Why?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 153

What was the least successful? Why?

[IF SBCC not mentioned]: Can you tell me a bit about your work with SBCC? [PSI ONLY]

- Are you reaching your targets?

- What has worked well?

- What has not worked well? Why?

What are some of the challenges ISM has experienced?

Will these challenges keep ISM from reaching its targets by the end of 2017?

02 Can you tell us a bit about the method you have used to set your targets? (Baseline data used? 1 Sources? Etc.) [PSI AND ISM PARTNERS ONLY]

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 154

At which point do you involve the Mission? [PSI ONLY]

How likely are you to achieve your targets?

Why?

Why not?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 155

03 Let’s talk about integration within the ISM project. In implementing ISM, can you tell us how you 2 linked SBCC with service delivery and commodities?

(from planning to implementation; improvement processes; review of results)

Examples?

What does it take to create these linkages? Maintain them?

Can you give examples of where linking these three components has led to synergies?

Are there reasons you think it would be better to focus on only one or two? (If so, which one, or two?)

Overall, what proportion of resources went to each of the three components:

In terms of money:

______SBCC ______Private healthcare ______commodities

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 156

In terms of time:

______SBCC ______Private healthcare ______commodities

04 For the ISM target population (women of reproductive age, children under five, youth age 15 to 24), 3 how were "underserved" and "rural" operationally defined?

(probe: market-driven vs. wealth quintiles (lower wealth))

(probe: explore definition of urban, peri-urban, rural)

What data sources did you use, if any?

Did program activities reach these populations? (document proof)

Why or why not?

Are you likely to reach your targets for [each target group] by December 2017?

How do you know?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 157

Why?

Why not?

07 We understand ISM has done a significant amount of research. What have they researched? Can 4 you please describe your research process, from start to end?

(probe: who initiates, who conducts, who analyzes, who presents, who benefits? Involvement of USAID?)

08 Can you tell us about your use of the research data?

09 What do you do to disseminate your research findings? 4

Is this applied to every research project? If not, which ones does it apply to? Are there any exceptions?

10 Outside of the ISM project, is the ISM data being used? For what? By whom? 4

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 158

What has been ISM’s role in promoting this use?

11 On a scale of 1-10, with 1 being not at all and 10 being completely, how prepared do you 5 think the Government and the private sector are to continue to increase commercially available health products and services without ISM or USAID support?

Family planning

Pilplan ______

Confiance ______

Yes youth condoms ______

Protector Plus ______

Feeling condom ______

Sayana Press ______

Norlevo ______

Implanon ______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 159

Jadelle ______

IUDs (Copper T) ______

IUDs (Zarin) ______

Chlomexidine ______

Rojo-branded cycle beads ______

Malaria

LLINs ______

ACTs ______

RDTs ______

ITP ______

Child Survival

ViaSur (or equivalent) ______

HydraZinc (DTK) ______

Sur'Eau Pilina Tablets ______

Pneumox ______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 160

Zaza Tamady-branded micronutrient powder (MNP) ______

12 Are all of these products legally registered in Madagascar?

13 Who owns the brands/trademarks (ownership)?

14 On a scale of 1-10, how much capacity would you say the ISM project's distribution partners have to advocate for socially marketed products?

______

Please explain:

15 On a scale of 1-10, do the ISM project's distribution partners have the physical infrastructure for socially marketed products?

Space ______

Equipment ______

Transport ______

Access to relevant information ______

16 On a scale of 1 to 10, please rate the extent to which you believe the ISM project's distribution partners have adequate staff to implement ISM activities:

______

Please explain:

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 161

16 On a scale of 1-10, do the ISM project's distribution partners have the organizational infrastructure for socially marketed products?

Clear internal roles and responsibilities ______

Marketing capacity ______

Internal QA/CQI processes that are operational ______

Supply Chain Management ______

Financial management capacity ______

Overall, on a scale of 1 to 10, how strong would you say the ISM project's distribution partners organizational capacity is for social marketing? ______

17 On a scale of 1-10, what is the level of trust that the distributors/retailers have from their consumers?

______

18 5

On a scale of 1-10, with 1 being not at all and 10 being completely, do you think that Top Réseau, overall:

Shares the same understanding with the ISM project regarding its role in promoting health equity by providing high quality care to the poor as well as other clientele?

______

Takes full ownership and responsibility for ensuring the financial viability of the e-voucher (or other subsidy) system?

______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 162

Has a policy environment that supports its role as a provider of equitable services to the poor as well as other clientele?

______

Has the capacity to advocate for additional resources to support/maintain its role as an equitable and high-quality private service provider?

______

In particular, what is Top Réseau's advocacy capacity to solicit additional support or to create public-private partnerships that may benefit its catchment area?

Has sufficient staff numbers, types and capacity to ensure that it can successfully serve as a private provider of high-quality services?

Numbers: ______

Types: ______

Capacity: ______

Are Top Réseau clinics' staffing levels in general sufficient to manage the delivery of the necessary high-quality primary healthcare services in the communities they serve?

Yes/No: ______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 163

Why/why not?

What changes to staffing are needed/have been completed recently to support service delivery?

What are the plans and processes for continuous recruitment, education, orientation, retention, and evaluation?

Has sufficient access been given to the necessary technology and supplies to facilitate these clinics' role as high-quality primary healthcare providers in their catchment area?

______

What technology and supplies do they need that they do not currently have access to?

- What access do they have to computers, electricity, hardware/software, Internet? Who pays?

- What is their access to medicines, and who pays?

- What is their access to laboratory tests, and who pays?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 164

Do Top Réseau clinics have adequate financial resources outside of the ISM project for staffing, Internet, training and other resources to provide high-quality primary healthcare services in their catchment area?

______

Do they have the capacity to attract additional paying clients if/when necessary?

______

Please elaborate on their marketing strategy/strategies for obtaining new clients and retaining old clients:

Do they have mechanisms for recognizing and rewarding good performance within their team?

______

Do they have access to the medicines and supplies they need to provide high-quality primary healthcare to underserved and rural populations?

______

On a scale of 1 to 10, to what extent will Top Réseau clinics have the organizational infrastructure to continue to implement ISM activities in the absence of the ISM program?

Documented mechanisms for continuing education ______

Documented process for staff recruitment ______and retention ______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 165

Well-defined staff roles and responsibilities ______

Marketing capacity ______

Referral and counter-referral with larger (public or private) facilities to transfer more complicated cases ______

Internal QA/CQI processes that are operational ______

Routine data collection, analysis and reporting ______

Financial management capacity ______

On a scale of 1 (not at all) to 10 (a lot), please rate the extent to which Top Réseau clinics have positive media coverage at the community, district and provincial levels:

______

On a scale of 1(completely negative) to 10 (completely positive), please rate the communities' perception of Top Réseau clinics:

______

Does the public understand the concept of Top Réseau?

Is the public supportive of Top Réseau?

In your opinion, who does the public rely on as a credible source of expertise and credible health information?

[ ] Top Réseau

[ ] CHWs

[ ] Peer models

[ ] Radio

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 166

[ ] TV

[ ] Newspaper

[ ] Friends

[ ] Family

[ ] Other

What is the level of confidence in Top Réseau clinic services in their catchment area?

______

Why do you think this perception is accurate/not accurate?

Urban:

Rural:

19 Overall, on a scale of 1 to 10, how financially capable are Top Réseau clinics of to continue to provide services at lower or no cost to the poor without support from the ISM project or USAID?

______

Please detail what the ISM project funds for Top Réseau clinics:

______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 167

Of the 257 Top Réseau clinics, what proportion are likely to continue without additional support from the ISM team?

What distinguishes these from the other Top Réseau clinics?

Where are they located?

20 Do you have a close-out plan?

If yes, may we have a copy?

21 How will you/have you factor(ed) in sustainability in your close-out plan?

22 How would you describe ISM’s added value in Madagascar to: 4

The private clinic scene:

The essential commodity supply chain:

Community Health Workers:

Women of reproductive age?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 168

Youth:

- Males

- Females

Children under five and their mothers?

The Ministry of Health:

The Ministry of the Population, Social Protection and Promotion of Women:

Other USAID partners:

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 169

33 Now that you have lived through the implementation of ISM, what would you have done differently gen if you could?

34 How could the program design be adapted to improve sustainability?

35 In an ideal world, what do you think should be the main focus of a follow-on project? gen

Would you include all of these components together again?

If you had to remove at least one, which one would it be?

Would you add others?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 170

36 Do you have any other thoughts you would like to share?

Thank you very much! If you think of anything else, please do not hesitate to get in touch with us.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 171

2. Quantitative Survey Questionnaire Households with children under five, youth age 15 to 24 and women age 15 to 49 Hello. My name is ______and I work in the health sector in Madagascar. We are conducting a survey to improve community health services as part of a PSI project. You have been selected to be interviewed for this survey and we would be very grateful for your participation. The survey usually takes around 30 minutes. Your participation is voluntary and you can stop the interview at any time, or decide not to answer a particular question. Your answers will remain confidential. May we continue with the interview?

0 = No If no, stop here.

1 = Yes If yes, start the interview. /__/ CRITERIA

A. Do you have any children under five? YES NO B. Does a woman age 15 to 49 live in your house? YES NO C. Does a youth age 15 to 24 live in your house? YES NO If the participant answers NO to all three questions, thank him or her and END the interview. If the participant answers YES to any of the questions, continue the interview.

IDENTIFICATION

DISTRICT: /__/ ______/__/ COMMUNE: ______

FOKONTANY: /__/__/ ______

INTERVIEW

INTERVIEWER: ______/__ /__ /

DATE OF INTERVIEW (day/month/year) /__/__/ /__/__/ /__/__/

NAME OF INTERVIEWEE: ______/__/__/

OTHER INFORMATION

START OF INTERVIEW (time): /__/__/ : /__/__/

END OF INTERVIEW (time): /__/__/ : /__/__/

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 172

A. GENERAL CHARACTERISTICS

No. QUESTIONS AND FILTERS CODING OF RESPONSES SKIP TO How old are you? /__/__/ years old

Gender ⬜ 1. Male

⬜ 2. Female

What is your marital status? ⬜ 1. Married

⬜ 2. Divorced

⬜ 3. Separated

⬜ 4. Single

⬜ 5. Widowed

Do you know how to read and/or write? ⬜ 1. Yes If no, skip to

⬜ 2. No

What is the highest level of education ⬜ 1. Never went to school

you have completed? ⬜ 2. Primary

⬜ 3. Secondary

⬜ 4. University

⬜ 5. Functional literacy

How many people permanently live in /__/__/ your home?

How many children live in your home? /__/__/ How many boys? /__/__/ How many girls? /__/__/ How many children are under five? /__/__/

What is your primary occupation? ⬜ 0. Unemployed

⬜ 1. Farming

PRIMARY OCCUPATION – CHOOSE ONE. ⬜ 2. Livestock

⬜ 3. Fishing

⬜ 4. Trade

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 173

⬜ 5. Arts and crafts

⬜ 6. Day labor

⬜ 7. Mining and gems

⬜ 8. Salaried employee

⬜ 9. Freelance work (taxi driver, domestic worker, mason, etc.)

⬜ 99. Other (Specify)______

Is your spouse employed? ⬜ Yes

⬜ No

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 174

B. COMMUNICATION

No. QUESTIONS AND FILTERS CODING OF RESPONSES SKIP TO

List the following promotion campaigns and check the sources according to participant responses:

Mother and child health promotion ⬜ Yes, I have heard of it. campaign

Sources (where)?

⬜ a. Radio

⬜ b. Television

⬜ c. Friend

⬜ d. Relative

⬜ e. Public CSB (Basic Health Center)

⬜ f. Private clinic (name: ______)

⬜ g. Top Réseau

⬜ h. Mobile Video

⬜ i. Pharmacy

⬜ j. Community Health Worker

⬜ k. Other (Specify) ______

⬜ l. No response

Promotion of LLINs ⬜ Yes, I have heard of it.

Sources (where)?

⬜ a. Radio

⬜ b. Television

⬜ c. Friend

⬜ d. Relative

⬜ e. Public CSB (Basic Health Center)

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 175

No. QUESTIONS AND FILTERS CODING OF RESPONSES SKIP TO ⬜ f. Private clinic (name: ______)

⬜ g. Top Réseau

⬜ h. Mobile Video

⬜ i. Pharmacy

⬜ j. Community Health Worker

⬜ k. Other (Specify) ______

⬜ l. No response

Malaria treatment (ACTs, RDTs, ITP) ⬜ Yes, I have heard of it.

Sources (where)?

⬜ a. Radio

⬜ b. Television

⬜ c. Friend

⬜ d. Relative

⬜ e. Public CSB (Basic Health Center)

⬜ f. Private clinic (name: ______)

⬜ g. Top Réseau

⬜ h. Mobile Video

⬜ i. Pharmacy

⬜ j. Community Health Worker

⬜ k. Other (Specify) ______

⬜ l. No response

Yes, I have heard of it. Water treatment products (Sur'Eau in solution and tablet form)

Sources (where)?

⬜ a. Radio

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 176

No. QUESTIONS AND FILTERS CODING OF RESPONSES SKIP TO ⬜ b. Television

⬜ c. Friend

⬜ d. Relative

⬜ e. Public CSB (Basic Health Center)

⬜ f. Private clinic (name: ______)

⬜ g. Top Réseau

⬜ h. Mobile Video

⬜ i. Pharmacy

⬜ j. Community Health Worker

⬜ k. Other (Specify) ______

⬜ l. No response

Diarrhea treatment products (Zinc, ORS, ⬜ Yes, I have heard of it. ViaSur)

Sources (where)?

⬜ a. Radio

⬜ b. Television

⬜ c. Friend

⬜ d. Relative

⬜ e. Public CSB (Basic Health Center)

⬜ f. Private clinic (name: ______)

⬜ g. Top Réseau

⬜ h. Mobile Video

⬜ i. Pharmacy

⬜ j. Community Health Worker

⬜ k. Other (Specify) ______

⬜ l. No response

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 177

No. QUESTIONS AND FILTERS CODING OF RESPONSES SKIP TO Yes, I have heard of it. Promotion of family planning products (birth control pill, condoms)

Sources (where)?

⬜ a. Radio

⬜ b. Television

⬜ c. Friend

⬜ d. Relative

⬜ e. Public CSB (Basic Health Center)

⬜ f. Private clinic (name: ______)

⬜ g. Top Réseau

⬜ h. Mobile Video

⬜ i. Pharmacy

⬜ j. Community Health Worker

⬜ k. Other (Specify) ______

⬜ l. No response

Yes, I have heard of it. Promotion of family planning products (injectables and implants)

Sources (where)?

⬜ a. Radio

⬜ b. Television

⬜ c. Friend

⬜ d. Relative

⬜ e. Public CSB (Basic Health Center)

⬜ f. Private clinic (name: ______)

⬜ g. Top Réseau

⬜ h. Mobile Video

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 178

No. QUESTIONS AND FILTERS CODING OF RESPONSES SKIP TO ⬜ i. Pharmacy

⬜ j. Community Health Worker

⬜ k. Other (Specify) ______

⬜ l. No response

Yes, I have heard of it. Promotion of Top Réseau clinics

Sources (where)?

⬜ a. Radio

⬜ b. Television

⬜ c. Friend

⬜ d. Relative

⬜ e. Public CSB (Basic Health Center)

⬜ f. Private clinic (name: ______)

⬜ g. Top Réseau

⬜ h. Mobile Video

⬜ i. Pharmacy

⬜ j. Community Health Worker

⬜ k. Other (Specify) ______

⬜ l. No response

Rate the effectiveness of the following methods for communicating with people from 1 to 10:

National radio ads 1 2 3 4 5 6 7 8 9 10

Radio stories about family health 1 2 3 4 5 6 7 8 9 10

Local radio 1 2 3 4 5 6 7 8 9 10

Mobile video 1 2 3 4 5 6 7 8 9 10

Counseling at a Top Réseau clinic or 1 2 3 4 5 6 7 8 9 10 elsewhere

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 179

No. QUESTIONS AND FILTERS CODING OF RESPONSES SKIP TO Interpersonal communication with 1 2 3 4 5 6 7 8 9 10 Community Health Workers

Rate the quality of the following products and services from 1 to 10:

Long-lasting insecticidal nets 1 2 3 4 5 6 7 8 9 10

Sur'Eau in solution or tablet form 1 2 3 4 5 6 7 8 9 10

ViaSur diarrhea treatment products 1 2 3 4 5 6 7 8 9 10

PilPlan 1 2 3 4 5 6 7 8 9 10

Confiance, Implants 1 2 3 4 5 6 7 8 9 10

PneumoStop 1 2 3 4 5 6 7 8 9 10

ACT malaria treatment product 1 2 3 4 5 6 7 8 9 10

Top Réseau clinic 1 2 3 4 5 6 7 8 9 10

Rate access to the following products and services from 1 to 10:

Long-lasting insecticidal nets 1 2 3 4 5 6 7 8 9 10

Sur'Eau in solution or tablet form 1 2 3 4 5 6 7 8 9 10

ViaSur diarrhea treatment products 1 2 3 4 5 6 7 8 9 10

PilPlan 1 2 3 4 5 6 7 8 9 10

Confiance, Implants 1 2 3 4 5 6 7 8 9 10

PneumoStop 1 2 3 4 5 6 7 8 9 10

ACT malaria treatment product 1 2 3 4 5 6 7 8 9 10

Top Réseau clinic 1 2 3 4 5 6 7 8 9 10

Rate the affordability of the following products and services from 1 to 10:

Long-lasting insecticidal nets 1 2 3 4 5 6 7 8 9 10

Sur'Eau in solution or tablet form 1 2 3 4 5 6 7 8 9 10

ViaSur diarrhea treatment products 1 2 3 4 5 6 7 8 9 10

PilPlan, Condoms 1 2 3 4 5 6 7 8 9 10

Confiance, Implants 1 2 3 4 5 6 7 8 9 10

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 180

No. QUESTIONS AND FILTERS CODING OF RESPONSES SKIP TO PneumoStop 1 2 3 4 5 6 7 8 9 10

ACT malaria treatment product 1 2 3 4 5 6 7 8 9 10

Top Réseau clinic 1 2 3 4 5 6 7 8 9 10

What are the greatest barriers to ⬜ a. Never heard of them obtaining these socially marketed products and services? ⬜ b. Not easy to access

⬜ c. Too expensive

d. Sometimes out of stock

Rate your agreement with the following statements from 1 to 10:

I've never heard of Top Réseau. 1 2 3 4 5 6 7 8 9 10

I've heard of Top Réseau but I've never 1 2 3 4 5 6 7 8 9 10 used its services.

I've never gone because I live too far 1 2 3 4 5 6 7 8 9 10 away.

I've never gone because it's too expensive. 1 2 3 4 5 6 7 8 9 10

I've used these services because they 1 2 3 4 5 6 7 8 9 10 meet my needs.

I've used these services because I think 1 2 3 4 5 6 7 8 9 10 they're affordable.

I've used family planning services. 1 2 3 4 5 6 7 8 9 10

I've used health services for children 1 2 3 4 5 6 7 8 9 10 under five.

I would recommend Top Réseau clinics to 1 2 3 4 5 6 7 8 9 10 others.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 181

C. SERVICES

No. QUESTIONS AND FILTERS CODING OF RESPONSES SKIP TO Have you ever gone to a Top ⬜ Yes If no, skip Réseau clinic? to ⬜ No

If yes, when was the last time you went ⬜ 1. Less than 2 weeks ago to a Top Réseau clinic? ⬜ 2. 2 weeks to 1 month ago

⬜ 3. 1 to 3 months ago

⬜ 4. 3 to 6 months ago

⬜ 5. More than 6 months ago

⬜ 6. Never

⬜ 8. Other (Specify) ______

⬜9. No response

Why did you go to a Top Réseau clinic? ⬜ 1. To take a sick child

Age: ⬜ Less than 1 month old

1 to 12 months old

1 to 5 years old

⬜ 2. I was sick

⬜ 3. I was pregnant

⬜ 4. I went for a checkup

⬜ 5. For family planning

⬜ 6. Youth health services

⬜ 7. Lab or medical imaging services

⬜ 8. Other (Specify) ______

⬜ 9. No response

How did you find out about the Top ⬜ a. Radio Réseau clinic? ⬜ b. Television

⬜ c. Friend

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 182

No. QUESTIONS AND FILTERS CODING OF RESPONSES SKIP TO ⬜ d. Relative

⬜ e. Public CSB (Basic Health Center)

⬜ f. Private clinic (name: ______)

⬜ g. Top Réseau

⬜ h. Mobile Video

⬜ i. Pharmacy

⬜ j. Community Health Worker

⬜ k. Peer educator

⬜ l. Road sign

⬜ m. Other (Specify) ______

⬜ n. No response

Did you buy any products at the Top ⬜ Yes Réseau clinic? ⬜ No

If yes, what? ______

How did you find the consultation fee? ⬜ 1. Expensive

⬜ 2. Acceptable

⬜ 3. Inexpensive

Would you go back to the clinic again? ⬜ Yes

⬜ No

Rate your satisfaction with the quality 1 2 3 4 5 6 7 8 9 10 of service you received at your last visit to the clinic, from 1 to 10:

QUESTIONS FOR WOMEN AGE 15 to 49

Do you use family planning? ⬜ 1. YES If no, skip the next ⬜ 2. NO question

If yes, what method of contraception do ______you use?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 183

No. QUESTIONS AND FILTERS CODING OF RESPONSES SKIP TO Where do you normally get ⬜ 1. Public CSB (Basic Health Center) contraceptives? ⬜ 2. Private clinic (name: ______)

⬜ 3. Top Réseau

⬜ 4. Pharmacy

⬜ 5. Community Health Worker

⬜ 6. Other (Specify) ______

⬜ 7. No response

How much did you pay at the visit?

For the consultation ______Ar

For medications ______Ar

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 184

D. SOCIAL MARKETING

No. QUESTIONS AND FILTERS CODING OF RESPONSES SKIP TO Have you ever gone to a Top Réseau clinic? ⬜ 1. YES If no, skip to ⬜ 2. NO

If yes, when? ⬜ 1. This month

⬜ 2. Three months ago

⬜ 3. Six months ago

⬜ 4. More than one year ago

⬜ 5. Never

⬜ 8. Other (Specify) ______

⬜ 9. No response

Why did you go? ⬜ 1. Women's health/family planning

⬜ 2. Children's health/ IMCI

⬜ 3. Youth health

⬜ 4. Lab or medical imaging services

⬜ 5. Other (Specify) ______

⬜ 6. No response

How much did you pay at the visit? ⬜ 1. For the consultation ______

⬜ 2. For medications ______

⬜ 3. Don't remember

How satisfied were you with the services ⬜ 1. Very satisfied provided? ⬜ 2. Satisfied

⬜ 3. Somewhat satisfied

⬜ 4. Not satisfied

If no, why haven't you gone to a Top ⬜ 1. It's for young people Réseau clinic? ⬜ 2. It's too expensive

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 185

No. QUESTIONS AND FILTERS CODING OF RESPONSES SKIP TO ⬜3. Habit

⬜4. I didn't know it existed

⬜ 5. Other (Specify) ______

⬜ 6. No response

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 186

E. PREVENTION PRACTICES Look around the house to see if the household uses or does the following:

No. QUESTIONS AND FILTERS CODING OF RESPONSES SKIP TO What water source do you use? ⬜ 1. Faucet

⬜ 2. Surface water (river)

⬜ 3. Gravity system

⬜ 4. Well

What do you do to purify water before ⬜ 1. Nothing drinking it? ⬜ 2. Boil it

⬜ 3. Use Sur’Eau (observed)

⬜ 4. Filter it (observed)

⬜ 5. Other (Specify) ______

⬜ 6. No response

Where do you store/keep water in the ⬜ 1. Not applicable, because I house? use a faucet

⬜ 2. In a covered container (bucket, basin or other)

⬜ 3. In an uncovered container

⬜ 4. Other (Specify) ______

⬜ 5. No response

Observed ⬜

Do you wash your hands with soap? ⬜ 1. YES

⬜ 2. NO

Observed ⬜

Does your household have latrines? ⬜ 1. YES

⬜ 2. NO

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 187

No. QUESTIONS AND FILTERS CODING OF RESPONSES SKIP TO

Observed ⬜

How many of your children under five have ______had diarrhea in the last 14 days?

Do you sleep under an insecticide-treated ⬜ 1. YES net? ⬜ 2. NO

Do all children under five in the household ⬜ 1. YES sleep under a net? ⬜ 2. NO

Net available (observed) ⬜

Number of holes______

Is it dirty? ⬜

Is the courtyard of the home clear, with no ⬜ 1. YES standing water? ⬜ 2. NO

Observed ⬜

Thank you for your participation!

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 188

3. TELEPHONE INTERVIEW WITH TOP RÉSEAU MEMBERS Thank you for taking my call.

My name is ______, and I work for GH Pro, which USAID/Madagascar has asked to conduct an evaluation of the performance of the ISM project implemented by PSI. As part of the evaluation, we are contacting Top Réseau members like you by telephone.

Our questions are organized to obtain a good overview of how the ISM project functioned operationally.

All of the answers you provide will be kept confidential. Your identity will never be mentioned in our report. You are also free to not respond to any of our questions or stop the interview at any time. Our interview should not take more than 30 minutes. Before we go further, do you have any questions about this interview?

No. 1 May we continue with the interview? A. Yes B. No

No. 2 Clinic location A. Urban B. Peri-urban C. Rural

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 189

No. 3 Region where the clinic is located:

No. 4 District where the clinic is located:

No. 5 Commune where the clinic is located:

No. 6 This clinic was opened: A. Before 2005 B. Between 2005-2010 C. Between 2010-2013 D. After 2013

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 190

No. 7 When was this clinic recruited by Top Réseau? A. 2005-2010 B. 2010-2013 C. After 2013

No. 8 Who founded the clinic? A. A doctor B. SAF C. SALFA D. PSI E. Another NGO

No. 9 If it was another NGO, which one?:

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 191

No. 10 Why did you join Top Réseau? A. To get more patients B. To improve the quality of care C. To offer more services D. To get more publicity E. To provide care to much more vulnerable people F. Other G. To get more patients H. To improve the quality of care I. To offer more services J. To get more publicity K. To provide care to much more vulnerable people L. Other

No. 11 If you answered "Other," what was the reason?

No. 12 What is the most requested service at the clinic? A. Reproductive health B. Family planning C. Prenatal consultation D. Treatment for sick children E. Diarrhea F. Malaria G. Nutrition H. Other

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 192

No. 13 What other services?

No. 14 Which of these make up the majority of your patients? A. Children under five B. Youth (boys and girls age 15 to 24) C. Women of reproductive age (15 to 49) D. Pregnant women E. Men F. Other

No. 15 Who sets the price of consultations and services in this clinic? A. Clinic management B. PSI C. Other

No. 16 On average, how many patients do you see each month?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 193

No. 17 What motivates people to come to the clinic? A. They don't have many choices B. They have a voucher C. We are inexpensive (affordable or average prices) D. We have a good reputation E. People know the quality of our care F. We treat them well

No. 18 People hear about you: A. From Top Réseau advertising B. From Top Réseau's promotional workers (youth, etc.) C. From a friend or family member D. From a recommendation from a Community Health Worker E. Other

No. 19 Being part of the network has allowed us to improve the quality of our services. (1 = completely disagree, 5 = completely agree) A. 1 B. 2 C. 3 D. 4 E. 5

No. 20 Being part of the network has given us more publicity (1 = completely disagree, 5 = completely agree) A. 1 B. 2 C. 3 D. 4 E. 5

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 194

No. 21 Which NGO has provided you with technical support or training in the last 12 months? A. PSI B. IntraHealth C. Banyan Global D. SAF E. SALFA F. Other

No. 22 What type of training? A. Medical B. Family planning C. Technical D. Management E. Financial F. Counseling G. Quality assurance and support H. Promotional techniques I. Other

No. 23 Does Top Réseau hold you to minimum standards? A. Yes B. No

No. 24 What are those minimum standards?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 195

No. 25 Are there supervisory visits? A. Yes B. No

No. 26 How often do these supervisory visits occur? A. Every month B. Every other month C. Once every four months D. Once every six months E. Once a year F. Never

No. 27 Do you think the quality of your services has benefitted from your clinic's belonging to Top Réseau? (1 = not at all, 5 = greatly) A. 1 B. 2 C. 3 D. 4 E. 5

No. 28 Do you think your clinic's belonging to Top Réseau has increased the use of your services? A. Yes B. No

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 196

No. 29 Do you think your clinic's belonging to Top Réseau has allowed you to offer a greater variety of services? A. Yes B. No

No. 30 Do vouchers bring you new patients who would not otherwise have come to your clinic? A. Yes B. No

No. 31 If you raised your prices, would you lose patients? A. Yes B. No

No. 32 Do patients come from very far away? A. Yes B. No

No. 33 How far do most of your patients live from the clinic? A. 0-1 km B. 1-2 km C. 2-3 km D. 3-4 km E. 4-5 km F. More than 5 km

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 197

No. 34 How well do you understand the long-term vision of PSI's ISM project for Top Réseau clinics? (1 = not at all, 10 = very well) A. 1 B. 2 C. 3 D. 4 E. 5 F. 6 G. 7 H. 8 I. 9 J. 10

No. 35 How much influence does PSI have in your planning process? (1 = none, 10 = a lot) A. 1 B. 2 C. 3 D. 4 E. 5 F. 6 G. 7 H. 8 I. 9 J. 10

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 198

No. 36 How much do government policies support private health services? (1 = not at all, 10 = a lot) A. 1 B. 2 C. 3 D. 4 E. 5 F. 6 G. 7 H. 8 I. 9 J. 10

No. 37 How much have you engaged with the community in your area of influence? (1 = not at all, 10 = a lot)) A. 1 B. 2 C. 3 D. 4 E. 5 F. 6 G. 7 H. 8 I. 9 J. 10

No. 38 Do you conduct promotional activities to engage the community? A. Yes B. No

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 199

No. 39 How much access do you have to the technology (computers, electronic files, applications, equipment, telephones, etc.) you need to implement ISM/PSI activities? (1 = none, 10 = always accessible) A. 1 B. 2 C. 3 D. 4 E. 5 F. 6 G. 7 H. 8 I. 9 J. 10

No. 40 How much capacity do you and other Top Réseau members have to advocate for government support of your activities? (1 = none, 10 = a lot) A. 1 B. 2 C. 3 D. 4 E. 5 F. 6 G. 7 H. 8 I. 9 J. 10

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 200

No. 41 How likely would you be to be able to continue to provide the same quality of care as you do now in the absence of ISM/PSI? (1 = not likely, 10 = very likely) A. 1 B. 2 C. 3 D. 4 E. 5 F. 6 G. 7 H. 8 I. 9 J. 10

No. 42 How much recognition do you get from the Government of Madagascar of your contribution to providing health care to the population? (1 = none, 10 = a lot) A. 1 B. 2 C. 3 D. 4 E. 5 F. 6 G. 7 H. 8 I. 9 J. 10

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 201

No. 43 How capable are you of finding other sources of funding or creating other partnerships? (1 = not capable, 10 = very capable) A. 1 B. 2 C. 3 D. 4 E. 5 F. 6 G. 7 H. 8 I. 9 J. 10

No. 44 How likely would you be to have the medical equipment you need to continue your activities in the absence of PSI? (1 = not likely, 10 = very likely) A. 1 B. 2 C. 3 D. 4 E. 5 F. 6 G. 7 H. 8 I. 9 J. 10

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 202

No. 45 How likely would you be to have the staff you need to continue your activities in the absence of PSI? (1 = not likely, 10 = very likely) A. 1 B. 2 C. 3 D. 4 E. 5 F. 6 G. 7 H. 8 I. 9 J. 10

No. 46 Has the clinic received good publicity in the community as a member of Top Réseau? (1 = not at all, 10 = completely) A. 1 B. 2 C. 3 D. 4 E. 5 F. 6 G. 7 H. 8 I. 9 J. 10

No. 47 Would you be able to continue to provide the same volume of services in the absence of PSI? A. Yes B. No

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 203

No. 48 If you no longer had PSI's support, which of the following services could you continue to offer? A. Contraceptive sales B. Sales of other health products C. Reproductive health D. Family planning E. Maternal and child health F. Prenatal consultations G. Diarrhea H. Nutrition I. Malaria J. Other

No. 49 The ISM/PSI will end soon and will be replaced by another project. What type of support would you like to have from the next program?

No. 50 Has the clinic received good publicity in the community as a member of Top Réseau? (1 = not at all, 10 = completely) A. 18-25 B. 26-35 C. 36-45 D. 46-55 E. 56-65 F. 65+

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 204

No. 51 What is your position or role in the clinic? A. Doctor B. Nurse C. Administrator D. Other

No. 52 Your gender: A. Male B. Female

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 205

4. EXIT INTERVIEW WITH TOP RÉSEAU CLINIC USERS

Question/Information Required

00 Date of interview: ______/______/______

Thank you for making the time to talk with me today.

You were suggested as a key person to inform this activity and we greatly appreciate your perspective, experiences and views.

Before we begin, I want to let you know that any information or examples we gather during this interview process will not be attributed to any specific person or institution, unless you tell us that you would be willing to have your responses either quoted by you in the report, or otherwise attributed to you. You are also free to not respond to any of our questions or stop the interview at any time.

Our interview will take about one hour.

Before we go further, do you have any questions about this interview?

[ ] Consent provided ______[Interviewer/Recorder initials]

Do you have any questions before we begin?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 206

May we continue with the interview?

[ ] Yes [ ] No

01 Male [ ] [ ] Female

Marital status: Single ______Married______Other______

Age: 15-24______25-35______35-49______

Distance from the clinic: Very close _____ Fairly close ______Far ______Very far _____

02 In general, how many people live in your home?

[___|___]

03 How many children do you have?

[___|___]

04 What is the highest level of education you have completed?

[ ] None

[ ] Primary

[ ] Secondary

[ ] Vocational certificate

[ ] High school/University

[ ] Master's degree

A5 What is your occupation?

[ ] Unemployed

[ ] Farmer

[ ] Fisherman/woman

[ ] Trade

[ ] Domestic worker

[ ] Small business

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 207

[ ] Taxi driver

[ ] Professional (teacher, nurse, doctor, lawyer, accountant, etc.)

[ ] Police, security guard

[ ] Tailor

[ ] Barber

[ ] Hairdresser

[ ] Artist

[ ] Other ______

06 Does your spouse work?

[ ] Yes

[ ] No

07 Is this your first visit to the clinic?

[ ] Yes

[ ] No

08 How often do you come to the clinic?

[ ] More than once a month

[ ] Once a month

[ ] Once every three months

[ ] Once every six months

[ ] Once a year

[ ] Less than once a year

09 What is the reason for your visit?

[ ] Sick child Child's age: [ ] 0-30 days [ ] 31 days-1 year [ ] 1-5 years

Child's gender: [ ] Male [ ] Female

[ ] You are sick

[ ] Pregnant

[ ] Checkup

[ ] Family planning

[ ] Other

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 208

10 How long did you have to wait to see a doctor?

[ ] 0 to 15 minutes

[ ] 15 to 30 minutes

[ ] 30 to 60 minutes

[ ] 1 to 2 hours

[ ] Other ______

11 How long did it take you to get to the clinic?

[ ] 0 to 15 minutes

[ ] 15 to 30 minutes

[ ] 30 to 60 minutes

[ ] 1 to 2 hours

[ ] Other ______

12 Did anyone complete a medical record today or at a previous visit?

[ ] Yes

[ ] No

13 Did you receive a detailed explanation of your problem?

[ ] Yes

[ ] No

14 Did you receive good advice?

[ ] Yes

[ ] No

15 How would you rate the quality of care you received today? (1 = very bad experience, 10 = very good experience)

______

16 How did you hear about this clinic?

[ ] Friend

[ ] I saw it from the road

[ ] Radio

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 209

[ ] TV

[ ] CHWs

[ ] Community agency

[ ] Peer educator (role model, mother/father, youth group, etc.)

[ ] Another clinic

17 Did you come here because you know the doctor?

[ ] Yes

[ ] No

18 Did you hear an advertisement for the clinic on the radio?

[ ] Yes

[ ] No

19 Did a health worker tell you to go to the clinic?

[ ] Yes

[ ] No

20 Did a friend or family member tell you about the clinic?

[ ] Yes

[ ] No

21 Would you recommend the clinic to a friend or family member?

[ ] Yes

[ ] No

22 Did you buy anything at the clinic?

[ ] Yes

[ ] No

23 If yes, what did you buy?

______

24 How did you find the consultation fees?

[ ] Expensive

[ ] Acceptable

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 210

[ ] Inexpensive

25 Would you go back to the clinic if you needed to?

[ ] Yes

[ ] No

5. FGD with Female parents age 15-49 of children under five

Question/Information Required

00 Date of interview: ______/______/______

Name of FGD moderator______

Name of note taker______

Name of observer______

Thank you for making the time to talk with me today. USAID/Madagascar has asked GH Pro to conduct an evaluation of the performance of the ISM project implemented by PSI. This evaluation is meant to serve a dual purpose: (1) to learn to what extent the project’s objectives and goals—at all result levels—have been achieved; and (2) to inform the design of a new social marketing project. We are meeting with four groups of young women, including yours.

Before we begin, I want to let you know that any information or examples we gather during this interview process will not be attributed to any specific person or institution, unless you tell us that you would be willing to have your responses either quoted by you in the report, or otherwise attributed to you. You are also free to not respond to any of our questions or stop the interview at any time.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 211

Our interview will take about one hour.

Before we go further, do you have any questions about this interview?

[ ] Consent provided ______[Interviewer/Recorder initials]

Do you have any questions before we begin?

May we continue with the interview?

[ ] Yes [ ] No

FGD session participants profile

Ages______

# Males ______

# Females______

Let’s begin by talking about the ISM Project.

01 Evaluation Question 1

1a What do you know about the promotion of good health habits to parents of children under five, including the prevention and treatment of malaria, diarrhea, pneumonia and other aspects of family health (Sur’Eau, Hydro-Zinc, LLIN)? How were they promoted (national radio, local radio, television, Mobile Video Units, interpersonal communications)?

1b What has been your experience with the promotion of short-term Family Planning methods like PilPlan and long-term methods like injectables, implants and IUDs? What communication channels reached you (national and local radio, Mobile Video Units or interpersonal communications, including Family Planning Counselors and Community Health Workers)?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 212

1c If you obtained these products, where did you obtain them and what motivated you to do so?

02 Evaluation question 2 1

2a What motivated you to obtain these products and services to prevent and treat illnesses of children under five? Was it possible to easily obtain these commodities and services?

2b What has been your experience with the promotion of Top Réseau clinics through the mass media (radio and TV), Mobile Video Units, and interpersonal communications by outreach workers and others?

2c What has been your experience, if any, using Top Réseau clinics, and what services did you access? How would you assess the access, affordability and quality of the services and medications available there?

03 Evaluation question 3 4

Treating illness

When your child under five had diarrhea most recently, how did you take care of it?

When your child had a fever most recently, how did you take care of it?

Preventing illness in children under five

What are some ways you know to prevent diarrhea?

How often do you practice them? What are the barriers?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 213

What are some ways you know to prevent malaria?

How often do you practice them (sleeping under a net)? What keeps you from doing the practices you know you need to do?

Which messages or who encouraged you to take action? (focus on healthy behavior(s))

04 Evaluation question 4

4a Do you think the communications are well-suited for reaching parents of children under five? Were the communications appropriate, and did they have a clear understanding of what parents of children under five need and like in order to motivate them to take action?

05 Evaluation question 5 4

5a What do you think of the availability and cost of Family Planning and other health-related commodities made available by Community Health Workers, Top Réseau clinics and commercial sources? What would the impact be if the cost of these commodities increased?

Thank you very much! If you think of anything else, please do not hesitate to get in touch with us.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 214

6. FGD with male parents age 15-49 of children under five

Question/Information Required

00 Date of interview: ______/______/______

Name of FGD moderator______

Name of note taker______

Name of observer______

Thank you for making the time to talk with me today.

Before we begin, I want to let you know that any information or examples we gather during this interview process will not be attributed to any specific person or institution, unless you tell us that you would be willing to have your responses either quoted by you in the report, or otherwise attributed to you. You are also free to not respond to any of our questions or stop the interview at any time.

Our interview will take about one hour.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 215

Before we go further, do you have any questions about this interview?

[ ] Consent provided ______[Interviewer/Recorder initials]

May we continue with the interview?

[ ] Yes [ ] No

FGD session participants profile

Ages______

# Males ______

# Females______

Let’s begin by talking about the ISM Project.

01 Evaluation question 1

1b What has been your experience with the promotion of short-term Family Planning methods like PilPlan and long-term methods like injectables, implants and IUDs? What communication channels reached you (national and local radio, Mobile Video Units and interpersonal communications, including Family Planning Counselors and Community Health Workers)?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 216

1c If you obtained these products, where did you obtain them and what motivated you to do so?

02 Evaluation question 2 1

2a What motivated you to obtain these products and services to prevent and treat illnesses of children under five? Was it possible to easily obtain these products and services?

2b What has been your experience with the promotion of Top Réseau clinics through the mass media (radio and TV), Mobile Video Units and interpersonal communications by outreach workers and others?

03 Evaluation question 3 4

Treating illness

When your child under five had diarrhea most recently, how did you take care of it?

When your child had a fever most recently, how did you take care of it?

Preventing illness in children under five

What are some ways you know to prevent diarrhea?

How often do you practice them? What are the barriers?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 217

What are some ways you know to prevent malaria?

How often do you practice them (sleeping under a net)? What keeps you from doing the practices you know you need to do?

Which messages or who encouraged you to take action? (focus on healthy behavior(s))

04 Evaluation question 4

4a Do you think of the communications were well-suited for reaching parents of children under five? Were the communications appropriate, and did they have a clear understanding of what parents of children under five need and like in order to motivate them to take action?

05 Evaluation question 5 4

Thank you very much! If you think of anything else, please do not hesitate to get in touch with us.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 218

7. FOCUS GROUP DISCUSSION GUIDE FOR MALE YOUTH 15 TO 24

Question/Information Required

00 Date of interview: ______/______/______

Name of FGD moderator______

Name of note taker______

Name of observer______

Thank you for making the time to talk with me today.

Before we begin, I want to let you know that any information or examples we gather during this interview process will not be attributed to any specific person or institution, unless you tell us that you would be willing to have your responses either quoted by you in the report, or otherwise attributed to you. You are also free to not respond to any of our questions or stop the interview at any time.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 219

Our interview will take about one hour.

Before we go further, do you have any questions about this interview?

[ ] Consent provided ______[Interviewer/Recorder initials]

May we continue with the interview?

[ ] Yes [ ] No

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 220

FGD session participants profile

Ages______

# Males ______

# Females______

Let’s begin by talking about the ISM Project.

01 Evaluation Question 1

1a What do you know about communications for youth on pregnancy prevention, malaria and other health issues? What were the primary channels that were used to communicate directly to youth about pregnancy prevention and malaria (national radio, local radio, etc.)?

1b What has been your involvement, if any, with Community Health Workers and Ministry of Youth peer educators promoting pregnancy prevention services and other aspects of health specifically to youth?

1c What has been your involvement, if any, with the “Education Through Listening” campaign that links youth to Top Réseau clinics with vouchers and promotes empowering young women to make reproductive health decisions?

1d What has been your experience, if any, with the Tanora 100% (“100% Youth”) initiative that promotes delaying early marriage and delaying first births and links youth to Top Réseau clinics, through radio spots and peer educators?

1e Considering all of these different ways of communicating with youth, what were the most effective in reaching and motivating youth when it comes to pregnancy prevention and other aspects of improving

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 221

health? Considering all of these different ways of communicating with youth, what were the most effective in reaching and motivating youth?

02 Evaluation question 2 1

2a What do you know about the promotion of the “Yes With You” condom to youth? How well did the style of the communication on the radio and other places attract your attention? What do you think of the price and availability of it?

2b What do you know about the promotion, specifically to youth, of short-term methods of Family Planning, like the pill, and long-term methods like injectables and implants like “Confiance”? Where are they available? If you obtained these products, what inspired you to do so?

2c What do you know about Top Réseau clinics? Where did you hear about them and what is your experience with them? What do you know about Top Réseau's special services for youth and vouchers for free visits?

What has been your experience to date using a Top Réseau clinic?

03 Evaluation question 3 4

3a Considering all of these different ways of communicating with youth, what were the most effective in reaching and motivating youth to use products and services? What is your experience, if any, with

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 222

mobile services (FP, HIV testing) targeting youth?

Where did you receive information about FP, HIV, etc.?

Who provided you with this information?

04 Evaluation question 4

4a Do you think of the communications were well-suited for youth? Were they appropriate, and did they understand the needs of young people?

05 Evaluation question 5 4

5a What do you think of the availability and cost of Family Planning and other health-related commodities made available by Community Health Volunteers, Top Réseau clinics and other sources? Are these commodities and services available and affordable in other locations or organizations?

Let’s talk about demand. Do youth want [condoms or other FP products]? Do they know where to get them? What happens if they don’t find them in [place A]? Where do they go?

Thank you very much! If you think of anything else, please do not hesitate to get in touch with us.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 223

8. FOCUS GROUP DISCUSSION GUIDE FOR COMMUNITY HEALTH WORKERS

Question/Information Required

00 Date of interview: ______/______/______

Name of FGD moderator______

Name of note taker______

Name of observer______

Thank you for making the time to talk with me today.

Before we begin, I want to let you know that any information or examples we gather during this interview process will not be attributed to any specific person or institution, unless you tell us that you would be willing to have your responses either quoted by you in the report, or otherwise attributed to you. You are also free to not respond to any of our questions or stop the interview at any time.

Our interview will take about one hour. Before we go further, do you have any questions about this interview?

[ ] Consent provided ______[Interviewer/Recorder initials]

May we continue with the FGD?

[ ] Yes [ ] No

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 224

1) What is your role obtaining commodities from the “Point d’Approvisionnement,” promoting their use and selling them?

2) What is your role in conducting interpersonal communications (IPC) about healthy behaviors not directly related to a product?

3) What is your experience with interpersonal communication sessions, either one-on-one or in small groups, and what training did you get in how to do them?

4) What has been your experience promoting Top Réseau clinics, if any?

5) What specific Information Education and Communication (IEC) materials have you used (job aids, brochures, leaflets, photonovela, T-shirts and caps, product samples, etc.)?

6) What was the result of your interpersonal communication efforts in terms of inspiring positive changes in behavior?

7) Who were the target populations for the communications and what was done to specifically reach those who are underserved (youth, rural populations, etc.)? Have you been involved in the promotion of the Yes condom to youth and encouraging them to prevent unwanted pregnancies?

8) To what degree are the services and commodities promoted accessible and affordable to the target populations? To what degree have you been able to reach sufficient numbers of the target populations to have an impact on behavior?

9) When the ISM program ends, how will you be able to continue to use interpersonal communications?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 225

Community Based Distribution LOCATION: DATE:

Key information to be completed by the interviewer/FGD facilitator Hello. I am here to do an observation and to ask you a few questions to evaluate PSI's social marketing program. I would like your permission to do the observation and maybe take some photos. May I ask you a few questions?

Consent provided______Initials 1- Site

Urban

Peri-urban

Rural

2- Appearance, Signage and IEC materials display Cleanliness

Posters

Danglers

Stickers

Other

3- Product and Price

Product Price Product Price Pilplan ACT

Confiance IRDT

Protector Plus ITP

YES for you Hydrazink

Sayana press Sur'Eau pilina

Norlevo Pneumox

Implanon

Jadelle

Copper T

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 226

Zarin

7- Stock Can you show me your stock? When was the last time you were out of stock?

Can I take a picture of a stock card?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 227

9. Focus Group Discussion Guide for Female Youth 15 to 24

Question/Information Required

00 Date of interview: ______/______/______

Name of FGD moderator______

Name of note taker______

Name of observer______

Before we begin, I want to let you know that any information or examples we gather during this interview process will not be attributed to any specific person or institution, unless you tell us that you would be willing to have your responses either quoted by you in the report, or otherwise attributed to you. You are also free to not respond to any of our questions or stop the interview at any time.

Our interview will take about one hour.

Before we go further, do you have any questions about this interview?

[ ] Consent provided ______[Interviewer/Recorder initials]

May we continue with the interview?

[ ] Yes [ ] No

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 228

FGD session participants profile

Ages______

# Males ______

# Females______

Let’s begin by talking about the ISM Project.

01 Evaluation question 1

1b What has been your involvement, if any, with Community Health Workers or Ministry of Youth peer educators promoting pregnancy prevention services and other aspects of health specifically to youth?

1d What has been your experience, if any, with the Tanora 100% (“100% Youth”) initiative that promotes delaying early marriage and delaying first births and links youths to Top Réseau clinics through radio spots and peer educators?

1e Considering all of these different ways of communicating with youth, what were the most effective in reaching and motivating youth regarding pregnancy prevention and other aspects of improving health? Considering all of these different ways of communicating with youth, what were the most effective in

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 229

reaching and motivating youth?

02 Evaluation question 2 1

2a What do you know about the promotion of the “Yes with You” condom to youth? How well did the style of communication on the radio and other places attract your attention? What do you think of the price and availability of it?

2b What do you know about the promotion, specifically to youth, of short-term methods of Family Planning, like the pill, and long-term methods, like injectables and implants like “Confiance”? Where are they available? If you obtained these products, what inspired you to do so?

What has been your experience to date using a Top Réseau clinic?

03 Evaluation question 3 4

Where did you receive information about FP, HIV, etc.?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 230

Who provided you with this information?

04 Evaluation question 4

4a Do you think of the communications were well-suited for youth? Were they appropriate and did they show an understanding of youth needs?

05 Evaluation question 5 4

5a What do you think of the availability and cost of Family Planning and other health-related commodities made available by Community Health Volunteers, Top Réseau Clinics and other sources? Are these commodities and services available and affordable in other locations or organizations?

Let’s talk about demand. Do youth want [condoms or other FP products]? Do they know where to get them? What happens if they don’t find them in [place A]? Where do they go?

Thank you very much! If you think of anything else, please do not hesitate to get in touch with us.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 231

10. Key Informant Interviews with Distributors

Question/Information Required

00 Date of interview: ______/______/______

[ ] PSI

[ ] PSI partner ______

[ ] USAID

[ ] Other USAID partner

[ ] GoM

[ ] Other GoM partner

Which ISM component are you responsible for?

[ ] SBCC [ ] Private service delivery [ ] Social Marketing [ ] Not an ISM implementer

Thank you for making the time to talk with me today.

You were suggested as a key person to inform this activity and we greatly appreciate your perspective, experiences and views.

All of the answers you provide will be summarized and included in our report. We will also be conducting interviews and focus group discussions with other stakeholders, and will be

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 232

triangulating all findings. The final report will be shared with you through USAID.

Before we begin, I want to let you know that any information or examples we gather during this interview process will not be attributed to any specific person or institution, unless you tell us that you would be willing to have your responses either quoted by you in the report, or otherwise attributed to you. You are also free to not respond to any of our questions or stop the interview at any time.

Our interview will take about one hour.

Before we go further, do you have any questions about this interview?

[ ] Consent provided for interview ______[Interviewer/Recorder initials]

May we continue with the interview?

[ ] Yes [ ] No

Let’s begin by talking about the ISM Project.

01 What is the objective of the ISM project? (probe – what services does ISM offer/support, to whom, and how? How do you work with DISTRIBUTORS, WHOLESALERS, RETAILERS OF SOCIALLY MARKETED PRODUCTS? CHWs? Supply chain and distribution actors?)

Of the three components (SBCC, social marketing and private services), what was/is the most successful? Why?

What was the least successful? Why?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 233

What are some of the challenges ISM has experienced?

Will these challenges keep ISM from reaching its targets by the end of 2017?

02 Overall, what proportion of resources should go to each of the three components: 2

In terms of money:

______SBCC ______Private healthcare ______commodities

Why?

03 Did program activities reach these populations? (document proof) 3

Urban-

Rural-

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 234

Why or why not?

4 Outside of the ISM project, is the ISM data being used? For what? By whom? 4

What has been ISM’s role in promoting this use?

5 On a scale of 1-10, with 1 being not at all and 10 being completely, how prepared do you 5 think the Government and the private sector are to continue to increase commercially available health products and services without ISM or USAID support?

Family planning

Pilplan ______

Confiance ______

Yes Youth condoms ______

Protector Plus ______

Feeling condom ______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 235

Sayana Press ______

Norlevo ______

Implanon ______

Jadelle ______

IUDs (Copper T) ______

IUDs (Zarin) ______

Chlomexidine ______

Rojo-branded cycle beads ______

Malaria

LLINs ______

ACTs ______

RDTs ______

ITP ______

Child Survival

ViaSur (or equivalent) ______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 236

HydraZinc (DTK) ______

Sur'Eau Pilina Tablets ______

Pneumox ______

Zaza Tamady-branded micronutrient powder (MNP) ______6 On a scale of 1-10, how much capacity would you say large distributors (importers, agents, large pharmacies) have to advocate for social marketing commodities?

______

Please explain:

7 On a scale of 1-10, do the ISM project's distribution partners have the physical infrastructure for socially marketed products?

Space ______

Basic Equipment ______

Transport ______

Access to relevant information ______8 On a scale of 1 to 10, please rate the extent to which you believe the ISM project's distribution partners have adequate staff to implement ISM activities:

______

Please explain:

9 On a scale of 1 to 10, how far are DISTRIBUTORS, WHOLESALERS, and RETAILERS OF SOCIALLY MARKETED PRODUCTS to clinics in the application of an efficient financial management system?

______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 237

10 On a scale of 1-10, do the ISM project's distribution partners have the organizational infrastructure for socially marketed products:

Clear internal roles and responsibilities ______

Marketing capacity ______

Internal QA/CQI processes that are operational ______

Supply Chain Management ______

Financial management capacity ______

Overall, on a scale of 1 to 10, how strong would you say is the ISM project's distribution partners' organizational capacity for social marketing? ______

11 On a scale of 1 to 10, what is the level of trust that distributors/retailers have from their consumers?

______

12 On a scale of 1 to 10, with 1 being not at all and 10 being completely, do you think that 5 DISTRIBUTORS, WHOLESALERS, and RETAILERS OF SOCIALLY MARKETED PRODUCTS, overall:

Share the same understanding with the ISM project regarding their role in promoting health equity by providing high-quality care to underserved and rural populations as well as other clientele?

______

Take full ownership and responsibility for ensuring the financial viability of the social marketed products they sell?

______

Are operating within a policy environment that supports their role as a provider of equitable services to the poor as well as other clientele?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 238

______

Would it be useful to have a coordinating body for all private sector health franchises?

What are franchises' capacity to advocate for additional resources to support/maintain their role as equitable and high-quality private service providers?

______

-- In particular, what is the advocacy capacity of DISTRIBUTORS, WHOLESALERS, and RETAILERS OF SOCIALLY MARKETED PRODUCTS to solicit additional support or to create income- generating partnerships that may benefit their catchment area?

Do individual DISTRIBUTORS, WHOLESALERS, and RETAILERS OF SOCIALLY MARKETED PRODUCTS have sufficient staff numbers, types and capacity to ensure they can successfully serve as private providers of high-quality services?

Numbers: ______

Types: ______

Capacity: ______

Are staffing levels at DISTRIBUTORS, WHOLESALERS, and RETAILERS OF SOCIALLY MARKETED PRODUCTS in general sufficient to manage the delivery of the health commodities needed in rural and underserved communities?

Yes/No:

______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 239

Why/why not?

Do they have sufficient access to the necessary technology and socially marketed commodities to facilitate their role as a high-quality primary healthcare provider for their catchment area?

______

On a scale of 1 to 10, 1 being not at all and 10 being completely, do DISTRIBUTORS, WHOLESALERS, and RETAILERS OF SOCIALLY MARKETED PRODUCTS have adequate financial resources outside of the ISM project for staffing, Internet, training, and other resources in order to provide socially marketed products for sale?

______

On a scale of 1 to 10, 1 being not at all and 10 being completely, do they have the capacity to attract additional paying clients if/when necessary?

______

Please elaborate on their marketing strategy/strategies for obtaining new clients and retaining old clients:

On a scale of 1 to 10, to what extent do DISTRIBUTORS, WHOLESALERS, and RETAILERS OF SOCIALLY MARKETED PRODUCTS have the organizational infrastructure to continue to implement ISM activities in the absence of the ISM program?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 240

Marketing capacity ______

Internal QA/CQI processes that are operational ______

Routine data collection, analysis and reporting ______

Financial management capacity ______

On a scale of 1(completely negative) to 10 (completely positive), please rate communities' perceptions of DISTRIBUTORS, WHOLESALERS, and RETAILERS OF SOCIALLY MARKETED PRODUCTS:

______

Does the commercial sector understand the concept of SOCIALLY MARKETED PRODUCTS?

Is the commercial sector supportive of DISTRIBUTORS, WHOLESALERS, and RETAILERS OF SOCIALLY MARKETED PRODUCTS?

In your opinion, who does the public rely on as a credible source of expertise and health information?

[ ] DISTRIBUTORS, WHOLESALERS, and RETAILERS OF SOCIALLY MARKETED PRODUCTS

[ ] CHWs

[ ] Peer models

[ ] Radio

[ ] TV

[ ] Newspaper

[ ] Friends

[ ] Family

[ ] Other

How much confidence does the public have in your distribution network for socially marketed

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 241

products?

______

Why do you think this perception is accurate/not accurate? Is it different in urban and rural areas?

Urban:

Rural:

13 Overall, on a scale of 1 to 10, how financially capable are DISTRIBUTORS, WHOLESALERS, and RETAILERS OF SOCIALLY MARKETED PRODUCTSs of continuing to provide socially marketed products to underserved and rural populations without subsidies from the ISM project or USAID?

______

14 How would you describe ISM’s added value in Madagascar: 4

15 In an ideal world, what do you think should be the main focus of a follow-on project? gen

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 242

16 Do you have any other thoughts you would like to share?

Thank you very much! If you think of anything else, please do not hesitate to get in touch with us.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 243

11. KII SOCIAL MARKETING PERSONNEL AND SALES FORCE Location:

Date:

Key information to be completed by the interviewer 1-Type

[ ] Distributor [ ] Retailer

[ ] Point of sale [ ] Pharmacy [ ] Shop 2 - Site [ ] Urban [ ] Peri-urban [ ] Rural 3 - For retail and PA [ ] Cleanliness [ ] Signage and IEC materials display [ ] Product display [ ] Stock cards (if so, take picture) 4 - Gender [ ] Male [ ] Female

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 244

11. KEY INFORMANT INTERVIEWS SM PERSONNEL AND SALES FORCE

Question/Information R

00 Date of interview: ______

Thank you for making the t USAID/Madagascar has aske ISM project implemented b learn to what extent the pr achieved; and (2) to inform

You were suggested as a ke perspective, experiences an

Our questions are organize operationally, and to answe questions about 1) project

integration of program resu project’s definition/operatio project’s target population; learning; and 5) social and e

If there are staff members w appreciate the opportunity with them separately.

All of the answers you prov be conducting interviews an triangulating all findings. Th

Before we begin, I want to this interview process will n tell us that you would be w or otherwise attributed to stop the interview at any ti

Our interview will take abo about this interview?

[ ] Consent provided ____

May we continue with the i

[ ] Yes

Age ______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 245

01 FOR DISTRIBUTORS AND WHOLESALERS ONLY Besides PSI, does someone else sell you contraceptives and/or other health products?

02 Please list the products you sell and the prices you sell them for: 1

03 What are your monthly sales targets, and do you frequently meet, exceed or fall short of your 1 targets?

04 Please list the people you sell to: 3

05 COMMERCIAL AGENTS ONLY 1

How many clients do you receive every day?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 246

06 POINTS OF SALE ONLY 1

Please describe your interaction with health workers.

07 ALL 2 What are the most common questions clients ask you?

08 Beginning with the one that attracts the most clients, list the advertising and promotional 2 methods you use (radio, point of sale materials, PSI Mobile Video Units, person-to-person education, events and/or campaigns):

09 ALL 2

Please list the training you have received from PSI, MAHEVA and MIKOLO:

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 247

10 RETAIL ONLY 2

In your opinion, how do your clients learn about and decide to buying your products?

11 What do you think of the product prices? 2

[..] Expensive

[..] Acceptable

[..] Inexpensive

12 ALL 3

Please list the problems you may have encountered with stock:

13 Please list other contraceptives or products similar to PSI’s that you sell, if any:

Do you sell these at full commercial price?

13A

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 248

14 POINTS OF SALE ONLY 5

Do you receive malaria products from the government that you also sell to health workers?

15 In your opinion, are the PSI products that you sell profitable or not? 3

16 Distributors, wholesalers and retailers: no POINTS OF SALE 5

How do the volumes of socially marketed products compare to other, similar products (for example, essential household goods like soap or over-the-counter medicines)?

17 Where is the closest Top Réseau clinic? Do you use it? Do you send clients there? 3

18 Qs 18 to 21 ARE FOR WHOLESALERS and DISTRIBUTORS ONLY 2

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 249

Do you think there is a favorable policy environment for your business in Madagascar? For health and Family Planning products specifically? Are there any taxes and tariffs that limit your actions?

19 On a scale of 1 to 10, 1 being very low and 10 being very high, how would you rate your capacity to advocate for your business?

______

20 On a scale of 1 to 10, how willing would you be to pay to acquire the trademark of a socially 2 marketed brand like Pilplan or Confiance?

______

21 On a scale of 1 to 10, 1 being very poor and 10 being excellent, please rate: Your physical infrastructure ______

You Staff ______

Your organization ______

22 ALL 3

Are your clients:

[ ] Urban

[ ] Rural

[ ] Very poor

[ ] Poor

[ ] Middle Class

[ ] Rich

23 On a scale of 1 to 10, 1 being not possible and 10 being very possible, if you were to pay full 3,5 commercial price for a product like "Pilplan," "Yes with You," or "Confiance," how likely would you be to be able to sell it with the necessary profit margins?

______

How much extra (from 10% to 100%) would you be willing to pay, and for which product(s)?

______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 250

______

24 On a scale of 1 to 10, 1 being impossible and 10 being very possible, if there were no socially 5 marketed products available, how likely would you be to continue to sell these products at commercial prices?

______

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 251

12. KII WITH LOCAL RADIO STATION MANAGERS

Question/Information Required

00 Date of interview: ______/______/______

Name of interviewer:______

Thank you for making the time to talk with me today.

Before we begin, I want to let you know that any information or examples we gather during this interview process will not be attributed to any specific person or institution, unless you tell us that you would be willing to have your responses either quoted by you in the report, or otherwise attributed to you. You are also free to not respond to any of our questions or stop the interview at any time.

Our interview will take about one hour.

Before we go further, do you have any questions about this interview?

[ ] Consent provided ______[Interviewer/Recorder initials]

May we continue with the interview?

[ ] Yes [ ] No

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 252

FGD session participants profile

Ages______

# Males ______

# Females______

Let’s begin by talking about the ISM Project.

01 Evaluation question 1

What was your relationship with ISM and PSI in regards to the diffusion of programs and spot advertisements promoting positive health behaviors?

What was the involvement of your station in the broadcasting of ISM radio dramas, if any?

02 Evaluation question 2

What was your experience in promoting the following products and services:

- Malaria prevention and treatment (LLIN, rapid treatment)

- Diarrhea treatment

- Top Réseau clinics

- Short-term methods of Family Planning commodities (pills and condoms)

- Long-term Family Planning methods (injectables and implants)

- Other commodities and services

03 Evaluation question 3 4

3a What was done to promote ISM messages, products and services specifically to youth and rural populations on the radio? What percentage of the district's population does your radio station reach?

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 253

3b Who were the target populations for the communications, and what was done to specifically reach those who are underserved (youth, rural populations, etc.)?

3c To what degree are the services and commodities promoted accessible and affordable to the target populations?

04 Evaluation question 4

4a How did ISM's operational research inform the creation of the communication messages and materials?

05 Evaluation question 5 4

05 What is your radio station's capacity to produce its own programs and spots targeting youth, rural populations, and the poor?

When the ISM program ends, how will you be able to continue to use your station to communicate about health messages?

Thank you very much! If you think of anything else, please do not hesitate to get in touch with us.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 254

13. KEY INFORMANT INTERVIEWS WITH TOP RÉSEAU CLINIC STAFF

Question/Information Required

00 Date of interview: ______/______/______

Thank you for making the time to talk with me today.

We are visiting six Top Réseau Clinics, and yours was selected.

Before we begin, I want to let you know that any information or examples we gather during this interview process will not be attributed to any specific person or institution, unless you tell us that you would be willing to have your responses either quoted by you in the report, or otherwise attributed to you. You are also free to not respond to any of our questions or stop the interview at any time.

Our interview will take about one hour.

Before we go further, do you have any questions about this interview?

[ ] Consent provided ______[Interviewer/Recorder initials]

May we continue with the interview?

[ ] Yes [ ] No

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 255

00 Age: ______Gender: [ ] Male [ ] Female Job Title:______

Can you please tell us a bit about what you understand about the goals of the ISM program?

01 Describe when and how this clinic became a Top Réseau Member: 1

02 What are the main reasons for becoming a Top Réseau clinic, and what are the advantages? (more 1 clients, improved quality, less expensive commodities, expanded services)

03 List the new services offered under Top Réseau to your clients, their response, and the effect 1 adding these services had on the number of clients coming to the clinic.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 256

04 Describe your clients' profile (age, gender, economic status, urban or rural): 3

05 How is price of each service determined? 1

06 Describe what kind of technical support you have received and from whom (medical, financial and 2 administrative, counseling, supply management)?

07 Describe the flow for a patient the first time he or she comes to your clinic, starting with the first 1 contact until he or she sees the doctor. Please give us detail on how and who creates the medical record (histoire clinic).

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 257

08 Describe the same flow for a repeat patient. 1

09 Please describe the medical record keeping system. 1

10 How would you describe the counseling provided to the patient? (Who does it? When is it done?)

11 Describe the basic elements of the Top Réseau minimum standards. 1

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 258

12 To your understanding, how important is the application of the minimum medical standards? 1

13 In what manner and to what degree has belonging to Réseau Plus helped the clinic in improving the 2 quality of its services?

14 Please describe the Information, Education and Communications (IEC) materials used by the clinic 2 (brochures, posters, danglers, signs, other). Who provides them and how often?

15 Describe the kind and frequency of detailing visits from PSI supervisors and detailers. 2

16 Please list the other organizations that visit the clinic for supervision or promotion purposes. 2

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 259

17 List the promotional events, like Women’s Day or Malaria Day, in which the clinic has participated 2 and the role it played.

18 To your understanding, where do your first-time clients hear about the clinic? 2

19 Why do you think the clients come to the clinic? 2

[ ] Clinic is close to their home

[ ] Radio advertising

[ ] Recommended by friend or family

[ ] Price

[ ] Voucher

[ ] Doctor reputation

[ ] Good quality

[ ] Referral by Agent Communautaire (CHW or CHV)

[ ] Other ______

20 Please list the products that you sell and those that you give away for free: 2

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 260

21 Please list the products that you buy from PSI: 5

22 Please list the other public or private medical services in your area: 3

23 If PSI was not supporting the clinic, would it be able to continue providing the same services at the 5 same prices? Or what would you need to do and would you continue to provide these same services?

We are almost done. Now let’s talk about sustainability.

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 261

On a scale of 1-10, with 1 being not at all and 10 being completely, do you think you (Top Réseau clinic):

Understand the objective of the ISM program in supporting the expansion of Top Réseau franchisees?

______

What is your understanding of it?

On a scale of 1 to 10, with 1 being not likely and 10 being very likely, how likely are you to continue to provide medical care to underserved and rural populations in the absence of ISM and USAID?

______

Do you have any other questions for us?

Thank you very much for your time!

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 262

14. TOP RÉSEAU SITE VISIT GUIDE Location:

Date:

Basic information to be completed by the interviewer The day that the key person interview is to take place, it is recommended that you arrive 10 to 15 minutes ahead of time and just sit in the waiting area and observe the flow. It will help you direct the conversation and obtain as much information as possible about the culture of quality assurance and client satisfaction. In your opinion, is this an urban, peri-urban, or rural site? General appearance of the clinic: Cleanliness: Signage and IEC materials display: How many people were in the waiting room? To the best of your knowledge, how would you describe the patients in the waiting room in terms of their socioeconomic status? How many staff work in the clinic? Write down any other details of interest that relate to your perception of quality upon arrival at the clinic: Throughout the whole process, think, "Would I bring my husband/wife/children to this clinic?"

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 263

ANNEX V. SOURCES OF INFORMATION

DOCUMENTS REVIEWED:

General ISM

1. ISM Cooperative Agreement

2. ISM Implementation Plan fy2017 & Q1 Fy2018 (R) 3. PSI implementation areas 4. ISM FY13 Workplan Narrative approved 03-12-2013 5. ISM FY14 workplan 6. ISM FY 15 Workplan 7. ISM FY 16 Workplan 8. ISM M&E plan and RF (R) SEPT 2016 9. Ministere de L’Interieur et de la Decentralisation, Decret no. 2015-592. Portant classement des Communes en Communes urbaines ou en Communes rurales.

10. ISM Annual Report and Annexes FY 15 11. ISM FY 16 Q 1 REPORT AND ANNEXES 12. ISM FY 16 Q2 REPORT AND ANNEXES 13. ISM FY 16 Q3 REPORT AND ANNEXES 14. Madagascar Final Report 2016 (ICSF) 15. Madagascar MIS 2013 FRENCH

16. List of Research Studies to date, ISM 17. Madagascar MICS 2012 (French) 18. Madagascar DHS 2008_9 (French)

19. Madagascar MIS 2011 (French)

Top Réseau Service Documents and Statistics

20. Données par Centres Top Réseau FY13 a FY16

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 264

21. Données clients PF sur TR 22. Madagascar Exit Interview Summa Report Clients Audience

23. Liste des centres SALFA 2016 24. Liste des centres SAF 2016

25. Strategie d’extension en zones rurales

26. Strengthening Madagascar’s Top Reseau Social Franchise, 2016 27. Madagascar General Motivation Among Top Reseau, 2013

Social Marketing 28. Madagascar Summa Report New Distribution System 2014 final 29. ISM_-Distribution Activity Level Indicator FY13-Fy16 (Sales) 30. Structure des Prix Update 31. PSI Madagascar FP COGS Analysis 11.29.2016

SBCC

32. Madagascar Household Nutrition Study June 2014 Final 33. Madagascar Trac Summary Reports: Diarrhea 2014 34. Madagascar Trac Summary Reports: Pneumonia 2014 35. Madagascar Trac Summary Reports: FP 2015

Sub Agreements

36. Subagreement by Population Services International (PSI) to Sampan’Asa Fampandrosoana Fiangonani Jesoa Kristy Eto Madagasikara Saffjkm, a non-US, Non- Governmental Organization

37. Subagreement by Population Services International (PSI) to Sampan’Asa Loterana Momba Ny Fahasalamana Salfa, a Non US, Non-Governmental Organization

DATABASES

1. Top Reseau Phone Interview 2. Household Survey 3. Top Reseau Client Exit Interviews

4. Retailer Checks

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 265

ANNEX VI. DISCLOSURE OF ANY CONFLICTS OF INTEREST

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 266

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 267

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 268

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 269

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 270

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 271

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 272

FINAL PERFORMANCE EVALUATION OF USAID/MADAGASCAR ISM PROGRAM / 273

For more information, please visit ghpro.dexisonline.com

Global Health Performance Cycle Improvement Project 1299 Pennsylvania Avenue NW, Suite 1152 Washington, DC 20006 Phone: (202) 625-9444 Fax: (202) 517-9181 http://ghpro.dexisonline.com/reports-publications