Independent Final Programme Evaluation Report

DFID-funded programme “to improve maternal health and infant survival rates through supporting women and children in Badin District, Pakistan”

March 2017

DFID

* Cover Picture: A participant from Health Literacy class demonstrating her newly acquired skills to write and fill in a basic form in Village Shevo Kolhi, Badin District. (Photo Credit: GLOW Consultants)

AUTHORS This independent final programme evaluation report was commissioned by Feed the Minds and was supported by its partner National Rural Development Programme. It was produced by GLOW Consultants Pvt Limited Pakistan. For further information, please contact Mr. Saeed Ullah Khan ([email protected]).

The contact point for Feed the Minds is Ms. Albha Bowe ([email protected]).

GLOW CONSULTANTS (PRIVATE) LIMITED SECP Registration No: 0088603

Address: GLOW Consultants Private Limited, 4th Floor, Software Technology Park-1, F 5/1, Islamabad, Pakistan / Phone: +92-51-2828 948 / +92 345 85 75 974 Email us: [email protected]

i | P a g e Final Programme Evaluation

DFID

Figure 1: Discussions with Health Literacy Facilitator belonging to Hindu minority engaged in FTM programme in Badin District, (Photo Credit: GLOW Consultants)

PREFACE The evaluation team would like to thank everyone who participated in and supported the undertaking of this evaluation. This includes the communities the team visited as well as the management and field staff of National Rural Development Programme.

EVALUATION TEAM Mr. Saeed Ullah Khan (Lead Consultant) Dr. Ejaz Ahmed (Consultant) Mr. Zaki Ullah Khan (Consultant) Dr. Ms. Salma Khalil (Technical Expert) Ms. Sobia Sattar (Consultant); Ms. Qurat Ul Ain Nawaz (Consultant)

ii | P a g e Final Programme Evaluation

DFID

CONTENTS Authors ...... i Preface ...... ii Evaluation Team ...... ii Contents ...... iii List of Figures ...... v List of Tables ...... v List of Key Acronyms ...... v Executive Summary ...... vi SECTION 1-INTRODUCTION ...... 1 1.1 Purpose of the Evaluation ...... 1 1.2 Organisation context ...... 2 1.3 Overview of UK Aid Direct funded activities ...... 2 SECTION 2: EVALUATION METHODLOGY ...... 4 2.1 Evaluation Plan ...... 4 2.1.1 Evaluation Geographical Coverage...... 4 2.1.2 Evaluation Inquiry Techniques ...... 5 2.1.3 Sample Size for FGDs and KIIs ...... 5 2.1.4 Additional Data Sources ...... 6 2.1.5 Gender Consideration ...... 7 2.1.6 Data Collection and Quality Assurance ...... 7 2.2 Strengths and weaknesses of selected design and research methods ...... 7 2.2.1 Strengths ...... 7 2.2.2 Weaknesses ...... 7 2.3 Summary of problems and issues encountered ...... 7 SECTION 3: PROGESS TOWARDS OUTCOMES ...... 8 3.1.1 Women Health Committees (WHC) ...... 8 3.1.2 Men’s Support Groups (MSGs) ...... 13 3.1.3 Community Midwives (CMs) ...... 15 3.1.3 Health Literacy Facilitators (HLFs) ...... 19 3.1.4 Health Literacy Class Participants ...... 21 3.1.5 Other Stakeholders ...... 22 3.2 Relevance ...... 23 3.3 Effectiveness ...... 25 3.4 Efficiency ...... 26 3.5 Sustainability ...... 26 3.6 Impact ...... 27

iii | P a g e Final Programme Evaluation

DFID

SECTION 4: CONCLUSIONS ...... 30 4.1 Summary of achievements against evaluation questions ...... 30 4.2 Summary of achievements against rationale for UK Aid Direct funding ...... 33 4.3 Overall Impact and Value for Money of UK Aid Direct Funded Activities ...... 37 SECTION 5: LESSONS LEARNT...... 38 SECTION 6: RECOMMENDATIONS ...... 39 6.1 Number of WHC Members ...... 39 6.2 Refresher training of WHC Members ...... 39 6.3 Numbers and Membership MSG ...... 39 6.4 Capacity Building of MSGs ...... 39 6.5 MSG Support to WHC for Advocacy and Lobby ...... 39 6.6 Improving Linkages with Government Health Facilities ...... 39 6.7 Training of Community Midwives ...... 40 6.8 Refreshers Related to Usage of Smart Phones ...... 40 6.9 Enhancing Skills/ Qualification Requirement for HLF and Class Duration...... 40 6.10 Capacity Building on Awareness and Lobbying Activities and its Logic ...... 40 6.11 Policy Level Engagements ...... 41 6.12 Increase Scale of the Programme ...... 41 Annex 1 Evaluation Matrix ...... b Annex 2: Programme villages and phases ...... e Annex 3: FGD details ...... j Annex 4: Evaluation Plan ...... k Annex 5: Quality Assurance Plan ...... l Annex 6: Data Collection tools ...... m Annex 6.1-FGD: Women Health Committee ...... m Annex 6.2-FGD: MSG ...... p Annex 6.3-FGD: General Community ...... s Annex 6.4-FGD: Health Literacy Class Participants ...... u Annex 6.5-KII – Community Midwives ...... w Annex 6.6-KII – Health Literacy Facalitators ...... z Annex 6.7-KII – Government Health Department ...... bb Annex 6.8-KII – BHU Staff ...... dd Annex 6.9-KII – NRDP Staff ...... ff Annex 6.10-KII – District Government Official ...... hh Annex 7: Evaluation Criteria Table ...... jj

iv | P a g e Final Programme Evaluation

DFID

LIST OF FIGURES

Figure 1: Discussions with Health Literacy Facilitator belonging to Hindu minority ...... ii Figure 2: Discussions with Women Health Committee members in FTM programme in Badin District . 1 Figure 3: Map of DFID funded FtM Programme Evaluation District ...... 5 Figure 4: A Community Midwife demonstrating her skills in FtM programme in Badin District ...... 8 Figure 5: A Key informant interview with a FTM programme participant in Badin District ...... 18 Figure 6: A Community Midwife explaining different items of her midwifery kit in Badin District ...... 24 Figure 7: Key Informant Interview with BHU staff in Badin District ...... 28 Figure 8: Key Informant Interview with a doctor who is also part of mobile referral system ...... 38

LIST OF TABLES

Table 1: Sample from the List of 100 villages (Phase-I & Phase-II) ...... 4 Table 2: Type and Number of Key Informants ...... 6 Table 3: Information about Focus Group participants ...... 6

LIST OF KEY ACRONYMS

CSOs Civil Society Organisations CMs Community Midwives DfID Department for International Development FTM Feed the Minds FGDs Focus Group Discussions GPAF Global Poverty Action Fund HANDS Health & Nutrition Development Society HLF Health Literary Facilitator HDDS Household Dietary Diversity Score IDDS Individual Dietary Diversity Score IPC Integrated Food Security Phase Classification MSS Marie Stopes Society MSGs Men’s Support Groups MAM Moderate Acute Under-Nutrition NRDP National Rural Development Programme NRSP National Rural support Program NSP Nutrition Strategic Plan PFF Pakistan Fisher Folk Forum PC1 Planning Commission from 1 SAM Severe Acute Under-Nutrition ToR Terms of Reference NNS 2011 The National Nutrition Survey 2011 WHC Women’s Health Committee

v | P a g e Final Programme Evaluation

DFID

EXECUTIVE SUMMARY

Sindh Province is the second most populous province of Pakistan. Badin District, which is part of Sindh province, has high levels of infant and maternal mortality. To respond to the health needs of women and children in this area a programme was initiated with the support of the Department for International Development. The programme, “to improve maternal health and infant survival rates through supporting women and children in Badin District”, was implemented over 2.5 years. It began in October 2014 and ended in March 2017. This final programme evaluation was commissioned by Feed the Minds and conducted by GLOW Consultants Private Limited. The evaluation is based on the criteria of the Organisation for Economic Co-operation and Development’s Development Assistance Committee or specifically: relevance, efficiency, effectiveness, sustainability and impact.

The programme was implemented in 100 villages in Badin District where there were no trained government health service providers available. It directly supported 5,400 community members to improved access to health services. The overall objective was to increase the survival rate and health of pregnant women and their children in rural communities in the targeted villages. By the time the final programme evaluation was conducted all of the activities were concluded. For this evaluation, the team randomly selected eleven villages representing 11% of the sample size. A total of 28 Focus Group Discussions were conducted with 258 participants (190 women) as well as 21 Key Informant Interviews.

Key Findings:

Women’s Health Committees (WHCs): A total of 100 Women’s Health Committees were established in the 100 programme villages with a total of 700 members. Dropout of members of committees was around 7-10% and was mainly as a result of committee members moving to another area due to marriage or their husband finding another job. This turnover was mostly observed with younger women and those belonging to poor households as they were more likely to change their homes/villages in search of livelihood opportunities. WHCs were provided with one main induction training followed by several awareness sessions over the course of the programme. WHC members found the training and sessions useful. They identified multiple occasions on which they were able to apply the training. WHCs were meeting at least once in a month. Besides these regular meetings WHCs also met on an ad-hoc basis whenever a specific need arose to respond to a MCH case or emergency. WHCs selected Health Literacy Facilitators and Community Midwives and supported them in their work. A key success of the WHCs was the setting-up of a fund to support poorer women to access MCH services. On average current fund amounts within committees was around PKR 1,000. With the help of this fund 4-6 women were directly supported in each of the villages visited during the evaluation.

Men’s Support Groups (MSGs): A total of 100 Men’s Support Groups were established in all 100 programme villages, each with 20 members or a total membership of 2,000. These men had an active interest in contributing to the betterment of women and children’s health in their villages. Unlike the WHCs, turnover in the MSGs was comparatively high, averaging at around 30% of the membership. This turnover was associated with the migration of men to other areas to seek employment opportunities. The evaluation team did not come across cases where new members were inducted to fill the gap. Thus, as a whole, no replacements were engaged and MSGs continue to function with an operational membership of 15 or so. At the same time, whenever the former members returned to their homes on holidays or for good, they re-joined the MSGs.

vi | P a g e Final Programme Evaluation

DFID

MSG members were provided with training on Mother and Child Health (MCH), advocacy and lobbying. Also at the start of the programme training was organized for MSGs on their roles and ways in which they could play their role effectively. MSGs were meeting monthly, usually on Fridays after Muslim prayers when most businesses were closed. Different issues of common interest were discussed in these meetings. The MSG members were contributing PKR 10 to PKR 100 per member per month to the fund for MCH cases. The main challenge MSGs faced related to their interactions with government officials and politicians during their lobbying and advocacy activities. Specifically individuals in positions of authority were not taking them seriously due to their poverty and lack of social clout in the community.

Community Midwives (CMs): The programme engaged and trained a total of 100 Community Midwives. On average CMs had completed ten years of schooling. Once CMs were selected they were provided with specialized training. The training had a theory based component and a practical component. The WHCs played an important role in community acceptance of CMs by advocating for their role and the health benefits derived from visiting a CM. This facilitated CMs to perform their functions with ease. The CMs were provided with smart phones for the recording of patient data and for referrals to government health services. The CMs found the phones useful but had experienced some difficulties using them. CMs were referring patients to BHUs and hospitals. At the end of a pregnancy, at the time of successful delivery, CMs were given on average PKR 2,000 voluntarily by families. 3,521 women benefited from ANC services, 1,688 women received PNC services and 1,943 women were supported with family planning services by CMs trained through the programme. The CMs mentioned it was initially difficult to talk about family planning especially in smaller families with three to four children compared with larger families of six and above. However, over the course of the project, CMs became more comfortable talking about family planning with families of all sizes. The presence of a number of boys also helped in convincing women to adopt family planning methods.

Health Literacy Facilitators (HLFs): A total of 100 Health Literacy Facilitators were engaged and trained under this programme, one HLF per village. HLFs were selected by WHCs and on average had completed 10 years of formal schooling. Once selected, HLFs were provided with training through the programme. HLFs with a strong background in reading and writing were comfortable with the HLF training duration and contents. Overall HLFs were of the opinion that all of the modules covered in the health literacy course were good, however they found modules 1, 3 and 4 more useful. HLFs mentioned that module 2, on pregnancy related issues, was technical and difficult to follow. This view was also expressed by learners. HLFs spent from half an hour to three hours preparing prior to classes. The duration of preparation depended on the level of difficulty of the content of the class. Some HLF’s felt class size should be reduced from 32 to 20 students so that individual attention could be provided to each student. Each of the classes had 32 students including the 7 members of the WHC. 1 in 5 of the HLFs interviewed by the evaluation team had continued to deliver literacy classes beyond the end of the six month health literacy course without financial support.

Health Literacy Class Participants: A review of programme data and conversations with participants confirmed there were 32 learners in each health literacy class, thus making a total of 3,200 learners. Health literacy classes were aimed at women and girls aged 14-49 to increase their knowledge about safer health practices. The learners attended classes before noon when they were already finished with other domestic work or after 2 PM. Average attendance was around 80% with more women absent from the class during harvesting season or on festival days. Dropout was observed only when women had to move from the village. The main difficulties participants mentioned were related to their ability to balance competing household demands with class timings. Therefore class timings were scheduled in line with requests from

vii | P a g e Final Programme Evaluation

DFID literacy learners. Overall participants were of the opinion their HLF had sufficient knowledge to teach them, even though some other participants differed with these views. To assess the level of skills learners had acquired, participants were asked during the evaluation to narrate examples of information learned in class in relation to prenatal, delivery and postnatal care and to complete a health form. Overall all of the learners who participated in the evaluation were able to do so and write their names and at least 50% of learners were able to fill-out the form in an acceptable manner, reflecting the effectiveness of their learning.

Other Stakeholders: From discussions with health service providers and with indirect beneficiaries, it was evident the programme had made a positive contribution to the health of communities. Health service providers expressed the view that communities had more access to MCH services especially in non LHW coverage areas and also observed a positive impact on vaccination coverage. Health department officials also mentioned that qualified health opinion is now available to expecting and lactating mothers in communities where that was not the case before. The District Health Official mentioned the reduction in diarrhoea incidence in programme villages as a result of this intervention. Local communities who indirectly benefited from the programme mentioned that family members can access health services more easily and have more information about health related issues in their communities.

Relevance: The programme was relevant to the needs of the targeted beneficiaries and contributed towards the achievement of the Millennium Development Goals (MDGs) in the district.

Effectiveness: The programme helped in reducing infant mortality and maternal mortality. Infant mortality for programme villages reduced to 40 / 1,000 live birth as compared with 106 deaths per thousand live births (Sindh Multi Indicator Cluster Survey data for 2014). Maternal mortality for programme areas was calculated at around 100 / 100,000 live births as compared with a Maternal Mortality Ratio for Sindh Province overall of 314. The programme achieved 100% of its key targets such as: 3,521 women of reproductive age, pregnant and post-natal women were provided with one or more ANC, 1,688 women with one or more PNC and 1,943 women were supported with family planning services.

Efficiency: In terms of overall efficiency the programme delivered all results mostly on time and within the agreed budget. This is despite the fact that activities took slightly longer than planned to be fully rolled out at the start of the programme. The programme delivered results presenting high value for money and with limited time and resources was able to extend its interventions to 100 villages.

Sustainability: The programme was able to leverage additional financial resources from communities. This enabled the targeted communities to provide health related services to mothers and children in their neighbourhoods. From a long term sustainability perspective some MSGs and WHCs will continue to grow and may become a Civil Society Organisations. New learning and knowledge gained through this programme will be further disseminated to other people by programme participants. In terms of sustainability of health literacy interventions, it is likely that learners will retain their newly acquired literacy skills.

Impact: The programme provided women with an opportunity to manage their own affairs and to make decisions by themselves in matters which affected them. As access to mother and child healthcare was a major need, the programme helped in delivering this service. Neither the government nor any other NGO was delivering these services to women and children in these areas.

viii | P a g e Final Programme Evaluation

DFID

Lessons Learnt and Recommendations

. The total number of WHC members (i.e. seven) was sufficient in comparatively smaller villages up to 70 households. In large villages or in villages with internal community disputes it was not sufficient. In the future the programme may wish to increase the number of WHC members to ten or create another WHC thus ensuring reach to all the population in target villages.

. Quarterly two-day refresher training should be provided for WHC members.

. Repetition of key messages for MSGs should take place at least quarterly after the first three months, so that the messages could be reinforced.

. In future programme design there is a need to include specific measures to tackle the weak linkages between the programme and government health facilities. This could include issuance of health care to beneficiaries from programme areas which would result in instant recognition and access to the history of the patient in the hospital and at other government health services.

. CMs and other stakeholders, especially doctors, were of the view that the duration of the practice component of CM training was short and should be extended at least by another two weeks. CMs should also be provided with additional refresher training. During the practical part of their training CMs should be given more opportunity to do things by themselves instead of mainly relying on observation.

. Further training and refresher training should be provided to CMs on how to use the mobile phones. Changing the language to Sindhi would also help in increasing the usage of smart phones both for recording data on patients and for seeking support from doctors.

. HLFs were selected on the basis of the number of years of education with the expectation that this would correspond with a particular level of literacy. In practice this was not the case as the quality of education in Sindh is poor and grades do not necessarily correspond with actual reading and writing skills. There is a need to review selection criteria for Health Literacy Facilitators to include proficiency in reading and writing and therefore ensure they have the necessary literacy level to teach effectively in class. Further, increasing duration of the health literacy course from six to nine months could further improve learners’ ability to read and write.

. There is a need to revisit the design of activities related to lobbying and re-evaluate the whole approach. Recommendations include: providing additional training on lobbying to communities, simplifying the concepts used, linking activities with local MCH issues of relevance and building networks and alliances with other like-minded actors to support lobbying of government.

. Awareness raising activities helped in improving community understanding of vital issues surrounding their health situation. However, the evaluators believe awareness raising activities covered a whole range of topics which included general health, hygiene, nutrition, access to water, access to health services, polio campaigns, school enrolment, prevention of hepatitis etc. Having fewer topics that were directly linked to

ix | P a g e Final Programme Evaluation

DFID

programme would have helped made the activities more focused and have enabled communities to have deeper knowledge of mother and child health issues.

. The programme should increase its policy level engagements by participating in provincial level MCH related meetings. These include meetings of the Provincial Technical Coordination Committee on MCH, the Provincial Training Coordination Committee to guide family planning training covering clinical and non-clinical and the Nutrition Cluster at the Provincial level in Sindh amongst others. This would facilitate sharing of information about the programme with other stakeholders and potentially enable the programme to be scaled up to other parts of Sindh.

. Keeping in mind the success of the model, the programme team should disseminate information about it and advocate with other donors to expand the programme to other areas within the district and beyond.

x | P a g e Final Programme Evaluation

DFID

SECTION 1-INTRODUCTION

1.1 Purpose of the Evaluation

GLOW consultants was commissioned by Feed the Minds (FTM) to evaluate the Department for International Development (DFID) funded programme “to improve maternal health and infant survival rates through supporting women and children in Badin District”. This DFID- funded programme began in October 2014 and ended in March 2017 (implemented over a 2.5 year period). The evaluation assessed the overall effectiveness of the programme intervention and the following key questions:

 To what extent has the programme met its objectives?  What are the lessons learned (what has worked well and what has not worked so well)? And why?  Are there any improvements in approach, methodology or activities that could be applied in future programmes?  Are there any unexpected outcomes?  What is the level of stakeholder engagement?  How does the programme rate in terms of relevance, sustainability, impact, community contributions and ownership, effectiveness and efficiency?

Figure 2: Discussions with Women Health Committee members in FtM programme in Badin District (Photo Credit: GLOW Consultants)

1 | P a g e Final Programme Evaluation

DFID

1.2 Organisation context

FTM is an international Non-Governmental Organisation (NGO) working in partnership with NGOs in developing countries. FTM responds to local needs and aspirations and uses education and empowerment to improve lives and livelihoods of communities. By working at a local level FTM believes it can reach the most marginalized people and also build the capacity of partner NGOs to improve the impact of their work in the future.

The partner organisation implementing this programme is National Rural Development Program (NRDP). NRDP is an indigenous NGO with 22 years of experience of working with rural communities at a grassroots level in Pakistan. This programme was implemented in Badin District in Sindh Province.

Sindh Province is the second most populous province of Pakistan. The 2013 UNDP report on the status of the Millennium Development Goals (MGDs) highlights high rates of maternal mortality in Badin District. In Pakistan, the maternal mortality rate is 275 per 100,000 live births with haemorrhage being the leading cause of death. These rates are 314 and 275 for Sindh and Badin respectively. The recently released data from Sindh Multi Indicators Cluster Survey (MICS)1 indicated infant mortality was 82 per 1,000 live births and under five morality was 104 deaths per 1,000 live births. Infant mortality was 106 deaths per thousand live births for rural areas in Sindh. The percentage of ever-married women aged between 15 and 49 years who reported that a Lady Health Worker2 visited and provided them with health care services was 76%3. Lack of health workers not only negatively affects mothers’ health, but also severely affects the health of their children.

To help address these issues, FTM started a programme to improve the health and survival rates of pregnant women (MDG5) and that of their children (MDG4) in Badin District. The programme aimed to train women as community midwives and peer health educators, empower them (MDG3) to access health services and to make positive changes by improving health behaviours and encouraging others to do the same. The overall aim of the programme was to increase the survival rate and health of pregnant women and their children in rural communities in Badin District.

1.3 Overview of UK Aid Direct funded activities

The overall outcome of the programme was to improve access to health services and make positive changes in health in a total of 100 villages in Badin district by directly supporting over 5,400 community members. Further, 58,000 men and women benefited from this programme. The indirect beneficiaries include men and women in the communities who benefited from programme activities either through increased awareness of mother and child

1 http://sindhbos.gov.pk/wp-content/uploads/2014/09/01-Sindh-MICS-2014-Final-Report.pdf 2 Lady Health Workers (LHWs) are government funded health staff based in the communities. They are paid, trained and supervised by predefined criteria. A trained LHW can provide all basic health care facilities to her own community. Supervisors are appointed to supervise the work of LHW. 3 http://sindhbos.gov.pk/wp-content/uploads/2014/09/01-Sindh-MICS-2014-Final-Report.pdf

2 | P a g e Final Programme Evaluation

DFID health and general health issues or through improved health services which otherwise they would have not received. The 100 targeted villages were divided into two cohorts, each with 50 villages. Activities were carried out in the first 50 villages during Phase 1 of the programme and then expanded to include another 50 villages in Phase 2. The evaluation covered villages from both of these phases.

The programme had specific outcomes in six areas related to this overall outcome as follows:

1. Number and proportion (%) of women of reproductive age, pregnant and post-natal women who receive the following MCH services with a trained provider: (a) 1 or more ANC ; (b) 1 or more PNC ; (c) family planning; 2. Number and proportion (%) of total births attended and referred by trained health personnel; 3. Number and proportion (%) of total women and girls consuming at least two meals a day that contain protein and iron-rich vegetables; 4. Total women and children, disaggregated by boy/girl who have received vaccinations and type of vaccination – Polio, BCG (children), Tetanus (women). 5. Number of villages reporting improvements of resources/services for women’s reproductive health attributable to WHC/MSG initiatives; 6. Number and proportion (%) of (a) total households that purify drinking water and wash hands before food preparation and eating; (b) No. of cases of children, disaggregated boy/girl, with diarrhoea reported in the last 30 days.

The programme had five outputs as follows:

 Establishment of 100 proactive village-based Women’s Health Committees (WHCs) that coordinate health education programmes and access to community midwifery services for the most marginalised women.  Women and girls are more knowledgeable about options for safer health practices.  Establishment of a network of trained community midwives who provide antenatal, delivery and post-natal care to pregnant women in their villages  Men’s Support Groups (MSGs) are established as a forum for awareness raising for men on MCH.  Communities have strengthened capacities to advocate for improved health services, particularly related to maternal and child health.

To capture progress against these outputs and outcomes, indicators were identified and targets set for each output and outcome indicator. Using the programme logical framework, the evaluation team assessed the degree to which programme indicators were achieved in terms of overall improvements in the community health. Please refer to Annex 1 for detailed questions derived from OECD DAC criteria which were the basis for the evaluation of this programme.

3 | P a g e Final Programme Evaluation

DFID

SECTION 2: EVALUATION METHODLOGY

2.1 Evaluation Plan The team evaluated the programme on the basis of following OECD Development Assistance Committee (DAC) criteria:  relevance,  efficiency,  effectiveness,  sustainability and  impact.

To perform this evaluation, an evaluation plan was developed as outlined below.

2.1.1 Evaluation Geographical Coverage The evaluation took place in eight Union Councils of Badin District. A map of Badin District is provided below (Figure 3) reflecting its position within the country and province. Within these Union Councils, 11 villages (list given below) out of the total 100 programme villages were randomly selected where each village had an equal chance of selection. Of these villages, six were from Phase 2 of the programme and five were from Phase 1. The evaluation team visited all 11 villages that is 11% of the total villages where programme activities were implemented.

Table 1: Sample from the List of 100 villages (Phase-I & Phase-II) Programme: HELP for Poor, Badin-Sindh Sample from the List of 100 villages (Phase-I & Phase-II)

S. # Village UC Tehsil Phase

1 Ishaque Khaskheli Seerani Badin P-2 2 Pandi Ghirano Seerani Badin P-2 3 Saleh Panwhar Khalifo Qasim Tando Bago P-2 4 Ali Muhammad Soomro Pahar Mari Tando Bago P-2 5 Ali Turk Khalifo Qasim Tando Bago P-2 6 Geno Kolhi Dehi Jarks Tando Bago P-2 7 Shaheed Mubarik Soomro MKB Badin P-1 8 Khalifo Hashim Soomro Abdullah shah Badin P-1 9 Illyas Soomro MKB Badin P-1 Sheevo Kolhi P-I 10 Kadi Kazia Badin Saleh Norhio P-I 11 Abdullah shah Badin

4 | P a g e Final Programme Evaluation

DFID

Figure 3: Map of DFID funded FTM Programme Evaluation District

2.1.2 Evaluation Inquiry Techniques

The following inquiry methods were used for this evaluation:

 Briefing with FTM and NRDP team  Desk study / literature review, review of secondary data and documents, including the programme proposal, beneficiary data, logical framework etc.  Semi-structured interviews with 21 key informants. Details are provided in Table 2 below.  28 Focus Group Discussions with 258 direct and indirect beneficiaries. Details are provided in Table 3 below.

2.1.3 Sample Size for FGDs and KIIs The evaluation team conducted a total of 21 KIIs with key beneficiaries such as: Health Literacy Facilitators, Community Midwives, health staff, other government officials and members of the NRDP project team.

Information about the key informants is provided below.

5 | P a g e Final Programme Evaluation

DFID

Table 2: Type and Number of Key Informants Key Informant Interview (KII) Index S.No Type Number Union Council Names / Village Names / Additional Information 1 BHU Staff 4 Union Council Khalifo Qasim, Union Council Pahar Mari and Union Council Abdullah Shah 2 Community Midwives 5 Khalifo Hashim Soomro, Geno Kolhi, Shaheed Mubarik Soomro, Ali Muhammad Soomro and Shevo Kohli 3 Health Literacy Facilitator 5 Haji Saleh Nahario, Ali Muhammad Soomro, Geno Kolhi, Saleh Panwhar and Ishaq Khaskheli 4 Social Welfare Department 1 District Administration 5 Executive District Officer 2 Health Department of Sindh Health and team - Badin 6 NRDP team 4 Field and Management team Total 21

As noted above, a total of 28 FGDs took place: seven were male only FGDs and 21 were female only FGDs. 7 FGDs were held with general community members and the remainder were with direct beneficiaries of the programme. A total of 258 persons participated of whom 74% were female. Details of FGD participants are outlined below.

Table 3: Information about Focus Group participants Focus Group Discussion (FGD) Index Total Number of FGDS Covered S. No Type of FGD Participants of FGD Interview Group # of Male # of Female Total # of # of # of Total No of FGD FGD FGDs Male Female Members per village 1 General Community 2 5 7 21 55 76 (Not direct beneficiary) 2 Health Literacy Class 0 6 6 0 74 74 Participants (excluding WHC) 3 Women Health 0 10 10 0 61 61 Committee 4 Mens Support Group 5 0 5 47 0 47 Grand Total 7 21 28 68 190 258

The names of villages where FGDs took place are provided in Annex 3. 2.1.4 Additional Data Sources The consultants collected data from other additional data sources. The most important amongst these was data from local Basic Health Units (BHUs). BHUs maintain records of Out Patient Departments and can provide a good overview of the health status of the population. Three BHUs were visited in Union Council Khalifo Qasim, Union Council Pahar Mari and Union Council Abdullah Shah and data on diarrhoea, vaccination coverage, antenatal care and postnatal care services was collected.

6 | P a g e Final Programme Evaluation

DFID

2.1.5 Gender Consideration All of the evaluation methods used captured gender sensitive data. All data collected in the field was consolidated and analysed from a gender perspective.

2.1.6 Data Collection and Quality Assurance The evaluation team ensured data quality through the following measures. Activity In Field Support Data Collection Tools Tools were developed by the Consultants in consultation with FTM and Development were pre-approved before use in the field Briefing with FTM The evaluation team had a briefing with FTM to understand the programme and their expectations. Field Testing of Tools The evaluation team conducted field testing of the tools on the first day in the field. No changes were made. Field Data Collection and The data was collected by the lead consultant and co-consultants. For Validation female FGDs, two expert females were engaged to help with accessing women beneficiaries. Team Composition The team consisted of two male and two female staff members. The female staff members were able to speak local languages.

A detailed quality assurance plan and the data collection tools used are in Annexes 4, 5 and 6.

2.2 Strengths and weaknesses of selected design and research methods 2.2.1 Strengths . All programmatic areas were considered and different stakeholders were consulted through interviews and focus group discussions. . Focus on the quality of the activities helped the evaluation to go beyond numbers of people trained to consider how people used their new learning. . Customised tools helped the evaluation team to ask tailored questions and to focus on the different kinds of interventions received by different segments of the community. . The potential for bias in the findings was addressed by triangulating data from multiple sources. . The evaluation team consulted different segments of the society, including Hindu minority groups. . The presence of female team members and team members with local language skills eliminated the need for a translator.

2.2.2 Weaknesses As different phases of the programme were implemented at different periods of time, in some places activities were more recent as compared to others. On the one hand this created issues with comparability, however, at the same time it gave an opportunity to contrast the two phases of the programme and reflect on overall sustainably and impact.

2.3 Summary of problems and issues encountered

There were no problems and issues encountered during the evaluation.

7 | P a g e Final Programme Evaluation

DFID

SECTION 3: PROGESS TOWARDS OUTCOMES 3.1 Overall Results Key Findings

A total of 100 Women’s Health Committees, 100 Men’s Support Groups, 100 Community Midwives, 100 Health Literacy Facilitators and 3,200 Health Literacy Learners were directly engaged and trained under this programme in the targeted 100 villages. All of these achievements were in line with the programme’s overall targets. Selection of members of WHCs and MSGs was based on their engagement in their local community and their ability to motivate people. CMs and HLFs were selected by WHCs in consultation with the programme team. They were mostly locals from the targeted communities. More information about these beneficiary types and findings specific to each are outlined below.

3.1.1 Women’s Health Committees (WHCs)

Retention Of the 100 WHCs established through the programme, 50 were established in Phase 1 and 50 in Phase 2. Each of these committees have seven members (all women) led by the committee president, general secretary and a treasurer. All of the members were 18 to 40 years old when they joined the committee. The average time since Phase 1 committees were established was almost two years whereas for Phase 2 villages it was almost one year.

Figure 4: A Community Midwife demonstrating her skills in FtM programme in Badin District (Photo Credit: GLOW Consultants)

8 | P a g e Final Programme Evaluation

DFID

Based on discussions with WHCs, the level of dropout amongst their members was found to be around 7-10% and was mainly as a result of committee members moving to another area due to marriage or their husband finding another job. This turnover was mostly observed with younger women and those belonging to poor households as they seemed to be more likely to change their homes/villages in search of livelihood opportunities. None of the WHC members in leadership positions in the eleven villages visited had moved out of their village. This could also be attributed to the fact that women with responsibilities in the WHCs were mostly middle aged and well-established in life and were thus less likely to move.

The evaluation team expected some members of WHCs to have opted out due to a lack of interest in the activities, however, this was not the case. The primary reasons for this were continuous engagement by the programme team and the frequency of WHC meetings which helped in creating bonds amongst the members. The importance and relevance women attached to their work as WHC members also played a role. The evaluation team found that WHC members enjoyed independent-decision making and the ability to take their own initiative. This gave them greater self-actualisation and increased their motivation to make their committee a success, as demonstrated through their continued regular meetings.

Selection Process As part of the selection process for WHC members, the NRDP programme team held introductory meetings, separately organised for men and women, with communities in each village. The team introduced the programme, its objectives, the committees to be formed as part of the programme, their structures, functions and the support the committees would receive. The discussions took place in Sindhi language and were facilitated by male and female programme staff. Once the programme team left, the communities in all 100 villages, including the evaluation villages, continued their discussions to identify and select WHC and MSG members. These discussions spanned from one day to one week. Age, ethnic diversity, income diversity and willingness to contribute to committee activities were amongst the factors considered as part of the selection process. At the end of these consultations, each of the programme villages had selected seven WHC members. Once this selection process concluded each of the WHCs selected or elected, depending on whether or not there was a consensus within the group, the office bearers. As each of the seven numbers of the WHC was representing different segments of the village the WHCs reflected overall diversity in their communities.

As the programme was working mostly in relatively small villages (up to 70 households) and sub villages, the total number of members per village was sufficient. At the same time, social dynamics and local conflicts within a village did play a role in the functioning of the WHCs. In the case of larger villages (above 100 households) or the presence of unresolvable conflicts within a community it was not possible for the WHC to operate optimally. Based on the villages visited, the evaluation team believe this was the case in approximately 30% of programme villages. However the evaluation team believes one of the reasons WHCs were more active and successful as compared to MSGs (discussed later in the report), was their smaller number of members. The high number of tasks and outputs they were expected to deliver given their size, required them to be on their toes and contributed to their engagement. Further, by comparison the role of MSG members was much more supportive in nature.

9 | P a g e Final Programme Evaluation

DFID

Training Once WHC members were selected, they were provided with training in the following topics: community mobilization, management skills, resource mobilization, leadership skills, advocacy, networking and building public opinion. After the main training, several two hour follow up sessions were spread throughout the programme period. WHC members mentioned they found the training useful and identified multiple occasions in which they were able to apply it. This was especially in the case of book keeping with regard to managing financial contributions made by different individuals, including MSG members, and expenditures incurred for the treatment of women and children. In terms of community mobilisation training, WHC members were of the opinion they already knew how to talk to women, however they were not aware of how to have a collective voice for a common cause. It was the first time they were able to apply this skill within a specific framework. WHC members highly appreciated management and leadership training they received as part of this programme. These skills were used to organise health literacy classes and monitor the work of Health Literacy Facilitators and Community Midwives.

During the evaluation WHC members clearly demonstrated the usefulness of these trainings as they were maintaining meeting minutes and financial records. When participants were asked which aspects of the training they did not like, they indicated components of the training which involved reading and writing. This was primarily due to the fact many of the WHC members did not know how to read and write when programme activities started. At the same time, these women mentioned even though they did not like taking notes it did not mean these training were not important. To overcome the literacy issue associated with writing meeting minutes or financial book keeping, they assigned these responsibilities to members of the group who could read and write with ease. Most of the notes were taken in Sindhi language.

Organisation

All the WHCs which the team visited met at least once in a month. Some of the WHCs which the evaluation team interacted with had set aside a specific day of the month for the meeting (e.g. 1st Tuesday of every month). Beside these meetings WHCs also met on ad-hoc basis whenever a specific need arise. Meetings were called by the president of the committee. At regular meetings WHC members discussed different issues such as the health of mothers and children, immunisation, needs in the area, hygiene related issues etc. Unlike regular WHC meetings, which covered a variety of subjects, emergency WHC meetings were mostly related to giving financial approval for the use of funds to cover expenses related to specific cases of mothers and children or to discuss the way forward for an emergency situation. The evaluation team noticed even though meeting of the minutes were maintained for regular meetings, for emergency meetings no minutes were maintained. The programme team later explained that adding this requirement for emergency meetings may delay the response such as the release of funds.

Throughout the intervention, the programme team provided continuous support to WHCs and visited on average once in a month. During their visits the team discussed the committees’ progress, challenges they were facing, current and future plans and how to make their interventions more relevant and effective. They also facilitated information sessions on different topics such as health and hygiene, the importance of boiling water, hand washing techniques. In almost all these visits, interaction with WHCs was led by female staff whereas towards the end of the intervention, when rapport was built, male team members of the

10 | P a g e Final Programme Evaluation

DFID programme were also able to interact with WHCs and other female beneficiaries. Visits by the programme team acted as a stimulus for the WHCs to continue their activities and strive for better performance. It also provided an opportunity for the WHCs to discuss their ideas with programme staff. Thus, on the whole, these visits strengthened interventions in the targeted villages and helped the newly founded committees to ground themselves before programme support could be withdrawn.

Public Awareness & Fundraising Activities One of the functions of WHCs was to organise public awareness activities. These public awareness activities were on health and hygiene and linked with events like government polio campaigns, International Women’s Day etc. As such these activities were initiated by external stakeholders including the programme team, government officials and health department staff amongst others. The evaluation team did not come across any example of a public awareness campaign self-initiated by a WHC. When WHC members were asked why they were not initiating a public awareness activity, they mostly linked it with the lack of knowledge and resources. In addition to this, the evaluation team believe it could be linked with the lack of importance associated with these public awareness activities by the committee members as they could not see an immediate benefit from it.

During FGDs with general community members, WHCs and MSGs, however, it was evident that the dissemination of health information through informal means was taking place. This was reflected particularly in terms of knowledge acquired by community members who were not direct beneficiaries of the programme. Examples of information disseminated through informal means included knowledge around mother and child health issues, the importance of hygiene for the maintenance of health, the importance of vaccination for mothers and children, the support available through WHCs and MSGs for expecting and lactating mother in the communities etc.

A key success of WHCs was the setting-up of a fund to support poorer women to access MCH services. Contributions by WHC members were made monthly and ranged from PKR 10 to PKR 50 per person per month. The contributions were collected and recorded in the register. This register was accessible to everyone. The contribution itself was kept either by the president, general secretary or the treasurer. The evaluation team has seen financial records where up to PKR 5,000 was collected in the fund before it was given out to cater for the needs of other community members. On average, current funds with the committees was around PKR 1,000.

Decisions on who should receive the funds was mainly taken in emergency committee meetings. The money was used to cover initial costs with regard to transportation, doctors’ fees and medicine amongst other things before a woman’s family could make alternative arrangements such as selling livestock or arranging a loan for the medical treatment. In the fund allocation decision, no specific criteria were used and it was based on the need of the family. The evaluation team asked WHC members what if someone should get the money that may not need it, the response from the WHC members was unanimous saying they were locals and knew best who was poor and in need of support. The evaluators believe the current practice should continue as the community is the best judge as to whom should receive funding or in-kind support. Beside the cash support, the in-kind support provided was mainly related to transportation. On average in each village with the help of this fund, 4-6 women were directly provided with in-cash support. The evaluation team also noted there were no

11 | P a g e Final Programme Evaluation

DFID issue with regard to trust deficit amongst committee members. This suggests if WHC members continue asking for contributions this activity is likely to continue in the future.

Support to HLFs & CMs WHC members played an active role in the identification and selection of Health Literacy Facilitators. As programme villages have low literacy levels, WHC members mentioned it was difficult to find suitable HLFs. In at least one case, the HLF was selected from the neighbouring village. In all the villages visited WHCs were happy with the HLF selection process as well as with the selected HLF. Once the HLF was selected, WHCs played an active role in establishing and supporting literacy classes in their villages. WHC members went from house to house to mobilise women and girls to join the classes and select learners from the aspiring candidates. WHCs also selected the location for the health literacy class, helped the HLF in agreeing on a schedule for the classes with the students and monitored her performance. In all the villages visited, WHC members were also part of the classes, which gave them first-hand knowledge of the performance of the HLF and progress made in the classes. One of the challenges which WHCs faced with the administration of the classes was ensuring continued attendance of learners especially during key farming seasons such as the harvesting season. Also individual women had different preferred timings depending for example whether they had children in school, work commitments of their husbands etc. This required a flexible schedule to enable the maximum number of participants to attend.

WHC members also played a key role in the selection of Community Midwifes in all 100 villages. This involved identifying potential candidates who were: within the age bracket of 18 to 40, could read and write and had an interest in becoming a midwife. Those with prior experience were given preference. Once CMs were selected and trained, WHCs facilitated their work by identifying potential MCH cases and also supported CMs and patients to get to doctors when needed. A key challenge with regard to the work of CMs was related to establishing their position in the village as qualified advisors on health related issues. As some of these CMs had no prior experience in MCH and had never handled pregnancy cases, it took specific advocacy by WHCs to convince community members to consult CMs.

Changes in Health Provision Commenting on the major changes in the community’s behaviour in terms of maternal health or health in general as a result of the programme, members of WHCs were of the opinion that seeking timely advice from doctors and CMs were the major changes in their communities. The WHCs also mentioned they had no one to consult with regard to MCH related issues previously and had to either travel to neighbouring villages or rely on their own experience. However this was not the case any longer with the presence of CMs. From the perspective of changes in the government’s maternal health or general health provision in the targeted areas, in general there were no changes. Changes were found in only two to three of the 11 villages the team visited. These changes were in relation to: increased attendance of a doctor in the health facility in terms of the number of hours spent as well as his/her presence at the health facility, increase in the frequency of immunisation campaigns and in the availability of medicine at the Basic Health Unit.

The biggest challenge mentioned by WHCs in relation to improving maternal health in the targeted villages was the absence of lady doctors in the closest health facilities as women were reluctant to consult male doctors for gynaecology related issues. WHCs also mentioned that the non-availability of doctors at night made the situation worse. The absence of proper

12 | P a g e Final Programme Evaluation

DFID transport facilities in the villages to transfer patients at night was another key challenge which still hinders their ability to deliver effective support to pregnant mothers. At the same time, progress towards resolving these issues, or at least knowing possible solutions to address these challenges (for example having the phone number of an ambulance or the contact details of a doctor) are some of the major achievements which WHCs were happy to claim. When asked about their future, WHCs were of the opinion they would continue to function.

3.1.2 Men’s Support Groups (MSGs)

Retention A total of 100 Men’s Support Groups were established in all 100 programme villages, each with 20 members. Members had to be between 18-40 years of age and have an active interest in contributing to the betterment of health for women and children in their villages. The selection process for members and the executive (a president, general secretary and treasurer) was the same as that used for WHCs. Unlike WHCs, turnover in the MSGs was comparatively high, averaging at around 30% of the membership. This turnover was associated with the movement of men to other areas to seek employment opportunities. The evaluation team did not come across cases where new members were inducted to fill the gap. Thus, as a whole, no replacements were engaged and MSGs continue to function with an operational membership of 15 or so. At the same time, whenever these original members returned to their homes on holidays or for good, they re-joined the MSGs. Overall MSGs were representative of different segments of society.

Training At the start of the programme, training was organized for MSGs on their role and how to play their role effectively. This included training on how to take minutes of meetings and how to record funding contributions. Unlike with the WHCs, it was comparatively easy to find one to two men who could read and write and thus it was easy for them to maintain their records. That said the evaluation team found no significant difference in quality between the minutes of the meetings of MSGs as compared to WHCs. MSGs were also provided with training on advocacy and lobbying. This training lasted for two days. The focus was on identifying and agreeing common asks, identifying key decision makers, approaching key decision makers and lobbying techniques amongst others. This training was to provide the basis for the MSGs’ advocacy and lobbying work. Further, these MSGs participated in information sessions around health and hygiene, mother and child health care, water and sanitation, pregnancy amongst others. These sessions usually took place once a month and lasted for around two hours. The time of each session was optimal as the evaluation team believe expanding it further would have resulted in the loss of interest from the membership.

MSGs were of the opinion the training and information sessions were good as they provided an opportunity to learn new things especially about issues which affect their lives. Insistence on using boiled water for personal consumption is one practice which could be clearly attributed to the training received from the programme. The ability and willingness to help women and children who were in need of assistance was another measure which could be attributed to this programme. Aspects of the training MSGs did not like mainly stemmed from the use of difficult language and concepts which they could not fully understand. An example they provided was the training and information sessions on lobbying. They were also finding it difficult to conceptualise how their collective voice could bring about change.

13 | P a g e Final Programme Evaluation

DFID

Organisation MSGs were mostly meeting monthly, usually on Fridays after Muslim Friday prayers when businesses were closed. The meetings were called by the president. The date of the meetings, in most cases, was predetermined. Different issues of common interest were discussed in these meetings such as health and hygiene, immunisation campaigns amongst others. Emergency meetings were called in cases where urgent support was required for a woman or child in the village. These meetings usually happened at the request of WHCs and led to a specific action towards the end of the meeting. MSGs mentioned the importance of the support role they play in enabling women seek timely assistance.

Activities From the perspective of organising and participating in public awareness activities, MSGs’ activities were mainly related to the general public campaigns run by the government or other institutions such as the programme itself. The evaluation team did not come across any notable example where an MSG took the lead in organising a public awareness campaign. This suggests weak capacity to self-initiate public awareness campaigns without any support from external sources. From a lobbying and advocacy perspective, the main challenge MSGs faced related to their interactions with government officials and politicians. MSGs mentioned that government officials and politicians were not taking them seriously despite their continuous efforts.

As was the case with the WHCs, the MSGs were regularly contributing to a fund to support poorer women to access MCH services. MSG members were contributing PKR 10 to PKR 100 per member per group. Some unemployed members were exempt from the contribution. The average contribution was PKR 50 per person per month. Once collected, the money was handed over to the WHC or was kept by the treasurer. Decisions on the allocation of funds were taken in consultation with WHCs and funds were mainly used to cover transportation, medicine and other related costs. MSGs also had developed an emergency response plan for pregnant women. The plans were very simple in nature and contained: phone numbers of the drivers who were supposed to transport women in cases of emergency, the address of the nearest health facility where support could be solicited and the contact numbers of doctors.

MSGs engaged in informal dissemination of health information, especially to other men in the community. This dissemination related to issues of health and hygiene, the importance of health services for women and the immunisation of children amongst others. MSG members also mentioned that they discuss health related issues with women, especially their wives. This included family planning which they now felt more confident to discuss than before.

Support to WHCs With regard to support from Men’s Support Groups to the Women Health Committees, there were two different types of support WHC members received. One related to contributions to the fund. As outlined above, MSG members were contributing regularly to this. In addition the money donated by women was also obtained from men as they usually hold cash within the household. Even in households where women were keeping cash for their husbands/ fathers/sons, women were seeking permission from them before contributing to the fund. Thus support from MSGs was crucial for the success of the fund. The second type of support which MSGs provided was logistical support for women when the WHCs requested it. This included arranging transportation as well as accompanying women to hospital who may not

14 | P a g e Final Programme Evaluation

DFID necessarily be their close family members4. One area where MSGs could have supported WHCs more but did not was related to awareness campaigns. This could be linked to lack of interest by both WHCs and MSGs in awareness related activities as they could not see any immediate benefits arising from them. At the same time, this cooperation took place in cases of broader public awareness activities such as supporting government run polio campaigns.

Feedback on the Programme When MSGs were asked if CMs were giving a good service, they provided positive feedback. The justification MSGs provided for this was the fact CMs were now available within their own villages and women do not have to travel to other villages to consult with traditional midwives. MSG members mentioned four advantages of having a CM based in their village. Firstly it reduces time required to consult them; secondly it is more cost effective; thirdly it ensures timely availability of advice as CMs were easily accessible; and finally, it increases the frequency of consultations women can have in relation to their health. When MSGs were asked if they believe the mobile phone referral system was working well, they were of the opinion it was a good initiative, however, it could not be fully utilised as CMs were finding it very difficult to operate the phone and to write messages in English. MSG members said whenever and wherever it possible, they provided support to CMs to draft and send messages.

In terms of key changes in the communities behaviour related to maternal health or health in general as a result of the programme, MSGs were of the opinion this was mainly in recognising the importance of maternal health. MSGs believed programme communities were more aware of why, when and how to respond to the needs of women, especially during pregnancy and after delivery. At the same time despite these improvements, the absence of basic health services continues to be challenging as government was unable to make a meaningful difference in terms of service delivery in these villages. According to MSG members, these persistent challenges include lack of availability of health facilities, of doctors in general and lady doctors in particular.

MSGs acknowledged the support they received from the programme team. This included a visit every month, training and awareness sessions on different topics which also usually took place every month, encouragement to participate in different events organised by government, support in relation to record keeping amongst others. MSGs were of the opinion these visits should continue, even if only once in a quarter, so that they would have an opportunity to consult with the programme team when in need of support.

3.1.3 Community Midwives (CMs)

Selection & Training The programme engaged and trained a total of 100 Community Midwives, one per village. On average, CMs engaged in Phase 1 were part of the programme for the last two years, whereas CMs engaged in Phase 2 were part of the programme for the last year. On average,

4 Normally, in the local cultural context, only a woman’s husband or immediate family members of her husband may accompany the pregnant women to a hospital, especially during delivery.

15 | P a g e Final Programme Evaluation

DFID

CMs had ten years of schooling but still faced problems in relation to literacy. In some cases CMs had prior experience in MCH where they worked as assistants to their relatives who were Traditional Birth Attendants. 2 out of 5 CMs the evaluation team met with had no prior experience and knowledge of MCH related issues. As outlined above, CMs were identified and engaged by WHCs members. In cases of more than one qualified candidate, WHCs conducted an interview with the potential CMs to understand their interest in the role and also consulted the programme team. CMs identified their personal and professional motivations as being to learn a new skill and learn a livelihood while at the same time contributing to the welfare of their community. At the start of the programme one of the challenges with the selection of CMs was the literacy level, however, this was balanced with previous experience of MCH. In addition the programme team explained that where low literacy levels were identified in relation to some CMs, WHCs collaborated with the HLF to support the CM or a relative was identified to support the CM to effectively use their smartphones, draft text messages and enable them to keep proper record of MCH services.

Once the CMs were selected, they were provided with specialised training with a theory based component and a practical component. On the theory side, CMs learned basic knowledge of MCH related issues. On the practical side, they were provided with orientation on how to do physical examinations and conduct deliveries. The practical component of the training took place at the main hospital in Badin district. The training for CMs was only for one month whereas government provides similar training to Lady Health Workers for 18 months. During discussions key informants from the government as well as doctors suggested refresher training should be added so that CMs would have the opportunity to build on their new learning. This is in addition to the current refresher training which is part of the existing programme. The CMs were provided with a midwifery kit. There were no missing items in the kits checked during the evaluation.

Smartphones & Referrals The CMs were provided with smart phones for keeping records of patients as well as for making referrals. Referrals of patients were made via phone where patient data and health questions were shared with doctors who provided a response via SMS messages. The CMs found it useful, even though some mentioned they were facing difficulties using the phones. As noted above, CMs experiencing challenges received help from other family members who could use the phones to keep records and send referral information. During key informant interviews, doctors who were receiving these messages said they found CMs very enthusiastic about smart phones and keen to use them, however, they were making many mistakes while drafting the messages. On average, a CM was using their mobile phone for referrals at least twice in a week. The CMs were receiving responses to all their queries from within a few minutes to a few hours. The issue of no responses to SMS messages was not raised by CMs during the evaluation, even when they were specifically asked about it. Not surprisingly some of the CMs were finding it difficult to read the messages which they received in response to theirs. To overcome this issue, family members of some of the CMs were trained by the programme on the smartphone referral system, which reduced challenges associated with it. The programme team also spoke several times with CMs to make them at ease with the phones, but this was only partially successful. Other issues which affected usage of the smart phones were battery life and the ability of CMs to buy credit for sending text messages to doctors.

16 | P a g e Final Programme Evaluation

DFID

CMs were referring patients where needed to BHUs and hospitals. During the evaluation, it was revealed that the five CMs interviewed conducted 51 deliveries with a further 26 high risk deliveries being referred to BHUs or hospitals. At the BHUs and hospitals, there was no follow up on the referrals made. However, whenever possible the referring CM would also accompany women to the health facility. At the health facility, the patient would be treated like other visiting patients. At the same time, the referred pregnant woman would be in a better position to explain her condition as compared to other patients due to prior consultations with the CM. The CMs mentioned they had no formal communication or referral system with the local BHU or hospital, however with passage of time they were recognized by the BHU and hospital staff and also had become more familiar with the processes at these facilities. Thus it helped them navigate the administrative processes more easily as compared to those visiting for the first time. All deliveries in the villages were handled by trained professionals.

At the end of a pregnancy, at the time of successful delivery, each CM was given on average PKR 2,000 voluntarily by the family. The amount of money depended on family income status, whether or not it was the first child in the family, whether a boy or girl, the level of support the CM provided to the family amongst other factors.

MCH Services Provided When CMs were asked how many times they visited someone or someone visited them the average was at least once in a day. These visits included a full check-up as well as informal discussions. Once a woman becomes pregnant, a CM visits her at least once in two weeks. During Antenatal Care (ANC), CMs described providing support with regard to: folic acid, vitamin D, general nutrition and diet, food hygiene and lifestyle factors that may affect a woman’s health or the health of her baby, such as hard labour in the home or on the farm. CMs also look for any complications and refer these patients to the doctor for at least two full check-ups. A key challenge with regard to ANC was the affordability of ultrasound by the communities. The CMs were of the opinion if a discounted rate for ultrasound could be offered for mothers it would further improve screening. They also take the history of patients such as any complications or infections in a previous pregnancy or delivery and help women to monitor high blood pressure. The CMs also keep an eye as to whether there are any babies in the family with an abnormality.

17 | P a g e Final Programme Evaluation

DFID

Figure 5: A Key informant interview with an FTM programme participant in Badin District (Photo Credit: GLOW Consultants)

Once a baby is delivered in hospital or at home, the CMs are actively engaged in Postnatal Care (PNC). As part of this care, CMs check the health of the mother and new-born and counsel the mother on: family planning methods, breast feeding, nutrition, hygiene, rest and infant care, including cord care. The CMs also advise the mother on danger signs such as excessive bleeding, fever, painful urination amongst others and help with child immunisation. On average, according to feedback from communities visited, every seven out of ten women and children in areas covered by the CMs has received immunisation. Previous to the programme intervention, this percentage was in the range of 40%. As such there were no specific challenges with PNC. In none of the villages the evaluation team visited, were there any reported cases of a child’s or mother’s death over the last three years.

CMs mentioned it was difficult to talk about family planning, especially in smaller families of three to four children, however, it was comparatively easier to talk about it in larger families of six or more. The presence of a number of boys also helped in convincing women to adopt family planning. The CMs mentioned men were shy to talk about family planning whereas women were even willing to talk about it in the presence of other women. As a whole, CMs believe there is a changed attitude amongst families with regard to discussing family planning and practicing it. Currently at least one in every five women dealt with by CMs was opting for family planning and there were no major difference across the villages visited by the evaluation team. Poverty, cultural taboos, lack of understanding and the desire

18 | P a g e Final Programme Evaluation

DFID to have more children, especially sons, were amongst the key factors which stopped women adopting family planning.

CMs were of the opinion their work would not have been possible without active support from WHCs and MSGs. This support included: convincing women to consult CMs, helping with finding a suitable place in the communities where CMs could provide services to women, arranging financial support for women who needed it and covering some of the costs needed for CMs’ work. Where needed, WHCs and MSGs were accompanying CMs to the hospital along with the patient. As far as the programme team was concerned, they were visiting CMs usually once in a month which provided an opportunity for CMs to discuss their work as well as brainstorm on issues they was facing. The programme team also helped in creating networks with doctors for the CMs. As there was a clear and demonstrated need in the community and the services provided by CMs were well received and also paid, the CMs were of the opinion they would continue their services beyond the end of the programme.

3.1.3 Health Literacy Facilitators (HLFs) Selection & Training A total of 100 Health Literacy Facilitators were engaged and trained under this programme. Each village had one HLF. On average, HLFs engaged in Phase 1 were part of the programme for the last two years, whereas HLFs engaged in Phase 2 of the programme were part of the programme for the last year. The 5 HLFs interviewed during the evaluation had completed on average ten years of schooling. Specifically one had completed 14 years of schooling, one 12 years of schooling, two had completed eight grades and 1 HLF who had completed just five grades was selected as there was no alternative. None of the HLFs the evaluation team met with had any prior experience in MCH. Once selected, HLFs were provided with training. HLFs with a strong background in reading and writing were comfortable with the training duration and content. In each phase the HLFs received 22 days training: 12 days initial training with 6 days and 4 days follow up training. Within each phase the 50 HLFs were divided in two batches of 25 students each. The content of the training focused on: adult literacy techniques, health issues covered in the health literacy course, reading, writing and challenges of class management. HLFs were also trained on post assessment of literacy learners, how to attain feedback from learners on classes and on overall activities delivered. Based on the HLFs’ feedback, the training duration was not sufficient for those with lower grades or weak academic backgrounds.

Health Literacy Classes Preparation for each of the health literacy classes taught took from half an hour to three hours depending on the level of difficulty of the content of the module. Overall, HLFs were of the opinion all modules were good; however they found Modules 1, 3 and 4 more useful. These modules dealt with: health, hygiene and sanitation; antenatal care and general MCH issues including rights and services available to mothers and children; issues related to family planning, postnatal care and the dietary requirements of women and girls. The modules were taught over a six month period. They believed students found Module 2, which dealt with pregnancy related issues, more difficult to follow. This view regarding the difficulty of Module 2 was expressed by learners also. The primary reason for this was the fact it was very technical. From a richness of content perspective, HLFs were of the opinion that all important

19 | P a g e Final Programme Evaluation

DFID topics were covered and there was no need for any additional topics to be included in the health literacy course. When asked about specific difficulties faced when facilitating classes HLFs responded by saying some of the students arrived late which affected other learners as well as the tempo of the class. To overcome this issue, class timing was changed at least twice to accommodate competing priorities of students and ensure their timely arrival. The timing of the classes varied from village to village but mostly classes took place around 11 am or 2 pm. Average attendance was around 80% which is an acceptable level of attendance with reference to an adult literacy programme and also in the context of Badin where the majority of women are busy in fields, doing chores in their homes or other activities. More women were absent from the class during harvesting season or during an ongoing festivity in the village.

Another challenge HLFs faced was that different participants had varied learning abilities and different prior skills in reading and writing. This made facilitation of classes much more difficult. Some of the HLFs also mentioned having 32 students in the class was on the higher side and this number should be reduced to 20 students so that individual attention could be provided to each one of them. This individual attention is an important consideration as the learners were receiving special support from their husbands to stay out of the house and sit in class5.

Challenges & Sustainability The HLFs acknowledged how the WHCs helped them to address different issues they faced and that the presence of WHC members in classes as learners, helped them find timely solutions. This included: changing the venue of the class to a place with less disturbance, agreeing on a time which was suitable for the majority of the learners, motivating learners to follow class timings amongst others.

Once the six months course concluded, 1 of the 5 HLFs interviewed and visited by the evaluation team had continued their literacy classes even without any support from the programme. The class included mostly new learners as well as some of the original learners who wanted to further improve their reading and writing skills. Unlike the original class which had 32 students, the new class had around 15 students. The continuation of classes was made at the specific request of students as well as the personal motivation of the HLF. Some of the HLFs were of the opinion they would continue with their activities in the future and may even charge a fee for it. One recommendation which HLFs made was in relation to having separate practice books. Currently the four modules have workbooks inbuilt into the textbooks. As the learners were new to writing, this resulted in significant deterioration in the quality of the books and reduced their lifespan. According to HLFs by separating the textbooks from the workbooks, it would be possible to re-use the textbooks beyond the current six month period.

5 In the local context, a woman has to obtain permission from her husband or the male head of the household to leave the house or take a break from household duties. Once classes were finished, women were expected to return home immediately in order to complete remaining work in the home or in the fields.

20 | P a g e Final Programme Evaluation

DFID

3.1.4 Health Literacy Class Participants

A review of programme data and conversations with participants confirmed there were 32 learners in each of the health literacy classes, thus making a total of 3,200 learners. Health literacy classes were aimed at women and girls aged 14-49 to increase their knowledge about options for safer health practices. Overall class size was adequate given the number of eligible, interested learners. However in some of the larger villages there were 15-20 potential learners who were unable to join due to the lack of space in a class. As such there were no issues from the families in relation to women and girls joining the health literacy classes, however, women and girls faced some problems adjusting their schedules so that they could join the class and perform other responsibilities at home and in the field. According to some participants, this created tensions at home on some occasions but they were able to talk to their family members and find a solution. As a whole it was possible, even though not necessarily easy, to adjust to class hours along with other engagements at home.

The learners indicated that classes usually started before noon but after they had already finished with other domestic work or after 2 p.m. when they had already served food to all the family members and thus had no pending tasks at home. Health literacy class participants also mentioned the support and flexibility they received from HLFs who adjusted class timing and thus enabled them to join the classes. In most villages the six month health literacy course started in May/June and ended by December. Each class lasted for around two hours. The participants mentioned they preferred not to miss any classes as it was difficult to catch up later. When they missed a class due to guests arriving at home, social activities in the villages or work in the fields, they consulted fellow learners to help them catch up. A key challenge with this approach was the fact that fellow students were not very strong and the learners also did not have their books with them at home. (Reasons for this are outlined below). This affected their ability to work on their own after class time. When participants were asked if they are aware of any dropouts from the classes, their response overall was negative. At the same time, they mentioned that at the very start of the course some women did drop out as they were finding classes very difficult and had also expected to receive some benefit. These learners were replaced immediately. Afterwards, there were no dropouts and all learners continued with the classes.

Health literacy class participants referred to the four books used in the classes. They were of the opinion books 1, 3 and 4 were easy whereas book 2 was difficult. Participants explained they did not take the 2nd book outside of the class for the fear of loss and given it has some pictures related to sexual health. At the same time they mentioned they had no issue with using the key learnings in their practical life. In terms of usefulness, participants found these books addressed the practical issues they faced and this was a key strength which enhanced their utility. One of the main difficulties participants mentioned was how difficult they found it to learn to read and write and that they needed more practice. Overall, participants were of the opinion their HLF had sufficient knowledge to teach them. Some other participants differed with these views and shared even though HLFs were more informed then them, the programme should have brought in someone from outside the local area with higher capacity. This was especially true for villages which had no qualified female even though in those cases HLFs had been selected from other nearby villages. In terms of politeness and punctuality, there were no complaints. In the villages where literacy classes continued after the 6 month course, participants mentioned these additional classes were started by the HLF

21 | P a g e Final Programme Evaluation

DFID at the request of some of the learners who either could not join in the first place or moved out of their villages at the start of the course.

To assess the level of skill learners acquired, during the evaluation participants were asked to narrate examples related to prenatal, delivery and postnatal care that were covered in class. Overall all of the participants who participated in the evaluation were able to do so. Some of the messages they mentioned included: the importance of professional medical advice before pregnancy, vaccination related information, possible complications with delivery, important considerations during delivery, care of mothers and babies after delivery amongst others. As a final part of the skills assessment during the evaluation, the learners were asked to fill out basic health information about family members as well as writing their names. During this practical demonstration, at least 50% of the learners were able to fill out the form in an acceptable manner, reflecting the effectiveness of their learning. Even though this percentage is lower than the target, we believe it is still a major achievement and that in fact the target set in the proposal might be over-ambitious. The proportion of participants who could write their names only was higher and that in itself is a high achievement. These learners still needed more practice and time before they could achieve the desired proficiency level. Learners from the health literacy classes participated in different campaign activities related to the programme such as immunization campaigns.

There was no direct support provided by the programme team to these learners. However during programme team visits to villages, they met with the leaners and listened to them to understand how they were progressing and to find out if they were facing any specific challenges.

3.1.5 Other Stakeholders

From interviews with key informants like BHU staff and Health Department staff and FGDs with indirect beneficiaries, it was evident that the programme had made a positive contribution to the health of the communities. Government officials and BHU staff were of the opinion they now have more access to communities especially in non-LHW coverage areas. They also mentioned that they had observed a positive impact on vaccine coverage. One specific aspect health professionals mentioned was the availability of qualified health opinion to expecting and lactating mothers in communities which was not the case before. The District Health official mentioned that the programme should be expanded to the other parts of the district, as it is not covering needs in all areas. They also mentioned that some of the Union Councils of the District are facing high prevalence of Hepatitis C but the programme was not addressing this emergency-like situation through provision of medicines and other supplies as it was focusing only on awareness raising around health and hygiene.

Both the doctors and health officials mentioned that further capacity enhancement of the CMs would help them cover the gap arising from lack of coverage of government LHWs and that it would be advantageous to scale up the programme’s mother and child healthcare services including both ANC and PNC. Doctors in BHUs specifically referred to the increase in vaccination coverage as well as in parents who are seeking assistance to improve the nutritional status of their children. The BHU data indicated an increase in the number of patients seeking MCH services. This included vaccinations for mothers and children, ANC and PNC. There were even requests from women in programme communities for provision of folic acid and multi-vitamins. There has also been a reduction in diarrhoea cases as could be seen from BHU data when compared on a year to year basis.

22 | P a g e Final Programme Evaluation

DFID

Community members who benefited indirectly from the programme stated that family members can access health services more easily than before. They mentioned having more information about health related issues in their communities. The wider community was aware of the programme including WHCs and MSGs and individuals were able to narrate the main activities facilitated by these committees including CMs and health literacy classes. They mentioned these committees were providing financial and in-kind support to needy households and to those in need of external assistance especially with reference to mother and child healthcare. Communities mentioned that there is now easier access to and availability of health services in the village and to mobile referrals for specialized advice from doctors. Moreover, communities unanimously mentioned that there is reduction in diarrhoea incidence and that they are more aware of the risk associated with Hepatitis C and how it is spread from person to person. Community members indicated that there is no change in hand washing practices as they were already doing it as part of their religious obligations, however, at the same time, community members mentioned they are boiling water now before consumption. Those who mentioned that they are still not purifying their water identified the lack of purifying pills or firewood for boiling water as the two main reasons for not opting for safe water.

3.2 Assessment of Accuracy of Reported Results

The evaluation team cross-verified results reported by the programme team with the randomly selected sample from the eleven villages it visited. The evaluation team did not come across any discrepancies or inconsistencies in data which could put into question the accuracy of reported results. From a data quality assessment perspective, the evaluation team believe that data for all reported results had the necessary validity and reliability. 3.2 Relevance

The programme was relevant to the needs of targeted communities and contributed towards the achievement of the Millennium Development Goals (MDGs) in the district. The programme helped in promoting women’s empowerment (MGD 3) through the provision of training to local women as community midwives and peer health educators to promote access to health services.

This programme targeted the poor and marginalised communities of the district, including Hindu minorities. Hindu minority communities made up almost 15% of the total beneficiary group. This programme was relevant to the needs of the targeted communities as it was implemented in the poorest and less served areas of Badin and was catering for the needs of women and children. The programme mainstreamed gender equality in the design and delivery of activities. Overall 90% of the direct beneficiaries were women and children; men were only engaged through Men’s Support Groups.

23 | P a g e Final Programme Evaluation

DFID

Figure 6: A Community Midwife before explaining different items of her midwifery kit in Badin District (Photo Credit: GLOW Consultants)

The programme targeted only those areas without a Lady Health Workers programme which is a key intervention by the Government of Pakistan to improve mother and child healthcare in the province. The targeted population under this programme in general did not have the ability to meet the medical needs of their families and had limited abilities to save. Women also did not have easy access to cash when they needed money for medical care.

One of the main needs of communities in the targeted villages was timely access to health care services for women and children. To respond to this need, the programme focused on: organising communities, increasing community based knowledge of mother and child health issues, improving access to health services through collective action and enhancing communities own capacities to respond to the needs of women and children. This included establishing emergency plans and a support fund to enable women cover costs associated with accessing health care. Through this programme, the availability of CMs within the communities increased the likelihood of women having check-ups on a regular basis, thus helping in early deduction of complications in pregnancy and immediate referrals as needed. The availability of professional advice through CMs also helped women seek timely support from qualified doctors. The needs of the communities did not evolve over the programme life as they continue to face a similar challenging context. At the same time, the communities’ acceptance of different programme interventions did evolve, such as their willingness to know more about available family planning options.

24 | P a g e Final Programme Evaluation

DFID

3.3 Effectiveness

The programme helped in reducing infant mortality and maternal mortality. Based on data from the mid-term and final evaluation, collected during FGDs with WHCs and general communities and KIIs with CMs, the infant mortality rate across all evaluation villages averaged at 40 / 1,000 live births while the maternal mortality for programme areas was calculated at 100 / 100,000 live births. This is compared with an infant mortality rate of 106 deaths / 1,000 live births for rural areas of Sindh as reported in the Sindh Multi Indicator Cluster Survey (MICS)6 2014 and a Maternal Mortality Ratio of 314 for Sindh Province overall7.

At community level, programme stakeholders had continuous communication with government vaccination teams which allowed for the creation of synergies like in relation to vaccination campaigns. Data collected from three different sources, the BHUs, FGDs with WHCs and interviews with CMs, indicates that vaccination coverage in programme areas increased from 40% at the start of the programme to 75% by the time of the final evaluation. As the government had not started any new programme or interventions, if DFID funding had not been available it would not have been possible to extend healthcare services to mothers and children in the targeted villages.

The programme adapted its intervention strategy throughout the implementation period as and when needed. Specific examples of this included: organising additional training for CMs on smart phone usage, training family members of CMs on smart phone usage and dropping the kitchen garden activity in Phase 2 of the programme intervention. These amendments helped the programme to apply learning from the early stages of implementation to improve delivery.

From an outcomes perspective, the programme achieved most of its targets and exceeded some significantly. For example 3,521 women of reproductive age, pregnant and post-natal women were provided with one or more ANC; 1,688 women with one or more PNC and 1,943 women were supported with family planning services. All of these are in excess of targets. 2661 women had their births attended and referred by trained health personnel. 1,566 women (as per project M&E data at 31st December 2016), 15,198 boys and 15,969 girls have received vaccinations as a result of the efforts of this programme. Further, based on project M&E data & FGDs with communities, 3,637 households were purifying drinking water and washing hands before food preparation and eating whereas before the programme over 98% of the households were not doing so. Finally the number of cases of children with diarrhoea reported in the last 30 days decreased by 80%.

The key barriers that affected delivery of the programme were social where government and other decision makers were unwilling to take demands from the community seriously because of community members’ low capacity and education base. The communities needed more time to fully engage government officials in this programme as they would have become more capable, empowered and informed in their demands over a longer time period.

6 http://sindhbos.gov.pk/wp-content/uploads/2014/09/01-Sindh-MICS-2014-Final-Report.pdf 7 Pakistan Demographic and Health Survey (PDHS)

25 | P a g e Final Programme Evaluation

DFID

3.4 Efficiency

The programme delivered results presenting high value for money. With limited time and resources, the programme was able to extend its interventions to 100 villages. Some training components were conducted using government facilities. Local CMs and HLFs were used which reduced the chances of dropout and turnover. Technical resources related to senior management, including financial management, were jointly shared with another FTM programme in Narowal thus reducing costs assigned to this DFID funded programme. In addition, procurement of medical kits for CMs was done jointly for both programmes which helped in achieving economies of scale. From an administrative perspective, the ratio of staff to beneficiaries / villages was slightly below the desired level. This limited the frequency of programme team visits to a village to only once a month. Staff salaries were in line with the market price for national level organisations for similar positions. The stipends given to CMs during training were also appropriate and were not above the overall average for similar activities. For operational costs, the programme was using its own vehicles and was also sharing vehicles with other programmes which helped in reducing charges.

In terms of overall efficiency, the programme delivered all the results mostly on time and on budget against agreed plans as described in programme documents. This is despite the fact that activities took a slightly longer time than planned to be fully rolled out, especially at the start of the programme. The programme was able to achieve all its targets, as articulated in the logical framework. The programme was able to fully understand cost drivers and manage them effectively with no significant impact on overall performance requirements. For example, staff numbers were kept at a minimum and staff were offered market salaries which neither created a pull factor nor forced staff to actively search for other work opportunities. Similarly, as noted above, procurement was centralized and done in bulk thus bringing down prices. Each vehicle was assigned a specified quantity of fuel which was duly monitored through a logbook and helped keep fuel costs under control.

3.5 Sustainability

Over its duration, the programme was able to leverage additional financial resources from the communities. This enabled the targeted communities to provide health related services to mothers and children in their neighbourhoods. Committees reviewed during the evaluation were collecting funds on a voluntarily basis from their respective communities and had collected up to PKR 5,000 per committee. In terms of in-kind support, those members who had space in their house or compound gave it up for free for the delivery of health literacy classes. Some of the members were able to provide transportation to mothers and children to health facilities. As communities were seeing a clear and visible positive impact from their contribution to the fund for timely provision of assistance to mothers who need it, the evaluation team believes this practice will continue. In terms of sustainability of the health literacy interventions, it is likely learners will retain their newly acquired literacy skills in relation to reading and writing especially those who continue to practice. There is willingness in the wider community to maintain good health practices. This was reiterated by community members during the evaluation. These factors are likely to contribute to the sustainability of health literacy knowledge. Further, 1 in 5 of the programme villages visited is maintaining literacy classes.

26 | P a g e Final Programme Evaluation

DFID

The programme trained three government doctors to support with the smartphone referral system and this will help with its sustainability. At the same time, sustainability of the system will depend on the willingness of the doctors to respond to queries from CMs. During the evaluation, doctors indicated they will continue to do so if they believe there is an incentive for them (e.g. more private patients) or may even continue in the spirit of volunteerism. Women who were trained as community midwives were from the villages in which they work and are likely to continue to work in these areas. They will thus provide sustainability to the intervention. It is also likely that as local women are now more aware of the importance of health, they are likely to continue availing of health services with the help of CMs and other health service providers in the main population centres in Badin District. The existing linkages between the CMs and government health workers and services will add value to the sustainability of this programme. Thus it is likely programme activities will go beyond the programme duration. Women and men from the wider community who participated in the evaluation explicitly stated their expectation that the WHCs and MSGs will continue their activities in the long term. From a long term sustainability perspective, some of the MSGs and WHCs will continue to grow and may become Civil Society Organisations. The evaluation team believes this will depend firstly on the interest and pro-activeness of the community leads of these committees. Secondly there are other organisations that could come to these villages and continue to interact with these established groups thus giving them new opportunities to develop and grow. As the evaluators did not come across any self-initiated example of a public awareness activity by the WHCs and MSGs interviewed during FGDs, it may be less likely they will initiate such activities unless there is change in the activity design. This would include for example making the activities more focused and relevant to MCH so that communities can see direct linkages between raising awareness and potential benefits for them.

3.6 Impact

Capacity building of community groups was a prominent and important feature of this programme. The programme capacity building measures focused on two aspects – management related issues such as how to lead, run meetings, take minutes etc. and awareness related topics such as the importance of mother and child health, the need to boil water, how hygiene could contribute to reducing disease prevalence. The programme provided women in particular with an opportunity to manage their own affairs and to make decisions by themselves in matters which affected them.

27 | P a g e Final Programme Evaluation

DFID

Figure 7: Key Informant Interview with BHU staff in Badin District (Photo Credit: GLOW Consultants)

As access to mother and child healthcare is a major need, the programme helped in delivering this service. Neither the government nor any other NGO was delivering these services to women and children in these areas. As noted above, there were no Lady Health Workers in the targeted villages. Similarly, BHUs did not have any female doctors and there were no locally available professional health advisers. Consultations with doctors were not possible as they were based in bigger towns and required a fee for their services. The programme made health information more available and improved access to health services. The availability of financial support for mothers and their children was another important achievement of this programme as it made it possible for people to access health services which otherwise they would not have been able to afford.

Lastly the programme helped in increasing awareness of and preventative action on Hepatitis C. There was a very high prevalence of Hepatitis C (around 30%) in programme villages but

28 | P a g e Final Programme Evaluation

DFID it was not considered as a key public health priority during the programme design stage. However once communities learned about the disease in the course of programme implementation, they took the issue very seriously and took measures to help with further prevention.

29 | P a g e Final Programme Evaluation

DFID

SECTION 4: CONCLUSIONS

4.1 Summary of achievements against evaluation questions

S. No Evaluation Criteria and Evaluation Questions Key Achievements 1 Relevance 1.1 To what extent did the grantee support achievement The programme supported MDGs 3, 4 and 5. towards the MDGs, specifically off-track MDGs? 1.2 To what extent did the programme target and reach the The programme assisted poor and marginalised communities of Badin district poor and marginalized? including Hindu communities. This included 2,000 men who were supported through MSGs, 700 women through WHCs, 100 CMs, 100 HLFs and 3,200 participants in Health Literacy classes. There were numerous other women and children who were assisted by Community Midwives. 1.3 To what extent did the programme mainstream gender Over 90% of the programme’s direct beneficiaries were women and children. equality in the design and delivery of activities (and or Hindu minority communities made up almost 15% of the total beneficiary other relevant excluded groups)? group for this programme. 1.4 How well did the programme respond to the needs of The programme focused on organising communities, increasing community target beneficiaries, including how these needs evolved based knowledge of mother and child care issues, improving access to and over time availability of health services in the communities through collective action and enhancing communities own capacities to respond to the needs of women and children. This included the establishment of emergency plans and a support fund to enable women to access health care who otherwise would be unable to do. The availability of professional advice through smart phones helped women seek timely support from qualified doctors. 2 Effectiveness 2.1 To what extent are the results that are reported a fair and All the results related to the programme present a fair and accurate record of accurate record of achievement? the achievements. They accurately reflect programme activities, outputs and outcomes as described in the logical framework and different quarterly reports. 2.2 To what extent has the programme delivered results that With limited time and resources, the programme was able to extend its

30 | P a g e Final Programme Evaluation

DFID are value for money? Specifically: services to 100 villages. Training for CMs was conducted using government facilities. Local CMs and HLFs were used which reduced the likelihood of  How well did the programme apply value for dropout and turnover. Technical resources relating to senior management, money principles of effectiveness, economy, including financial management, were jointly used with another FTM efficiency in relation to delivery of its outcome; programme in Narowal thus reducing costs assigned to this DFID funded  What has happened because of DFID funding that programme. The ratio of staff to beneficiaries / villages was slightly below wouldn’t have otherwise happened; and the desired level, which limited the team’s ability to visit a village more than  To what extent has the programme used learning once in a month. Staff salaries were in line with the market price for national to improve delivery? level organisations for similar positions. The stipends given to CMs during training were also appropriate and were not above the overall average for similar activities. As the government did not start any new health interventions in this area, if DFID funding had not been available it would not have been possible to extend healthcare services to mothers and children in the targeted villages. The programme adopted its approach throughout the implementation period as and when needed. Specific examples of this included: organizing additional training for CMs on smart phone usage, training of family members of CMs on smart phone usage and dropping the kitchen garden activity in Phase 2. These amendments enabled the programme to apply learning to improve delivery. 2.3 What are the key drivers and barriers affecting the The first key barrier that affected delivery of the programme was social: delivery of results for the programme? where government and other decision makers were unwilling to take demands from the community seriously because of the low capacity and education base of community members. Thus the communities needed more time to engage government officials. A longer time period would have enabled community members to become more capable, empowered and informed in their demands. 3 Efficiency 3.1 To what extent did the grantee deliver results on time and In terms of overall efficiency, the programme delivered all the results mostly on budget against agreed plans? on time and on budget against agreed plans and as described in programme documents. This is despite the fact that the activities took slightly longer to be rolled out than planned, especially at the start of the programme.

31 | P a g e Final Programme Evaluation

DFID 3.2 To what extent did the programme understand cost drivers The programme was able to fully understand cost drivers and manage them and manage these in relation to performance effectively with no significant impact on overall performance. Some of the requirements? key costs were staff salaries, procurement, vehicle fuels and stipends. To cater for these costs, staff numbers were kept at a minimum, staff were offered market salaries and procurement was centralized to get better prices. 4 Sustainability 4.1 To what extent has the programme leveraged additional The programme was able to leverage additional financial resources from resources (financial and in-kind) from other sources? communities through cash contributions to emergency funds and in-kind What effect has this had on the scale, delivery or support through provision of space for project activities and transport for sustainability of activities? women to health facilities. This enabled the targeted communities to provide health related services to mothers and children in their neighbourhoods. 4.2 To what extent is there evidence that the benefits From a long term sustainability perspective some of the MSGs and WHCs delivered by the programme will be sustained after the established through the programme will continue to grow and may become programme ends? Civil Society Organisations. Literacy learners are likely to retain their newly acquired skills in relation to reading and writing especially those who continue to practice. Government doctors trained through the programme on the smart phone referral system will contribute to its sustainability. Though it will depend on the willingness of the doctors to respond to queries from CMs, the doctors interviewed stated they would continue to do so. New learning and knowledge gained through this programme will continue to be held within the community and will be further disseminated to other people. 5 Impact 5.1 To what extent and how has the programme built the The capacity building of civil society was a prominent and important feature capacity of civil society? of this programme. The capacity building measures focused on two aspects: management related issues such as how to lead, how to run meetings, how to take minutes etc. and knowledge about MCH and general health issues. 5.2 How many people are receiving support from the All the people receiving support through this programme would not have programme that otherwise would not have received received this type of support without it. support? 5.3 To what extent and how has the programme affected The programme helped in increasing awareness of Hepatitis C which has high people in ways that were not originally intended? prevalence (around 30%) in these communities. Hepatitis C was not

32 | P a g e Final Programme Evaluation

DFID considered as a key public health priority during the programme design stage. However over the course of the programme as communities were made aware of this and other diseases, they began to take the issue of Hepatitis seriously and undertook measures to help with further prevention

4.2 Summary of achievements against rationale for UK Aid Direct funding

The programme aimed at improving the health situation of women and children in Badin by increasing knowledge of MCH. Prior low take up of MCH services was the result of a combination of factors including: lack of literacy skills, lack of access to qualified health service providers and lack of access to information. Women were keen to gain health knowledge and basic literacy skills to enable them to understand and implement measures to reduce sickness and death, particularly from preventable conditions. However they did not have relevant opportunities to do so prior to the programme. Similarly once provided with opportunities to access family planning information and services through the programme, women were keen to take this up as they were aware that large family sizes were a contributing factor to high levels of poverty. The programme provided 3,521 women of reproductive age, pregnant and post-natal women provided with one or more ANC; 1688 women with one or more PNC and 1943 women were supported with family planning services.

Further health and poverty are inextricably linked. By reducing maternal mortality, FTM ensured fewer children were orphaned and in need of support from extended families. By reducing anaemia and improving women’s health during and following pregnancy, women were able to continue with their existing responsibilities, caring for family members and engaging in individual or collective family activities to bring in household income. There were also fewer instances of loss of household income due to costly emergency visits to hospitals and medication. By ensuring pregnant women and children were vaccinated (for tetanus, measles and TB), the incidence of these conditions was reduced thus enabling valuable household resources to be used elsewhere. As programme interventions targeted the most marginalised among the communities, it contributed to improved health of the poorest. Two of the main achievements of the programme were both the increase in the provision of and take up on MCH services within target villages.

The table below outlines achievements to December 2016 against milestones for each of the programme indicators. This is based on data compiled by the programme team. This data was cross verified by the evaluation team during the field visit.

33 | P a g e Final Programme Evaluation

DFID

Sr.# Progress Indicators Milestone 3 Achievements till Dec 31, 2016

Outcome: Women and children in Badin district have greater access to reproductive health services and adopt health-seeking and disease-prevention behaviour

Outcome 1: Number and proportion (%) of women of reproductive age, pregnant and a. 1,072 w, 80% a. 3521 w, 100% post-natal women who receive the following MCH services with a trained provider: (a) b. 670 w, 50% b. 1688 w, 100% 1 or more ANC ; (b) 1 or more PNC ; (c) family planning c. 670 w, 50% c. 1943 w, 100% Outcome 2: Number and proportion (%) of total births attended and referred by 804 w, 60% 2661 W, 100% trained health personnel Outcome 03: Number and proportion (%) of total women and girls consuming at least 9,000 w and 3,850 g, 50% of total 7802 w, 3260 g, 85% of total women

two meals a day that contain protein and iron-rich vegetables women and girls & Girls

Outcome 04: Total women and children, disaggregated by boy/girl, who have received

vaccinations and type of vaccination b. 1566 w, 15198 b, 15969 g, 100% b. 7,740 w, 3,543 b, 3,339 g, 43%

20 villages (11,600 people or 36,00 w, 48,00 m, 1640 b, 1, 560 g) (final Outcome 05: Number of villages reporting improvements of resources/services for 25 villages (14,500 people or 4,500 survey is still pending, and the results women’s reproductive health attributable to WHC/MSG initiatives w, 6,000 m, 2,050 b, 1,950 g) may lead to achievement of this target)

a. 3637 hh (32733 people), 100% b. Incidence of diarrhoea reduced to Outcome 06: a) Number and proportion (%) of (a) total households that purify a. 2,700 hh (26,100 people), 45% 20% in 100 villages in last 30 days drinking water and wash hands before food preparation and eating; b) No. of cases of b. Incidence of diarrhoea reduced to (Other contributing factors like children, disaggregated boy/girl, with diarrhoea reported in the last 30 days 8% in 100 villages in last 30 days access to clean drinking water, Open Defecation Free status of the village which prevented achievement of this target. Output 01: WHCs in each village established and functioning

34 | P a g e Final Programme Evaluation

DFID

Output 1.1: Number and proportion (%) of WHCs: (a) set up with full complement of a. 100 WHCs (700 w), 100% a. 100 WHCs (700 w), 100% members; (b) have in place health literacy classes/sessions and community midwifery b. 100 WHCs (700 w), 100% b. 100 WHCs (700 w), 100% services; (c) facilitating >9 meetings per year c. 60 WHCs (350 w), 60% c. 60 WHCs (420 w), 60% Output 1.2: Number and proportion (%) of WHCs (a) trained in community mobilization and management skills, leadership skills, resource mobilization, a. 100 WHCs (700 w), 100% a. 100 WHCs (700 w), 100%

networking, advocacy and building public opinion and (b) able to demonstrate b. 80 (560 w), 80% b. 80 (560 w), 80% maintaining meeting minutes and financial literacy Output 1.3: Number of (a) WHCs setting up systems for poorer women to pay for CM a. 80 WHCs (560 w), 80% a. 100 WHCs (700 w), 100%

services and (b) marginalised women supported by (i) cash and/or (ii) in-kind charity b. (i) 50 w and/or (ii) 450w b. (i) 128 w and/or (ii) 1,017 w Output 02: Women and girls (aged 14-49) are more knowledgeable about options for safer health practices.

Output 2.1: Number and proportion (%) of women (a) trained as health facilitators; a. 100 w, 100% a. 100 w, 100% (b) delivering a full 6 month programme of integrated health and literacy; (c) b. 90 w, 90% b. 100 w, 100%

undergoing/undergone follow-up training and (d) delivering 3 out of 4 thematic areas c. 100 w, 100% c. 100 w, 100% developed d. 85 w, 85% d. 100 w, 100%

Output 2.2: (a) Number and proportion (%) of women and girls attending health a. 2,666 w and 534 g, 100% a. 2,666 w and 534 g, 100% literacy classes; (b) Number of women and girls able to provide 3 recommendations of b. 2,132 w and 426 g, 80% b. 2,132 w and 426 g, 100% good practice related to 3 of the 4 thematic areas covered Output2.3: Number and proportion (%) of women and girls (a) able to write own a. 1,896 w and 389 g, 75% (75% for name and fill out basic health information about family members and (b) able to try out a. 1,866 w and 373 g, 70% writing names, and around 50% for

2 new recommended practices in relation to the 4 thematic areas covered in 6 months b. 1,999 w and 400 g, 75% filling the form) of course b. 1,999 w and 400 g, 75% Output 2.4: Number of (a) examples of women and girls taking collective action on health; (b) women and girls participating in collective action such as awareness walks, a. 120 examples a. 113 examples corner meetings, etc. to raise awareness and knowledge of others in their village; or to b. 2,000 w and 400 g b.2100 w and 460 g lobby government officials to provide extra support; or to take special action in the local community that improves the status of MCH issues. Output 03: Network of trained CMs established and functioning a. 100 w, 100% a. 100 w, 100% b. 100 villages (58,000 people or Output3.1: (a) Number and proportion (%) of women trained as CMs; (b) number of b. 100 villages (65356 people or 18,000 w, 7,760 g, 8,240 b, 24,000 villages covered by CMs and (c) number and % of women attending follow-up training 21500 w, 9760 g, 9,540 b, 24,556 m) m) c. 100 w, 100% c. 100 w, 100%

35 | P a g e Final Programme Evaluation

DFID

a. 9, 270 w, 55% (Target will be most Output3.2: Number and proportion (%) of total women (a) who can name their a. 10,800 w, 60% likely achieved once this quarter’s community midwife and (b) give two reasons why CMs should visit her when pregnant b. 7,200 w, 40% results are available). b. 7,802 w, 45% Output 3.3: (a) Number of referrals requests made by CMs through Smartphone a. 800 a. 2661

system and (b) proportion of referrals responded by doctors b. 40% b. 40% Output 04: MSGs are established as a forum for awareness-raising of men in relation to MCH a. 100 MSGs (2,000 m), 100% a. 100 MSGs (2,000 m), 100% Output 4.1: Number and proportion (%) of MSGs (a) set up; (b) receiving training on b. 100 MSGs (2,000 m) 100% b. 100 MSGs (2,000 m) 100% MCH; (c) receiving training on advocacy and lobbying and (d) meeting >9 times per c. 100 MSGs (2,000 m) 100% c. 83 MSGs (1660 m) 90% year d. 50 MSGs (1,000 m) 50% d. 100 MSGs (2,000 m) 100% Output 4.2: Number and proportion of trained men who can identify and explain (a) a. 1,800 m, 90% a. 1,890 m, 95% two key issues affecting health of mothers and children and (b) two actions they can do b. 1,800 m, 90% b. 1,810 m, 90% to support their wives/female relatives

Output 4.3: Number and proportion (%) of MSGs which have in place an emergency 80 MSGs (1,600 m), 80% 100 MSGs (2000 m), 100% response plan for pregnant women Output 05: Communities have strengthened capacity to advocate for improved health services, particularly for mother and child health a. 250 workshops (or similar event) a. 287 workshops (or similar event) Output 5.1: Number of (a) MCH knowledge-sharing workshops with local b. 6,250 participants (2,000 w, 1,750 b. 8, 610 participants (2960 w, 2,350 stakeholders and (b) workshop participants disaggregated by men, women, boys, girls g, 2,250 m, 250 b) g, 2,950 m, 350 b) a. 56 Output 5.2: Number of (a) lobbying initiatives organised by communities with local a. 40 b. 55 government for improved/additional health services (b) where WHCs and MSGs have b. 40 (This includes all kinds of public jointly organised the event activities). Output 5.3: Number and proportion (%) of community members (disaggregated 29,400 adults (12,600 w and 16,800 32,497 adults (14,700 w and 17,797 women/men) able to identify which local service improvements are being lobbied for m), 70% m), 100% by their WHC and MSG

36 | P a g e Final Programme Evaluation

DFID

4.3 Overall Impact and Value for Money of UK Aid Direct Funded Activities

Overall Impact As outlined in Section 3 above, one of the main achievements of the programme was its contribution to reductions in infant and maternal mortality rates in targeted areas of Badin. Based on data from the midterm and final evaluations, collected during FGDs with WHCs and general communities and KIIs with CMs, infant mortality fell to around 40 / 1,000 live births as compared with 106 deaths per thousand live births in Sindh. While the Maternal Mortality Ratio fell to 100 / 100,000 live births as compared with 314 / 100,000 live births for Sindh Province. This reflects significant reductions.

Value for Money The evaluation team used the DFID standard framework of economy, efficiency, effectiveness, cost-effectiveness and equality to determine value for money.

Efficiency: From an efficiency perspective, the programme strived to maximise the impact of each pound spent in the field and obtain the maximum health benefits for poor people. For this purpose, the programme continuously adapted its implementation strategy based on evidence of what was working and what was not. This was reflected in the adoption of new measures to increase usage of smart phones and the cessation of kitchen gardening activities, to prevent allocation of resources on activities with little benefit. The programme allocated only the required input for the delivery of outputs. An example of this was the allocation of exact fuel amounts for vehicles which were calculated on the basis of the average distance from the targeted villages to the office.

Economy: In terms of economy, a number of different materials were centrally purchased thus securing reduced prices as a result of bulk buying. An example is the combined purchase of midwifery kits for this programme and another similar programme to reduce costs.

Effectiveness: From an effectiveness perspective, the programme’s theory of change clearly articulated how different interventions would help in achieving the desired outcomes on health. Where elements were not working necessary corrective measures were taken, as outlined above, to ensure desired results were achieved.

Cost-effectiveness: The programme took cost-effective measures to provide value for money. An example is staff salaries which were in line with the market prices.

Equality: All marginalised groups within targeted villages had access to programme interventions and their benefits irrespective of gender, ethnicity or religion. A case in point was the deliberate targeting of a number of Hindu villages as these communities experience higher levels of discrimination and poverty.

37 | P a g e Final Programme Evaluation

DFID

SECTION 5: LESSONS LEARNT Overall the programme approach of establishing committees for local women and men, training Community Midwives and running health literacy classes worked for improving knowledge of and services related to mother and child health. This was particularly true of the Women’s Health Committees and Community Midwives whose presence increased knowledge within communities of mother and child health issues and also made these services readily available. This was not the case prior to the programme implementation. The WHC members enjoyed independent-decision making which gave them greater self- actualisation. This led to an increase in their motivation to bring about positive change in their communities. A key lesson is the successful demonstration through this programme that it is possible to deliver these services to poor communities in a very cost effective manner. Thus this programme presents a workable model for scale up to other NGOs. The ability of committees to advocate and lobby was hindered by uneven power dynamics with government officials not taking committee members seriously. A key lesson learnt therefore relates to the need to create stronger linkages between decision makers and communities.

Figure 8: Key Informant Interview with a doctor who is also part of mobile referral system in Badin District (Photo Credit: GLOW Consultants)

38 | P a g e Final Programme Evaluation

DFID

SECTION 6: RECOMMENDATIONS

Below is the summary of the key recommendations to be considered for future programme design. 6.1 Number of WHC Members Having a total of seven members on a WHC was sufficient in comparatively smaller villages of up to 70 households. However, in large villages or in villages with internal community disputes it was not sufficient as members were unable to reach to all of the households. In similar programmes in the future increasing the number of WHC members up to ten or creating another WHC would ensure interventions reached all the population in target villages. 6.2 Refresher training of WHC Members The evaluators agreed with the view of WHC members that quarterly two-day refreshers should be held, with at least two of these refreshers on topics to help them further in their work. This would include an orientation on topics covered by the Health Literacy Facilitators and work done by Community Midwives. WHCs were of the opinion this would not only enhance their own skills but would also better equip them to understand what they should expect from HLFs and CMs. 6.3 Numbers and Membership MSG The evaluation team believes there is no need to increase the membership of MSGs any further as it is representative of the whole community. Rather considerations around choosing men who are more likely to stay in the village are more relevant to resolve the issue of loss of members due to male migration for work. Identifying men whose source of income is based in the village would reduce dropout from the groups as they would be less likely to leave the area. In addition men with a local source of livelihoods would be more able to contribute to the emergency fund. 6.4 Capacity Building of MSGs MSG members and the evaluators were of the combined opinion that there should be a repeat of key messages for MSGs at least once in three months so that they are reinforced. MSGs proposed that all new lessons should take place should take place in the first three months. They suggested that after this period each of the monthly two hour sessions should comprise: one hour focused on new learning and one hour focused on re-visiting key messages from previous sessions. For their future training needs MSGs requested training in financial literacy and livelihoods opportunities. 6.5 MSG Support to WHC for Advocacy and Lobby One area where MSGs could have supported WHCs more but did not do so was in relation to awareness campaigns. This lack of cooperation could be linked with lack of interest in these types of activities among both WHC and MSG members. At the same time, this was not the case with public awareness activities involving external actors like polio campaigns which were well supported by WHCs and MSGs. 6.6 Improving Linkages with Government Health Facilities In future programme design there is a need to include specific measures which would tackle the issue of weak linkages between the programme and government health facilities. This

39 | P a g e Final Programme Evaluation

DFID would improve healthcare services to the beneficiaries through increased coordination and collaboration between programme stakeholders and health services provided by government doctors/hospitals. 6.7 Training of Community Midwives Overall the CMs and other stakeholders, especially doctors, were of the view that the duration of training for CMs was short and should be extended by at least another two weeks. CMs and stakeholders were also of the opinion that the number of refresher trainings held should be increased and that during the practical part of their training CMs should be given more opportunity to do things by themselves instead of mainly relying on observations. This is to further enhance their technical skills around deliveries. The evaluators are in agreement with these recommendations. In addition CMs suggested that they should be provided with an examination bed so that they could examine women as it was difficult to do this in traditional beds and that they should be linked up with government hospitals for supplies which are essential for any service providers. 6.8 Refreshers Related to Usage of Smart Phones MSG and WHC members were of the opinion that more training should be provided to CMs on how to use the mobile phones and refreshers should be organised. Changing the language to Sindhi would also help in increasing usage of the smart phones, both for recording data on patients and for seeking support from doctors. If the language were changed from English to Sindhi, it would significantly reduce issues faced by the CMs with usage of smart phones. 6.9 Enhancing Skills/ Qualification Requirement for HLF and Class Duration. The HLFs were selected on the basis of number of years of education with the expectation that a specific number of years of schooling would correspond with a particular level of reading and writing skill. In practice, this was not the case as the quality of education in Sindh is poor and grades do not necessarily translate to actual reading and writing ability. Therefore there is a need to link the minimum requirements for HLFs with the actual literacy skills they possess. This will ensure that selected HLFs have the necessary skills in reading and writing to effectively teach a class. Further, increasing the duration of the health literacy course from six to nine months could further improve learners’ ability to read and write. As per feedback from the evaluators, HLFs and WHCs, nine months would be more than sufficient to cover all of the text books in an efficient and effective way. Although both HLFs and learners found the textbooks more helpful than any other books they had seen before, they were of the opinion that using more graphics would make them even more user friendly. 6.10 Capacity Building on Lobbying Activities and its Logic There is a need to revisit the design of activities related to lobbying and re-evaluate the whole approach. This would include providing additional training to build the capacity of communities on lobbying activities, enhancing their skills to identify and agree on common asks, mobilising communities for these key asks, building up networks with other like- minded actors for a common cause amongst others. Some of the concrete efforts to make lobbying more effective would involve simplifying the concepts involved and linking activities with local issues especially issues relating to women’s and children’s health which are of interest and concern to these communities. Formation of alliances with other stakeholders to get the attention of government officials and politicians will yield positive results in relation to outcomes of lobbying efforts. This will also help motivate communities

40 | P a g e Final Programme Evaluation

DFID in general and WHCs/MSGs in particular as they will able to see direct linkages between lobbying efforts and improvement in service provision. 6.11 Formalising Awareness Activities Awareness raising activities helped in improving community understanding of vital issues surrounding their health situation. The evaluators believe awareness raising activities covered a whole range of topics which included general health, hygiene, nutrition, access to water, access to health services, polio campaigns, school enrolment, prevention of hepatitis etc. This lack of focus diluted the significance and potential impact of awareness activities. If they had focused on a fewer topics which were directly linked to programme this would have helped the awareness activities to be more focused. It would also have enabled communities to have deeper knowledge of mother and child health issues instead of having limited knowledge of many different topics. Further to make public awareness activities more successful, it will be important to give communities a better understanding of the logic behind them and reasons why they should be conducted in the first place. These efforts will make awareness raising activities more focused on MCH activities and make it more likely that communities will take action on their own initiative.

6.12 Policy Level Engagements The programme should increase its policy level engagements by participating in provincial level MCH related meetings. These include meetings of the Provincial Technical Coordination Committee on MCH, the Provincial Training Coordination Committee to guide family planning training covering clinical and non-clinical and the Nutrition Cluster at the Provincial level in Sindh amongst others. Organising regular workshops and commissioning research would also enable the programme to share its learning with other stakeholders and potentially enable the programme to be scaled up to other parts of Sindh.

6.13 Increase Scale of the Programme Keeping in mind the success of this model, the programme team should document, disseminate and advocate with other donors to expand the programme to other areas within the district and beyond. This will help in providing access to better health care services at reduced cost and thus help in reaching to more women and children with fewer resources.

41 | P a g e Final Programme Evaluation

DFID ANNEX 1 EVALUATION MATRIX Data Collection Instruments Desk Review of Observatio Evaluation Criteria Evaluation Questions NRDP/FTM Beneficiaries Key Informant Programme n through Staff Interviews FGDs Interviews Records Field Visits To what extent did the grantee support achievement towards the MDGs, specifically off-track MDGs?

To what extent did the programme target and Relevance: To reach the poor and marginalized? what extent does the programme To what extent did the programme mainstream respond to the priorities and gender equality in the design and delivery of needs of the target activities (and or other relevant excluded population?. groups)?

How well did the programme respond to the needs of target beneficiaries, including how these needs evolved over time

Effectiveness: To what extent are the results that are reported a What are the key fair and accurate record of achievement? drivers and barriers (factors) To what extent has the programme delivered affecting the ability of WHC & results that are value for money? To include but MSG to achieve not limited to: the overarching objective of  How well the programme applied value

b | P a g e Final Programme Evaluation

DFID

Data Collection Instruments Desk Review of Observatio Evaluation Criteria Evaluation Questions NRDP/FTM Beneficiaries Key Informant Programme n through Staff Interviews FGDs Interviews Records Field Visits poverty for money principles of effectiveness, alleviation? economy, efficiency in relation to delivery of its outcome;  What has happened because of DFID funding that wouldn’t have otherwise happened; and  To what extent has the programme used learning to improve delivery?

What are the key drivers and barriers affecting the delivery of results for the programme?

To what extent did the grantee deliver results on time and on budget against agreed Efficiency: To what extent has the plans? programme achieved its objectives? To what extent did the programme understand cost drivers and manage these in relation to performance requirements?

Sustainability: How To what extent has the programme sustainable are the leveraged additional resources (financial activities funded by the programme and and in-kind) from other sources? What has the programme effect has this had on the scale, delivery or

c | P a g e Final Programme Evaluation

DFID

Data Collection Instruments Desk Review of Observatio Evaluation Criteria Evaluation Questions NRDP/FTM Beneficiaries Key Informant Programme n through Staff Interviews FGDs Interviews Records Field Visits been successful in sustainability of activities? leveraging additional interest and To what extent is there evidence that the investment? benefits delivered by the programme will be sustained after the programme ends?

Impact: What has To what extent and how has the programme the programme built the capacity of civil society? achieved that would not have How many people are receiving support from been achieved without external the programme that otherwise would not have funding? received support?

To what extent and how has the programme affected people in ways that were not originally intended?

d | P a g e Final Programme Evaluation

DFID ANNEX 2: PROGRAMME VILLAGES AND PHASES

National Rural Development Program (NRDP) Project: HELP for Poor, Badin-Sindh List of 100 villages (Phase-I & Phase-II) District: Badin

S. # Village UC Tehsil Phase

1 Ishaque Khaskheli Seerani Badin P-II

2 Murad Ali Gopang Seerani Badin P-II

3 Muhammad Khan Pussio Seerani Badin P-II

4 Bachal Jakhro Seerani Badin P-II

5 Ahmed Abad Seerani Badin P-II

6 Pir Aalam Shah Seerani Badin P-II

7 Suliman Pusio Seerani Badin P-II

8 Haji Ismail Mallah Seerani Badin P-II

9 Ramzan Sheedi Seerani Badin P-II

10 Noor Mohammad Ghrinao Seerani Badin P-II

11 Pandi Ghirano Seerani Badin P-II

12 Hoat Moosa Photo Seerani Badin P-II

13 Misri Mandro Seerani Badin P-II

14 Umar Mallah Seerani Badin P-II

15 Mohsin Sheedi Seerani Badin P-II

16 Umar Khaskheli Bhugra Memon Badin P-II

17 Buxo Dero Bhugra Memon Badin P-II

18 Darya Khan Khaskheli Bhugra Memon Badin P-II

19 Daleel Ahmedani Khalifo Qasim Tando Bago P-II

e | P a g e Final Programme E v a l u a t i o n

DFID

20 Geno Kolhi Dehi Jarks Tando Bago P-II

21 Saleh Panwhar Khalifo Qasim Tando Bago P-II

22 Saindad Lund Khalifo Qasim Tando Bago P-II

23 Ali Murad Rind Khalifo Qasim Tando Bago P-II

24 Gul Mohammad Kapri Khalifo Qasim Tando Bago P-II Ghulam Mohammad 25 Khalifo Qasim Tando Bago P-II Chadhro 26 Qadir Bux Hingorjo Pahar Mari Tando Bago P-II

27 Abbass Sehto Pahar Mari Tando Bago P-II

28 Haji Majnu Sheedi Pahar Mari Tando Bago P-II

29 Essa Mallah Khalifo Qasim Tando Bago P-II

30 Muhammad Khan Laghari Khalifo Qasim Tando Bago P-II

31 Ali Muhammad Soomro Pahar Mari Tando Bago P-II

32 Allah Bux Solangi Khalifo Qasim Tando Bago P-II

33 Lano Machi Khalifo Qasim Tando Bago P-II

34 Gul Muhammad Khoso Khalifo Qasim Tando Bago P-II

35 Khalifo Hashim Laghari Khalifo Qasim Tando Bago P-II

36 Sajan Turk Khalifo Qasim Tando Bago P-II

37 Dilawar Ahmedani Khalifo Qasim Tando Bago P-II Haji Jan Muhammad 38 Khalifo Qasim Tando Bago P-II Kolachi 39 Giyan Chand Kolhi Khalifo Qasim Tando Bago P-II

40 Manu Bheel Khalifo Qasim Tando Bago P-II

41 Ali Turk Khalifo Qasim Tando Bago P-II

42 Ahmed Udhejo Khalifo Qasim Tando Bago P-II

43 Shahdad Khan Rind Khalifo Qasim Tando Bago P-II

44 Kandhero Bajeer Khalifo Qasim Tando Bago P-II

f | P a g e Final Programme E v a l u a t i o n

DFID

45 Budho Unar Khalifo Qasim Tando Bago P-II

46 Muhammad Younis Junejo Khalifo Qasim Tando Bago P-II

47 Ali Muhammad Khaskeli Khalifo Qasim Tando Bago P-II

48 Fazal Ahmedani Khalifo Qasim Tando Bago P-II

49 Veerji Kolhi Khalifo Qasim Tando Bago P-II

50 Soomar Junejo Khalifo Qasim Tando Bago P-II

51 Bachal Khokhar Pharmari Tando Bago P-I

52 Qasim Soomro Pharmari Tando Bago P-I

53 Haji Shafi Soomro Pharmari Tando Bago P-I

54 Gul Muhammad Mallah Pharmari Tando Bago P-I

55 H Ahmed Mallah Pharmari Tando Bago P-I

56 Jaffar Mallah Pharmari Tando Bago P-I

57 Tayab Halepoto Pharmari Tando Bago P-I

58 Pharmari Tando Bago P-I Bacho Halepoto 59 Saeed Khan Jamali Pharmari Tando Bago P-I

60 Paro Kolhi Pharmari Tando Bago P-I

61 Shaheed Mubarik Soomro MKB Badin P-I

62 Imam Bux Khoso MKB Badin P-I

63 Ali Muhammad Khoso MKB Badin P-I

64 Piyaro Khoso MKB Badin P-I

65 Sobo Kolhi Kadi Kazia Badin P-I

66 Sheevo Kolhi Kadi Kazia Badin P-I

67 Peetho Kolhi Kadi Kazia Badin P-I

68 Mir Hoat Kadi Kazia Badin P-I

69 Haji Muhammad Soomro Abdullah shah Badin P-I g | P a g e Final Programme E v a l u a t i o n

DFID

70 Sallah Norhio Abdullah shah Badin P-I

71 Khalifo Hashim Soomro Abdullah shah Badin P-I

72 Ghulam Ali Jamali Abdullah shah Badin P-I

73 Ghulam Rasool Jamali Abdullah shah Badin P-I

74 Soomar Jamali Abdullah shah Badin P-I

75 Nabi Bux Bhurghri Abdullah shah Badin P-I

76 Saleh Norhio Abdullah shah Badin P-I

77 Kando Jamali Abdullah shah Badin P-I

78 Ali Murad Chandio Abdullah shah Badin P-I

79 Sikno Soomro MKB Badin P-I

80 Moulvi Hussain Soomro MKB Badin P-I

81 Illyas Soomro MKB Badin P-I

82 Faqeer Muhammad Soomro MKB Badin P-I

83 Majeed Jaat Abdullah shah Badin P-I

84 Bachal Soomro Abdullah shah Badin P-I

85 Dariya Khan Warar Seerani Badin P-I

86 Saleh Warar Seerani Badin P-I

87 Jan Muhammad Warar Seerani Badin P-I

88 Mossa Samejo Seerani Badin P-I

89 Majno Khaskeli Seerani Badin P-I

90 Ghafor Mallah Seerani Badin P-I

91 Makhan Korejo Seerani Badin P-I

92 Seerani Badin P-I Soomar Mallah 93 Allahyo Chang Seerani Badin P-I

h | P a g e Final Programme E v a l u a t i o n

DFID

94 Ramzan Jakhro Seerani Badin P-I

95 Haji Parhiyar Seerani Badin P-I

96 Soomar Parhiyar Seerani Badin P-I

97 Humer Parhiyar Seerani Badin P-I

98 Allah Bachayo Parhiyar Seerani Badin P-I

99 Dariya Khan Parhiyar Pharmari Tando Bago P-I

100 Suleman Khaskeli Pharmari Tando Bago P-I

i | P a g e Final Programme E v a l u a t i o n

DFID ANNEX 3: FGD DETAILS

Focus Group Discussion (FGD) Index Total Number of FGDS and The Village Names where it took place S.No Type of FGD Village Name Interview Group # of # of Total # of Male Female FGDs FGD FGD 1 General Community (Not 2 5 7 Khalifo Hashim Soomro, Illyas direct beneficiary) Soomro, Ali Muhammad Soomro, Saleh Panwhar, Ishaq Khaskheli and Ali Turk 2 Health Literacy Class 0 6 6 Khalifo Hashim Soomro, Ali Participants (excluding Muhammad Soomro, Shevo WHC) Kohli, Pandi Ghirano, Saleh Panwhar and Ali Turk 3 Women Health 0 10 10 Ali Muhammad Soomro, Ishaq Committee Khaskheli, Pandi Ghirano, Saleh Panwhar, Saleh Hozio, Shevo Kohli, Khalifo Hashim Soomro, Ali Turk, Geno Kolhi and Shaheed Mubarik Soomro 4 Men Support Group 5 0 5 Pandi Girano, Khalifo Hashim Soomro, Shevo Kohli, Geno Kohli Grand Total 7 21 28

j | P a g e Final Programme E v a l u a t i o n

DFID ANNEX 4: EVALUATION PLAN

• Discussions/Meeting FTM/NRDP Network Partners in Islamabad/Over skype. Briefing

• Final Review of Proposal, Beneficiary Data, Progress Reports, Other relevant Reports etc (already shared with GLOW team) Desk Review

• Agree on Final List of Key Stakeholders and Explain any remaining questins on Evaluation Methodology and data Collection Tools Inception • Finalise the Timeline Meeting

• Travel to the field and conduct interviews with relevant team members Field Travel and Interviews

• Focus Group Discussions Field Data Collection - • Interviews with government officials, partners and/or cluster/working group leads FGDs Interviews

• Debrief workshop with FTM and/or NRDP Debriefing

• Draft Report • Final Report based on FTM Feedback Draft and Fianl Report

k | P a g e Final Programme E v a l u a t i o n

DFID ANNEX 5: QUALITY ASSURANCE PLAN Following are the quality assurance steps, which were part of GLOW evaluation plan.

Quality Assurance Steps

1. Data Collection Tools Development or Review  Tools will be developed or reviewed by GLOW Sectoral Experts i.e. Saeed/Zaki 2. Team Training for collection of quantitative and qualitative data  Qualified and experienced field team members will be engaged.  50% of the field enumerators will be females.  Enumerators from Punjab and Sindh will be engaged to ensure they have knowledge of particular sectors. A sector expert will lead the training. He / she will be facilitated by co-trainer and other experts from GLOW. 3. Testing of Tools  The data collection tools will be tested as part of the training through active participation of the data collection teams. 4. Field Data Collection and Validation  The quantitative and qualitative data will be collected by the trained survey teams.  Team Leader will supervise will monitor the field team and visit all field locations.  All filled questionnaires will be reviewed in the evening to ensure data is collected as per the agreed protocols. Repeat visits will take place to the respondents for any missing information. If a field team member is making frequent mistakes, s/he will be replaced.  Random phone calls will be made to the respondents to ensure their concerns, if any, are recorded. 5. Data Entry and Cleaning

 The syntax for data entry will be developed by GLOW.  Data entry activity will be carried out by the trained data entry operators in excel at one centralized location.  Data validator will review the whole data entry process. 6. Data Analysis / Checks

 To ensure quality of data entry / Data validators / data entry supervisors will run cross tabs.  Excel data will be shared with Concern  All reports will be analysed by GLOW team leader and Deputy Team Lead / Sr. Manager

l | P a g e Final Programme E v a l u a t i o n

DFID ANNEX 6: DATA COLLECTION TOOLS

Annex 6.1-FGD: Women Health Committee Women Health Committees Focus Group Discussions Guide Final Evaluation of DFID-funded programme “to improve maternal health and infant survival rates through supporting women and children in Badin District, Pakistan”

Dated: / / 2017

A1. Moderator name A2. District

A3. Village A4. UC A5. Tehsil

A6. Number of participants A7. Gender Male Female (Please check)

Introduction: My name is………………………….. I work for a private organization based in Islamabad. We are conducting a study to assess the benefits and lesson learnt of completed programme in your area such as (mention the name of an activity taken place in the area by NRDP) of villages). The interview will take approximately 60-75 minutes. Nothing you say will be publicly attributed to you, and your names will not be given to anyone.

Do I have your permission to record the interview? (Yes…No)

If all participants say yes, continue the interview. If no, try to motivate participants by answering their questions and explaining the importance of recording the interview.

1. When was this WHC established? How many members are there? 2. Have you all been members of the WHC from the beginning? If no, get details. 3. How were you selected as members? Please tell us more about the membership selection process? 4. Was there anyone else whom you believe should be a member? Were the number of members were sufficient? 5. Did you receive any training as WHCs? What topics were these trainings on? Which of these training have you found most useful/interesting? Which training has been the least useful/interesting? Are there areas that you need more training/skills development for WHC? 6. How did you use the training? Give us an example? Please tell us how we can improve the training?

m | P a g e Final Programme E v a l u a t i o n

DFID 7. How many times do you meet in a month? Who organizes those meetings? 8. What do you discuss or do at those meetings? 9. What kind of records do you keep and who keeps your records? 10. Have you organized public awareness activities? If yes, get details. If no, why not? 11. Have you set up a fund to support poorer women to access MCH services? If yes, please ask how this works including details of a) contributions (who contributes, how much, how often, etc.). and b) how the fund is managed (how potential women for support are identified/selected, decisions made, criteria applied). 12. How many marginalized women were supported by (i) cash and/or (ii) in-kind charity (and get details of the specific assistance given for ‘in-kind’)? What in the current amount available in the fund? 13. Have you received any support from MSG? If yes, what kind of support? If no, why not? 14. Did you organize any lobbying initiatives with local government for improved/additional health services? Did you organize any joint activity with MSG? 15. What role did you play in selection of the HFL and setting up the literacy classes? 16. Did you have a role in supporting the literacy classes once they were established? If yes, get details. 17. Did you experience any challenges with selection of the HLF and running the literacy classes? If yes get details of challenges and how they responded. Is there anything the programme could have done better in relation to this? 18. How was a Community Mid-Wife selected for the village and what was your specific role? 19. What role have you played in supporting the Community Mid-Wife once she was trained? 20. Did you experience any challenges in selecting the CMW and supporting her in her work? If yes, get details of challenges and how they responded. Is there anything the programme could have done better in relation to this? 21. What kind of support did you receive from NRDP programme staff? For example, how often did they visit and what did they do during visits. Is there anything they could have done better or differently? 22. Do you know about any examples when mobile phone referral was use? What were the main advantages of this system? What are the main challenges associated with this system? How can we make it better? 23. Have you noticed any changes in the community’s behavior in terms of maternal health or health in general as a result of the programme? Get details. 24. Have there been any changes in the government’s maternal health or general health in your area as a result of your activities or the activities or the MSG? If yes, get details of the changes and how those changes came about. 25. What are the biggest challenges you continue to face in improving maternal health and general health in your community? 26. What do you think is the WHC’s biggest achievement? 27. What are your plans for the future?

n | P a g e Final Programme E v a l u a t i o n

DFID 28. If this programme was to take place again, what changes you would like to make to it? 29. Any other comments?

Notes:

i) Fill out attendance form. ii) Check records/ Take pictures of the available document. Ask to see records for group minutes, financial books, logbooks to evidence Q.9 iii) Take pictures of any document presented iv) Note down case studies, if any.

Name of Literacy Level # Gender (M/F) Age Participants

1

2

3

4

5

6

7

8

9

10

11

12

o | P a g e Final Programme E v a l u a t i o n

DFID

Annex 6.2-FGD: MSG Men’s Support Group Focus Group Discussions Guide Final Evaluation of DFID-funded programme “to improve maternal health and infant survival rates through supporting women and children in Badin District, Pakistan”

Dated: / / 2017

A2. Moderator name A2. District

A3. Village A4. UC A5. Tehsil

A6. Number of participants A7. Gender Male Female (Please check)

Introduction: My name is………………………….. I work for a private organization based in Islamabad. We are conducting a study to assess the benefits and lesson learnt of completed programme in your area such as (mention the name of an activity taken place in the area by NRDP) of villages). The interview will take approximately 60-75 minutes. Nothing you say will be publicly attributed to you, and your names will not be given to anyone.

Do I have your permission to record the interview? (Yes…No)

If all participants say yes, continue the interview. If no, try to motivate participants by answering their questions and explaining the importance of recording the interview.

1. When was this MSG established? How many members are there? 2. Have you all been members of the MSG from the beginning? If no, get details. 3. How were you selected as members? Please tell us more about the membership selection process? 4. Was there anyone else whom you believe should be a member? Were the number of members were sufficient? 5. How many men left the groups? Can you tell us why men were leaving the group? 6. How many new members joined the group? Did new members experience any challenges when they joined? 7. Were there any challenges in having a change in membership and how was this dealt with?

p | P a g e Final Programme E v a l u a t i o n

DFID 8. Did you receive any training as MSG members? What were these trainings? Can you tell us more about the training on MCH? Received training on advocacy and lobbying? How many training took place? What was the duration of these training? 9. Was the quality of the training good? Were there any ways in which the quality of training or other aspects of the training could have been improved? 10. What aspects of the training did you find most useful? 11. What aspects of the training did you find least useful? 12. Are there any areas where you feel you need additional training/skills development? 13. How did you use the training? Give us an example? Can you tell me about some of key issues that affect health of mothers and children? 14. Can you tell me about some of the actions you have done or that other men can do to support the health of their wives/female relatives? 15. How many times do you meet in a month? Who organizes these meetings? 16. What do you do or discuss at those meetings? 17. Do you keep any records? If yes, who within the group keeps records? 18. Have you organized or participated in public awareness activities? If yes, please let us know? If no, why not? (If not covered above, ask about informal education activities in which they engage – nature, how often, etc.) 19. Have you set up a fund to support poorer women to access MCH services? If yes, please ask how this works including details of a) contributions (who contributes, how much, how often, etc.). and b) how the fund is managed (how potential women for support are identified/selected, decisions made, criteria applied). 20. How many marginalized women were supported by (i) cash and/or (ii) in-kind charity (and get details of the specific assistance given for ‘in-kind’)? What is the current amount available in the fund? 21. Do you have an emergency response plan for pregnant women? If yes, please tell us more about it? If no, why not? Have you ever used it? 22. Have you received any request for support from WHCs? If yes, what kind of support they requested for it? If no, why not? 23. Did you organize any lobbying initiatives with local government for improved/additional health services? Did you organize any joint activity with WHCs? 24. What are the biggest challenges you have faced in your work in the MSG? How did you deal with these? 25. Have there been any improvements in government’s maternal health or general health services in your area as a result of your activities or the activities of the WHC? If yes get details of improvements and how they came about. 26. Do you think the CMW is providing a good service? Do you think the mobile phone referral system is working well? Are there any areas where these could be improved? 27. Have you noticed any changes in the community’s behavior in terms of maternal health or health in general as a result of the programme? Get details. 28. What do you think is your biggest achievement? q | P a g e Final Programme E v a l u a t i o n

DFID 29. What are the challenges you continue to face in improving health in your community? 30. What are your plans for the future? 31. If this programme was to take place again, what changes you would like to make to it? 32. What kind of support did you receive from NRDP programme staff? For example, how often did they visit and what did they do during visits. Is there anything they could have done better or differently? 33. Any other comments?

Notes:

i) Fill out attendance form ii) Check records. Take pictures of the available document iii) Take pictures of any document presented iv) Note down case studies, if any.

Name of Literacy Level # Gender (M/F) Age Participants

1

2

3

4

5

6

7

8

9

10

11

12

r | P a g e Final Programme E v a l u a t i o n

DFID Annex 6.3-FGD: General Community General Community (Not Direct Beneficiary) Focus Group Discussions Guide Final Evaluation of DFID-funded programme “to improve maternal health and infant survival rates through supporting women and children in Badin District, Pakistan”

Dated: / / 2017

A3. Moderator name A2. District

A3. Village A4. UC A5. Tehsil

A6. Number of participants A7. Gender Male Female (Please check)

Introduction: My name is………………………….. I work for a private organization based in Islamabad. We are conducting a study to assess the benefits and lesson learnt of completed programme in your area such as (mention the name of an activity taken place in the area by NRDP) of villages). The interview will take approximately 60-75 minutes. Nothing you say will be publicly attributed to you, and your names will not be given to anyone.

Do I have your permission to record the interview? (Yes…No)

If all participants say yes, continue the interview. If no, try to motivate participants by answering their questions and explaining the importance of recording the interview.

1. Do you know about the mother and child health programme in your village that is taking place? If no, please tell them about the programme? 2. Are you aware of the women committees (WHC)? Are you aware of men committees (MSGs)? Do you know about the activities of these committees? If yes, can you please share your understanding with us? Can you also tell us how were they selected as a member? Did they present the collective views of the communities? 3. In your opinion, is there any improvement in the health of mother and child in your village as a result of this programme? If yes, why? If no, why not? Can you give some examples? 4. In your opinion, is there any improvement in the awareness around mother and child related issues in your village as a result of this programme? If yes, why? If no, why not? Can you give some examples?

s | P a g e Final Programme E v a l u a t i o n

DFID 5. Has there been any improvement in the general health of people in your village as a result of this programme – for example in relation to things like diarrhea, malaria? Please explain your answer. 6. In your families do you purify drinking water? For those who answer yes, can you explain how you do this? (Also establish if they started doing this as a result of the programme or for other reasons) For those who answer no, ask them to talk about why they don’t). 7. In your families do you wash your hands before preparing and eating food? (Establish if they started doing this as a result of the programme or for other reasons. For those who answer no, ask them to talk about why they don’t). 8. Are you aware of the health services offered by CM? If yes, were they useful? Are you or someone in your family using these CMs? Are they charging any fee? 9. There are community health fund to support pregnant women? Have you ever used it? Are you aware of it? 10. In your opinion, is there any improvement in the health services in the BHU at your village or UC? If yes, why? If no, why not? Can you give some examples? 11. Have you participated in any lobbying initiatives organized by communities with local government for improved/additional health services? 12. Have you participated in any other community activities to improve maternal health or overall health in the village? Get details. 13. Have there been any improvements in health or other government services in your area as a result of the programme? Ask for more details. 14. In your family do the women and girls consume at least two meals a day with protein and iron rich vegetables (you may need to explain this, give examples)? For those who answer yes, establish when they began to do this and why. For those who answer no, establish whether it is due to lack of knowledge or other factors (like poverty). 15. For women participant only: Can you name your community midwife 16. For women participant only: can you give two reasons why CMs should visit pregnant women? 17. If we have to implement this programme again, what in your opinion we should do differently? 18. Any other comments?

Notes:

v) Fill out attendance form vi) Take pictures of the available document vii) Take pictures of any document presented viii) Note down case studies, if any.

t | P a g e Final Programme E v a l u a t i o n

DFID Annex 6.4-FGD: Health Literacy Class Participants Health Literacy Class Participants (excluding WHC) Focus Group Discussions Guide Final Evaluation of DFID-funded programme “to improve maternal health and infant survival rates through supporting women and children in Badin District, Pakistan”

Dated: / / 2017

A4. Moderator name A2. District

A3. Village A4. UC A5. Tehsil

A6. Number of participants A7. Gender Male Female (Please check)

Introduction: My name is………………………….. I work for a private organization based in Islamabad. We are conducting a study to assess the benefits and lesson learnt of completed programme in your area such as (mention the name of an activity taken place in the area by NRDP) of villages). The interview will take approximately 60-75 minutes. Nothing you say will be publicly attributed to you, and your names will not be given to anyone.

Do I have your permission to record the interview? (Yes…No)

If all participants say yes, continue the interview. If no, try to motivate participants by answering their questions and explaining the importance of recording the interview.

1. Are you all participants of health literacy class? What was the total number of learners in your class? How were participants selected? 2. Did you know other women who wanted to join the class but could not do so as there was no space? 3. Did all of your family agree with the idea of you attending health literacy classes? Did they support you? 4. When did your six month health literacy course start and end? At what time of day was the class held? Duration of class? 5. Was it easy to adjust your class hours along with your other engagements at home? 6. Did any of you miss classes for a period of time, why? Why did you return? 7. Are there any women present or not present who dropped out of the class completely? Why was this?

u | P a g e Final Programme E v a l u a t i o n

DFID 8. How many books you used in the classes? Were these books useful? Easy to understand? Was there any way in which the books could be improved? 9. Did you use these books outside of class? If yes, how. If not, were there reasons why? 10. What did you like most about the classes? 11. Was there anything about the classes you didn’t like or found difficult? 12. Were all important topics covered? Any important topics which you might have been missed out? 13. Did the HLF have sufficient knowledge to teach you? Was she polite? Was she punctual? Any other feedback about her? 14. Have the literacy classes been continued after the initial 6 month training period? If yes, when was this initiative started, who runs the classes, how often are classes held and for how long? 15. How many people attend the classes and have any new people joined? 16. What are the content of the classes – do you focus on literacy skills at all? 17. Do you find these classes as good, better or not as good as the original classes and why. 18. Have any of the original learners chosen not to attend the extra classes? If yes, why? 19. Can you provide one recommendations of good practice related health, hygiene and sanitation to cover in your class? 20. Can you provide one recommendations of good practice related pregnancy and antenatal care in your class? 21. Can you provide one recommendations of good practice delivery and postnatal care to cover in your class? 22. Please let us know any of the recommended practices you applied to your own lives after literacy training (do you continue to do this). 23. Do have the ability to fill out basic health information about family members as well as writing names? If can you can please fill out the form for us (take some sample forms with yourself). 24. Did you participate in any campaigns or other volunteer activities for MCH? 25. What kind of support did you receive from NRDP programme staff? For example, how often did they visit and what did they do during visits. Is there anything they could have done better or differently? 26. 28. If this programme was to take place again, what changes you would like to make to it? 27. Any other comments?

Notes:

ix) Fill out attendance form x) Take pictures of the available document xi) Take pictures of any document presented xii) Note down case studies, if any.

v | P a g e Final Programme E v a l u a t i o n

DFID

Annex 6.5-KII – Community Midwives COMMUNITY MIDWIVES Key Informant Interviews Tools Final Evaluation of DFID-funded programme “to improve maternal health and infant survival rates through supporting women and children in Badin District, Pakistan”

Introduction:

Assalamoalekum, my name is ….. We are conducting a survey in (name of the district) on community’s views about NRDP Programme. We will be interviewing different key informants to help us improve the program we are implementing in your community. I thank you for participating in this survey. I assure that your answers will remain confidential and we would not keep record of your name or other personal information. I also want to assure you that you have the right to stop this interview at any time and feel free to skip questions that you don’t want to answer.

Let me also share with you that there are no right or wrong answers. The more candid your answers are, the better we shall understand your views on the issue of women and economy.

Interview will take about 30-40 minutes to complete. Your participation is voluntary and your opinion will be very helpful.

Do you have any questions?

Do I have your consent for this interview? (TICK ACCORDINGLY)

[ ] DOES NOT AGREE TO BE INTERVIEWED THANK & WIND UP

[ ] AGREES TO BE INTERVIEWED

1. How long you have been a CM? 2. Please tell us about yourself? Have you received any education? If yes, till what grade? 3. Do you have any prior experience in MCH? If yes, please tell us more about it? If no, what were you doing before? Are any of your family members associated with MCH? 4. Please tell us about your selection in this programme? How were you selected? What motivated you? 5. Have you received the kit? How useful is the kit? Is there any item missing in the kit? What about quality of the kit? Are all items in the kits still working/functional? 6. Did you receive any midwifery training? If yes, where? 7. Did you receive any hands on midwifery training? Where was this training?

w | P a g e Final Programme E v a l u a t i o n

DFID 8. What were the key strengths of your training? What was the key weakness of your training? Was the training sufficient? How could it be improved? 9. Have you received any refresher training? Do you believe there is a need for refresher training? 10. Did you or a family member receive training on using the smartphone app? 11. If you received training, how did it help you in your work? Has it led to any additional challenges? If a family member received training, how did it help you in your work and has it led to any additional challenges? 12. Are you or a family member using the Smartphone App to document patient information i.e. register patient information, consultation? If the smart phone system is not being used, why not? Any issues with technical knowhow? Language of the smart phone? 13. Did you receive any technical support after the training? Who did you receive technical support from? 14. If you were unsure of how to handle a case what did you do/ who did you ask? 15. Are you using mobile to refer? Can you tell us how referrals requests you have made through Smartphone system? 16. How many of the referrals responded by doctors? How soon you received a response usually? Are the feedback you are getting is useful? How can we make this referral more useful? 17. Have you made any referral of pregnant women? If yes, what happened next? 18. Do you have any communication or referral system with the local BHU or hospital? If yes, please tell us more about it? If no, why not? 19. What do you know about assisted deliveries in your community? What percentages of deliveries in your village are handled by trained professional? 20. In a week how many times you visit someone or someone visits you? How many times you visit a pregnant woman in a month? 21. How many deliveries you have conducted? Do you receive any money from the communities for the services you provide? 22. Please let us know about your work related to pregnancies & referred deliveries? Do you follow up on women who were referred for delivery? How? What advice and support do you provide to pregnant women? 23. Is there any death of child or mother over the last three years in your village? If yes, what was the reason behind it? 24. Please let us know about your work related to ANC? How many cases of ANC you were dealing with? What kind of antenatal care do you provide? Are your care recommendations followed? Any particular challenges you faced? 25. Please let us know about your work related to PNC? How many cases of PNC you were dealing with? Is your advice followed? How many pregnant women she supports receive PNC? Any particular challenges you faced? 26. What do you know about immunization coverage in your community? What percentage of women and children in your opinion has received the necessary immunization? 27. How many women you have served has taken up family planning methods. If the number is low, why it is low? Do you expect more women to take up family planning, what they think are the reasons for this? 28. Did you receive all required support from WHC? If yes, what kind of support you got from them? Please give examples? 29. Did you receive all required support from MSG? If yes, what kind of support you got from them? Please give examples? x | P a g e Final Programme E v a l u a t i o n

DFID 30. Have you ever requested for health fund for one of your patient? If yes, when was it and if you got it? If no, why not? 31. Did you receive any support from NRDP? What was the biggest challenge for you in this job? 32. Once the programme closes, do you think you will continue with the activities? 33. If we have to do this programme in another area, any suggestions how we can make it better? 34. Any other comments?

The CMWs are supposed to maintain logbooks so should have accurate records for all of the above in terms of numbers of cases dealt with to-date, number of on-going cases etc. – please also ask to see their records. There is a difference between the use of the smart phone to record patient information and referral of the patient through the smart phone. Please make sure you understand this difference as the proposed questions merged for the two concepts. Answers should be clearly and separately reported for the two concepts.

Thank You

y | P a g e Final Programme E v a l u a t i o n

DFID Annex 6.6-KII – Health Literacy Facalitators

Health Literacy Facilitator Key Informant Interviews Tools Final Evaluation of DFID-funded programme “to improve maternal health and infant survival rates through supporting women and children in Badin District, Pakistan”

Introduction:

Assalamoalekum, my name is ….. We are conducting a survey in (name of the district) on community’s views about NRDP Programme. We will be interviewing different key informants to help us improve the program we are implementing in your community. I thank you for participating in this survey. I assure that your answers will remain confidential and we would not keep record of your name or other personal information. I also want to assure you that you have the right to stop this interview at any time and feel free to skip questions that you don’t want to answer.

Let me also share with you that there are no right or wrong answers. The more candid your answers are, the better we shall understand your views on the issue of women and economy.

Interview will take about 30-40 minutes to complete. Your participation is voluntary and your opinion will be very helpful.

Do you have any questions?

Do I have your consent for this interview? (TICK ACCORDINGLY)

[ ] DOES NOT AGREE TO BE INTERVIEWED THANK & WIND UP

[ ] AGREES TO BE INTERVIEWED

1. How long you have been a HLF? 2. Please tell us about yourself? Have you received any education? If yes, till what grade? Do you have any prior experience in MCH? If yes, please tell us more about it? If no, what were you doing before? Are any of your family members associated with MCH? 3. Please tell us about your selection in this programme? How were you selected? What motivated you? 4. Did you receive any training? If yes, where? Was this training sufficient? Where was this training? What were the key strengths of this training? What was the key weakness of this training? 5. Have you received any refreshers? Do you believe there is a need for refreshers? 6. Of the four modules, which module do you think is the best? And why? z | P a g e Final Programme E v a l u a t i o n

DFID 7. Of the four modules, which module do you think is the weakest? And why? 8. Which module did your students seem most interested in? Which module your students seem least interested in? And why? 9. Were there any topics that weren’t covered in the six month health literacy course that you felt should have been included? 10. What kind of preparation you do prior to the classes? 11. Was there anything about the classes you didn’t like or found difficult? 12. How many students did you have in the six month health literacy course? What time of day was the class held? What is the average duration of your class? 13. Did any of the women have low attendance, how many. Do you know why?? Did any women drop out of the class completely, do you know why. 14. Did women receive support from their husbands for the classes? 15. What was the biggest challenge or challenges for you in this job as HLF? What did you do to address this/these challenges? 16. Did you receive all the support you need as a HLF from the WHC and/or other sources? If yes, please tell us more about the support you receive? If no, what additional support you would like to have had? 17. Did you continue the literacy classes are after the initial 6 month training period? If yes, -when did this initiative start, how frequent are the classes, how long are the classes? Approximate number of people who attend (and if it includes new people)? 18. From where did the idea of additional HLF classes came from? 19. Once the programme closes, do you think you will continue with the activities? Please provide details. 20. If we have to do this programme in another area, any suggestions how we can make it better? 21. Any other comments?

Thank You

aa | P a g e Final Programme E v a l u a t i o n

DFID Annex 6.7-KII – Government Health Department

Health Department Staff Key Informant Interviews Tools Final Evaluation of DFID-funded programme “to improve maternal health and infant survival rates through supporting women and children in Badin District, Pakistan”

Introduction:

Assalamoalekum, my name is ….. We are conducting a survey in (name of the district) on community’s views about NRDP Programme. We will be interviewing different key informants to help us improve the program we are implementing in your community. I thank you for participating in this survey. I assure that your answers will remain confidential and we would not keep record of your name or other personal information. I also want to assure you that you have the right to stop this interview at any time and feel free to skip questions that you don’t want to answer.

Let me also share with you that there are no right or wrong answers. The more candid your answers are, the better we shall understand your views on the issue of women and economy.

Interview will take about 30-40 minutes to complete. Your participation is voluntary and your opinion will be very helpful.

Do you have any questions?

Do I have your consent for this interview? (TICK ACCORDINGLY)

[ ] DOES NOT AGREE TO BE INTERVIEWED THANK & WIND UP

[ ] AGREES TO BE INTERVIEWED

1. Are you familiar with NRDP Programme? Yes No If no, please give him a short brief on the programme, thank you him and conclude the interview. 2. Please tell us about your engagement in the programme 3. Do you think the programme l was addressing major needs of the targeted communities? If yes, why do you say so? If not, why not? 4. Can you identify three major successes of the programme? 5. Can you help us identify three major weakness of the programme? 6. Have you done some special activity as a result of lobbying from this programme?

bb | P a g e Final Programme E v a l u a t i o n

DFID 7. Did you participate in any of the lobbying activity from this programme? If yes, can you tell us more about it? 8. Were there other programmes of similar nature in the communities? If yes, was there any duplication? 9. How will the programme activities continue after the closure of the programme? 10. If the same programme has to take place again, what key changes you will recommend? 11. What do you know about immunization coverage in your district? 12. What do you know about assisted deliveries in your district? 13. What is the mortality rate for child or mother over the last three years in your district? If yes, what was the reason behind it? 14. Any other suggestion? Feedback? Thank You

cc | P a g e Final Programme E v a l u a t i o n

DFID Annex 6.8-KII – BHU Staff BHU Staff Key Informant Interviews Tools Final Evaluation of DFID-funded programme “to improve maternal health and infant survival rates through supporting women and children in Badin District, Pakistan”

Introduction:

Assalamoalekum, my name is ….. We are conducting a survey in (name of the district) on community’s views about NRDP Programme. We will be interviewing different key informants to help us improve the program we are implementing in your community. I thank you for participating in this survey. I assure that your answers will remain confidential and we would not keep record of your name or other personal information. I also want to assure you that you have the right to stop this interview at any time and feel free to skip questions that you don’t want to answer.

Let me also share with you that there are no right or wrong answers. The more candid your answers are, the better we shall understand your views on the issue of women and economy.

Interview will take about 30-40 minutes to complete. Your participation is voluntary and your opinion will be very helpful.

Do you have any questions?

Do I have your consent for this interview? (TICK ACCORDINGLY)

[ ] DOES NOT AGREE TO BE INTERVIEWED THANK & WIND UP

[ ] AGREES TO BE INTERVIEWED

1. Are you familiar with NRDP Programme? Yes No If no, please give him a short brief on the programme, thank you him and conclude the interview. 2. Please tell us about your engagement in the programme 3. Do you think the programme l was addressing major needs of the targeted communities? If yes, why do you say so? If not, why not? 4. Can you identify three major successes of the programme? 5. Can you help us identify three major weakness of the programme? 6. Is there any increase in the women who are availing MCH services at your BHU as a result of this programme? 7. Is there any increase in the children who are availing health services at your BHU as a result of this programme? dd | P a g e Final Programme E v a l u a t i o n

DFID 8. Because of this programme, coverage of immunisation amongst children/mother increased? 9. Have you done some special activity as a result of lobbying from this programme? 10. Did you participate in any of the lobbying activity from this programme? If yes, can you tell us more about it? 11. Do you know CM from this programme? Is there any kind of referral system established with her? If yes, did they ever refer any patients to you? 12. Were there other programmes of similar nature in the communities? If yes, was there any duplication? 13. How will the programme activities continue after the closure of the programme? 14. If the same programme has to take place again, what key changes you will recommend? 15. What do you know about immunization coverage in your Union Council? 16. What do you know about assisted deliveries in your Union Council? 17. What is the no. of cases of children, disaggregated boy/girl, with diarrhea reported in the last 30 days in your UC? 18. What is the mortality rate for child or mother over the last three years in your Union Council? If yes, what was the reason behind it? 19. Do you believe infant mortality (under 5) in your district reduced as a result of your programme? If yes, how come? 20. Do you believe the programme has a visible impact on the levels of preventable diseases and conditions (anemia, malaria, tetanus, jaundice, etc.) among pregnant women, neo-natal infants and children under 5 years? If yes, how come? 21. Any other suggestion? Feedback? Thank You

ee | P a g e Final Programme E v a l u a t i o n

DFID Annex 6.9-KII – NRDP Staff

NRDP Staff Key Informant Interviews Tools Final Evaluation of DFID-funded programme “to improve maternal health and infant survival rates through supporting women and children in Badin District, Pakistan”

Introduction:

Assalamoalekum, my name is ….. We are conducting a survey in (name of the district) on community’s views about NRDP Programme. We will be interviewing different key informants to help us improve the program we are implementing in your community. I thank you for participating in this survey. I assure that your answers will remain confidential and we would not keep record of your name or other personal information. I also want to assure you that you have the right to stop this interview at any time and feel free to skip questions that you don’t want to answer.

Let me also share with you that there are no right or wrong answers. The more candid your answers are, the better we shall understand your views on the issue of women and economy.

Interview will take about 30-40 minutes to complete. Your participation is voluntary and your opinion will be very helpful.

Do you have any questions?

Do I have your consent for this interview? (TICK ACCORDINGLY)

[ ] DOES NOT AGREE TO BE INTERVIEWED THANK & WIND UP

[ ] AGREES TO BE INTERVIEWED

1. How long you are engaged in this programme? Please tell us more about your engagement in the programme? 2. Do you think the programme l was addressing major needs of the targeted communities? If yes, why do you say so? If not, why not? 3. Please let us know what support did you get from FtM? Was this support sufficient? If yes, why? If no, what more FtM could have done to make it better for you? 4. Did you receive any training for better implementation of this programme? If yes, in case of training, please tell us more about the content? Also let us know how the training was delivered? Are you happy with the training? How the training can be improved? How the training was used? 5. Did they coordinated activities with you? If yes, please give us an example? ff | P a g e Final Programme E v a l u a t i o n

DFID 6. Have you taken up any lesson learnt from this programme as a good practice and expanded it to district level? 7. Is there duplication with other actors? 8. To what extend the community has taken up NRDP learning as part of their routine activities. 9. Please let us know about involvement of communities in the formation and executing phases of the programme of the targeted geographical area? 10. Can you identify three major successes of the programme? 11. Can you help us identify three major weakness of the programme? 12. What is your opinion about CM? Tell us in detail about the design, any weaknesses in it and how to make it better next time? 13. What is your opinion about HL Classes? Tell us in detail about the design, any weaknesses in it and how to make it better next time? 14. What is your opinion about HLF? Tell us in detail about the design, any weaknesses in it and how to make it better next time? 15. What is your opinion about MSG? Tell us in detail about the design, any weaknesses in it and how to make it better next time? 16. What is your opinion about WHC? Tell us in detail about the design, any weaknesses in it and how to make it better next time? 17. What is your opinion about kitchen gardening? Tell us in detail about the design, any weaknesses in it and how to make it better next time? 18. What is your opinion about mobile referral? Tell us in detail about the design, any weaknesses in it and how to make it better next time? 19. What is your opinion about community health fund? Tell us in detail about the design, any weaknesses in it and how to make it better next time? 20. What is your opinion about training component – for CMs and HLF? Tell us in detail about the design, any weaknesses in it and how to make it better next time? 21. Can you tell us about the lobbying activities conducted under this programme? 22. What was your coordination with other relevant stakeholders in the district? Tell us in detail? 23. Was budget allocation sufficient for all activities? Or would you like to propose any changes? 24. Do you believe maternal mortality in your district reduced as a result of your programme? If yes, how come? 25. Do you believe infant mortality (under 5) in your district reduced as a result of your programme? If yes, how come? 26. Do you believe the programme has a visible impact on the levels of preventable diseases and conditions (anemia, malaria, tetanus, jaundice, etc.) among pregnant women, neo-natal infants and children under 5 years? If yes, how come? 27. Do you believe programme activities will continue beyond programme life period? If yes, why and how? If no, why not?

Thank You

gg | P a g e Final Programme E v a l u a t i o n

DFID Annex 6.10-KII – District Government Official

District Government Officials Key Informant Interviews Tools Final Evaluation of DFID-funded programme “to improve maternal health and infant survival rates through supporting women and children in Badin District, Pakistan”

Introduction:

Assalamoalekum, my name is ….. We are conducting a survey in (name of the district) on community’s views about NRDP Programme. We will be interviewing different key informants to help us improve the programme we are implementing in your community. I thank you for participating in this survey. I assure that your answers will remain confidential and we would not keep record of your name or other personal information. I also want to assure you that you have the right to stop this interview at any time and feel free to skip questions that you don’t want to answer.

Let me also share with you that there are no right or wrong answers. The more candid your answers are, the better we shall understand your views on the issue of women and economy.

Interview will take about 30-40 minutes to complete. Your participation is voluntary and your opinion will be very helpful.

Do you have any questions?

Do I have your consent for this interview? (TICK ACCORDINGLY)

[ ] DOES NOT AGREE TO BE INTERVIEWED THANK & WIND UP

[ ] AGREES TO BE INTERVIEWED

1. Are you familiar with NRDP Programme? Yes No If no, please give him a short brief on the programme, thank you him and conclude the interview. 2. Please tell us about your engagement in the programme 3. Do you think the programme l was addressing major needs of the targeted communities? If yes, why do you say so? If not, why not? 4. Can you identify three major successes of the programme? 5. Can you help us identify three major weakness of the programme? 6. Have you done some special activity as a result of lobbying from this programme?

hh | P a g e Final Programme E v a l u a t i o n

DFID 7. Did you participate in any of the lobbying activity from this programme? If yes, can you tell us more about it? 8. Were there other programmes of similar nature in the communities? If yes, was there any duplication? 9. How will the programme activities continue after the closure of the programme? 10. If the same programme has to take place again, what key changes you will recommend? 11. What do you know about immunization coverage in your district? 12. What do you know about assisted deliveries in your district? 13. What is the mortality rate for child or mother over the last three years in your district? If yes, what was the reason behind it? 14. Any other suggestion? Feedback? Thank you

ii | P a g e Final Programme E v a l u a t i o n

DFID ANNEX 7: EVALUATION CRITERIA TABLE

Rating 1-5 (1 Low, 5 High) Evaluation Criteria Rationale 1 2 3 4 5 The programme supported MDGs 3, 4 and 5. Infant mortality for programme areas was 40 / 1,000 live birth whereas according to Sindh MICS 2014, infant mortality rate in Relevance rural areas of Sindh was 106 deaths per thousand live births. Maternal mortality for programme areas was calculated around 100/100,000 live birth the Maternal Mortality Ratio in Sindh it was 314. All the results related to the programme present a fair and accurate record of the Effectiveness achievements. With limited time and resources, the programme was able to extend it services to 100 villages. In terms of the overall efficiency of the programme, it delivered all the results mostly on time and on budget against Efficiency agreed plans as described in the programme document.

The programme was able to leverage additional financial resources from the communities. From the long term Sustainability sustainability perspective of the programme, some of the programme MSGs and WHCs will continue to grow and may become a Civil Society Organisations. Capacity building of the civil society was a prominent and important feature of this programme. All the people receiving Impact support through this programme would not have received this support without this programme.

jj | P a g e Final Programme E v a l u a t i o n