Evaluation of the ECHO Funded Tearfund Integrated WatSan and Public Health Promotion Programme in Nimba County, Liberia
August 2007
Evaluation Team Leader
Dr. James Webster
Cranfield University Bedfordshire MK45 4DT UK Email: [email protected] Contents
Glossary ...... i
Acknowledgements...... i
1 EXECUTIVE SUMMARY...... 1 1.1 General conclusions ...... 1 1.2 General recommendations...... 1 1.2.1 Sustainability...... 1 1.2.2 On-going support...... 2 1.2.3 Future projects...... 2
2 INTRODUCTION...... 3 2.1 Background ...... 3
3 METHODOLOGY...... 5
4 FINDINGS, CONCLUSIONS AND ASSESSMENTS...... 7 4.1 Effectiveness: in achieving the intended individual project objectives ...... 7 4.1.1 The achievement / likely achievement of the project outputs ...... 7 4.1.2 The achievement / likely achievement of the project purpose...... 9 4.1.3 The contribution / likely contribution towards the project goal ...... 9 4.1.4 The major factors influencing the achievement / non-achievement of the project objectives...... 9 • Best practice...... 10 • Sustainability ...... 12 • Logistics and coordination...... 17 4.1.5 Conclusions and assessment...... 18 4.2 Impact: of projects on beneficiaries and non-beneficiaries...... 18 4.2.1 The intended and unintended effects (social, physical, environmental, economic, spiritual), both positive and negative, of the project on beneficiaries and non- beneficiaries...... 18 4.2.2 The major factors influencing the impact of the project on beneficiaries and non-beneficiaries...... 19 4.2.3 Conclusions and assessment...... 19 4.3 Relevance: of the programme to the priorities of the target group, the partner and Tearfund ...... 19 4.3.1 The validity of the project objectives ...... 19 4.3.2 The relevance of the individual projects with the needs and vulnerabilities of the target group...... 19 4.3.3 The use and enhancement of the existing skills, knowledge and coping strategies of the target group ...... 21 4.3.4 The consistency of the projects with the vision, values, strategy and resources of Tearfund ...... 21 4.3.5 The major factors influencing the relevance of the project to the priorities and policies of the target group and Tearfund...... 21 4.3.6 Conclusions and assessment...... 21 4.4 Efficiency - in achieving the intended objectives...... 22 4.4.1 The cost-efficiency of project activities ...... 22 4.4.2 The achievement of objectives to time and to budget ...... 22 4.4.3 The comparison of project practice against accepted good practice for the various project sectors...... 22 4.4.4 Comparison of the implementation of the projects with alternatives ...... 23 4.4.5 The major factors influencing the efficiency of the projects...... 23 4.4.6 Conclusions and assessment...... 24 4.5 Sustainability - of project benefits after the end of the projects ...... 24 4.5.1 The continuation of project benefits after funding by Tearfund has ceased...... 24 4.5.2 The major factors influencing the sustainability of the project ...... 24 4.5.3 Key recommendations on effective exit (including options for handover to other agencies, community level sustainability mechanisms etc) for Tearfund from Nimba...... 25 4.5.4 Conclusions and assessment...... 27 4.6 Coordination and coherence – integration of projects with the activities and priorities of other agencies and organisations (including local and national government)...... 27 4.6.1 The linkages and relationships between the target group, Tearfund, and other agencies and organisations ...... 27 4.6.2 The coherence of the projects with national and local policies ...... 28 4.6.3 The major factors influencing coordination and coherence of the projects with the activities and priorities of other agencies and organisations ...... 28 4.6.4 Linkages to the WRC funded Food Security Project...... 28 4.6.5 Conclusions and assessment...... 28 4.7 Participation: the extent to which the project stakeholders were involved in the assessment, planning, implementation and evaluation phases of the project ...... 28 4.7.1 The quality of relationship between Tearfund and the project stakeholders ...... 28 4.7.2 The level of participation of the stakeholders in all stages of the project cycle ...... 29 4.7.3 The involvement of all the beneficiaries, ensuring gender balance and inclusion of the vulnerable...... 29 4.7.4 Conclusions and assessment...... 30
5 SPECIFIC ACTIONABLE AND PRIORITISED RECOMMENDATIONS FOR FUTURE DEVELOPMENT OF PROJECT IN LIBERIA...... 31
ANNEX 1: TERMS OF REFERENCE FOR THE EVALUATION ...... 32
ANNEX 2: EVALUATION SCHEDULE...... 39
ANNEX 3: WATER SAFETY PLAN...... 40
ANNEX 4: DETAILS OF OUTPUTS ACHIEVED (OBSERVATIONS AND REPORTS TO 18/08/07)...... 45
ANNEX 5: DOCUMENTS AND FIELD DATA CONSULTED AND USED DURING THE EVALUATION ...... 46
ANNEX 6: PERSONS PARTICIPATING IN THE EVALUATION...... 46
ANNEX 7: BIBLIOGRAPHY...... 47
ANNEX 8 (FOR TEARFUND ONLY): LEARNING (ASPECT 8)...... 49 • The extent to which lessons learned from the Bomi DCI funded WatSan evaluation in 2005 have been been incorporated into this project...... 49 • Conclusions and assessment...... 49
Glossary
ACF Action Against Hunger (Action Internationale Contre la Faim) CDC Community Development Committee CHV Community Health Volunteer CWA Community Water Attendant DDC District Development Committee DMT Disaster Management Team (Tearfund) GOL Government of Liberia IDP Internally Displaced People KAP Knowledge, attitude and practice MOH Ministry of Health NGO Non Governmental Organisation PHE Public Health Education ToR Terms of Reference WATSAN Water and Sanitation
Acknowledgements
The evaluator would like to express sincere thanks to the individuals consulted and the communities visited in the course of the evaluation, for their very warm welcome and openness, and often patience in the face of detailed questioning.
In particular, members of the evaluation team (DMT Nimba staff, EQUIP Regional Supervisor and Dr. Jen Smith from Cranfield University) and the DMT Monrovia Programme Director and Deputy Director are thanked for their fellowship, flexibility and frank, open participation in this evaluation.
Whilst the evaluation was conducted in a participatory manner with considerable reflection and cross- checking of information, the opinions, Figure 1: Members of the evaluation team. Left-right: Jen Smith, conclusions and recommendations James Webster, Timothy Owhochukwu, Badio William, Matthew Carlquist, Sahr, Joseph Flomo (holding George, RIP), non team expressed in this report are ultimately member, Ambaye Zekewos the responsibility of the evaluation team leader.
i 1 Executive Summary
1.1 General conclusions
The planned activities and outputs for two phases of the programme were reasonably ambitious but achievable in the original 18 months timeframe. Logistical issues caused delays in implementation which necessitated a two month extension (although the total time lost to logistical issues exceeded two months). Allowing for this extension, at the time of the evaluation, in all areas witnessed, activities were on target for completion by the end of August, largely due to the high commitment of staff to the programme. Construction progress has been/will be accomplished in accordance with those planned: potable water has been provided in sufficient quantities to 34 communities (45 new and 17 rehabilitated hand-dug wells with handpumps, 10 bio-sand filters); 652 latrines have been constructed, provisionally one per maximum of 20 users. Public health education (PHE) has resulted in clear improvements in public and private health knowledge, attitudes and practices (KAP). Communities have been empowered to implement their projects and largely provided with socio-economic skills to manage and sustain these inputs, although issues of sustainability exist (in particular community capacity building for maintenance of hand-pump soak pits, use of open water containers, and non-usage of pit-latrines in some locations). There is much evidence of communities being empowered to resolve conflicts and address issues of governance. The programme activities have been achieved in a cost-effective manner, with high levels of professionalism evident throughout the implementation of the programme. Overall, the programme is of a very high standard, with Government and users expressing high levels of satisfaction. Especially in light of the on-going logistical problems, programme staff are thoroughly congratulated for their achievements, and the professional and participatory manner in which they are being achieved. However, in order to ensure these benefits are sustained, on- going support is required.
1.2 General recommendations
Although the programme is meeting its objectives, purpose and goals, this evaluation was undertaken in a manner to identify any areas that could be improved upon. The following were identified where the effectiveness, efficiency, impact and sustainability of the work could be improved. These recommendations are presented below in priority order and in summary form only: reference is made to the appropriate sections of the main text for the full discussion.
1.2.1 Sustainability
It is generally too early to tell if the programme is sustainable, although the following recommendations require action in order to move towards this: - Additional funds to allow jerry cans to be purchased and distributed to Phase 1 communities to replace the UNICEF buckets (4.1.4). - Construction in each village of demonstration ‘organic’ VIP latrines, using only locally available materials to ensure replicability. Experimentation with demonstration latrines only in one new village or a cluster of villages, preferably remote from currently served villages. High levels of social marketing and monitoring required. (4.1.4). - Experimentation with the use of handrails in the walls of latrines, a rope or ropes to hang on to, or a box to sit on for the elderly and disabled (4.1.4). - The provision of a hand-washing facility at latrines, for example tippy taps or containers (drums) with taps (4.1.4).
1 - Development of a water safety plan for Nimba County (which could provide the basis for a WSPs throughout the organisation) (4.1.4). - Continued social marketing and promotion of latrine use with appropriate monitoring of usage (4.1.4). - Continued promotion of ash for reduction of odour from latrines, as well as the use of mosquito nets (4.3.2). 1.2.2 On-going support
In order to ensure project benefits are sustained, on-going support is required (and ideally further implementation occurs)1. The following recommendations are made: - In the short-term (during the implementation time period), follow up training of trained personnel through an intensive one day of one-to-one interaction is recommended. In the long-term, on-going support to provide follow-up and refresher training (4.1.4). - Monitor the evolution of Government structure and relationship with EQUIP, and Tearfund reach consensus through systematic assessment as to the relative capacity of, and pros and cons of partnering with EQUIP, Government and the Liberian National Red Cross Society. (4.5.3). - Long-term monitoring of the ownership and maintenance (and where appropriate replication) of project inputs (4.7.1). 1.2.3 Future projects
Implementation - Provision of modified VIP latrines with short length of female vent pipe socket embedded in the slab (4.1.4). - Ensure sufficient resources (time, personnel, finances and transport) are made available for promotion of income-generation projects (4.1.4). - Pre-selection screening of CWAs, CHVs and CDCs to assess levels of required literacy, or preferably communication of required teaching in an accessible and effective format. (4.1.4). - Assessment of the ties trained personnel have to the community. Family (especially children) and land are the main two (4.1.4). - Establish a dedicated HIV/AIDS sector (4.3.4). Logistics - To address maintenance issues for vehicles and plant, one of the following: use of DMT approved dealers other than Toyota; decentralisation of maintenance where capacity exists down country; and employment of a dedicated fleet mechanic. (4.4.5) - Improved resourcing and training to enable appropriate purchasing by Nimba logs (4.4.5). - Recruit a dedicated warehouse keeper (4.4.5). Best practice - Ensure on-going reflection and adaptation of programmes through continuing to formally schedule events that facilitate this (4.1.4). - Improved communication and understanding between Tearfund and its partners (4.4.5).
1 Tearfund is aware of this need and is actively seeking funds, currently through a proposal to Irish Aid to meet unmet needs in the specific areas already covered in Nimba, and to backstop particularly on PHE and CDC.
2 2 Introduction
At the request of Tearfund, to satisfy ECHO’s requirement of funding for this project and Tearfund’s corporate commitment to learning from practice, an evaluation of ECHO funded Water/Sanitation and Public Health work in Nimba County, Liberia was conducted with the aims:
• To evaluate the effectiveness of the programme and the extent to which the programme purposes and results, as laid out in the programme proposals, have been achieved.
• To identify lessons learned from this projects and make recommendations for similar programmes in other parts of Liberia.
Field visits between 10th and 19th August were conducted at selected locations with participation and facilitation by Tearfund’s West and Central Africa Team’s Disaster Management Team (DMT) staff. This report describes the background to and history of the programme, the methodology adopted, the findings, conclusions, assessments and recommendations.
The main conclusions and recommendations are drawn together and summarised in Chapter 1. The ToR and other relevant indicative information are provided in the Annexes.
2.1 Background
The programme background, findings of previous reports and evaluations, and the DMT profile, history and current activities in Liberia are detailed in the ToR (Annex 1). There seems little value in repeating them here. The following is therefore a brief summary of the background to the programme.
Nimba County (Figure 2) was one of the counties worst affected by the civil war, with devastation to infrastructure and decimation of population. Most who survived the war sought asylum in neighbouring countries. In February 2005 Nimba was among the last of the counties to be declared safe for the return of refugees and IDPs. Although official repatriation only occurred recently, large numbers returned spontaneously.
The international community response is mostly concentrated along the main access routes around Ganta and Saclepea. Tearfund focussed its work amongst largely unreached populations in the South and South East of the county. Two Phases of funding from ECHO occurred.
• Phase 1 (January - August 2006) focussed on 15 communities in the remoter areas surrounding Saclepea. • • Phase 2 (September 2006 - June 2007 with a two month extension to August 2007) focussed on 19 remote communities around Tappita (and 3 additional communities for WATSAN interventions only) (Figure 2).
The aim was to improve access to and consumption of safe water, and sanitation and hygienic practices. This was to be achieved through working with village communities to develop the socio-economic skills required to manage and sustain these inputs and address issues of governance, inclusiveness and reconciliation within the communities. Parallel World Relief Canada/Canadian Foodgrains Bank funding aimed to improve food security.
3
Figure 2: Project site locations (adapted from NIMAC, 2007)
4 3 Methodology
From the outset, this evaluation was conducted in a spirit of reflective learning for all participants. As such, openness was actively encouraged at all times. The positives of the programme were focused on and creative thinking promoted, whilst negatives were addressed where necessary through constructive critique. The aim was to consolidate and enhance programme gains and lessons learned, for the benefit of the current and future programmes.
A qualitative approach to data gathering was adopted in order to elicit more rich, in-depth information than would be possible through quantitative approaches. Therefore semi-structured interviews (SSIs) were used in preference to, for example, questionnaires. In addition to the team leader using SSIs, after a brief training, the DMT and EQUIP staff of the evaluation team also conducted SSIs independently in the communities visited. This served to nurture reflective learning amongst programme staff, and to give them experience both in the use of SSIs and in conducting evaluations. Therefore the DMT/EQUIP evaluation team members were both interviewees and interviewers, involved in the provision, collection and analysis of data. Other DMT staff (Nimba and Monrovia) were involved as interviewees only.
The evaluation team leader fed back findings to the Nimba and Monrovia staff for cross- checking and further discussion. This enhanced open expression of opinions and ideas.
It was decided by the evaluation team to visit two sites each day in the field - six sites in total (see Annex 2 for full details of sites visited and the evaluation schedule). Site selection was based on accessibility and performance. Consequently, good or average sites that had already learned from those that had underachieved were visited to facilitate the participatory learning approach. As stated, the primary focus was on what improvements could be made with a secondary focus on analysis of reasons for underachievement. Therefore new issues, approaches and technologies were discovered, thought about and planned for, rather than old ones simply re- hashed, important though those are. The findings seemed to have justified this approach.
Sites were therefore pre-selected by DMT staff. The disadvantages of this approach (communities likely to put their best foot forward) were outweighed by the advantages (preparedness and openness), and again, such an approach enabled both positives to be identified and negatives to be discussed in a spirit of openness. In each location, the evaluation started with a group discussion with various end-users. In all towns/villages this consisted of women, children, men, Town Chiefs, elders, community development committees (CDCs), community health volunteers (CHVs), public health educators (PHEs), community water assistants (CWAs or pump mechanics). Additionally, in certain locations, individuals such as the Township Commissioner, Zone Chief, Clan Elder, Justice of Peace, Township Clerk, Pastors and Teachers were present. Although such group interviews are limited in their benefits - thanksgiving speeches and further requests tend to predominate – they do serve to elicit a sense of overall end user (dis)satisfaction. More importantly, they contribute to the generation of acceptance and an open atmosphere upon which subsequent individual interviews prove more fruitful.
Both group discussions and individual interviews were based on indirect questioning through SSIs, although other PRA techniques were used such as needs priority ranking through dividing piles of stones into proportions (Cornwall and Jewkes, 1995). These data gathering techniques covered the Aspects for Assessment as stated in the ToR.
5 Individual interviews were conducted by random transect walks through the communities to interview users, with a bias towards interviewing the main collectors of water and primary teachers of health and hygiene in the home, i.e. women. Children were also targeted for interview, especially with regards latrine use. The translators used had excellent spoken English, secondary school education, and had worked with expatriates for a number of years, therefore the validity of data from interviews was deemed acceptable.
Additionally, SSIs were conducted with relevant representatives of local government.
Despite the constraints of time, the amount of travel involved, and the number of individuals to be consulted, the evaluator feels that the intensive use of SSIs with individuals and groups representing different stakeholders gave a fair and representative impression of the performance of the programme. This was evident through reaching a point of theoretical saturation whereby a consensus of opinions and issues were emerging time and again through differing lines of enquiry with the different stakeholder groups. A quantitative and statistically sound assessment would not have yielded the same depth or richness of information as the more qualitative approach adopted.
6 4 Findings, Conclusions and Assessments
The level of detail and the comprehensive nature of the seven Aspects of the evaluation in the ToR (Annex 1) allows the findings, conclusions, recommendations and assessments to be substantially presented through addressing these Aspects as laid out in the ToR.
4.1 Effectiveness: in achieving the intended individual project objectives
4.1.1 The achievement / likely achievement of the project outputs
From the log-frame, the 3 proposed outputs (or results) with associated activities were:
• Potable water provided for up to 34,554 people of sufficient quality and quantity that is both accessible and reliable (revised from the original estimate of 30,000 by PHE censuses in October 2006 and March 2007 of community leaders who did not have access to recent and/or accurate census data): o Sensitisation and consultation with communities, identification of well locations. o Mobilisation for hand digging of wells and gathering of local materials. o Construction of 45 wells and rehabilitation of 17 existing wells. o Identify appropriate cost recovery system for water supply. o Train designated Community Watsan Attendants. o Provision of Community Tools Package for community projects. o Install 10 bio-sand filters and promote them
• Fully functional latrines accessible by at least 13,000 people in project communities: o Sensitisation and consultation with communities regarding latrine design and locations. o Construction of 650 family latrines.
• Public health knowledge, attitudes and practices improved in relation to key health and hygiene messages, including HIV/AIDS, amongst 26,700 beneficiaries (including women, children and youth) and underlying issues of inclusiveness and reconciliation within community addressed: o Knowledge, attitudes and practice survey undertaken. o Community Health Volunteers (CHVs) selected and trained. o Community-wide health education led by trained CHVs. o Establishment of 4 youth groups serving 17 communities and meeting on a fortnightly basis to deal with reintegration and sexual health issues through sports activities. o Distribution of public health items which complement the UN assistance package. o Establishment of Children’s Health Clubs. o Formation and training of CDCs through workshops in each village, relating to self-reliance, governance, gender, reconciliation.
Within the time constraints of such an evaluation, it was not possible to independently and quantitatively verify all outputs as per the agreed sources of verification (e.g. procurement and distribution records: waybills signed by beneficiaries for materials received). However, from a combination of observed quantitative data, Tearfund DMT reports and interviews with ‘independent’ stakeholders (such as Government personnel and community members), the
7 evaluator is confident that activities have been undertaken in 37 communities (3 with WATSAN interventions only), and achieved at least 100% of proposed outputs through the following:
• Construction of 40 wells with an additional 5 on-going, and rehabilitation of 18 existing wells (97% coverage of target population); • Installation of 10 bio-sand filters; • Construction of 652 latrines (>40% coverage of target population); • Training of 133 CHVs; formation of 4 youth clubs (serving 17 communities) and 16 children’s health clubs (serving 33 communities); distribution of NFIs to 4,913 family heads, 1,010 vulnerable groups, 400 children and 8 schools; • Training CDCs in each village, with >90% meeting on a monthly basis to address issues of inclusiveness and reconciliation.
KAP surveys of Phases I and II report: • 85% of targeted youth, adults and children (6-10 years Figure 3: Traditional water source old) have an understanding of water borne disease, the and collection methodology in Nimba link between hygiene and health, and the causes and prevention of common diseases and infestations (an increase from 45%); • 60% of the adults and 45% of the youth show a basic awareness of HIV/AIDS (an increase from 50%); • 188% coverage of dish racks and clothes lines (i.e. approximately 2 per household), and 85% coverage of garbage pits (an 85% increase) amongst 1,300 households; • Increased levels of washing hands after defecation, fruit and vegetable before eating, and the wearing of slippers or shoes to prevent worm infestation.
Observations during the evaluation support these reports: • Universal presence of clothes line and dish racks at the household level; • Low levels of rubbish, overgrown vegetation and animal faeces around and between dwellings; • Fencing of completed wells by sticks, ‘live’ fencing or masonry/concrete walls.
These achievements are significant (see Figure 3) especially given some of the constraints (see 4.1.4). However, in spite of this, it cannot be conclusively said that the following outputs that are stated in the proposal have been achieved. These are again discussed in 4.1.4:
• Reliability of potable water. In 4 of the 6 communities visited, the sustainability of hand- pumps is not ensured due to concerns over maintenance capacity and revenue collection. The degree of concern varied across these communities. • Improvements amongst all beneficiaries in public health knowledge, attitudes and practices. Non-use of pit latrines amongst some intended end-users was reported in half the communities visited. Use of open containers for water collection was evident in Phase I sites. These practices therefore indicate a lack of improvement amongst some beneficiaries in some health aspects. There are also questions as to the extent and effectiveness of the HIV/AIDS training, despite the project reports, as a planned advisor for Phase II was not able to be recruited.
8 4.1.2 The achievement / likely achievement of the project purpose
As per the log-frame, the project purpose (or specific objective) is:
‘to ensure that re-settlers in targeted communities of Nimba County have improved access to and consumption of safe water, improved sanitation, hygienic practices and the socio-economic skills required to manage and sustain these inputs and address issues of governance, inclusiveness and reconciliation within their communities.’
As mentioned in 4.1.1, there are some concerns regarding the sustainability of water sources and the improvements in hygienic practices.
However, it is worth emphasising at this point the contribution made through the reconciliation training and pilot income-generation projects. Many individuals and communities (especially Kwendin) placed great value on the (increased) skills they have acquired in these areas (see 4.1.4).
Therefore, whilst there are some unknowns especially with regards sustainability, with the majority of outputs having been achieved and with the integrity of the log frame, the purpose/specific objective has also been substantially achieved.
4.1.3 The contribution / likely contribution towards the project goal
Again, from the log-frame, the goal (or principal objective) is:
‘Prepare a foundation for long term rehabilitation and rebuilding of sustainable livelihoods of communities in Nimba County which have been devastated following fourteen years of civil conflict.’
The comments in the previous section regarding the integrity of the log frame equally apply here. In several communities, specific increases in levels of empowerment were observed which have already provided an excellent foundation for the long-term achievement of the goal. Again, the programme is congratulated on this significant achievement.
If the concerns regarding sustainability of hardware and software are addressed it is reasonable to assume these gains will be seen throughout all project areas.
4.1.4 The major factors influencing the achievement / non-achievement of the project objectives
There are many overlaps between the 7 Aspects under consideration in this evaluation. Many influencing factors discussed below are relevant to more than one Aspect, to varying degrees. To avoid unnecessary duplication and fragmentation, these influencing factors are fully considered here, and referred back to in subsequent sections, or where necessary supplemented.
Dealing firstly with the positives, there are several indicators that the achievements made have been possible through the following factors, which loosely may be grouped as best practice:
9 • Best practice
o Staff commitment and coordination
Tearfund DMT has a reputation amongst Government and NGOs for working in remote areas where no other NGOs work. The Township Commissioner for the Kwendin area – a presidential appointment - stated that other NGOs had been to Kwendin and surrounding villages with hollow promises, but Tearfund delivered.
Working in these remote locations in conjunction with the need for staff to work from 0600-1900 to accomplish Phase II activities and outputs that effectively started in March 07 due to logistical problems (see later in this section), placed high demands on DMT staff. A parallel World Relief Canada (WRC) programme, whilst benefiting the ECHO projects (see 4.6.4), also stretched staff and their resources to the limit.
Having skilled staff from the previous Tearfund programme in Bomi and other NGOs, combined with good coordination from country level teams that sent the necessary additional vehicles when the programme was stretched, enabled achievement of objectives.
Again, the programme it commended for its dedication and sheer hard work.
A related expression of this is Tearfund’s willingness (and ability) to pre-finance mobilisation despite delays in funding from ECHO.
o A learning, adaptive, end-user focussed culture amongst staff
Many lessons from previous DMT projects in Liberia have been incorporated into this programme. Additionally, as the two Phases have progressed, experiences have been learned from. For example:
- Prior to the construction of hand-dug wells (HDWs), the majority of Public Health Education (PHE) and latrine construction occurred. Thus accessible and potable water was the initial incentive for PHE attendance and latrine construction. Evidence from the evaluation showed these became valued in their own right (see 4.3.2). In former DMT projects, HDWs were initially constructed with resulting low PHE attendance. - The number of Community Water Attendants (CWAs) to maintain handpumps was increased from 2 to 3 in Phase II as awareness of the migration of CWAs to Monrovia in particular once trained became apparent from Phase I projects. (15 community members are selected to assist in handpump construction. Through a practical and written test, the best 3 are Figure 4: Children’s health club, Kwendin chosen as CWAs).
10 - A similar adaptation occurred with the training of 3-5 CHVs per community, typically 2 male, 2 female. Although husbands didn’t initially allow women to travel to distant places to be CHVs, with appropriate sensitisation by DMT staff, this was eventually accepted. - In Yarsonah village, 3 latrine slabs were placed too close to the latrine door by field staff, but these were quickly noticed by the sector manager and staff ‘retrained’. - Weak Phase I UNICEF containers ($3.5) were replaced with much more durable locally purchased jerry cans ($5) for Phase II. - In response to low attendance to day-time PHE seminars, DMT staff recognised the need for house to house training as farming during the day is a priority. - Staff awareness during the course of the programme of the effectiveness of PHE training in schools has resulted in an increased focus on this methodology (Figure 4. It is estimated that 60-80% of children aged 5 and above attend school.).
Recommendation: ensure on-going reflection and adaptation of programmes through continuing to formally schedule events that facilitate this
o Effective and appropriate networking with and participation of stakeholders
Again, there are several indicators of this:
- One-day project awareness workshops with senior Government staff2 to inform them of Tearfund activities has helped to ensure high levels of support from the Government. Similarly good relationships with NGOs in Nimba were evident, and Tearfund staff reported the same with the ECHO monitor in Monrovia. - Projects were designed for appropriate, optimal community participation rather than maximum contributions (Dahanayake, 2004). Signed partnership agreements with communities ensured clarity over those contributions. - In Phase I and part of Phase II different NGOs were trying to form separate CDCs in each community for each donor project. Through very good network relationships with other NGOs and Government, the DMT programme manager was able to stop that, enhancing cohesion within communities as well as between other stakeholders.
o Promotion of skills to enhance sustainable livelihoods
The f o llowing are two examples of how capacity building communities has enhanced sustainable livelihoods: - Soap making was very successfully trialled amongst 50 women in Mehnpa (Phase I) with minor inputs. It was cited by many women as one of the main benefits of the project. The income generated from the soap making project will be used to establish a skill training school in tailoring and tie-dyeing. However, there was not enough time to implement such income-generation projects in Phase II. - In Yarsonah (Phase I), as a result of the concept of forming a CDC to raise revenue, the community formed another group and has raised funds and started building a school.
Recommendation: ensure sufficient resources (time, personnel, finances and transport) are made available for promotion of income-generation projects
2 Government District Superintendent and Development Superintendent, District Commissioner, Township Commissioner, Clan Chiefs, Paramount Chiefs, District Health Officers and City Mayor.
11
Moving onto the negatives, the factors that influenced or are likely to influence the (potential) non-achievement of project objectives may are essentially related to the issues of sustainability and logistics. Whilst sustainability is a separately considered aspect (see 4.5), it is at the heart of programme objectives, therefore the factors that influence it need to be discussed here.
It should be emphasised that the programme is aware of the need for sustainability, and its links with participation and ownership (e.g. Narayan, 1995). To this end, for example, four training sessions were conducted for the CDC’s between September 2006 and June 2007 in leadership and management for the sole purpose of sustainability.
Nevertheless, long-term sustainability is very difficult if not impossible to achieve through short- term programmes. There are no short cuts and there is a considerable body of literature emphasising that sustainability of project benefits requires on-going support by a backstopping agency (e.g. Webster et al, 1999; Abrams, 2001; Carter et al, 1996 & 1999; Bagamuhunda and Kimanzi, 1998).
Thus the validity of evaluating a short-term programme against the achievement of long-term objectives such as sustainability (that require long-term inputs) is not entirely reasonable. Nevertheless, an assessment of the likelihood of those long-term objectives being achieved is useful: lessons can be learned and useful comments made.
• Sustainability
o Project duration and behaviour/attitude change
Several observations from Phase I locations indicate that the behaviour change stated in the obj ecti ves was either not being fully achieved or had proved difficult to achieve within the given implementation timeframe:
- In Mehnpa, a group interview representing 13 households revealed that only 5 poured stored water in the homes from jerry-cans, the remaining 8 finding it easier to use bowls and dip. - Farmers putting dirty clothing on the drying racks when coming in from the field was a problem commonly cited by the CHVs. Some progress has been made through house-to-house visiting although contamination of utensils remains a concern in some households. - There was an expectation in Kpeikpoa that Tearfund would supply hand-pump spares, with no funds having been collected for maintenance by the CDC. However, there had been no need for spares yet, the pump had been re-fenced every 3-4 months (4 times in total) and the pump was well maintained with a female pump attendant/caretaker. - A recent returnee (Kima Dona) in Yarsonah stated that as she wasn’t present when beneficiaries were selected for NFI distribution, she felt bad and didn’t participate in the PHE training. She did however contribute to fencing the pump and casting the slab for her latrine, making bricks (3 days work) and purchasing timber for the roof. - Again in Yarsonah, the community complained that one off training in conflict resolution was inappropriate as time is needed to assimilate and accommodate such changes in behaviour.
12 Similar problems in Phase II locations are attributed by staff to the remoteness of Tappita compared to Saclepea where many NGOs work. As the communities are simply not used to the concept of NGOs, Tearfund and other NGOs who have worked in the region (e.g. World Vision) experience a marked slowness in mobilising communities for rehabilitation and development.
Phase II was acknowledged by DMT staff both in the UK and Liberia as having even more ambitious targets than Phase I for the given time-frame. Nevertheless, in Kwendin (Phase II) several community members expressed that behavioural practices that initially they found hard to change, had changed over time. This is a credit to DMT staff, and perhaps an indication again that lessons had been learned from Phase I, thus increasing the effectiveness of the programme.
Generally, though, where a core component of a relatively short-term programme is long-term behaviour change through PHE work, it is essential to have long-term support mechanisms in place to ensure this.
In a similar vein, attitude changes may need to occur. It is common for returnees to have developed attitudes of dependency during time in camps, etc. Some project practices are in danger of reinforcing this, such as the requirement of CDCs to submit monthly meeting reports of the positive impacts they are having within their communities. Whilst the desired behaviour change may be worthwhile, its sustainability is questionable. If the change is encouraged through extrinsic (i.e. external) motivators such as DMT requirements, the behaviour may not be assimilated into an intrinsic motivator that is internal and owned by the CDCs (Webster, 2006 and Wysoki, 2007).
Recommendations to ensure long-term, sustainable changes in behaviour are made at the end of the discussion on exit strategy (4.5.3).
o Maintenance of hand-pump soak pits
Relat ed to the above, whilst all handpumps observed were functioning (and the ACF pump in Yarsonah still functioning after 6 years), in four of the six towns and villages visited, the soak pits intended to drain surplus water from the hand-pump collection aprons were not functioning as intended. Stagnant water was therefore present to differing degrees, but in all cases providing a potential breeding site for mosquitoes (Figure 5).
It is intended that maintenance is by CWAs. Failure was due to two main reasons: lack of commitment of CWAs; and a mismatch of training to learning ability. On the job training is given to the 15 individuals that assist in construction. 3 CWAs are then selected through tests. Finally some formal (2-3 days x 3 times) is given. However, in Zahnzayee (Phase I), illiteracy amongst CWAs prevented them learning the taught topics, so they only attended one day of training. Although the CDC said they would re-appoint, this had not occurred at the time of the evaluation. Figure 5: Stagnant water in clogged soak Similar problems exist with both CWAs and CHVs leaving pit, Geapa (Phase II) to Monrovia to seek work once trained: in Mehnpa (Phase
13 I), out of 4 CHVs trained only one remained.
Recommendation: it is essential to ensure that training of all community personnel – CWAs, CHVs and CDCs – fits their abilities. This may include either pre-selection screening to assess levels of required literacy are present, or preferably communication of required teaching in an accessible and effective format. The details of this need careful consideration, although more pictorial representation of messages, perhaps including video, and role play/drama is likely to be effective.
Recommendation: in many projects, good indicators of likely future commitment of trained personnel are the ties they have to the community. Family (especially children) and land are the main two. These have successfully formed pre-selection criteria elsewhere, and are recommended for DMT projects, although some representation of, for example youths in the CDC, is necessary.
Recommendation: some movement of trained personnel to outside the project area is inevitable. Training is relatively inexpensive, so initially the numbers trained can be increased from 2-4 to 6. In the short-term (during the implementation time period), follow up training through an intensive one day of one-to-one interaction is recommended. In the long-term, on-going support (see later) is essential to provide follow-up and refresher training.
o Non-usage of pit-latrines
Severa l reasons were given by community members for latrines not being used as intended.
Some may be simply a case of users needing to get used to a different environment. For example, DMT staff reported initial concerns amongst some of the elderly over slabs breaking. Over time, staff claimed that people had got used to the slabs, especially through the promotional PHE work by children. During the evaluation, no concerns were voiced by community members with regards the safety of slabs. Thus, recognition that sanitation is, to a large extent, a social phenomenon, rather than a technical one (Wegelin-Schuringa, 2000), continuing social marketing and promotion of latrine use is likely to continue to solve this and similar problems, although this requires careful monitoring.
Other problems that require different solutions were:
¾ The heat rising through the latrine holes was reported by several individuals to prevent them using latrines. ¾ It was reported in two communities, including a women’s group in Kpeikpoa (Phase I), that where latrines are shared between two or more families, they are often not used due to the odour of other’s faeces. ¾ Similarly, sharing of latrines is resisted in many instances for reasons of dignity. Elderly people reported that they feel disrespected if a child comes and asks “Who’s in the toilet?” Females feel a similar disrespect if the toilet is shared with males from another family. ¾ Some elderly and disabled reported difficulty in squatting during defecation. ¾ Some concerns were raised in Yarsonah (Phase I) concerning the closeness to swamps of latrines, the swamps being breeding grounds for mosquitoes in the rainy season. DMT staff were aware of this and wherever possible, located latrines as far away from swamps and no more than 50m from dwellings. In some locations however, graves caused latrines to be located up to 70m from dwellings and closer to swamps.
14
Several of these concerns may be resolved through either increased numbers of latrines and/or different designs, as discussed below.
With regards increased numbers of latrines, although the project achieved its design of 650 latrines for 13,000 people, there was evidence that in some cases more than 20 individuals were using each latrine (see 4.4.3). It is widely accepted in the sector that latrines provided on a household basis result in a higher sense of local ownership, correspondingly higher usage, maintenance and cleanliness, and thus enhanced sustainability of health improvements. A simple design that relies on local inputs and can be readily replicated is therefore desirable (e.g. Sinclair, 2004).
The evaluator recommends the construction in each village of demonstration ‘organic’ VIP latrines of a design that may be easily replicated. Only local materials would be used - timber and mud floors, ‘thatch’ walls and roof, and either a wooden or maze door. Large diameter bamboo (100mm +) is locally available. This may be used for the vent pipe if either augured or split in half and the nodes cleared, and re-sealed using clay and binding. Although 100mm diameter pipe achieves the desired updraft (if located in a suitable location) with smooth pipes (e.g. PVC), for bamboo a minimum diameter of 150mm is recommended. Likely issues are cleaning mud floors and corrosion of the vent pipe (due to a combination of noxious gases and moisture). Only the former may be of serious concern - the pipes may be replaced. The use of concrete latrine slabs would overcome this but negate the concept of ‘organic’.
Alternatively, modified traditional VIP latrines are possible. Traditional VIP latrines are expensive to replace due to the PVC or equivalent vent pipe being cast into the slab and invariably breaking if attempts are made to move it. The evaluator suggests cutting off a short length of the female socket end and embedding in the slab (Figure 6). The straight length of pipe (min 100 mm dia.) may then be readily removed and the slab relocated when necessary.
Figure 6: VIP latine with Open door latrines (i.e. no door but a maze type entrance to provide removable vent pipe privacy) will reduce odours and are cooler if the opening faces the prevailing wind direction. However snakes and other creatures are likely to be a problem. Furthermore, experiences in Bomi II communities were that people were concerned for the safety of women and girls if, for the sake of privacy, latrines faced away from the community and into the bush. Further promotion of the use of ash to reduce odours is likely to be necessary.
To facilitate the use of latrines by the elderly and disabled, handrails in the walls of latrines, a rope or ropes to hang on to (being aware of the likelihood of children using them as swings), or a box (wooden or concrete) to sit on are potential remedies (see also Jones and Reed, 2005).
There are important issues concerning different types of latrines and especially the different levels of end-user versus Tearfund contribution. In a given community, if Tearfund provides all or some materials for ‘a chosen few’, with those not served expected to replicate demonstration organic latrines, evidence suggests strong tensions are likely to result, with subsequently low or non-uptake of the organic latrines (related to a major benefit of latrines being the high end-user status afforded to them). Social marketing may change this. Equally, evidence both from Tearfund’s work in Bomi and elsewhere, indicates that tensions over differing types of latrines and contributions between communities of end-users, as opposed to within a single community, are soluble through social marketing. Thus for new communities, it seems reasonable to expect
15 that with adequate social marketing, the provision of demonstration latrines only can result in high uptake, even where neighbouring communities have different Tearfund inputs (i.e. slabs and possibly iron sheets). It is more likely that ‘competitive tensions’ will be lower the more remote the new communities are from ones that have received inputs in the past. However, given the highly variable nature of the social dynamics involved, a high degree of both social marketing and monitoring, with a flexible approach would be essential to ensure uptake is as desired.
Summarising the above, the following recommendations are made:
Recommendation:
Recommendation: construction in each village of demonstration ‘organic’ VIP latrines, using only locally available materials to ensure replicability. Experimentation with demonstration latrines only in one new village or a cluster of villages, preferably remote from currently served villages. High levels of social marketing and monitoring required.
Recommendation: provision of modified VIP latrines with short length of female vent pipe socket embedded in the slab.
Recommendation: further promotion of the use of ash to reduce odours.
Recommendation: experimentation with the use of handrails in the walls of latrines, a rope or ropes to hang on to, or a box to sit on for the elderly and disabled.
Recommendation: continued social marketing and promotion of latrine use with appropriate monitoring of usage.
o Handwashing facilities at latrines
Although it was reported that hand-washing was widely practiced, it is likely that with latrines being located up to 100m from houses, hand-washing after defecation may be practiced less than desired: children in particular may be distracted before reaching the hand-washing facilities. The provision of tippy-taps also have the benefit of allowing two handed hand-washing (Figure 7).
Recommendation: the provisi on of a hand-washing facility at latrines, for example tippy taps or containers (drums) with Figure 7: Tippy-tap (SW Uganda: taps. Photo: Richard Carter)
o Collection and storage of water in open containers
In Phase 1, UNICEF resettlement collapsible jerry cans were provided as part of the NFI package to communities for collection of water from source and storage. These have proved relatively weak, with many end users reporting breakages. As a result, in Phase I sites it was observed that water was being collected and stored in open containers, with subsequent (potential) contamination prior to drinking. The problem of dipping for water from open storage containers has been mentioned.
16
For Phase II Tearfund replaced the UNICEF containers ($3.5) with much more durable locally purchased jerry cans ($5). These provide greater protection. The UN no longer considers Liberians as returnees, but as entering a development phase, thus NFIs are not encouraged. Furthermore, some individuals cannot afford jerry cans.
Recommendation: additional funds be provided to allow jerry cans to be purchased and distributed to Phase 1 communities to replace the UNICEF buckets. Tearfund could either distribute themselves or through other NGOs/UNHCR.
o Water safety plans
Linked to many of the above issues is a recommendation that Tearfund develops a water safety plan (WSP), many aspects of which it is already implementing. WSPs are currently being vigorously promoted by the World Health Organisation (WHO) and are seen as the most effective method for ensuring safe drinking water. Many NGOs are ‘jumping on the bandwagon’ of what is likely to become a core feature of required best practice. Annex 3 gives further details of WSPs and specifics for Nimba, but in summary, a WSP considers all aspects of water safety from the water source to the consumer and includes:
¾ hazard and risk identification (e.g. open buckets of stored water); ¾ operational monitoring (e.g. assessment of HH using covered storage containers); and ¾ management and documentation (e.g. education of risks associated with uncovered water storage).
In the context of Nimba county, Tearfund could develop a pictorial WSP for each community which could be positioned on a board next to the pumps. The WSP would need to include information such as the name and location of the CWA, CDC chairman, supporting NGO, as well as a full description of the supply chain (groundwater to HH storage) with hazards clearly identified. The ‘control measures’ (e.g. soak pits free of standing water) should also be shown and the actions needed to be taken if the control measures are breached (e.g. clean out silt from soak pits).
Recommendation: Tearfund develop a water safety plan for Nimba County (which could provide the basis for a WSPs throughout the organisation).
o On-going support
As previously mentioned (1.2.2), Tearfund recognises that to sustain the benefits of the ECHO funde d phases beyond the current funding period on-going support by a backstopping agency is required. Initial community enthusiasm wanes and members change, thus refresher training, re- training and encouragement of communities are critical. Furthermore, there is always a need for a back-stopping agency for major repairs that are beyond the capacity of the community to undertake. Thus calls by, amongst others, Shordt & Snel (2002) to increase project funding by donors beyond the currently typical 3-4 years are pertinent here.
Tearfund’s exit strategy (planned for the end of 2008) is discussed in exit strategy (4.5.3-).
• Logistics and coordination The majority of logistical issues that negatively impacted effectiveness are discussed under Efficiency (4.4), where recommendations are made. Additionally:
17
- The County Capital, Saniquellie, is located a seven hour drive from Tappita (Phase II base) and 5 hours from Saclepea (Phase I satellite base). The FC therefore needed to spend approximately 10% of his time attending (necessary and valuable) monthly meetings. - Running an ambitious ECHO programme in parallel with a WRC programme, whilst having some benefits, inevitably stretched staff. There were huge pressures and problems with purchasing, and sector managers lamented the lack of time for coordination between sectors over what messages are given to communities, for example, who should fence pumps.
4.1.5 Conclusions and assessment
It is a credit to programme staff that the ambitious outputs of the programme have been substantially achieved, and the longer-term objectives are in a position to be achieved. However, this achievement of the core objective of sustainability is potentially threatened by the relatively short term nature of the programme: in this regard, the ECHO funded programme is no different from any other. DMT is aware of the need to give sufficient attention to an appropriate exit strategy: this will require the presence and resourcing of on-going support and a back-stopping agency (see 4.5.3).
4.2 Impact: of projects on beneficiaries and non-beneficiaries
4.2.1 The intended and unintended effects (social, physical, environmental, economic, spiritual), both positive and negative, of the project on beneficiaries and non- beneficiaries
In that the programme has been largely successful in achieving its objectives, the effects on beneficiaries have been as intended. During the evaluation, individuals cited the following instances in addition to those already mentioned of how the project had positively impacted them:
- Conflict resolution: in Yarsonah (Phase I), one man stated the main project benefit was receiving advice on how to relate to his family, cope with angry children, and manage his own temper. Similarly, in Kpeikpoa (Phase I) a man stressed that conflict resolution training has seen an end to court cases as disputes are settled before reaching the courts. - Change in sexual behaviour: an elderly man in Geapa (Phase II) stated that after the HIV training, he decided to stop sleeping with his 4 wives and others indiscriminately. (It was not clear whether this individual subsequently chose only one wife to sleep with. There are associated issues of inclusion/exclusion with this). - Esteem: in Geapa the community felt overlooked that no-one ever came to their village previously to discuss matters with them, but felt important and expressed gratitude that Tearfund had done so. - Development and commerce: Jesuit Refugee services (JRS) wait for Tearfund to build bridges and improve roads (under the WRC funded work) before sending materials for schools in several communities. Similarly local merchants are benefiting from the improved access to communities. The community in Yarsonah has already been mentioned for building a school as a result of the CDC training.
18 4.2.2 The major factors influencing the impact of the project on beneficiaries and non- beneficiaries.
Again, much of what has been written in 4.1.4 applies here. Additionally:
- Recognition of CDCs by Government: CDCs are becoming recognised bodies by the Government, with a Government instigated chain of CDC → DDC (District Development Committee) → Commissioner (Sub-District) → Superintendent (District). - Gender sensitivity: in CHV and CDC selection, attempts are made to attain a 50/50 gender mix. T-shirts are given to husbands of CHVs which has often led to them voluntarily helping their wives with CHV activities. In financial issues women are generally trusted more and more committed to the success of projects, thus enhancing project success. - HIV/AIDS and health education in schools: schools are easily reached and very effective for dissemination of messages through songs. In all communities visited, children continuously repeated such messages.
4.2.3 Conclusions and assessment
The influences that threatened to negatively impact the programme have been considerable: programme staff have worked hard to militate against them. There is still work to be done, particularly in the relational aspects of firstly working with communities in the social marketing latrine structures and the need for a participatory development attitude, and secondly with other NGOs and Government to ensure uniformity of approach.
The fact that the programme has achieved its outputs underlines how hard staff have worked. The longer term impact depends upon successfully addressing several sustainability issues.
4.3 Relevance: of the programme to the priorities of the target group, the partner and Tearfund
4.3.1 The validity of the project objectives
Section 2 and the ToR (Annex 1) detail the current situation in Liberia and the background to it. Given the circumstances of Liberia’s returnees, from a humanitarian perspective, there is no question that the programme objectives are entirely valid and necessary. The likelihood that they will remain so beyond Tearfund’s planned exit time is addressed under 4.5.3.
4.3.2 The relevance of the individual projects with the needs and vulnerabilities of the target group
Initial needs assessments and stakeholder analyses conducted by Tearfund and partners (see Annex 1) provided the basis to enable the projects to be relevant to the target group.
Although not statistically valid, broad confirmation of this was gained through an exercise with some interviewees during the detailed semi-structured interviews. Having stated needs before and valued things now, individuals were given a pile of ten stones to prioritise and rank their responses. The results shown in Figure 9 confirm that water and latrines were the highest felt
19 needs before the project, and health afterwards. It is noticeable that prior to the project, health seems to have been associated with water and latrines.
The following are further indications of the projects’ relevance:
- The CDCs make legally binding by-laws to ensure the maintenance of pumps, cleaning of communal areas, etc; - Many cited a major benefit of latrines being that they used to go to the bush but are now sitting comfortably and safely in latrines; - Each community is given time to discuss think before signing a partnership agreement detailing contributions to be made. All communities have made the Figure 8: Walled pump collection area, requested contributions; Kwendin (Phase II) - In Kwendin (Phase II) the pump area was walled by the community meeting full costs - 4 bags of cement and 260 bricks (taking 3 days to make – Figure 8); - Communities are given a list of spares needed for pump maintenance, and their prices in Monrovia. In the past people have been exploited being charged L$50 for labour plus parts.
Priority needs before project (n=8) Most important thing now (n=14) 1st 2nd 3rd Score Rank 1st 2nd 3rd Score Rank choice choice choice choice choice choice Health, mosquito 8 1 43 1 nets, medication Water 3 2 2 23 1 Roofing materials 1 2 1 12 3= 2 2 16 3 Latrines 1 2 1 12 3= Food 2 1 1 14 2 Roads 1 3 6 School/education 1 5 5 3 1 1 19 2 Peace 1 1 8 4 Income generation 2 6 5 NFIs 1 1 6
Figure 9: Ranked prioritisati on of ne eds before the proje cts and th ing most valued no w
The following question the relevance of some a spects:
- Although mosquito nets were distributed to the vulnerable, no-one in Geapa (Phase II) bought additional mosquito nets. There were no results for other communities. - In Yarsonah (Phase I), many desired something other than ashes to keep the smell from latrines down. Issues related to heat, dignity, closeness to swamps, and difficulty in squatting for the elderly and disabled have been mentioned in 4.1.4.
20 - Again in Yarsonah, many of the tin sheets supplied for latrine doors were sold and replaced with wood (costing the same L$250) amidst claims that wood is more durable. However, in other communities the same is said for zinc. - Details have already been given of the suitability of CWA training in 4.1.4.
Recommendation: Continued promotion of ash for reduction of odour from latrines, as well as the use of mosquito nets.
4.3.3 The use and enhancement of the existing skills, knowledge and coping strategies of the target group
Indicators of this have been:
- Other NGOs often contract out construction (e.g. rings for HDWs, latrine slabs and roofs) but DMT has resisted this to ensure community capacity for sustainability and replicability where appropriate. Carpenters have been trained for latrine infrastructure construction. - All communities wanted to provide bricks for the walls of latrines. - All beneficiaries accepted the project management/promotion groups formed within communities (CDCs, CHVs and CWAs). The selection of these groups from various sectors within the community and the representation of both men and women have facilitated this acceptance.
4.3.4 The consistency of the projects with the vision, values, strategy and resources of Tearfund
From the ToR (Annex 1), the only slight inconsistencies are with the greater emphases placed on HIV/AIDS and food security in Tearfund’s Region/Country strategy.
The PHE sector manager amongst others emphasised the need for a separate HIV/AIDS sector to address this key area appropriately – this was accomplished as of 1st September with the employment of a HIV project manager. Although food security is not directly addressed through the ECHO funded work, the parallel WRC funded programme adequately addresses it.
4.3.5 The major factors influencing the relevance of the project to the priorities and policies of the target group and Tearfund
Beyond the key factors already mentioned - multi-stakeholder baseline surveys and needs assessments, staff commitment, and a learning, adaptive, end-user focussed culture - Nimba is known for its female leadership. From group discussions it was evident that whilst there are still gender issues to be addressed, there is not as much gender divide as elsewhere in sub-Saharan Africa.
4.3.6 Conclusions and assessment
Although the stated priorities of the partner ECHO were not available, it can be assumed that through continued funding, the project objectives are of relevance to ECHO. With regards Tearfund and the target groups, whilst there are some minor inconsistencies and concerns, overall the programme is highly relevant to both, and through adaptation has remained so.
21
4.4 Efficiency - in achieving the intended objectives
4.4.1 The cost-efficiency of project activities
There is insufficient data to be certain of the exact numbers of beneficiaries of the various activities (Annex 4). For example, some of project sites had existing HDWs; for others population data does not exist.
However, the project sought to provide:
- potable water for 25,600; - functional latrines accessible by at least 13,000; - PHE for 26,700 beneficiaries.
With a total projected cost of €678,000 (end August 2007), the cost of €25 per beneficiary of PHE, with the majority (96%) having potable water and half having latrines, is very reasonable, and thus it may be concluded that overall, the project has been cost-effective.
4.4.2 The achievement of objectives to time and to budget
Every indication is that the project will achieve its objectives to time and budget (although the issue of long-term sustainability needs addressing).
Within the budget, whilst many items were over budget - notably CD/PHE training materials (+86 to 138%), children’s health club/youth sports club kits (+92%) and office stationery (+106%) – their values of over expenditure were relatively small.
Only vehicle running costs (+110%) represented significant over expenditure (approximately +€17,000).
4.4.3 The comparison of project practice against accepted good practice for the various project sectors
Specific areas of best practice have been detailed in 4.1.4 (see also Figure 11). The programme meets the ICRC Code of Conduct. With regards Sphere standards, Figure 10 summarises observations during the evaluation and concerns already raised. Recommendations have already been made to address these concerns.
Standards Key indicators Hygiene promotion See Excreta disposal, Vector control and Ditto Drainage, below Water supply Some collection/storage in open Soap not always present; some coliforms containers (typically 1/100ml) present (Phase I) Excreta disposal Some concerns re. number of toilets and >20 users per latrine; some latrines >50m usability by elderly/disabled from dwellings; Vector control Some latrines close to mosquito Only the vulnerable have mosquito nets breeding sites Solid waste management Good Good Drainage Most HDWs observed have some standing water Figure 10: Summary of programme practice against Sphere standards and key indicators
22 Further detail can be found in previous reports, notably the Phase I and Phase II Pre-Final Reports. Details of the outputs of HDWs and latrines (Annex 4) indicate that whilst it was planned to construct 650 latrines for 13,000 people, if communities (and latrines) where no population figures exist are ignored, 621 latrines have been constructed in communities with a total population of 25,158. Although it is unlikely that all community members would share these latrines equally, if they did there would be an Figure 11: Clothes lines and drying racks as a result of the PHE work, Kwendin average usage of 41 people per latrine. This emphasises the need for demonstration latrines made in 4.1.4.
In summary, allowing for the recommendations already made, the standards of practice are generally very high, with extremely committed and dedicated staff achieving targets.
4.4.4 Comparison of the implementation of the projects with alternatives
Alternatives such as latrine structures, hand-washing facilities, and CWA training have already been recommended in 4.1.4.
4.4.5 The major factors influencing the efficiency of the projects
Factors contributing to efficiency were:
- Storage of fuel in Monrovia at Tearfund’s supplier’s warehouse, allowing sufficient fuel to be brought by tanker to Tappita during in-country shortages. - A mechanic in Tappita enabling staff motorbikes to be maintained locally. - Coordination of vehicle use which according to sector managers improved greatly from that occurring at the outset between the sectors. - PHEs staying in villages during the week, enabling training at times to suit end- users, and expressing solidarity with them (Freire, 1996). - Training of trainers (i.e. CHVs) by PHEs with a resultant multiplier effect compared to training communities directly. - Shared purchasing and transport costs with WRC programme of materials from Monrovia. - Many staff recruited from Bomi, thus building on experiences gained there.
Factors that contributed to inefficiency were:
- The late realisation by Tearfund that ECHO purchasing requirements were different from those of most other organisations including Tearfund. This led to a 3 month delay in procurement while additional quotes were sought for item. During this time, software aspects of community sensitisation, mobilisation and PHE started.
23 - Poor access and poor quality roads. General remoteness, a 6-8 month rainy season, and road quality meant the few vehicles in the programme required frequent maintenance. - Heavy usage of the 2 dewatering pumps. Again, high maintenance resulted. - A centralised maintenance policy. Current policy is to only use the Toyota dealers in Monrovia who are swamped with work. Similarly, delays in maintenance in Monrovia of the dewatering pumps led to further delays as the sides of partially completed hand dug wells in Kwendin (Phase II) collapsed, requiring cleaning out. It is likely the HDWs in Phase II sites would have been finished on time if this was not the case. - Only one warehouse keeper per warehouse, who was also responsible for transport matters and therefore often not present when needed. No budget was available to recruit a dedicated warehouse keeper. - Very short time span of project. Issues that affected timeliness had a magnified impact. - Delays in purchasing due to centralisation i.e. all through Monrovia. Nimba logistics is poorly resourced with no computer, in part due to lack of capacity.
Recommendation: improved monitoring of internal systems by Tearfund to ensure compliance with partner requirements.
Recommendation: to address maintenance issues for vehicles and plant, one of the following: - use of DMT approved dealers other than Toyota; - decentralisation of maintenance where capacity exists down country; - employment of a dedicated fleet mechanic.
Recommendation: recruit a dedicated warehouse keeper.
Recommendation: improved resourcing and training to enable appropriate purchasing by Nimba logs.
4.4.6 Conclusions and assessment
There is much overlap between effectiveness (Aspect 1) and efficiency, and they share many factors that promote or militate against the programme achieving its objectives.
Generally, the programme has worked very hard to meet or exceed the outputs on time and to budget.
4.5 Sustainability - of project benefits after the end of the projects
4.5.1 The continuation of project benefits after funding by Tearfund has ceased
Relevant comments have been made in 4.1.4 or will be made in 4.5.3.
4.5.2 The major factors influencing the sustainability of the project
Again, the majority of relevant factors have been / will be discussed in 4.1.4 and 4.5.3. Additional factors were:
24 - Training of Nimba finance and logistics departments by Monrovia DMT staff. Two days training in core competencies including Tearfund and ECHO’s procurement systems was given (although the need for adequate resourcing remains). - Slow uptake of the concept of the biological sand filters (BSFs) by the communities Staff found it difficult to persuade them to drink ex-muddy water, highlighting how technologies cannot be seen in isolation from culture. - Local legislation. Johnson Zindo, the Town Chief of Nyasin (Phase I) is strict about barring those who have not contributed to the project from drinking from HDWs, and fines them L$50 for doing so. - Government concern for sustainability. This is demonstrated by their buy-in to the CDC structure. The DDCs (2 year tenure) evaluate the performance of CDCs (no fixed tenure) and try to evict some. DMT reviews the complaint and assesses whether to replace or (re)train the CDC through training elsewhere, although no funds or manpower are available for further training. This need and on-going support is discussed in the following section.
4.5.3 Key recommendations on effective exit (including options for handover to other agencies, community level sustainability mechanisms etc) for Tearfund from Nimba
This discussion primarily follows on from 4.1.4 where it was established there was a need for on-going support by a backstopping agency to sustain project benefits, especially behaviour changes. Such support would include discreet monitoring, refresher training and exchange visits between communities, and if a local partner were used, sufficient capacity building and resourcing of that partner to ensure sufficient competence (Batchelor et al, 1999 & 2001).
Essentially the question of exit strategy boils down to what needs to be done, and by who? One fundamental issue that needs addressing is who is driving the exit strategy, and with whose interests prioritised? Linked to this, what combination of on-going support, capacity building, and direct implementation should future involvement include?
With regards the drivers of Tearfund’s exit strategy, as stated, the UN has declared Liberia to be entering a development phase. The semantics of relief/development debates, and organisational structures that divide the two should not over-ride the needs of those currently unserved in Liberia. Donors are pressurising Tearfund and others to upscale, as evidenced by the recent large grant won by a Tearfund inspired DFID consortium. This highlights a disjoint between emergency and development wings in TF. As part of the consortium, Oxfam has the country mandate to build the capacity of local authorities, such as the Ministry of Health.
If an end-user focus is adopted, there is no doubt in the evaluator’s mind that the needs in Liberia generally and Nimba county specifically are still significant enough to warrant continued implementation. With Tearfund being the main NGO in Nimba county, a strategy is required that enables the benefits of the ECHO funded phases to be sustained and extended. It is therefore recommended that Tearfund considers continuation of activities to meet current needs for new projects and provide on-going support to completed ones (with on-going support potentially involving partner capacity building).
Assuming Tearfund exits as planned at the end of 2008, this could be achieved in several ways. The following are relevant factors intermingled with possible ways forward:
25 - There is currently a lack of capacity amongst (potential) local partners for the support (and/or continuation) of WATSAN activities. - Tearfund Monrovia staff recognise from experience that it takes considerable time to capacity build partners, especially with proposal writing and raising funds. For this reason, they are willing to work as collaborators in implementation/on-going support but reluctant to work as partners. - There are differing perspectives within Tearfund as to the capacity of its current PHE partner, EQUIP, to provide on-going support for the PHE component. Potentially, if adequately resourced with personnel (from existing staff), funds and equipment (e.g. motorbikes), it could provide the required support in the PHE sector, one that it has a proven track record in. - Again, opinions expressed during the evaluation as to EQUIP’s potential capacity for WATSAN implementation vary. On the one hand concerns were expressed by Teddington staff over EQUIP’s management structure, accountability, and their dependence on one individual’s vision and leadership. DMT field staff report improving relations with EQUIP management and increased confidence in their capacity to undertake WATSAN implementation (if adequately resourced) without undermining their PHE work. - The Government plans to create a structure in the Ministry of Water similar to that in the Ministry of Health where District Community Development (DDC) officials integrate with CDCs. If these plans materialise, Tearfund could build the capacity of personnel in such a structure in the same way as those in the DDC to provide sustainability and on-going support for WATSAN activities. This could be in conjunction with EQUIP taking responsibility for PHE. However, there are frequently logistical problems with Government departments, including problems with poor pay and resources. - As an alternative to capacity building the Government, Tearfund could build the capacity of the Liberian National Red Cross Society – a local partner to IRC. They use a very similar method to DMT for WATSAN construction and training, though would require personnel and funding. (In a similar way, WaterAid has recently broadened its horizons with regards local partners in Tanzania through advertising for any organizations capable of assisting in implementation. A number of private sector organizations now work alongside government and NGOs to good effect (Sinclair, 2004)). - A new local partner organisation consisting of (some) existing section heads and support staff, headed by a suitably proficient ‘programme director’, could be formed to meet PHE and WATSAN needs. With Liberia in a transition into a development phase, longer-term funding for development activities is recommended. Oxfam, ACF are amongst NGOs in Liberia who have formed such organisations from former staff to achieve this, although there have been challenges. However, Tearfund’s regional team and partners have requested DMT not to set up a new NGO, therefore this idea is unlikely to be implemented. - If the Tearfund Mano River programme partnerships in Sierra Leone reduce or close, an expatriate could be based in Monrovia as the Liberia Country Manager, (with a suitable overlap with DMT Liberia).
There are therefore many unknowns: - If the capacity of EQUIP and as importantly, the perception of its capacity continues to improve, they could provide: on-going support to either the PHE or PHE plus WATSAN activities; capacity to implement new PHE or PHE plus WATSAN activities;
26 - If the Government creates the Ministry of Water structure outlined above, they could provide on-going support and possibly implementation capacity to WATSAN; - If Tearfund were willing to capacity build/partner with the Liberian National Red Cross Society, they could provide the same outputs as the Government above.
In addition to these unknowns, there are key stakeholders, namely Tearfund’s regional team and partners, who were not included in the scope of the ToR, but who clearly would be affected by the exit strategy and have an influence in the decision.
Therefore to make feasible exit recommendations, due to the unknowns and the considerable variation in opinions between those consulted within Tearfund, it would be necessary to both monitor the evolution of Government structure and relationship with EQUIP, and consult more widely than the ToR scope of this evaluation allowed for.
Recommendation: monitor the evolution of Government structure and relationship with EQUIP, and Tearfund reach consensus through systematic assessment as to the relative capacity of, and pros and cons of partnering with EQUIP, Government and the Liberian National Red Cross Society.
4.5.4 Conclusions and assessment
It is too early to state with any certainty whether long-term sustainability is likely to be achieved for the reasons stated in 4.1.4. However, with the recommended on-going support (and ideally further implementation) there is every reason to believe sustainability will be achieved.
4.6 Coordination and coherence – integration of projects with the activities and priorities of other agencies and organisations (including local and national government)
4.6.1 The linkages and relationships between the target group, Tearfund, and other agencies and organisations
CDCs and to a lesser extent CHVs and CWAs are the focal points of linkages between Tearfund and the target group. Relationships are cited by these groups and individuals as extremely good, and gratitude was always expressed in villages visited for the work that Tearfund has undertaken.
There are several pieces of evidence of good linkages and coherence with other groups:
- Meetings with Government and coordination with other NGOs prevents overlaps of activities and ensure common practice. The need for this was evident in Kwendin where initially there were problems mobilising labour as ACF had formerly paid for digging of HDWs. - Both DMT staff and field staff of the local NGO EQUIP reported a very good relationship between Tearfund and EQUIP in the area of PHE, with Tearfund benefiting from initial contact and assistance with vehicles (motorbikes) and EQUIP from Tearfund’s status. Monthly meetings are held with EQUIP and the County in Saclepea. National meetings are handled through the Monrovia office. - In Tappita only the Jesuit Relief Service works so the Government is very appreciative of Tearfund’s presence, especially with Nimba being one of the priority
27 areas in the Government’s Country Framework Plan. An indication of this is the Government ensuring that Tearfund is present at the monthly District meetings.
4.6.2 The coherence of the projects with national and local policies
See above.
Additionally, during an interview with the Superintendent, Paramount Chief (Tappita District) and Development (Vice) Superintendent, all were very pleased with Tearfund’s liaison with the office, asking what the needs are, the way Tearfund fitted in with Government strategy, and their high quality work.
Tearfund is also certified by the Government to handle food security activities in Liberia and well recognised in others
4.6.3 The major factors influencing coordination and coherence of the projects with the activities and priorities of other agencies and organisations
The relevant factors have already been mentioned: staff commitment; professionalism; hard work; good coordination including attendance at key meetings.
4.6.4 Linkages to the WRC funded Food Security Project
There are several linkages between the ECHO and WRC programmes that are of mutual benefit:
- The WRC programme has enabled some of the achievements of ECHO through building bridges into joint WRC/ECHO funded communities. A total of 14 bridges have been constructed along with 40km of road side brushing in the ECHO project area. - WRC training on nutrition has compliment ECHO PHE work in particular, and vice versa. - The area coordinator and sector managers are the same for the ECHO and WRC programmes which ensures uniformity of approach and coordination of resources such as vehicles. In the field the two programmes dovetail well.
4.6.5 Conclusions and assessment
The programme exhibits extremely good coordination and coherence with all relevant stakeholders.
4.7 Participation: the extent to which the project stakeholders were involved in the assessment, planning, implementation and evaluation phases of the project
4.7.1 The quality of relationship between Tearfund and the project stakeholders
Further to comments made in 4.6, all stakeholders – Government, beneficiaries and local partners – spoke highly of the relationship with Tearfund. Additionally, there exist good relations with the UN and WFP.
28
Of note is high level of Tearfund’s accountability to the beneficiaries, as demonstrated through the Partnership Agreement and subsequent tracking sheets that are signed by communities to ensure materials are delivered and used according to the agreement.
The only possible concern that became evident during the evaluation was the frequent use of phrases by the communities such as “We did all we were told to do” (a male in Yarsonah referring to digging the latrine hole). The value placed on project inputs will only become evident in the long-term through (lack of) maintenance, although the replication of demonstration latrines may be another indicator.
Recommendation: long-term monitoring of the ownership and maintenance (and where appropriate replication) of project inputs.
4.7.2 The level of participation of the stakeholders in all stages of the project cycle
Participation was evident in the following areas:
Assessment and planning. - The Township Commissioner was very pleased with the process of involvement of communities whereby their water needs were expressed to the Government and UN and Tearfund sent an assessment team. - The communities themselves were not highly involved in the initial baseline and KAP surveys as they were settling having recently returned from camps and the bush. However, in subsequent planning and work, e.g. the siting of HDWs, they were.
Implementation. - Community contributions included unskilled labour and locally available materials. - Non-contributors are often ex-combatants (especially if now aged 18-30 thus former child soldiers) with little respect for authority and unable to go back to their original communities.
Monitoring and evaluation. - In recognition of the benefits of “professionals and the population learn(ing) together (to) synthesize new knowledge” (Wilson, 2006), during this evaluation, communities and DMT staff were encouraged to reflect on the projects through SSIs. - CHVs – community members themselves - undertake daily monitoring of community health through house to house visits, reporting weekly to Tearfund. - Exchange visits are planned between communities of Phase II project sites as well as visits by Government District officials.
4.7.3 The involvement of all the beneficiaries, ensuring gender balance and inclusion of the vulnerable
The above and section 4.2 detail the majority of involvement. Additionally:
- The elderly, under 5’s and pregnant women were prioritised with mosquito nets and food for work.
29 - Nimba is known for female leadership. From group discussions it was evident that there is not as much gender divide as elsewhere in sub-Saharan Africa. Especially in Kwendin (Phase II) where the town chief is female, very strong female leadership of the CDC was observed. The Township Commissioner remarked “women are the strength behind us when it comes to community development”. DMT staff remarked on how many women were involved in ‘manual’ labour in digging latrines and HDWs.
4.7.4 Conclusions and assessment
Generally, the levels of participation of the various stakeholders were appropriate throughout the project cycle, and the programme has achieved a good balance between inputs and contributions for the current state of the communities. Nevertheless, additional inputs of jerry-cans are necessary (see 4.1.4).
30 5 Specific Actionable and Prioritised Recommendations for future development of project in Liberia
The following recommendations (summarised from section 4) are prioritised firstly to achieve the objectives of the ECHO projects, secondly for future projects:
1. Additional funds to allow jerry cans to be purchased and distributed to Phase 1 communities to replace the UNICEF buckets (4.1.4). 2. Construction in each village of demonstration ‘organic’ VIP latrines, using only locally available materials to ensure replicability. Experimentation with demonstration latrines only in one new village or a cluster of villages, preferably remote from currently served villages. High levels of social marketing and monitoring required. (4.1.4). 3. Experimentation with the use of handrails in the walls of latrines, a rope or ropes to hang on to, or a box to sit on for the elderly and disabled (4.1.4). 4. The provision of a hand-washing facility at latrines, for example tippy taps or containers (drums) with taps (4.1.4). 5. Development of a water safety plan for Nimba County (which could provide the basis for a WSPs throughout the organisation) (4.1.4). 6. Continued social marketing and promotion of latrine use with appropriate monitoring of usage (4.1.4). 7. Continued promotion of ash for reduction of odour from latrines, as well as the use of mosquito nets (4.3.2). 8. In the short-term (during the implementation time period), follow up training of trained personnel through an intensive one day of one-to-one interaction is recommended. In the long-term, on-going support to provide follow-up and refresher training (4.1.4). 9. Monitor the evolution of Government structure and relationship with EQUIP, and Tearfund reach consensus through systematic assessment as to the relative capacity of, and pros and cons of partnering with EQUIP, Government and the Liberian National Red Cross Society (4.5.3). 10. Provision of modified VIP latrines with short length of female vent pipe socket embedded in the slab (4.1.4). 11. Ensure sufficient resources (time, personnel, finances and transport) are made available for promotion of income-generation projects (4.1.4). 12. Pre-selection screening of CWAs, CHVs and CDCs to assess levels of required literacy, or preferably communication of required teaching in an accessible and effective format. (4.1.4). 13. Assessment of the ties trained personnel have to the community. Family (especially children) and land are the main two (4.1.4). 14. To address maintenance issues for vehicles and plant, one of the following: use of DMT approved dealers other than Toyota; decentralisation of maintenance where capacity exists down country; and employment of a dedicated fleet mechanic. (4.4.5) 15. Improved resourcing and training to enable appropriate purchasing by Nimba logs (4.4.5). 16. Recruit a dedicated warehouse keeper (4.4.5). 17. Ensure on-going reflection and adaptation of programmes through continuing to formally schedule events that facilitate this (4.1.4). 18. Long-term monitoring of the ownership and maintenance (and where appropriate replication) of project inputs (4.7.1). 19. Improved monitoring of internal systems by Tearfund to ensure compliance with partner requirements (4.4.5).
31 Annex 1: Terms of Reference for the Evaluation
The following is verbatim from the Terms of Reference for the DMT (ECHO funded) rehabilitation programme in Liberia:
BACKGROUND Programme Title: Integrated WatSan and Public Health Promotion in Nimba County, Liberia
Background to the Programme: Nimba County is located in the Northeast of Liberia and shares a border with the Republic of Côte d’Ivoire to the east and the Republic of Guinea to the Northwest. Nimba was one of the counties worst affected by the civil war, its population was severely decimated and its infrastructure massively devastated. Most of its citizens who survived the calamity were forced to seek asylum in neighbouring countries or were displaced mainly in other counties. Pre-war population estimates for the County vary between 400,000 (UNICEF) and 800,000 (local government).
Nimba was among the last of the counties to be declared safe for the return of refugees and IDPs, this only happening in February 2005. The official repatriation process did not take place until recently. UNHCR's official figures as of 30th April show that only 756 out of 44,962 (1.7%) registered IDPs and refugees have resettled in the county. However large numbers have returned spontaneously and there are many war-affected people in need of assistance. The response from the international community has mainly been concentrated along the main access routes around Ganta and Saclepea, with large populations remaining untouched or with very limited assistance down in the South and South East of the county. Tearfund has focussed its work in these areas.
Tearfund began work in Nimba County in January 2006 with funding from ECHO. This project focussed on communities in Yarwein-Mehnsohnneh and Tappita Districts. The specific objective of the project is to ensure that re-settlers in targeted communities of Nimba County have improved access to and consumption of safe water, improved sanitation, hygienic practices and the socio-economic skills required to manage and sustain these inputs and address issues of governance, inclusiveness and reconciliation within their communities See logframes for project details). The project has been implemented in two phases (both of which will be evaluated), the cumulative outputs for both phases were as follows: • Potable water provided for up to 30,000 people of sufficient quality and quantity that is both accessible and reliable • Fully functional latrines accessible by at least 13,000 people in project communities • Public health knowledge, attitudes and practices improved in relation to key health and hygiene messages, including HIV/AIDS, amongst 30,000 beneficiaries (including women, children and youth) and underlying issues of inclusiveness and reconciliation within community addressed
Summary of findings of previous reports, evaluations An external evaluation of the DMT WatSan project in Sierra Leone commended Tearfund for the effectiveness of its community mobilization and local relationships, co-ordination within the NGO community and for the technical quality of the programme. Coliform testing on the protected water sources confirmed that all sources had less than 10 faecal coliforms per 100ml and the majority tested had zero coliforms.
A DEC evaluation of the AEL Watsan projects in IDP camps in August 2004, noted the shelter and advocacy work for their targeting and appropriateness. It also criticised AEL for the apparent poor effectiveness of their PHE
An external evaluation of the projects in Bomi in September 2005 found that the project had significant impact on the beneficiaries particularly in the area of hygiene promotion. The evaluation recommened further emphasis on capacity building and empowering communities to ensure project sustainability. It also suggested exploring the use of biological sand filters in the provision of clean water.
DMT profile, history and current activities Tearfund has been working in Liberia since 1995. From 1996 to 2000 DMT successfully completed a large-scale relief and rehabilitation programme in Lofa County and around the town of Greenville. Since
32 that time Tearfund has been working with its main local partner in Liberia, the Association of Evangelicals of Liberia (AEL). With the end of the 14-year civil war and the security conditions now in place to permit larger-scale return after the war, since October 2004 DMT has been undertaking a full operational programme in partnership with AEL. DMT is working in Bomi County, to continue to enable long-term rehabilitation and rebuilding of sustainable livelihoods of communities which have been devastated following the civil conflict. The project in Bomi has taken an integrated approach, with sectors addressing needs in the areas of food security, water, sanitation, public health education and socio-economic development. This programme is implemented in conjunction with AEL’s resettlement programme. In addition to Tearfund’s own funds, this work is funded by Irish Aid, the Jersey Government and Tearfund Switzerland. Tearfund also works in the districts of Tappita and Saclepea in Nimba County to improve food security, access to and consumption of safe water, and improved sanitation and hygienic practices. Tearfund works with village communities to develop the socio-economic skills required to manage and sustain these inputs and address issues of governance, inclusiveness and reconciliation within the communities. Tearfund’s work in Nimba is funded by ECHO, and World Relief Canada/Canadian Foodgrains Bank. At the end of 2006, Tearfund opened a third site in Sinoe County. Sinoe County remains one of the most under-served parts of Liberia and has been largely neglected by the humanitarian community since the cessation of civil conflict. The primary reason for this has been the difficulty in accessing this remote part of the country. ECHO, Unicef and UNFAO currently fund Tearfund’s work in Sinoe. The project is very similar to the aforementioned DMT project in Nimba County with a Watsan and food security focus and encourages more intensive yet sustainable agricultural techniques. Training in these techniques is complimented by building solidarity through the formation of farmers associations. Tearfund also works with the local administration and community leaders to address issues at policy level and at the practical level concerning the livelihoods and welfare of project beneficiaries. Summary of Region/Country strategy Goal: To support the development of sustainable peace in Liberia through the transformation of resettling and war-affected communities Purpose: To provide appropriate rehabilitation support to war-affected communities, and inform relevant local, national and international policies Outputs 1. Disaster Preparedness DMT prepared and enabled to respond rapidly and effectively to future conflict-affected people in Liberia and from surrounding countries, while operational in Liberia 2. Disaster Mitigation / Risk Reduction Within DMT target communities: • Communities empowered to undertake conflict transformation interventions leading to acknowledgement at the community level of their recent past and the need for peaceful solutions at the local and national level for future problems. • Individuals, and risk groups empowered to protect themselves against HIV infection, thereby reducing the impact of the AIDS pandemic on Liberia 3. Effective Assistance Conflict affected people receive timely and appropriate rehabilitation assistance in DMT core sectors of Water and Sanitation (Watsan) Public Health Education (PHE), and Food Security (FS) 4. Political Commitment Effective integration of advocacy approaches in project and programme activities on key selected issues to influence policy at local, national and international levels informed by field activities, aligned to DMT core sectors 5. Stakeholder Attitudes / Good Practice Acceptance and application of DM good practice, including individual sector good practice, beneficiary accountability, conflict sensitivity, gender disparity, child protection, HIV and Aids and Christian distinctiveness, with DMT projects and target communities 6. Partners / Local Church • Capacity of partners enhanced to implement quality rehabilitation projects and programmes in DMT specialist sectors • Local churches actively encouraged to be involved in DMT project community level activities
33 How the need for the requested assignment rose Donor requirement of Tearfund’s funding for this project.
PURPOSE
The aim of the assignment • To evaluate the effectiveness of the programme and the extent to which the programme purposes and results, as laid out in the programme proposals, have been achieved. • To identify lessons learned from this projects and make recommendations for similar programmes in other parts of Liberia [N.B An evaluation is the assessment at one point in time of the outcomes of a project. Outcomes refer to the effect the project has had in terms of the original situation and the original objectives. At a basic level, an evaluation asks whether project objectives have been achieved and whether the original aims and assumptions are valid. An evaluation is an integral part of the project cycle. If carried out well, an evaluation should increase transparency and allow all stakeholders to be able to influence the direction and emphasis of the project.]
Specific Objectives:
Aspect 1 – effectiveness Have the projects been effective in achieving the intended individual project objectives?
Areas for consideration include: • the achievement / likely achievement of the project outputs; • the achievement / likely achievement of the project purpose; • the contribution / likely contribution towards the project goal; • the major factors influencing the achievement / non-achievement of the project objectives.
Aspect 2 – impact What impact has the projects had on beneficiaries and non-beneficiaries?
Areas for consideration include: • the intended and unintended effects (social, physical, environmental, economic, spiritual), both positive and negative, of the project on beneficiaries and non-beneficiaries; • the major factors influencing the impact of the project on beneficiaries and non-beneficiaries.
Aspect 3 – relevance How relevant is the programme to the priorities and policies of the target group, the local church, the partner and Tearfund?
Areas for consideration include: • the validity of the project objectives; • the relevance of the individual projects with the needs and vulnerabilities of the target group; • the use and enhancement by the projects of the existing skills, knowledge and coping strategies of the target group; • the contribution of the projects to strengthening the local and national church in its mission and role to work with local communities; • the consistency of the projects with the vision, values, strategy and resources of Tearfund • the major factors influencing the relevance of the project to the priorities and policies of the target group and Tearfund.
Aspect 4 – efficiency Have the projects been efficient in achieving the intended objectives?
Areas for consideration include: • the cost-efficiency of project activities; • the achievement of objectives to time and to budget; • the comparison of project practice against accepted good practice for the various project sectors • comparison of the implementation of the projects with alternatives; • the major factors influencing the efficiency of the projects.
Aspect 5 – sustainability Will the benefits of the projects be sustained after the end of the projects?
34
Areas for consideration include: • the continuation of project benefits after funding by Tearfund has ceased; • the major factors influencing the sustainability of the project. • Key recommendations on effective exit (including options for handover to other agencies, community level sustainability mechanisms ertc) for Tearfund from Nimba
Aspect 6 – coordination and coherence How have the projects been integrated with the activities and priorities of other agencies and organisations (including local and national government)?
Areas for consideration include: • the linkages and relationships between the target group, the local church, Tearfund, and other agencies and organisations; • the coherence of the projects with national and local policies; • the major factors influencing coordination and coherence of the projects with the activities and priorities of other agencies and organisations. • Linkages to the WRC funded Food Security Project
Aspect 7 - participation To what extent were the project stakeholders involved in the assessment, planning, implementation and evaluation phases of the project?
Areas for consideration include: • the quality of relationship between Tearfund and the project stakeholders • the level of participation of the stakeholders in all stages of the project cycle • the involvement of all the beneficiaries, ensuring gender balace and inclusion of the vulnerable
Aspect 8 – Learning – Separate Annex for Tearfund only) To what extent have lessons learned from the Bomi DCI funded WatSan evaluation in 2005 been incorporated into this project?
METHODOLOGY The methodology will include: • A gender-aware, participatory approach by an Evaluation Team headed by the consultant • A review of pertinent documentation, held in Tearfund’s UK Head Office and with the Liberia programme field sites • Field visits to the relevant DMT operational sites in Nimba county and the programme head office in Paynesville, Monrovia • Interviews will be held with relevant Tearfund head office and field-based staff, key officials in co-ordinating agencies, local &/or central government representatives and beneficiaries. The beneficiaries will be questioned on their views of the assistance provided, the way they were selected and their overall views of the agency. Interviews with individuals may be complemented by discussions with groups of beneficiaries. CONSULTANT ABILITY
ATTRIBUTE/SKILL ESSENTIAL DESIRABLE EDUCATION/ Degree in Water/Sanitation & Health or related field Masters level qualification QUALIFICATIONS in Water/Sanitation/Health
EXPERIENCE Appropriate and relevant relief/ rehabilitation West African WatSan experience with international NGO experience Previous consultancy experience Previous experience with Tearfund
SKILLS/ABILITIES Good communicator Excellent interpersonal skills Training ability Report writing Proven evaluation skills
35 PERSONAL Flexible team member QUALITIES Able to use initiative Team player Organized Trustworthy SCHEDULING • Briefing & Prep 1 day • Prep & outbound flight 1 day • Field work & In-country travel 6 days (Including: In country Evaluation Team Briefing & Analysis Workshop & Report) • Report production & Return flight 1 day • Report production 2 days • Debriefing 1 day Total 12 days
ITINERARY
DATE TIME ACTIVITIES REMARKS No. of DAYS August 9 Briefing and prep 1 To include phone call with Andy Morgan in Teddington August 10 Depart Outbound flight from London 1 Heathrow 0700 Arriving Monrovia arrive Brussels Staff Interviews 0910/1040 arrive Monrovia 1750
August 11 Arrival in Nimba 1 August 12 Preparation 1 Staff Interviews August 13 8am- 8.45am Devotion The 1 9am- 9.30am Meeting with sector/Support programme communities mangers highlighted in bold are 10.00am Depart to Saclepea to visit Yarsonnon, easily Wehyeepa,Mehnpa and Back to Saclepea . accessible.
August 14 8am – 4pm Visit Zahn,Burtein,Dohn Same as 1 8am- 4pm and Kpeikpoa above Visit fleedin, Garwonpa, Gbeanpa, Guawin, New/ Old Nyasin and Nyao Back to Tappita. August 15 8am – 4pm Visit Ziah 1, Ziah 2, Cluster T&Z and Same as 1 Kpeletuo. Back to Tappita. above August 16 8am – 4pm Visit Kwendin, Gaepa Boyee, Zahnzaye, Same as 1 Gbulupea, Dorpa, Yeefan, Konwaypa and above Goekorpa. Visit Sahnpa, Dahnpa, Kpowin, Gbah, Klordin, August 17 8- 8:30am Devotion OR 1 8:45- 10am Debriefing at project level Further field work 10:30-5pm Travel Nimba-Monrovia Staff Interviews
36 August 18 Monrovia debrief (2 hrs) OR 1 Report production (0.5 Devotion day) Debriefing at project level Travel Nimba- Monrovia Staff Interviews August 19 Depart Report Production Monrovia debrief 1 Monrovia 1855 (2 hrs) arrive Brussels Depart Monrovia for Depart Monrovia 0535/0725 London for London arrive Heathrow 0745 August 20 Arrive London 1
August 1 (or 2 days) day report production 1 or 2 20/21 (dependent on itinerary above on 17-18-19) Report to be finally completed by 7 September 2007 TBC TBC UK debrief or presentation 1 TOTAL 14 or NO. DAYS 15 NB. The communities to be visited will be decided by the evaluator, based on the accessibility of the areas as indicated in the table. On average it could be possible to visit 2 communities. This also depends on the interest of the evaluator’s time utilisation plan per community.
MANAGEMENT OF VISIT • Jonas Njelango (Programme Director) and Andy Morgan (Operations Manager) are jointly commissioning and approving the work • At Field Site level the consultant will relate to Ambaye Zekewos (Area Coordinator – Nimba) • Responsibility for practical arrangements (travel arrangements, hotels, flights etc) is with the Tearfund UK Operations Department and Alfred Tamba, Human Resource Manager Liberia who will liaise with the DMT Nimba staff to make local arrangements
EXPECTED OUTPUT The expected output of this assignment is one report, no more than 25 A4 pages in length (excluding annexes) • Executive Summary (no more than two A4 sides) • Introduction / Background • Methodology • Context Analysis • Findings • Conclusions • Assessment • Specific Actionable and Prioritised Recommendations for future development of project in Liberia • Annexes (indicative) o Terms of Reference for the Evaluation o Evaluation Schedule o Protocols for the Evaluation o Documents consulted during the Evaluation o Persons participating in the Evaluation o Field data used during the Evaluation, including baselines o Bibliography
REQUIRED INPUTS • People to be involved (partners, community leaders, government officials) and their roles: DMT senior staff, DMT project staff, Paramount Chief, District Commissioner’s office, other WatSan NGOs, community representatives, community development committees. • Documentation: o DMT country framework o DMT ECHO proposal
37 o DMT ECHO interim report o DMT ECHO pre-final report o DMT ECHO final report (if available)
EVALUATION OF CONSULTANCY • The Senior Operations Manager and Liberia Operations Manager will review the report • A copy of the final report will be provided to ECHO
38
Annex 2: Evaluation Schedule
Day Date Activity 1 Wednesday 8th Briefing: Telephone briefing with Tearfund Teddington staff - Andy Morgan () and August Jane Chambers (Operations Officer, Liberia & DRC) 2 Friday 10th Travel: Fly Heathrow to Monrovia. August Meetings: Monrovia team meeting and briefing (Jonas Njelango (Programme Manager (PD), Catherine Karuno (HR Manager) and Patrick Kamara (Logisitcs Co- ordinator)). 3 Saturday 11th Travel: Drive Monrovia to Tappita (Nimba County). August Meeting: Nimba team meeting & briefing (Ambaye Zekewos (Area Co-ordinator) and Matthew Carlquist (Intern Officer)). 4 Sunday 12th Meetings: Nimba evaluation team meeting and planning (AC, Intern and Sector August Managers). Interviews: Nimba staff: Timothy Owhochukwu (PHE Manager); Sahr (WATSAN Manager); Joseph Flomo (Acting CD Manager); Badio William (M&E Manager) 5 Monday 13th Interviews: Nimba staff: Anita Coloba (CD Officer); David Woloba (PHE officer, August DMT); Joseph Kpadeh (Watsan Senior Technician) Travel: Drive Tappita to Yarsonah and Mehnpa Towns (ECHO Phase 1 sites, Saclepea) Community interviews: Group and individual interviews (Town Chiefs, elders, CDCs, CHVs, PHEs and individual users. Zone Chief in Yarsonah). Interview: Ambaye Zekewos (Area Coordinator (AC)) 6 Tuesday 14th Travel: Drive Saclepea to Kperkpor and New Nyesin (Phase 1 towns) August Community interviews: Group and individual interviews (Town Chiefs, elders, CDCs, CHVs, PHEs and individual users. PTA and Pastor in Kperkpor. Interviews: Mercy Lurlay (EQUIP Regional Supervisor); Ambaye Zekewos (AC) Travel: Drive Saclepea to Tappita 7 Wednesday 15th Travel: Drive Tappita to Kwendin and Geapa (Phase 2 towns) August Community interviews: Group and individual interviews (Town Chiefs, elders, CDCs, CHVs, PHEs and individual users. Township Commissioner, Zone Chief, Clan Elder, Justice of Peace & Township Clerk in Kwendin. Pastor and Teachers in Geapa). Interviews: Tappita District Superintendent, District Paramount Chief, and Development Superintendent; Nimba staff - WATSAN, PHE, CD and M&E Sector Managers. 8 Thursday 16th Interview: Matthew Carlquist (Intern). August Meeting: major recommendations discussed with evaluation team, (AC, Intern, and WATSAN, PHE, CD and M&E Sector Managers) Travel: Drive Tappita to Monrovia 9 Friday 17th Interviews: David Waines (Director EQUIP). Monrovia staff: Jonas Njelango August (Programme Manager (PD); Chantal Richey (Deputy PD); Belete Assefa (Finance Manager); Patrick Kamara (Logistics Co-ordinator) Meeting: major recommendations discussed with DMT Monrovia (PD and DPD). 10 Saturday 18th Interviews: Catherine Karuno (HR Manager); Bev Kaufeldt (WATSAN Program August Manager, Samaritan’s Purse). Write-up 11 Sunday 19th Write-up August Travel: Fly Monrovia to Heathrow 12 Monday 20th Write-up August 13 Tuesday 21st Write-up August NOTES: 1. All staff DMT unless stated otherwise. 2. PHE = Public health Education; CD = Community Development; Watsan = Water & Sanitation; CHV = Community Health Volunteer.
39 Annex 3: Water Safety Plan
Background Water safety plans (WSPs) are promoted by the World Health Organisation (WHO) and are seen as the most effective method for ensuring safe drinking water. WSPs fit within a framework of safe drinking water, which is driven by health based targets and verified by health surveillance (Figure 1). The rationale for implementing WSPs should, therefore, be related to the desire for improved beneficiary health.
Framework for Safe Drinking Water
Health Water Independent Based Safety Surveillance Targets Plans
Management System Operational plans, Assessment Monitoring Documentation, Communication
Figure 1: Framework for safe drinking water
Within a WSP, the hazards and risks of any water supply system are identified and assessed so that suitable actions and training can be carried out to prevent the consumption of unsafe drinking water. One difference between basic hazard identification and a WSP is that a WSP includes managerial, communication as well as hardware risk management. Tearfund’s WatSan and public health education (PHE) practices are already in line with many of recommended ‘preventative’ techniques for ensuring water safety and as such introducing WSPs would largely be an organisation of existing Tearfund practices with minimal introduction of new, different and difficult concepts.
WSPs are currently being developed and implemented throughout the developed world and are being introduced in developing countries, with examples of operating WSPs found in refugee camps (Thailand –IRC3), capital cities (Uganda – NWSC and WEDC4), schools (India – UNICEF5) and communities (Bangladesh – WaterAid).
As a minimum, WSPs consist of the three essential actions that are the responsibility of the drinking-water supplier. These are: (1) a system assessment, (2) effective operational
3 International Rescue Committee: Thai Him Refugee Camp 4 National Water and Sewerage Corporation, Kampala; and Water, Engineering and Development Centre, Loughborough: Water Safety Plans for utilities in developing countries – a case study from Kampala, Uganda 5 United Nations Childrens’ Fund:
40 monitoring, and (3) management (Figure 1). These are further broken down into 10 steps (Figure 2).
1. Assemble team
2. Describe supply system
3. Hazard 10. Document & assessment communicate
System Assessment 4. Risk 9. Management Monitoring assessment procedures Management
5. Identify control 8. Support measures programmes
6. Monitor control 7. Verify WSP measures
Figure 2: The ten steps of a water safety plan6
Introducing WSP to Tearfund and Nimba County The ultimate aim of a WSP is to protect public health by ensuring the provision of safe drinking water and this must be kept in mind at all times in order to avoid over complicating matters. This is true for all ten steps in the WSP, but should not necessitate the elimination of any one step entirely. The following text refers to each of the 10 steps:
Team Assembly: Typically the team responsible for developing and implementing a WSP contains people of authority and people with technical ability. Examples of suitable people might include WatSan Manager, PHE manager, local medical practitioner, town chief, and key members of the community development committee (CDC). Formal capacity building of Tearfund staff in the area of risk assessment and risk management would be required.
System Description:
6 Adapted from UKWIR news, Issue 35, June 2005
41 WSPs are designed to cover all aspects of water supply, from the catchment to the consumer. The description should consider each of the following aspects of the supply system: • Source water (e.g. shallow ground water) • Water treatment (e.g. none) • Water distribution (e.g. community hand pump) • Water collection (e.g. users > 8 yrs old, open buckets, jerry cans) • Water storage (e.g. closed buckets inside houses) • Special controls (e.g. water table fluctuates seasonally) • Quality requirements (e.g. WHO guidelines) • Users and uses (e.g. community; drinking, cooking)
This description should be tabulated and also depicted in a process flow diagram, with each stage clearly identified and the responsibility assigned. For example, safe water storage is the responsibility of the consumer; hand pump maintenance is the responsibility of the community water assistants (CWAs) and Tearfund.
Hazard and Risk Assessment – Control Measures Identification: The WSP team assess the entire water supply chain and identify existing or potential hazards, prioritise the risks associated with the hazard and identify what control measures exist (should exist) to prevent hazardous events from occurring. Frequently, steps 3 to 6 (Figure 2) are documented in table form. A typical table layout is shown in Table 1, with an example hazard documented.
Table 1: An example hazard, with assessed risk and corrective actions
Control Critical limits Monitoring Hazard Cause Risk Corrective event Measure Target Action What When Who Good size Lack of Good Ditch has Contamination diversion diversion drainage rubbish or Sanitary Repair and clean ditch. by surface Unlikely ditch, Monthly CWA ditches, around signs of inspection Increase size of ditch water clear of heavy rain. wellhead wear rubbish Hazard 2 Etc.
The WHO have defined a hazard as a biological, chemical, physical or radiological agent that has the potential to cause harm, and a risk as the likelihood of identified hazards causing harm in exposed populations in a specified time frame, including the magnitude of that harm and/or the consequences. The control measures are those steps in drinking-water supply that directly affect drinking-water quality and that collectively ensure that drinking-water consistently meets health-based targets. They are activities and processes applied to prevent hazard occurrence7.
Assessing and prioritizing the risk can be carried out using a standard 5x5 matrix of severity and likelihood (Table 2).
7 WHO: Guidelines for drinking water quality, 3rd Edition. Chapter 4
42 Table 2: Example of a simple risk scoring matrix for ranking risks8
Verify WSP: In the context of Tearfund Liberia, the verification stage may best be considered as part of health surveillance (Figure 1). Typically verification involves testing the water for faecal indicator organisms and hazardous chemicals and ideally should be carried out by an independent authority as well as the water supplier. Tests should be carried out along the whole supply chain and at different times to account for seasonal changes. If the capacity does not exist for this type of verification, it may be possible to use concepts from an independent health surveillance to indirectly verify the WSP. For example, the community health volunteers (CHVs) record the number of diarrhoea cases in the community before and after WSP implementation.
Support programmes: The links between Tearfund WatSan and PHE sectors are highlighted in this part of a WSP as supporting programmes are defined as actions that are important in ensuring water safety but so not directly affect water quality. Examples include public health education and controlling access to the well (fence construction). Tearfund Liberia already promote such programmes as part of normal operation and therefore implementation of this step is likely to focus on collating and documenting existing practices, promoting and auditing.
Management procedures: This step is very important as it deals with what should happen if things go wrong, as well as who should do what, when, how and why during normal operation. Community-managed or NGO-managed systems are prone to intermittent (or total) operational failure and if this occurs, it is likely that the alternative water supply available is unsafe for consumption. If this step of a WSP is given considerable attention, the sustainability of the project may be enhanced. Management procedures should consider and document how the community are informed (of technical breakdown, standard upgrading, water resources etc.) and how decisions are made as well as what corrective actions and incident response plans are in place.
Documentation and communication: The WSP documentation should include: • Description and assessment of the system • Operational monitoring and verification plan • Management procedures for normal operation and emergency situations • Description of supporting programmes • Risk assessment and defined ranking methodology Once a WSP is implemented and operational, the following should be documented: • Results and records of monitoring
8 WHO: Guidelines for drinking water quality, 3rd Edition. Chapter 4
43 • Training programmes (subject and attendees) • Incident details (situation and response) • Names (and ‘addresses’) of CHVs, CWAs, CDC chairperson and town chief
The WSP documentation should be kept with the Tearfund WatSan area manager and be passed on to national NGO partners once Tearfund exit the area. Another part of the documentation should include a summarised pictorial ‘water safety plan’ for the community. This could be drawn and posted on a board positioned by the well. The board could be used as further PHE but would also need to have contact details of key players responsible for ensuring water safety.
Summary A summary diagram designed to aid small water suppliers develop a WSP is shown in Figure 3. Tearfund could develop and implement WSPs without extensive changes to current practices of the WatSan and PHE sectors. Implementing WSPs would give a clear message to the international community that Tearfund consider water safety to be a priority.
Figure 3: A guide to water safety plan development and implementation9
9 WHO: Water Safety Plans- managing drinking water quality from catchment to consumer, 2005
44 Annex 4: Details of outputs achieved (observations and reports to 18/08/07)
Name of Pop- Existing New HDW Rehab- Bio Latrines Users Users Town ulation Pump ilitated Sand per per Wells Filter HDW latrine Saclepea District (Phase 1) Yarsonnoh 1,157 1 1 1 40 386 29 Wehyepa 314 1 5 314 63 Mehnpa 1,157 2 27 579 43 Kpeikpoa 1,014 2 20 507 51 Burlin 678 1 1 15 339 45 Dohn 782 1 1 20 391 39 Zahn- 876 2 15 438 58 Glesonia Loyee 1,798 1 2 40 599 45 Nyao 805 2 25 403 32 New Nyasin 460 2 10 230 46 Old Nyasin 585 1 10 585 59 Guarwin 725 1 3 20 181 36 Gbeanpa 146 1 5 146 29 Garwonpa 1,510 1 2 25 503 60 Fleeden 1,337 3 4 25 191 53 Total Phase 1 13,344 7 17 13 302 361 44 Tappita District (Phase 2) Tappita 3 complete 19 4 21 ** ** city††† 1 on-going Ziah 2 1,161 2 33 581 35 1 complete Ziah 1 1,508 30 754 50 1 on-going Kpeletuo 558 1 on-going 15 558 37 Cluster Z 300 5 ** ** Cluster T 1,250 5 ** ** Saye Town * 1 10 ** ** Kpowin 795 1 2 15 265 53 Klordin 155 1 on-going 8 155 19 Gbah 211 4 ** 53 Dahnpa 636 1 1 10 318 64 Sahnpa 815 1 2 35 272 23 Zahnzayee 1,610 1 3 48 403 34 Gbulupea 225 1 7 225 32 Boyee 633 1 1 12 317 53 Geapa 260 1 12 260 22 2 complete Kwendin 2,002 1 36 501 56 1 on-going Dorpa 470 1 15 470 31 Yeefan 137 5 ** 27 Konwaypa 112 5 ** 22 Goekorpa 526 1 19 526 28 Kaipa* 1 1 10 ** ** 28 Total Phase 2 13,364 26 4 10 350 406 38 (inc. 5 on-going) 45 Grand Total 26,708 33 17 10 652 380 41 (inc. 5 on-going)
††† Tappita city, Saye town and Kaipa populations excluded from totals as WATSAN interventions only. ** Incomplete data therefore numbers of HDWs and/or latrines excluded from calculations.
45 Annex 5: Documents and field data consulted and used during the Evaluation
Documents consulted: - ECHO narrative 3 feb 06 - Pre-final Report Narrative & Annex 2 Logframe Report - Liberia DMT Framework - approved in principle - Tearfund, Liberia, Nimba, Phase II extension proposal – 220806 - Tearfund Liberia ECHO Pre-final Report Jul06 - for submission - Field reports from CD, WATSAN and PHE sections
Field data consisted of interviews and meetings as detailed in Annex 2.
Annex 6: Persons participating in the Evaluation
Evaluation of DMT field sites: - Ambaye Zekewos (Area Co-ordinator) - Matthew Carlquist (Intern Officer) - Timothy Owhochukwu (PHE Manager) - Sahar (WATSAN Manager) - Joseph Flomo (Acting CD Manager) - Badio William (M&E Manager) - Mercy Lurlay (EQUIP Regional Supervisor) - Dr. Jen Smith (Cranfield University)
Feedback and discussion with DMT Nimba: - all DMT Nimba staff
Feedback and discussion with DMT Monrovia: - Jonas Njelango (PD) - Chantal Richey (Deputy PD)
Team leader: - Dr. James Webster (Cranfield University).
46 Annex 7: Bibliography
Abrams, L. (2001). “Understanding Sustainability of local water services”. http://www.africanwater.org/sustainability.htm. Access date 20th August 2007.
Bagamuhunda, G. and G. Kimanzi (1998). "In the Light of Experience - Water Policy and Usage in Uganda." Waterlines 16(3): 19-20.
Batchelor, S. J., K. McKemey, et al. (1999). Exit strategies for resettlement programmes. 25th WEDC International Conference: Integrated Development for Water Supply and Sanitation. Addis Ababa, Ethiopia, WEDC: 72-74.
Batchelor, S. J., J. Ngatshane, et al. (2001). Organisational exit strategies for water supplies. 27th WEDC International Conference: People and Systems for Water, Sanitation and Health. Lusaka, Zambia, WEDC: 83-86.
Carter, R. C., D. Demessie, et al. (1996). "Policy Development to Achieve Maximum Impact and Sustainability." Waterlines.
Carter, R. C., S. Tyrrel, et al. (1999). "Impact and Sustainability of Community Water Supply and Sanitation Programmes in Developing Countries." Journal of the Chartered Institution of Water and Environmental Management 13: 292-296.
Cornwall, A. and R. Jewkes (1995). “What is Participatory Research?” Social Science and Medicine.41: 1667-1676.
Dahanayake, K. (2004). Is there any optimum level for community participation in RWSS? – An Engineer’s view. 30th WEDC International Conference: People-centred approaches to water and environmental sanitation. Vientiane, Lao PDR, WEDC: 249-251.
Freire, P. (1996). Pedagogy of the Oppressed. London, Penguin Books Ltd.
Jones, H. and R. Reed (2005). “Water and Sanitation for Disabled People”. WEDC, Loughborough University. Available at: http://wedc.lboro.ac.uk/publications/log_pubs2.php
Narayan, D. (1995). The contribution of people’s participation: Evidence from 121 rural water supply projects. Environmentally Sustainable Development Occasional Paper Series No. 1. Washington DC, USA, World Bank.
NIMAC (2007). www.nimacliberia.org.
Shordt, K. and M. Snel (2002). "Building institutional capacity for sanitation." Waterlines 20(3): 27-32.
Sinclair, P. (2004). “Scaling up water supply – a WaterAid perspective”. Waterlines 23(2): 5-7.
Webster, J., G. Dejachew, et al. (1999). Sustainability of Rural Water and Sanitation Projects. 25th WEDC International Conference: Integrated Development for Water Supply and Sanitation. Addis Ababa, Ethiopia, WEDC: 266-267.
47 Webster, J. (2006) Culture’s Influence: Towards Understanding Stakeholder Interactions in Rural Water, Sanitation and Hygiene Promotion Projects. PhD thesis. Institute of Water and Environment. Cranfield, Silsoe College at Cranfield.
Wegelin-Schuringa, M. (2000). Public Awareness and Mobilisation for Sanitation. The Hague, IRC.
Wilson, G. (2006). “Beyond the Technocrat? The Professional Expert in Development Practice.” Development and Change. 37: 1-23.
Wysoki, M. (2007) Assessment of monitoring by end-users in the Diocese of Kigezi Water and Sanitation Programme, South West Uganda. MSc thesis. School of Applied Sciences. Cranfield.
48 Annex 8 (FOR TEARFUND ONLY): Learning (Aspect 8)
• The extent to which lessons learned from the Bomi DCI funded WatSan evaluation in 2005 have been been incorporated into this project
From the relevant, transferable recommendations made in the Bomi evaluation, Table 1 indicates the degree to which they have been incorporated into the ECHO programme:
Table 1: Incorporation of Bomi recommendations into the ECHO programme Fully incorporated Substantially incorporated Not incorporated 1. Water quality testing 1. Social marketing of latrines 1. Emphasising to 2. Training of communities on the 2. Monitor the use of ash for hand- communities the need need for a paradigm shift from washing and reduction of odour from to elect personnel for relief to rehabilitation/development latrines training that have thinking 3. Monitor the relative effectiveness of strong connections 3. Networking with Government and the various hygiene promotion and roots in the other NGOs methodologies and media used community 4. Quantification in the log-frame, and 4. The programme cycle commence after more care over OVIs harvest (October) and continue for full 5. A field office close to target 12 month cycles communities 5. Future proposals have the flexibility to 6. Proposals have the inbuilt provide one latrine per household flexibility, and staff the clear willing to provide the necessary procedures to withdraw from community contribution, including a uncooperative villages provision to return to communities of previous Phases to provide this
Lessons learned from Bomi include:
- PHE and construction of latrines before construction of HDWs - Construction of flat, reinforced slabs by staff to avoid delays and provide quality control (non-reinforced slabs were a problem in Bomi). - Emphasis on children for PHE dissemination, and good use of news groups e.g. churches. - Formation of Youth groups to deal with reintegration and sexual health issues through sports activities and training of Church leaders on sexual health issues - Added emphasis on community mobilization and sensitizations strategies. - Additional monitoring and report submission by CDC’s - A community entry strategy of a one day project awareness workshop for local leaders - Visible hygiene promotion and HIV/AIDS messages e.g. T-shirts and bill boards - Mainstreaming conflict transformation within project design
• Conclusions and assessment
The programme has already been commended in 4.1.4 for exhibiting a learning, adaptive culture. Although Table 1 generally underlines this, there are some aspects that require further effort, in particular the election of CWA sans CHVs with strong local roots to minimise migration.
49