EVALUATION JUNE 2013 Improving Living Conditions for Coastal Farmers in a Changing Environment, Thatta District, Sindh Province, Pakistan
Funded by Japanese Social Development Fund and World Bank By Ben Allen, ELA Unit, ACF-UK
This report was commissioned by Action Against Hunger | ACF International. The comments contained herein reflect the opinions of the Evaluators only. Acknowledgements
Thank you to all those ACF colleagues in New York and Pakistan who have made this evaluation possible. Special thanks go to the fieldwork team Khawar, Sikandar, Amir, Roshin and Wakeel, who made my time in Sindh both productive and enjoyable. Finally I am indebted to the farmers, the co-operative members and villagers for taking the time to talk to me and making endless cups of chai.
Ben Allen ELA Unit April 2013
ACF World Bank Programme Final Evaluation
Acronyms
ACF Action Against Hunger | ACF International CBOs Community Based Organisation CD Country Director FGD Focus-Group Discussion FSL Food Security and Livelihoods KPK Khyber Pakhtunkhwa Province LFA Log Frame Analysis M&E Monitoring & Evaluation NGO Non-Governmental Organisation NIAB Nuclear Institute for Agriculture and Biology PARC Pakistan Agricultural Research Council PCRWR Pakistan Council of Research in Water Resources PCSIR Pakistan Council of Scientific and Industrial Research PDM Post-Distribution Monitoring PHM Post-Harvest Monitoring PM Programme Manager SDU Solar Desalination Unit TMK Tando Muhammed Khan District UC Union Council WaSH Water Sanitation and Hygiene WB World Bank WWF World Wide Fund for Nature
Page 2 of 58
ACF World Bank Programme Final Evaluation
TABLE OF CONTENTS
Acknowledgements...... 1 Acronyms ...... 2 Executive Summary ...... 4 Introduction ...... 7 The Programme ...... 7 The Situation for Farming Families in Sindh Province ...... 8 The Evaluation ...... 8 Evaluation Methodology ...... 9 Evaluation challenges and limitations ...... 9 Findings and Analysis...... 10 Overall Appropriateness ...... 10 Comments on HR, Planning, Timing and internal coordination ...... 12 Training in Agricultural Techniques (Component A) ...... 13 The Research Element ...... 16 The Co-operatives ...... 17 The Marketing Activities (Component B) ...... 19 The WaSH Activities (Component C) ...... 22 Integration of the FSL and WaSH components ...... 25 Responding to Shocks ...... 28 Relationship and Interaction with local stakeholders ...... 29 The potential for Advocacy ...... 31 Learning from Mid-Term Evaluation ...... 32 Conclusions and Recommendations ...... 33 Annexes ...... 35 Annex A: Logical Framework of Analysis ...... 36 Annex B: Terms of Reference ...... 38 Annex C: Itinerary and List of Meetings (March 2013) ...... 50 Annex D: Bibliography ...... 54 Annex E: Best Practice ...... 56 Annex F: DAC Ratings Table ...... 57
Page 3 of 58
ACF World Bank Programme Final Evaluation
EXECUTIVE SUMMARY
The current report presents the findings and recommendations of the final evaluation of a 4 year Japanese Social Development Fund/World Bank funded ACF International programme known as Improving Living Conditions for Coastal Farmers in a Changing Environment, Thatta District, Sindh Province, Pakistan. The evaluation fieldwork was conducted in country over a 3-week period in March 2013. The evaluator conducted a number of interviews and discussions with staff from ACF-USA, ACF Pakistan and World Bank, relevant government staff at local and provincial level, and programme beneficiaries, including local farmers (land owners and sharecroppers), co-operative members, lead farmers, and beneficiary household members. One the most significant limitations to the evaluation was the inability of the (male) evaluator to meet with female household members.
The programme begun in 2009 in response to a World Bank assessment highlighting the challenge of soil salinity faced by communities in the coastal areas of Sindh Province. ACF designed a programme that aimed to address these challenges, and conduct research into effective solutions. The programme is referred to as a pilot project as research was a significant feature. The programme consisted of three main components. The first component (A) addressed the challenge farmers faced in cultivating high yields as a result of the salination in the soil. This included training in suitable agricultural techniques and research into saline resistant seed strains. Component B aimed to facilitate market access for the farmers through the creation of co-operatives and setting up markets. This would then enable farmers to increase profits when taking their produce to market. Component C also had a significant research element, which aimed to discover an effective and affordable method for desalinating water for human consumption. In addition to this ACF also implemented some hygiene promotion activities and borehole rehabilitation. The programme was implemented in 23 villages in the Keti Bunder and Kharo Chan Union Councils of Thatta district, covering a total of 425 farming households.
The programme was largely appropriate, addressing real needs of the coastal communities. The agricultural techniques component was well executed and well received. The marketing component however suffered from understaffing and poor planning.
The main strength of the FSL components combined was the setting up of farming co- operatives, and the lead farmer model. The co-operatives served as an effective base for sharing agricultural knowledge and eventually collaborating in market efforts. Each lead farmer enabled ACF to maintain a close relationship with the co-operatives and also ensure on-going support with new techniques.
The training in new techniques, including the setting up of demonstration pots in targeted villages was highly successful. Farmers were able to clearly see the difference in the new techniques and apply them. ACF produced materials (in Sindhi language) enabled each co- operative to have a reference for continued application of these techniques.
Page 4 of 58
ACF World Bank Programme Final Evaluation
ACF did not have an effective, assessment based strategy for dealing with government bodies. The approach to involving the government was ad-hoc and for the most part ineffective. Any development programme such as this should put local authorities at the centre of activities. It is recognised that dealing with an under-resourced provincial government is challenging, but in an effort to over come this lack of engagement ACF could have pursued an advocacy approach. This would have been both towards the government to lobby them to take ownership over the programme activities in order to ensure their on- going support after the programme period, and also towards large local landowners in aid of the sharecroppers (landless farmers).
The marketing element was marred by a delayed assessment and ill-conceived strategy. The marketing activities (training and setting up of markets) should have been implemented in line with the harvest cycle and at an earlier stage in the programme allowing at least 2 harvest cycles to happen. Floods in 2010 were a contributing factor to the inability of farmers to benefit substantially from the marketing activities. ACF provided transport (to market) and training, but the outstanding challenge farmers face in marketing their produce is related to credit.
The lack of available fair credit compels farmers to borrow from wholesalers (those who buy their produce) and hence keeping them in a cycle of debt. In order for farmers to invest in inputs (e.g. seeds) at the beginning of the harvest cycle, they need to borrow. The only available credit is from the wholesalers who negotiate a price from their produce based on this credit provision. ACF could have explored approaches to enable access to fair credit in order to relieve farmers from this cycle. This would have enabled farmers to farm more land, cultivate more produce and then negotiate a fairer price at market, as they would not have been tied to one wholesaler. The main strength of the marketing component was the training. Farmers reported being much better equipped to negotiate a better deal when going to market, and can do this in collaboration with their co-operative colleagues.
Another challenge that this programme did not address, which also significantly restricts the ability of farmers to increase profits is irrigation. Farmers are unwilling to take the risk of planting more seed in order to increase profits because they cannot guarantee suitable irrigation. As they are tied to an unfair credit arrangement, this is a risk they cannot afford to make. The programme would have benefited from greater integration with the WaSH department, in order to explore possible irrigation solutions.
The Solar Desalination Unit research was successful in so far as an effective model was discovered, designed and installed. ACF also ensured that the SDUs were eventually locally available. A challenge still remains in the affordability of these units.
Although the scope for integration between the programme components was limited, there was no integration at all. Both the WaSH and FSL elements were targeting the same villages and were both working with CBOs and training, therefore, at least from an efficiency point of view, their was some scope for sharing notes and collaboration.
ACF experienced great challenges in their relationship with World Bank. This is unfortunate as it is felt that this pilot programme discovered some effective approaches to the problem
Page 5 of 58
ACF World Bank Programme Final Evaluation
proposed and the coastal community would have benefited from sustained funding. However it is yet to be seen if the programme was able to produce enough hard evidence to support the effectiveness of certain approaches (SDUs or farming co-operatives for example) to support such funding. A number of individual monitoring activities took place, but there was no overall M&E approach that enabled ACF to measure the impact of the various programme elements.
Evaluation recommendations are made in terms of the programme that did take place (according to the programme proposal) and also to a wider programme that was maybe restricted by the proposal. These include the need for an assessment-based approach to dealing with authorities, and the formulation of an advocacy strategy based on the objectives of the programme. The availability of credit and reliable water resources were both significant obstacles to farmers increasing profit. ACF should have explored approaches to these, which would have substantially improved the impact of the marketing and agricultural technique activities. In future, such research programmes should allocate a more extensive period to research and assessment. This should include an integrated monitoring system that enables measuring of the individual approaches and activities. Finally, ACF should in future ensure that the different sectoral areas of programmes work together this would improve both impact and efficiency.
Page 6 of 58
ACF World Bank Programme Final Evaluation
INTRODUCTION
This report presents the findings of an evaluation of the programme referred to as “World Bank1” (hereafter referred to as “the programme”). The report will begin by detailing the context and evaluation, before presenting the findings, and finishing with conclusions and recommendations. Target users of the evaluation are primarily ACF-Pakistan and ACF-USA staff who are already familiar with the programme therefore details of the proposal and the programme itself will not be unnecessarily included in the main body of the report. For those readers more unfamiliar with the programme certain annexes have been included such as the LFA. The Evaluations, Learning and Accountability (ELA) Unit, ACF-UK conducted the evaluation, therefore, although not an external evaluation per se, the evaluator is neither a member of the ACF-Pakistan nor the ACF-USA teams, and was new to the programme from the outset.
The Programme
ACF responded to a call for proposals by the World Bank in response to a 2005 study highlighting above all the challenge farmers in the coastal regions of Thatta are faced with caused by soil salinity.2 In 2008, ACF wrote and submitted a proposal and, with relative ease, was granted funds. The World Bank allocated almost $2 million3 from the Japanese Social Development Fund to the programme. Originally a 3-year pilot programme, a no-cost extension of one year was granted bringing the programme period from March 2009 through to March 2013. According to the proposal the programme consists of the following 4 components: A. Increase autonomy/self sufficiency on agricultural inputs B. Reinforcing agricultural wholesale markets C. Renewed access to boreholes turned saline D. Increase knowledge and analysis in the livelihood sector for Thatta district coastal areas Component A and B consist of FSL interventions, and although include different activities address a common challenge of increasing household income. Component C, however, addresses the challenges in the provision of fresh drinking water and hygiene issues, thereby decreasing household burdens and expenditures. Component D relates to the implementation of a baseline and an endline survey in order to ascertain the progress made by the activities. For further details on the programme objective, intended results and activities, refer to the Logical Framework Analysis in Annex A.
The programme was implemented in 23 villages in the Keti Bunder and Kharo Chan Union Councils of Thatta district, covering a total of 425 farming households.
1 Full programme title is: “Improving Living Conditions in a Changing Environment of the Thatta Coastal Farmers” [in Sindh Province, Pakistan]. 2 Socioeconomic Study and Proposal for Livelihood Improvements: Badin and Thatta Districts, April 25, 2005 3 Exactly $1,977,650 see Japanese Social Development Fund – Grant Proposal
Page 7 of 58
ACF World Bank Programme Final Evaluation
The Situation for Farming Families in Sindh Province
According to the World Bank study4 (on which this programme was based) 40% (approximately 1.2 million acres) of farmland in Thatta and Badin Districts is affected by sea intrusion. This occurrence has increased the salinity of the soil proving a significant setback to the ability of farmers to make a sufficient living from the land.
High investments on inappropriate agricultural inputs, poor returns on agricultural activities and high health care costs (in part due to the high level of waterborne diseases) have created low standards of living for coastal farmers in the district of Thatta. A common coping mechanism for these farmers is to depend on loans to both enable agricultural investment and pay for other costs of living (healthcare costs for example). This borrowing is prone to engage families in a cycle of debt year after year. As farmers (and their families) are then at the mercy of the creditors (also the purchasers of their produce) it makes it very difficult for them to exit this cycle, and improve their standards of living.
Within the focus of the programme, the main cause of the targeted farmers misfortune is due to their inability to increase agricultural production. This is due to inappropriate approaches to the farming of saline land, the lack of appropriate saline tolerant seeds and the aforementioned cycle of debt. Furthermore, in the case of the land-less farmers (share- croppers), the dynamic between them and the owners of the land they farm, is not conducive to an ability to increase profit and improve their living conditions.
70% of farmers in Sindh are referred to as sharecroppers, meaning that they farm land owned by someone else and generally have unfavourable arrangements with the landowner. On average sharecroppers only receive 25% of the income generated from the land they farm. This renders them in an even weaker position in terms of increasing production, increasing marketing activities, leading to increased profits and an improvement in their living conditions.
Finally, it is pertinent to note that the target area is an area of changing climatic conditions. Not only is the area prone to changing conditions of the soil but is also at risk of cyclone, flood, and drought, as well as inconsistent rainfall. Considering the reliance on agriculture in Thatta, the challenge of increasing household income and improving their living conditions, is more acute and constantly varying.
The Evaluation
This end-term evaluation was commissioned in order to ensure the unique “World Bank” experience was analysed and documented. Being a more developmental programme institutional (ACF-USA) experience and knowledge of such programming is limited; therefore it was of the utmost importance for ACF to capitalise on this experience. Furthermore ACF- Pakistan is currently setting up a another long-term (4 year) development programme in
4 Socioeconomic Study and Proposal for Livelihood Improvements: Badin and Thatta Districts, April 25, 2005
Page 8 of 58
ACF World Bank Programme Final Evaluation
Dadu (funded by EuropeAid) and wishes to draw from the experience of the World Bank programme in a systemized manner to inform this programme and similar future programmes. Seeing as ACF Pakistan primarily implements emergency response, this evaluation serves as an important learning process for the upcoming programme. With a move towards more longer-term programming in ACF, such programmes are becoming a common feature of ACF country strategies.
The evaluation focused on assessing the effectiveness, appropriateness and sustainability of the overall programme approach (including all components). It was required however that there should be a more detailed analysis of the marketing activities (Component B) as this was deemed to be the most challenged area. The evaluation was also expected to assess how the programme team responded to the recommendations of the Mid-Term Evaluation.5 Finally, as with most evaluations, the evaluator was tasked with providing concrete recommendations for ACF. For further details refer to the Terms of Reference (see Annex B).
Evaluation Methodology
The evaluation took the traditional form consisting of first-hand interviews with HQ staff (New York), the coordination team (Islamabad) and the programme team (Thatta). The evaluator also visited the target area and spoke to beneficiaries of the programme including farmers (of which there were sharecroppers and land-owners), lead farmers, cooperative members, cooperative committee members and vegetable/fruit sellers in a local market. In addition various local authorities were interviewed along with national and international NGOs working in the Thatta District. For a full list of interviews and focus-group discussions see Annex C.
This primary research was complemented by a review of the extensive documentation relating to this programme, including monthly and quarterly reports, monitoring data, surveys, strategies, case studies and guidelines. ACF policy documents and additional materials related to the context were also reviewed, in order to sufficiently contextualize the programme. A full list of documentation can be found in Annex D. Information was cross-checked in order to ensure accuracy and recommendations were formulated in collaboration with the relevant ACF staff.
A debriefing based on preliminary findings was conducted with the country team in Islamabad before departure on 26th March 2013. The draft report was then shared with ACF-USA and Pakistan colleagues on 26th April 2013 for feedback.
Evaluation challenges and limitations
The evaluation was conducted over a 20-day period (not including report writing), with 11 days in the field visiting villages and other stakeholders. Even without assessing the impact of the programme this is an unusually short period for an evaluation of a 4-year programme.
5 The Mid-Term Evaluation was conducted in March 2012, 1 year before the end of the programme.
Page 9 of 58
ACF World Bank Programme Final Evaluation
However, with the support of the ACF staff, the evaluator has been able to gain comprehensive understanding of the programme and provide analysis, and recommendations.
The programme activities finished the week before the evaluator arrived in Thatta therefore most programme staff had already finished their contracts. Although the FSL and WaSH PMs were still under contract and other key staff were able to be re-recruited, the evaluation would have benefited from seeing activities underway and talking to a more extensive range of programme staff.
FINDINGS AND ANALYSIS
Overall Appropriateness
The overall objective of the programme was to improve living conditions by increasing household income and decreasing household expenditure. The chosen activities were geared towards increasing household income focussed on increasing agricultural production and profits, and those geared towards decreasing household expenditure focussed on health costs by way of providing suitable drinking water. In both cases the chosen activities were largely appropriate as they focussed on dealing with the main challenge that farmers in the region are faced with; salination.
However the focus on agricultural activities should be questioned. The Baseline survey showed that only 34% of families depend on agriculture for primary sources of income, and the World Bank assessment showing that only 11% of households (in Sindh) earned more than 80% of their income from agriculture6. That said, agriculture is still a principal income source, and is ‘largely underdeveloped’7 therefore in need of support in adapting to the changing conditions. The programme therefore responded well to the needs of the agricultural population. There are other income sources that the programme did not include that could have resulted in a greater impact in terms of increasing profits.
By relying on one source of income (agriculture) the programme encouraged targeted communities to invest a larger proportion of their resources (profits and labour) into agriculture than may have been appropriate. This makes them more susceptible to the impact of poor rainfall (for example), and without any back up in the advent of a drought. Some beneficiaries spoken to also noted the potential in other income sources such as fishing, livestock and casual labour, and in some cases make up a significant proportion of overall household income. Therefore ACF could have explored supporting communities in these areas. For example, livestock, is not so much affected by the level of rainfall, nor by cyclones, for example.
The irrigation challenge was also not addressed by this programme. Communities in the target area suffer from both lack of water and floods, which serve as a significant inhibitor in
6 Socioeconomic Study and Proposal for Livelihood Improvements: Badin and Thatta Districts, April 25, 2005 7 ibid
Page 10 of 58
ACF World Bank Programme Final Evaluation
increasing agricultural production. This is further explored in the section “Integration of the FSL and WaSH components” below.
The marketing approach was also not appropriate to the context and challenges faced. As further detailed below, a weak marketing survey and resulting strategy failed to take into account the socio-political realities of these communities – how is it possible in this ‘feudal’ context to create independent farmers within a very oppressive economic and social structure? This question was not sufficiently tackled. The baseline survey recommended marketing to be a central element of this programme. Instead there was a distinct focus on the production side.
The programme was also marred with budget and project design challenges. The need to adjust budget lines and seek approval from World Bank caused delays to activities. The need for such substantial changes stemmed from a poorly conceived programme plan and budget. The evaluator was unable to speak to the individuals directly involved in the writing the proposal but it has been reported that the proposal was submitted with haste. The ACF Pakistan mission was at a turning point and the need to get funds seemingly overshadowed the details of the proposal itself.
Many staff cited the difficulties in communicating with World Bank as a significant factor in causing delays to the programme, as any change in the proposal had to enter the World Bank “machine” and it took time to get answers. However the evaluation found that in fact communication with World Bank was very good at logistics level, and decision making related to the programme was made a by a handful of individuals at World Bank in Islamabad. This conclusion is made based on the evaluators experience in organising a meeting with World Bank, which proved very straight forward when facilitated through the ACF logistics department. During the meeting with World Bank8 it became apparent that the Islamabad team had considerable leverage in decisions around budget realignment and programme adjustments. Therefore it is of the opinion of the evaluator that ACF could have take a more flexible approach to communicating (and receiving validation on changes) with World Bank and this need not have been such a significant obstacle.
By taking a more flexible approach to the relationship with World Bank ACF could have worked more closely with them. Programmatic related communication tended to be made by CD to Programme Director (for all WB Asia programmes), where there was greater challenges, and financial and logistics related communication through ACF Logistics Coordinator/Admin Coordinator and the Procurement Manager, which was seemingly more fluid. By using the Procurement manager channel, ACF may have been able to make more progress on programmatic issues. This would have then enabled a more flexible approach to the programme design in order to achieve the overall objective. This would have provided space for ACF to deal with the problems in the programme proposal as soon as they were identified such as the lack of any real budget plan or time bound schedule. Furthermore, a fluid relationship such as this would have allowed adjustments to be made in light of the 2010 and 2011 floods9. It became apparent during the evaluation that this latency on the part of World Bank was all too often used as an excuse for programme delays
8 with both Rehan (Procurement Manager) and Imtiaz (Programme Director) 9 for example one of markets set up effected by the low yield of crops as a result of the floods in 2010.
Page 11 of 58
ACF World Bank Programme Final Evaluation
and adjustments. It was these apparent obstacles in communication with World Bank that caused a certain amount of reluctance within ACF to deal with the programme at all. A strong leadership of the programme and robust approach with the World Bank would have facilitated activities in being implemented on time, and redirected when necessary.
Finally, there are questionS around the identification of villages. Large parts of Southern Sindh suffer from the salination challenge, but some more than others. ACF decided to choose communities in Keti Bundar and Karo Chan, but according to a WWF assessment10 it is the creek villages (further south) that suffer from salination to a greater degree. It is unclear why ACF chose the 30 villages for the initial assessment. One could assume that that they posed less of a challenge in terms of access that the creek villages. In which case ACF should question such a rational.
Comments on HR, Planning, Timing and internal coordination
Timing and programme schedule As mentioned above the programme suffered from a lack of suitable planning. It is suprising that the World Bank did not require any timebound outputs but it is additionally surprising that ACF did not appear to have a defined schedule in place. Take the example of the market survey which was completed at the end of 2011 (2 years into a (at the time) 3-year programme). Such lack of suitable planning resulted in a number of missed opportunities, such as farmers being giving inputs too late and therefore missing the optimal time for planting seeds.
HR Challenges and Programme Leadership The programme was not always well furnished with all required staff, and suffered from a lack of technical expertise, and high turnover at both programme and coordination level. The only consisitent member of the team (programme or coordination) throughout the duration was the FSL PM. He did a fine job in keeping the programme going but suffered from a lack of leadership on a strategic level and lack of support from the WaSH team and marketing side, due to gaps in those respective positions. From a marketing perspective the programme never managed to employ someone with suitable expertise for a singificant period, a large contribution to the weakness of this component. The most significant HR ‘story’ however is that of the FSL Coordinator, the position tasked with overseeing the FSL side of the programme. It was not untill early 2012 (the last year of the programme) that the coordination paid any significant attention to the programme. A significant turnover of FSL coordinators from 2009-2012 and a focus on flood response activities meant that the World Bank programme was somewhat neglected. The FSLCoordinator from that time even reported that she did not receive a handover on the World Bank programme indicating that the predecessor did not pay any attention to the programme.Such neglect could be put down to two reasons firstly, the coordination team were overwhlemed with emergencies
10 WWF Synthesis Report: Building capacity on Climate Change Adaptation in the Coastal Areas of Pakistan (www.wwfpak.org/ccap)
Page 12 of 58
ACF World Bank Programme Final Evaluation
and secondly were probably not given the necessary support from HQ to see through the ‘fog of emergency’ and manage the on-going actvities in Sindh province.
Internal coordination and communication The coordination within the respective FSL and WaSH teams (at base and capital level) was good, even though it may not have been so strong between the two (see section on integration below). However it was reported that validation processess between Islamabad and New York, delayed activities. For example the formulation of the market strategy was delayed because of delays in validation of the market survey report.
Commuication with beneficiaries was a strength in this programme. ACF was based in the area of operation (Gahro) and was therefore able to visit field sites on a regular basis. A clear and defined system was establsihed whereby programme staff dealt with lead farmers who passed info on to village leaders/members. The quality of the interaction with communities was also very effective. Staff were recruited from the area and therefore spoke the local language, knew the area and were well respected. This was widely reflected during the evaluation, by beneficiaries and local authorities alike.
Training in Agricultural Techniques (Component A)
The training component of the programme was largely the most successful of the FSL activities. These trainings introduced a variety of farming techniques to enable farmers to increase production and to adapt to the changing condition of the soil. This comprised of the following activities:
1. Introduction of seed storage techniques. This included the provision of wooden platforms so that farmers could store seeds above ground level and away from vermin. By storing seeds farmers were able to make use of reproduced seeds and were also able to safely store seeds enabling them to plant at the optimum time. Prior to the programme farmers planted seeds accessed from the market, which were often only available after the optimum planting period and therefore decreasing potential production. 2. Introduction of bio-fertiliser. Farmers usually rely on the purchase and application of chemical fertilisers. Not only are these expensive to buy but they are also damaging to the land. Farmers were trained in the application of this fertiliser as well as the production. One farmer explained emphatically the potential in turning fertiliser production further into an income generating activity. Demonstration plots established by ACF showed farmers that the organic fertiliser was more effective. See Photo below. 3. Seed testing techniques.11 ACF introduced a technique to enable farmers to test the quality (and appropriateness) of seed before investing in large quantities for cultivation. This enabled farmers to be sure that they were planting the most suitable and high yielding seeds. Due to the changing condition of the seed they cannot rely on the same seed year-on-year.
11 For additional information see the ACF Learning Review 2012
Page 13 of 58
ACF World Bank Programme Final Evaluation
4. Identification and provision of seed. During the research component of the programme (further detailed below) ACF identified suitable strains of paddy, wheat and different vegetable seed. Small quantities of these seeds were then provided to farmers, who with the storage and extraction techniques were able (in theory) to continue production with those seeds year-on-year. 5. Land preparation and bed preparation. ACF conducted trainings in land preparation to facilitate good irrigation and suitable planting conditions. The demonstration plots in each village were used for practical demonstrations. 6. Seed extraction. Farmers were trained in the extraction of seeds from vegetables. This enabled farmers to garner seeds from one harvest in order to plant in the next. This combined with the seed storage enabled them to effectively manage seed production and availability.
Initially farmers were sceptical of the new techniques proposed by ACF, but the demonstration plots set-up in each village served to show them how alternative techniques to those they were traditionally used to will lead to more successful harvest and higher yields. The trainings drove change in practices highlighted by farmers interviewed who accurately recounted the techniques that they had learned from the trainings. Money was also saved from the seed techniques and introduction of bio-fertiliser. Whether or not the techniques resulted in increased production is hard to say. Farmers reported a 20% increase in production but hard data to support this change was not collected during the programme. However we can plausibly state that the trainings did result in change and did, to some extent, result in increased production. However they did not result in the farmers being able to cultivate more land, which would have a significant affect on production because they are still burdened by the A farmer from Harshim Sholani village explains the process problem of irrigation. for making bio-fertiliser.
The system of lead farmers was a strong point of the training component. Lead farmers were identified in each village as the focal point for trainings. They were identified on the basis of their willingness and also their ability to effectively transfer messages to other farmers in the village. ACF used these focal points to organise training sessions within the villages and also acted as the main liaison with other village lead farmers. The community mobilisation of this programme was a strong element, and the system of lead farmers was a main factor in facilitating this. However, due to the lack of any systematic documentation of good practice, this process is not yet well documented and an investigation into how ACF managed to mobilise these farming communities would be beneficial. Therefore this process has been identified as a ‘best practice’ and more details can be found in Annex E.
Page 14 of 58
ACF World Bank Programme Final Evaluation
The output indicators for the training component were all met (or exceeded). 400 households received certified saline tolerant seeds, 423 households received training on agricultural techniques and natural resource management, 23 communities received seed saving containers (or rather wooden platforms), 23 communities received training on seed saving techniques.
A principal weakness in the approach taken was the inability for ACF to quantitatively attribute impact of the trainings on production. The lack of an effective monitoring (and impact measurement mechanism) prevents ACF from supporting the reported change with data. A 3-year period should have allowed ACF to monitor at least 2 harvest cycles and therefore could have measured the actual effects of the trainings introduced and also the level of uptake by the farmers. This was not possible firstly due to delays in implementation, and secondly because there was no monitoring system was in place (neither in the programme design, nor in practice) to do so. ACF however did do well to document the trainings and this material will prove valuable when implementing similar trainings in the future.
There were a number of monitoring exercises that took place (PDMs, PHMs, etc.) but these did not serve to attribute change as a result of the trainings and inputs provided. Post training surveys were conducted but these were limited to demonstrating whether or not farmers were applying the learned techniques. The monitoring across the project was generally poorly planned. Each activity had some degree of monitoring in place and ACF did well to continue frequent visits to field sites, but there was no programme wide monitoring strategy in place and so monitoring was done on an ad-hoc basis. This could be attributed to the lack of capacity in the programme team as well as lack of continued support from the coordination team, but also the fact that an M&E strategy was not inscribed in the programme plan should be acknowledged. During the evaluation the perceived (or real) restrictions of the project plan we sighted as reason for not doing certain activities, even though they may have appeared the suitable way forward to the programme team.
Finally, integration with government structures was not very strong. In order to ensure that the new techniques continue to be passed on within the communities and they are rolled put beyond them, ACF needed to work in closer collaboration with the Agricultural Extension Department. Integration with government is explored further in a dedicated section below. However ACF did produce training manuals that illustrated all the newly learned techniques, and explained them in detail in the local language (Sindh). This manual has become a ‘bible’ for a number of farmers Lead farmer from Yaquoob interviewed and was highly praised by one government Memom with ACF Agricultural Techniques Manual official12.
12 Khuda Bukhsh Beharani (Dept. of Social Welfare)
Page 15 of 58
ACF World Bank Programme Final Evaluation
Farmers interviewed were largely very supportive of the techniques that they had learned and demonstrated that they were appropriate to the need of increasing production. However they also noted that these techniques were not sufficient to significantly increase production. Most farmers interviewed (and beneficiary data supports this) are only able to farm a small proportion of the land they own. This is because they are unwilling to borrow credit in order to afford enough seeds, because of the increased risk they would be taking, should the harvest not be substantial enough, to repay the creditor. Lack of suitable rainfall was the principal concern here. Due to the dependence on wholesale buyers (the farmers source of income) for credit this risk is enhanced. Therefore farmers explained how access to fairer credit would encourage them to plant more and take a greater risk. The lack of fair credit for farmers is explored in more detail below.
Secondly, and perhaps more significantly, farmers explained that the irrigation challenge was still, despite the new techniques, a major set-back to increased production. Even with the ability to invest in cultivating more land that credit would provide, the irrigation challenge would remain. ACF would have done well to support farmers in overcoming this challenge. A section below is dedicated to exploring how ACF could have supported farmers with this challenge.
The Research Element
The research element of the programme was part of both the WaSH and FSL activities. In terms of WaSH ACF looked into finding the most suitable Solar Desalination Unit (SDU) in order to enable communities to produce their own ‘sweet’ (drinking) water. As far as the FSL activities went, ACF conducted research in order to identify strains of seeds that were suitable for the salinised water.
Weak planning bogged down both the FSL and WaSH research components. For example on the WaSH side, it was not until early 2012 (originally the end of the programme) that an SDU was identified, and therefore ACF was only just able to install the 400 units as required in the proposal. This delay was due to a combination of factors, including the floods in 2010 when most of the WaSH team (and certainly the WaSH PM) moved from the programme to respond to the emergency (this accounted for about 3 months) and also the research took some unanticipated course (looking at distilleries) that did not ultimately contribute to the final product in any meaningful way. In addition the turnover and gaps in WaSH PMs must be considered as a factor in the delay. There was no time to either monitor usage or follow- up on the market activities that were intended to enable other households to purchase them. As far as the seeds went ACF did manage to identify suitable strains but came up against problems of the supply of one such seed, and ended up having to import from Bangladesh causing delays. It is assumed that, considering the efforts made to procure this seed, there was not any more practical and suitable alternatives.
There was not a clear and defined research methodology from the start. Although it is the nature of research that you may not always achieve what you set out to do, it is important
Page 16 of 58
ACF World Bank Programme Final Evaluation
to have time bound objectives to ensure that they can be achieved within the programme period. It is unusual that there were not any research indicators in the LFA, but ACF should have defined some early on in the programme in order to monitor performance.
Interviews with ACF programme staff also reveal that there is little clarity over the fact the programme was a ‘pilot’, and what that actually means. Had there been clarity on this, ACF could have reduced the number of outputs required in order to focus on trial and testing (and monitoring) certain approaches. This relates to the SDUs but also to the farming techniques and seeds. The coordination team therefore could be said to have not appropriately explained this essence of the programme, or alternatively that the programme team did not inform themselves of the basic rational for the programme. For example, some core member of the programme team were not able to supply the evaluator with the initial World Bank assessment, which suggests that background knowledge of the programme was not sought.
Interviewees reported that by working with Sindh University and PARC the results of the seed research remain in Sindh and therefore, in theory, are available to farmers should they need them. However research results have not really been mainstreamed. Stronger links with government bodies would have facilitated this, and also by making them available online (ACF-USA website) more accessible. The results from the research (including the selected seed strains and the SDU design), and also the training materials could be posted online. More specifically the following (or versions of) documents: - Paddy Cultivation Research Report 2010 - Paddy Research Report 2012 - Renew access to boreholes turned saline: Solar desalination research report - Agricultural techniques and SDU construction guidelines
Integration with government bodies is addressed in more depth later in the report.
Although delayed, one of the strongest achievements related to the research component is that of the SDUs. Prior to this project ACF’s knowledge on water desalination was limited. The research carried out and the documentation produced13 have served to build a body of knowledge within the organisation that can be made use of in other coastal areas were saline water is an issue.
The Co-operatives
During the programme 3 co-operatives (covering 23 villages) were formed. ACF mobilised farmers in the villages and supported them in the setting up of management structures. Co- operative members were for the most part beneficiaries of the programme however some non-beneficiaries, such as larger landowners, were included. It was impressive that the co- operatives were able to bring larger landowners to the table with the sharecroppers and smaller landowners. ACF did not however take suitable advantage of this in order to
13 for example Renew access to boreholes turned saline: Solar desalination research report, March 2012
Page 17 of 58
ACF World Bank Programme Final Evaluation
negotiate the sharecroppers a better deal. Increasing the profits of tenant farmers (those indentured to landlords) was a programme objective therefore such negotiations should have been part of the process.14 It was not apparent that facilitating such negotiations was part of ACF’s intended activities, and in any case, it could be said that such dynamics were too entrenched for ACF to influence. This is addressed further in the section ACF’s approach to sharecroppers and their (non-beneficiary) landowners below.
The cooperatives served as a very effective platform for sharing of agricultural knowledge, from changing market conditions to which crops to plant when, and also sharing experience on the application of new agricultural techniques. An important (yet perhaps unintended) feature of the cooperatives was that they brought farmers from different villages together. Farmers interviewed reported how before the programme they had limited communication with nearby villages, despite their proximity, and their new relationship helped overcome various conflicts. By bringing farmers together they were able to support each other in overcoming shared challenges.
ACF did well to ensure that the founding of the co-operatives was inline with co-operative laws and managed to support 2 of them in achieving official registration. One co-operative failed to achieve registration as they were reportedly impatient with the quantity of paperwork needed for the process, and also had a less strong foundation than the others. Co-operative Members from the Village of Suleman Mullah ACF conducted ToT trainings on leadership and co-op management to facilitate the continued functioning of the structures. Subsequently 68 sessions on field level agriculture marketing, calculating cost of production, record keeping and vegetable Post harvest management, were also conducted with the co- operatives.
Even though the marketing activities of the programme were on the whole weak, the co- operative structures did serve to improve marketing activities of the farmers to some extent. The success in marketing was limited to the empowerment of the farmers through the co-operatives and the knowledge gained through marketing trainings. The marketing component is explored in more depth below.
The establishment of these co-operative structures is one of the main strengths of this programme. The fact that ACF successfully mobilised farmers into forming strong management structures and created groups that meet regularly without ACF support, is impressive. The process of community mobilisation however was not well documented, something that would have hugely benefited ACF beyond the duration of this programme.
14 Sharecroppers constituted 203 of the 445 targeted households (beneficiary database 2010, as provided by FSL PM)
Page 18 of 58
ACF World Bank Programme Final Evaluation
The Marketing Activities (Component B)
The Marketing Challenge for farmers The marketing component of the programme was geared towards both training farmers to play markets to their benefit and to facilitating market conditions that were favourable to the farmers. Farmers face the challenge of not being able to get a fair price for their produce and make suitable levels of profit. This is due to a variety of contributing factors. Firstly, due to the remote location of the villages farmers are requried to travel long distances and therefore transport costs are high. Secondly, wholesale buyers at the markets do not facilitate fair competition; on the contrary they exploit the farmers with incomprehensible trading tactics. Thirdly, because the target farmers are small scale (1-10 acres), individually they have small amounts of produce and therefore are unable to either benefit from economies of scale or sell large quantities at a good price. Finally, competitition is inhibited because of the indebtedness of the farmers to the wholesale buyers (or middle men). The only availability of credit for the farmers is from the wholesale buyers who purchase their produce. This means that there is immediately a power imbalance and restrictions to market forces.
To enable farmers to overcome these challenges ACF conducted a market survey and drew- up a marketing strategy, put on a number of trainings in marketing, established 2 different vegetable markets and assisted farmers with transport in getting there.
Missing the Main Challenge (debt) In March 2011 (2 years into the project) a market survey was completed. This survey does well to highlight most of the challenges summarised above. It also provides some sound recommendations on establishing co-operatives (for collective-marketing) and doing trainings to better inform the marketing practices of the farmers. However there is one significant conclusion that is not followed by a recommendation that being the ‘rotating credit system involving producer and trader.’
Producers are required to use the wholesalers for sources of credit as they do not have access to traditional banking services and there are no other alternative forms of credit. Credit is needed to enable farmers to purchase inputs (seeds) in advance of cultivation. This credit is then given on the condition not only that he pay back the monetary value of the loan but also that the creditor (and wholesaler) is then given a good price for the produce. This good price tends not to be the market price, nor a favourable price for the producer. Therefore in getting credit in this way ‘the producer has lost the right to bargain and sell in an independent manner.’15
This credit system is the overriding obstacle that farmers face to increase profits but the marketing strategy does not recognise that. With this indenture in place other activities to improve the farmers’ deal, are going to have piecemeal effect. During the 4th year of the
15 Marketing Survey: A Baseline Survey of Agriculture Marketing System in Keti Bander & Karochan Talukas of District Thatta, Sindh, Pakistan
Page 19 of 58
ACF World Bank Programme Final Evaluation
programme, staff did begin to consider (and even discuss with a potential credit partner, Tameer Bank) facilitating better access to credit. ACF has a good relationship and successful passed experiences with Tameer Bank (albeit in cash transfer) and therefore a partnership was a strong likelihood. Tameer Bank however was not prepared to take the risk of providing credit to farmers with such volatile income sources, without the assurance of ACF. Had these discussions begun earlier however, then the programme would have been active for a suitable period for such a partnership to be realised.
Marketing Trainings The marketing trainings (68 sessions in total) were by all accounts well conceived and effective. There were trainings in co-operative leadership, record keeping, calculating cost of production post harvest management and agricultural marketing. Farmers were informed about the tactics of the wholesale buyers and were shown how to deal with them more to their advantage. Despite the fact that the content of the trainings was without doubt very relevant it is difficult to assess the impact of the trainings as there was no post training follow-up, to ascertain if the knowledge gained contributed towards an increase in income.
The trainings on marketing techniques included visits with beneficiaries to different fruit and vegetable markets to explore new opportunities for selling their produce, serving to empower beneficiaries in future market dealings.
Establishment of Markets On 3 occasions ACF organised for producers and wholesalers (3) to convene at a new site near Gahro. The location of this site was more favourable for the producers as it was more accessible and transport costs were reduced. In addition ACF contributed to the costs of transport.
At best these markets were a good demonstration of how co-operatives may set up a market by themselves in the future. There were too many flaws to the market establishment approach for it to be long lasting and contribute to a significant increase in profits. These flaws include the fact that the programme brought in wholesale buyers from Karachi where (according to farmers in one FGD) there is plenty of demand at local level. Therefore it would have been more suitable to focus on establishing a market form local supply and demand. Interviewees felt that ACF did not conduct a through enough assessment of the markets in the area and determine a suitable approach. For example there was a successful vegetable market established in Khuwaja during the programme period that the programme could have learnt from.
Although having little bearing on the overall approach, the 2011 floods and cyclones in 2012 did have an effect on the establishment of markets. Unusually low yields did not enable farmers to benefit frm the new markets nor the markets to be of suitable size. Nevertheless when formulating a marketing strategy the possibility of such shocks should have been taken into account. As this area is so prone to flooding and cyclones in order to establish (and measure the impact) an effective market system should be established over a number of years (and therefore harvest cycles) which would increase the posibility of progress, and allow for some less successful harvests.
Page 20 of 58
ACF World Bank Programme Final Evaluation
Ultimately ACF had no effect on the local markets system. The overall trends on selling farm produce have not changed during the baseline and endline period. The markets that were established resulted in very marginal increases in profits, which without the contribution towards transport costs from ACF would have been 0. Therefore the markets established will not continue beyond the project period. This was reiterated by beneficaries.
Conclusions It became evident during the evaluation that for most of the farmers interviewed the marketing of their produce is of secondary concern to increasing production for consumption and then sale.. This leads to conclusion that the programme should have focussed on the obstacles to increasing production; access to fair credit and irrigation (both to enable farmers to cultivate all (not a portion of) their land) and the agricultural techniques (included in component A).
The market survey16 looked at the economic value chain from production through to marketing, but in a superficial way, so did not provide the theoretical underpinnings for a robust marketing strategy. A suitable marketing strategy should have facilitated the establishment of markets by the co-operatives themselves in a more organic manner (when conditions were right – high production etc.) but instead ACF forced the establishment of these markets. The marketing strategy also has a focuss on capacity building not on materials to commercialise the harvest, access to which would have been possible with fair credit and/or increased production.
The provision of inputs, trainings, marketing activities and exit strategy were all based on ensuring farmers learnt and then continued implementing good practice (be it agricultural techniques or marketing) rather than addressing the overarching problem; a lack of initial resources. The executed approach did not tackle the base of the problem – investment and need to borrow. These farmers were not ready for market, and therefore a marketing strategy should have more effectively addressed the problem of productivity and credit, as well as creating market structures. The two-pronged approach (increasing production and access to markets) is based on the fact that households need to sell a portion of their product to meet other needs. Therefore if the goal was to maximise these sales, then facilitating access to fair credit, and in so doing taking out the need to borrow on unfair terms from middlemen, was crucial.
If farmers were able to increase production they would be able to first pay for transport to get to market, and second could relieve themselves of the need for credit (eventually) as they would have profit to invest the next harvest’s inputs. In order to reach this stage and to relieve themsleves from the cycle of indebtedness, access to fair credit is essential. Microcredit would allow farmers to increase inputs, and cultivated land, pay for transport themsleves, and then be free to ‘play the market’ without being bound to one particular trader. If the presmise of the programme was to address both production and marketing then a key element (in both cases) was missing; access to credit. This was not considered in
16 Marketing Survey A Baseline Survey of Agriculture Marketing System in Keti Bander & Kharochan Talukas of District Thatta, Sindh, Pakistan
Page 21 of 58
ACF World Bank Programme Final Evaluation
either the marketting nor the production strategy until it was too late to include (2012 when ACF had discussion with Tameer Bank).
Having said that the trainings were relevant and are likely to aid farmers in future marketing activities. Farmers are now better informed in how to deal with traders, and will be more likely to negotiate a better deal in the future. Should the new agricultural techniques and the co-operatives activities increase production and enable relief from wholesalers provided credit, the knowledge will exist to use the market to their benefit. The co-operatives are a great base from which to maximise efficiency and increase marketing potential.
Without a multi-demensional approach including credit, inputs and techniques future collective marketing is unlikely, as even in collaboration (in co-operatives) the power imbalance (as a result of indebteness) is such that it does not allow farmers to get a fair price.
ACF’s approach to sharecroppers and their (non-beneficiary) landowners
The programme beneficiaries consist of two different types of farmers, those that own the land on which they farm (225 beneficiaries) and those that do not (203 beneficiaries) known as, share-croppers, and tend to work for large (non-beneficiary) land-owners. The programme however did not take different approaches to the two groups. The way in which they could have been assisted is twofold.
As sharecroppers are more vulnerable to changing conditions, it would have been appropriate to explore alternative (more reliable) sources of income for these farmers. They are not contracted by farmers and therefore should the harvest fail are at risk of even more limited returns to those they would usually receive.
ACF should also have negotiated with their landlords more robustly, encouraging the landowners to offer a greater share of the revenue to the share-croppers. The programme did well to improve communication with landlords. The fact ACF was able to sit with them with the potential of improving the deal for share-croppers is impressive, however ACF made little progress in this regard. ACF could have promoted the idea that more loyal and long-term share-croppers would be advantageous (in terms of profits) for the landlords, and also that the new techniques would increase productivity and directly benefit the landlords. However ACF did not investigate how the programme activities may have benefited the landowners, officially non-beneficiaries. It is unlikely that the activities were never going to benefit the landowners but this increase in production (due to new techniques) could have been used to leverage a more favourable deal with them.
The WaSH Activities (Component C)
The levels of salinity in the soils of the Thatta District not only had an effect on the agricultural activities of the population (as addressed in the FSL components) but also on the drinking water. Typically communities either relied on water trucking, used previously installed boreholes for drinking water sources or more concerning used surface water from
Page 22 of 58
ACF World Bank Programme Final Evaluation
irrigation channels. Water trucking is expensive and unreliable, and due to changing conditions levels of salinity in borehole water was often unacceptably high.
ACF decided to respond with a package of WaSH activities. ACF renovated boreholes, conducted hygiene promotion sessions, distributed hygiene kits and built hand washing stations. However the most significant activity under the WaSH component was the research and installation of the Solar Desalination Units, and it is on this part that this evaluation will focus. With the exception of the SDU rollout phase, despite the significant time lag in many of the activities, the relevant expected results were either met or exceeded.
Solar Desalination Unit Research and Implementation Due to a number of reasons (including the floods and inappropriate staff) the research into a suitable style of SDU did not get fully underway until the arrival in November 2010 of the 3rd different WaSH PM since the programme began. In collaboration with the relevant government body in the region, the Pakistan Council of Scientific and Industrial Research (PCSIR), ACF tried and tested a variety of different designs in order to come up with the most cost effective design that could be locally produced. The results of this research are presented in the Renew access to boreholes turned saline: Solar desalination research report, March 2012.
By March 2012 ACF had identified 2 different models referred to as PCSIR and ACF models. The former has a high capacity and efficiency however is expensive and is more difficult to build. However, ACF implemented 150 of the PCSIR model which produced an average 7L water/day and cost $300. The ACF model is much cheaper at $250 per unit and produced an average 5L water/day. 250 ACF units ACF designed Solar Desalination Units in the village of Yaquoob Memon were installed. Other than production cost another distinct advantage of the PCSIR model is that is moveable and can be used to purify water. Therefore in the case of hurricane can be displaced, and continue to be used to provide desalinated and purified water. Nevertheless ACF pursued the ACF units for the most part because of the higher cost/production ratio.
The ACF model was identified as it was most efficient at turning saline water into drinking water, the materials were available locally and it was the cheapest. However it is still expensive for communities coming in at $250/unit. ACF did explore ways in which the cost could be brought down, however the project was coming to an end and installation had to be completed. Even though this model was identified in March 2012, delays in the
Page 23 of 58
ACF World Bank Programme Final Evaluation
procurement meant that construction of the 400 units did not begin until December 2012. The WaSH team then did a sterling job of finishing all construction before the end of the programme (March 2013).
Delays in the identification of a suitable unit and the installation of them did not allow for sufficient monitoring of unit usage, and therefore it is difficult to prove whether they will be continue to be used by the beneficiaries. When visiting the villages some had only been in place for a matter of weeks. Even so beneficiaries were able to demonstrate the procedures for maintaining the units and explained how they had improved their lives.
Having spoken to a number of beneficiaries of the SDUs the availability of suitable drinking water has increased. Although an improvement, and has enabled beneficiaries to save costs (for water trucking, in water purifiers, and in healthcare costs), this still does not meet Sphere standards.
Although it can be said that ACF has made some valuable progress in identifying an effective design, there is still considerable work to be done in making it more affordable. This refers to both decreasing the cost of the unit and increasing the ability of households to afford them.
In the first case ACF is continuing to trial alternative cost saving measures, including incorporating a chulli filter. By using a chuly filter to heat the water before it enters the unit production is increased, enabling the unit to be reduced in size and therefore reducing material costs. Currently water enters the unit at 30oC (air temperature) producing 10 L/day. By bringing the temperature of the water up so that it enters the unit at 70 oC production increases to at least 20L/day. Although ultimate production will not necessarily be increased the cost of the unit is halved due to the halving in material costs. These trials continue and will serve to inform an upcoming ECHO funded WaSH project in TMK and Badin, where SDUs will be implemented.
Even with a reduction in cost to $125, units are still barely affordable, and no beneficiary when asked said that given their current level of income could purchase one. Therefore if this approach is to be sustainable in the future ACF has to look at options in which to enhance the income of the households to enable them to afford the units. As explored in below it is here where there is a natural overlap in WaSH and FSL activities as the affordability of these units becomes a livelihoods issue. Due to ACF’s effective promotion of these units and the cross-village linkages made by the co-operatives even villages that were not a target of the WaSH activities are aware of the financial and health benefits of the units. Therefore it is not a question of awareness, nor of supply, but of affordability.
Most villages in the area have access to surface water in the irrigation channels. Whereas this is not yet purified it is low in salination. Therefore ACF should explore the option of providing filters to purify this water source. This is likely to be cheaper. At peak seasons this source may run dry at which time shallow boreholes penetrating the saturated bed of the irrigation channel should still yield water, or else water trucking could be considered over a short period. However this is still highly depended on a number of factors. Affordable SDUs still appear to be the most suitable approach. In any case, analysis needs to be made for the
Page 24 of 58
ACF World Bank Programme Final Evaluation
cost between SDUs and other options (such as paying for health care) in order to advocate households to invest in them.
As the initial baseline was weak, ACF took the initiative to produce another one before the installation of the units, enabling the endline to assess the effectiveness of the units in achieving the sole WaSH related indicator: “at least 10% decrease in healthcare costs.” When asked beneficiaries said that not only had the units saved them money on water purifiers, but also on visits to the doctor due to diarrhoea. Therefore this should also be considered when calculating the cost benefit of SDUs, and should be made apparent to the beneficiaries when considering purchase.
Had it not been for the delays, ACF could have finished the research in 2010, installed and monitored use in 2011 and then facilitated the roll-out of the unit (through a livelihoods approach) in 2012. That said the products of the research into the different SDUs will continue to benefit ACF in other programmes and the research report will be disseminated amongst relevant government departments, including PCSIR, PCRWR and the Public Health Department.
The ACF SDU design took into consideration the likelihood of floodwaters and therefore rather than being placed on the ground is raised by 3 ft. The PCSIR model however is mobile, and therefore can in theory be moved if necessary.
Integration of the FSL and WaSH components
One of the weakest elements of the programme was the integration between the FSL and WaSH components. The programme was not designed to be an integrated response, and at no stage throughout its duration were any significant steps taken in order to ensure the activities of both sectors complemented each other. Therefore it was not possible to achieve additional impact of the programme or to ensure optimum efficiency through the aligning of the two sectors. There were a number of missed opportunities for integration. In order to analyse these it is necessary to look at both what integration there could have/should have been considering the programme activities that were pursued, and then how a more appropriate programme design would have ensured an integrated response.
Missed Opportunities for Integration It was clear from initial discussions with FSL and WaSH staff that there was very minimal communication between the 2 teams at base-level. The rational for this presented by the team was that the WaSH activities rather than being irrigation related, were related to drinking water and hygiene therefore scope for integration with agricultural activities was very limited. It was considered therefore that integration could not extend beyond targeting the same villages.17 Indeed the design of the programme and the structures put in place did not lend themselves to an integrated approach, nevertheless the programme teams could have integrated on a number of levels to optimise the use of resources, and be more efficient and effective.
17 WaSH targeted 10 villages of the 23 FSL targeted villages.
Page 25 of 58
ACF World Bank Programme Final Evaluation
There was a couple of reported instances in the sharing of data (including targeting information) and coordination of fieldwork activities, however these were the exception and not the rule. At no occasion did the programme teams either take the initiative to put their heads together over the individual activities, nor was there the leadership from coordination level to ensure it happened. This did not necessarily require a single head of project, covering all components, (as suggested) but rather the initiative to discuss activities and see where benefits could be achieved by working together. At base-level meetings between the 2 teams only happened in regard to base management issues and not to discuss programme or logistical matters. It was even reported by one respondent that the two teams were taking separate cars to the same village on the same day without knowing. At the most basic level the necessity for the 2 teams to discuss where they are going when is paramount not only to save on transport costs, but also to ensure ACF appears as one to the beneficiaries, avoiding confusion.
The fact that there was a high turnover in WaSH PMs did not help the 2 teams to build a close working relationship. But had the intention existed amongst longer serving staff a spirit of close coordination on all matters could have been maintained. After all both teams were working towards the same objective.
On a programme level the potential for integration was fourfold; in trainings, in setting up CBOs (co-operatives and water management committees), in market analysis and creation, and in SDU funding.
Training Both WaSH and FSL included training, in the former case in regard to hygiene promotion and maintenance of SDUs, and in the later case in regard to agricultural techniques. Although the subject matters are different and therefore the scope for holding joint sessions (for example) may have been problematic, both experienced similar challenges in the training approach, the presentation of issues, the production of training materials and the mobilisation of attendees.
Setting-up CBOs Both teams experience similar challenges in the setting up and the maintenance of all CBOs. Information sharing on these issues could have benefitted both teams. Furthermore, the efforts in setting up these CBOs could have been halved had one body been set up for that covers all issues. Considering the farming households targeted for the SDUs were also beneficiary farmers the same structures could have been used for the farming co-operative and the SDU/borehole management committees.
Market Analysis and Creation As well as installing 400 SDUs, the WaSH team was tasked with creating a market for the units so that once the programme was over the knowledge remained within the communities to enable additional units to be purchased. The focus here was on the supply. The WaSH team did a commendable job in sourcing local materials for the units and identifying (and training) local contractors to build the units. This approach included a substantial market survey in order to identify strengths and weaknesses in the market. A
Page 26 of 58
ACF World Bank Programme Final Evaluation
large portion of the FSL activities were also market based, and therefore there existed scope for both teams to share information on local market information, and also market survey methodologies.
SDU funding Finally, and perhaps most significantly, the biggest challenge for the WaSH team in ensuring longevity in the introduction of SDUs into the communities was ensuring the continued demand for the units. It’s all very well ensuring the supply of the units but if there is no demand then benefits will not go beyond those supplied by ACF. As outlined below/above, it is unlikely that any additional SDUs will be bought because they are not affordable. Even with the introduction of cost-reducing measures the units will still be too expensive for communities, and with current income levels they will not purchase additional units. This therefore is a livelihoods issue. It is not that households do not appreciate the value of the units in reducing healthcare costs (on the contrary beneficiaries - even those without SDUs – understood well the benefits), it is that they cannot afford it. Therefore the teams could have discussed alternative ways of increasing household income and/or facilitating access to credit to enable households to make this initial investment. The failure of farmers to increase profits through agriculture has an impact on their ability to invest in the SDUs, and therefore highlighting a potential linkage between the FSL and WaSH activities of the programme.
Suggestions for a more integrated programme plan. Whilst retaining a focus on agriculture a more integrated WaSH component would have addressed the irrigation needs of the farmers. Farmers placed irrigation high on the list of priorities in enabling them to increase profits. For example, Syed Nazeer Shah, a beneficiary farmer, explained how although he owned 8 acres of land he was unable to cultivate more than 2 because he could not take the risk in investing in the seeds needed and there not being sufficient water to irrigate. Activities that dealt with the lack of water and irrigation capabilities for farming would have significantly contributed to the ability of farmers to increase profits and therefore improve living conditions.
In terms of water supply ACF could have explored community-based approaches such as rainwater harvesting, capturing of floodwaters and water saving. This would enable communities to deal the water challenge themselves without having to rely on the water authorities. However ACF should also have communicated with authorities on the unequal distribution of water resources (more on advocacy potential below). Farmers reported that at crucial times in the year the Irrigation Department would favour large landlords in the distribution of water. One farmer recounted that the situation was so desperate he was compelled to irrigate his land with saline water. Which is not only ineffective in the short- term but causes long-term damage to the land and further enhances the problem of salinity.
Aside from water supply, irrigation activities could have included the levelling of land, the introduction of drip and sprinkle irrigation, and maintenance of irrigation channels. ACF could have introduced Cash-For-work schemes in rehabilitating irrigation channels which would have also provided an additional (albeit short-term) income source. These suggestions are made bearing in mind the limits to flexibility posed by the programme plan
Page 27 of 58
ACF World Bank Programme Final Evaluation
and/or the difficulties of flexibility experienced with World Bank, and are made more in terms of lessons learned for future such programmes.
Responding to Shocks
During the course of the programme there were 3 significant natural disasters that affected the programme; floods in June-September 2010 and 2011, and cyclone (Cyclone Phet) in June, 2010.
Despite the floods in 2010 affecting over 7 million people in Sindh province targeted communities in Karo Chan and Keti Bundar did not suffer major losses in the floods of 2010 as they are located far south of the bund breaches, and in terms of FSL there was ‘negligible 18 direct impact’. However the mission’s priorities did change from the programme to responding to the emergency. This was demonstrated more notably in regards to the WaSH component activities to that of the FSL.
The WaSH programme team was entirely focussed on conducting assessments and then implementing emergency response activities, and therefore the scheduled activities did not take place. This however did not prove to be a significant gap in implementation (only 3 months from a 48 month) and the WaSH activities, due to delays in recruitment (for example) had not advanced much anyway. The transfer to emergency response activities was reportedly very rapid but was due to the defacto WaSH PM at the time having a greater motivation for these types of activities, rather than there being a specific mechanism in place for this change. Nevertheless the input from the World Bank WaSH programme staff was effective and boosted the response to the emergency19.
In the case of the FSL team there was not so much support given to the emergency response. Two FSL staff were seconded to conduct needs assessments but did not provide the necessary technical capacities to do a good job. This brings into question the suitability of staff on a development programme such as this to do emergency work such as rapid needs assessments. In addition, if a development programme team is to be compelled to divert activities towards an emergency response, this must be detailed in their employment contract. This would mean that programme staff, despite being contracted for specific activities, are contractually obliged to respond to more pressing needs, as and when they arise.
As detailed above ACF-Pakistan was a small mission at the beginning of 2010 managing only the World Bank programme in Sindh Province. Therefore when the floods affected the region the majority of ACF staff were dedicated to the World Bank programme, which did not include any component that would enable the flexibility to respond to the emergency. So where the programme did contribute to the emergency it was because of individual capacity and motivation, and on an ad-hoc basis. A formalised protocol inscribed within the
18 Pakistan Floods – Report on Impact to Ongoing Activities, December 2010 19 Interview with ACF-USA WaSH Advisor, March 2013
Page 28 of 58
ACF World Bank Programme Final Evaluation
programme proposal would ensure all activities are required to adjust (or be put on hold) to respond to the emergency.
Although the programme activities and target areas were not significantly effected by the floods, in future such programmes where community-based structures are built in disaster prone areas ACF should endeavour to use these groups for preparedness activities such as identification of evacuation sites, raising awareness about early warning signs, communicating with ACF on needs and allocation of roles (within the group) in the advent of an emergency20. Furthermore these groups provide the scope to build an early warning system with such structures – not only for fast onset (floods, cyclone) but also for sharing information on changing climatic conditions. In conclusion as stated in the Flood Impact Report “future programs in disaster-prone Sindh Province will need to build the resilience of communities to different forms of disaster and strengthen local response capacities while simultaneously reinforcing local systems of production and exchange.”
Relationship and Interaction with local stakeholders
Most ACF programmes involve some level of integration with local and/or national government. With long-term development programmes such as the World Bank programme integration of activities is paramount to ensure longevity of the intervention. Pakistan has a complex structure of local authorities, which proved a challenge for ACF to negotiate. It is commendable that ACF’s approach included the involvement of relevant government bodies, but collaboration was not always successful.
ACF began to collaborate with the PARC (Pakistan Agricultural Research Council) on the testing and verification of different seed varieties and soil testing. Not only were PARC the most suitable government department for this purpose, but they were also required to certify the selected seed variety. It is because of this that collaboration with them was detailed in the programme design. Working with PARC was not however very straight forward. Negotiating and signing an MoU was not easy and in the end ACF consulted PARC on seed varieties but never actually formalised a relationship and therefore was not able to ‘transfer’ knowledge in a systematic manner and guarantee the on-going application of the research findings. A similar partnership was also attempted with NIAB, however ACF was also unsuccessful in formulating a formalised relationship. Sindh University was also involved in the research (soil and seeds) and therefore to some extent the knowledge gained in soil and seed varieties will remain in Sindh, and has the potential to benefit famers in the future.
In terms of the agricultural practices and institutionalising the new techniques the most appropriate department was the Agricultural Extension Department. It is this department that ACF intended to involve in the training sessions so that they could provide similar support to further co-operatives. However ACF did not manage to gain the support of this department, as they were not in favour of some of the approaches involved, despite their proven effectiveness. For example, some Agricultural Extension officials were not in favour of promoting the use of organic fertiliser as they had an interest in selling the more expensive, less effective chemical alternative. Various attempts at some sort of
20 Refer to ACF Pakistan External Evaluation of the programme “Reducing vulnerability with DRM in KPK”
Page 29 of 58
ACF World Bank Programme Final Evaluation
collaboration were unsuccessful until early 2012 when a MoU was signed. This MoU however proved meaningless firstly as it came too late and secondly as it only requires ‘co- operation’ and no actual involvement, resulting in no practical collaboration.
The WaSH team also collaborated with PCSIR in the research of suitable SDUs. Having tested various designs and establishing the most efficient and cost effective version, in collaboration with PCSIR, this knowledge will remain with them and should benefit communities in the future. They also provided important technical advice on the features of the SDU. However it remains to be seen whether PCSIR would actively promote the use of the cheaper model SDU as they have developed and produce their own design. The WaSH team also worked with PCRWR on analysis of water produced by the SDUs. Initially it was surprising to find out the WaSH team did not work with the department concerned with drinking water, the Public Health Department, but it turned out that they were not ‘active’ in the target areas. Considering it is this department that is tasked with ensuring populations receive clean drinking water, ACF could have been in a good position to advocate for them to begin activities in Sindh.
Conclusions Overall there was some level of government buy-in and there was certainly support (with the exception of the Agricultural Extension Department) for ACF activities at local government level. ACF did well to ensure they respected all government protocols (in co- operative registration or example) and did achieve validation of some seed strains by PARC. However what was achieved in terms of integration is limited to this. It is very unlikely that any of these government departments will draw from ACF’s activities in any significant way nor that the working conditions for the targeted farmers will improve as a result of ACF’s approach to authorities. ACF did not have a well thought out or assessment based approach, and instead pursued a strategy that was ad-hoc at best.
Whereas it must be recognised that there is a corrupt-system at play in Pakistan and most notably in Sindh province, making interaction with government authorities challenging (maybe even in some cases impossible), ACF could have taken a more effective approach. This should have started with a thorough mapping of local authorities. This would have included an assessment of each relevant department, including their appropriateness for collaboration on certain activities, their capacity to do so, and their willingness.
Regardless of the challenges in changing government practice in favour of the targeted farmers, a strategy must be built, based on a thorough assessment. This would have enabled the programme to save time on waiting for ineffective MoUs and also enabled suitable relationships to be built in line with programme activities (may be even in advance).
Working with government bodies in Pakistan is very difficult due to the lack of resources and/or the inappropriate use of resources. Corruption is rife so civil servants do not necessarily operate in the interests of the electorate. Nevertheless in development you have to work with governments, and a strategy founded on a clear evidence base should be formulated. This may well include high-level negotations that require the invovlement of senior ACF staff members. Therefore should not be left to the PM but to FieldCo or even the Country Director.
Page 30 of 58
ACF World Bank Programme Final Evaluation
The relationship with PARC did not function because they required financial re-imbursement for field trips, and ACF as unwilling to meet these demands. ACF should consider why they did not pay PARC. It is normal practice to pay per diems for government employees to attend tranings and therefore ACF should have explored such incentives that would have increased their invovlement. This could then have also been translated onto the Agricultural Extension Department and other relevant government departments.
It is also recongised that by paying incentives and then departing activities may not continue. But the advantage in this approach is that ACF can use this programme period to gain government buy-in and advocacte for the allocation of funds to continue the activities. Another example is that ACF could have funded a position in government for seed exploration, certification, market building and pursuaded government to continue the funding after the programme period.
ACF should not have focussed on PARC. A thorough assessment and resulting strategy of government interaction would have directed ACF to work with the Seed Authorisation Authority, which, although apparently not very active in Sindh would have been another appropriate department to work with. Having identified the suitable strains ACF could have gone one step further and worked with them to build a market for the seeds, as well as identifying good strains.
The important point, whether ultimately feasible or not, is that these alternative approaches to working with government were not considered. It is well known that Pakistan authorities are challenging to deal with but ACF should have adapted a clear strategy based on a thorough assessment. Due to the unpredictable nature of dealing with authorities a fluid approach would have been appropriate, enabling ACF to act according to each failure and success. Such an approach was successful in other areas of Pakistan such as with the Livestock Authorities in KPK, whose support was gained by creating a network of support for the project.
Finally working with authorities should not be a question of ‘handover’ but itegration from the beginning. ACF is well versed in dealing with weak and apathetic governments, therefore has the capacity to formulate a coordinated and effective approach.
The potential for Advocacy
Advocacy was not an objective of the World Bank programme. But a successful development programme must work in conjunction with government authorities with aim of integration into government systems. As we have seen ACF had difficulty in taking such an approach successfully, which is both down to the lack of an appropriate strategy and the willingness of governments bodies to participate. It is this second point that would have benefitted from advocacy activities.
Areas where beneficiaries would have benefitted from change in government policy which ACF was in a strong position to apply pressure include with the water authority (in regard to the unequal distribution of water) and secondly, with landowners (through the government)
Page 31 of 58
ACF World Bank Programme Final Evaluation
in order to convince them to give sharecroppers a better deal. In terms of water availability there was even an instance when some farmers had mobilised through the co-operatives to protest against the unfair management of water resources. ACF could have capitalised on this momentum and support the protest at a higher level.
In regard to the landowners, there is scope for ACF to work with them and convince them that it is in their interest to give sharecroppers a better arrangement, as this could be aligned with an increase in production. The assumption is that better conditions (including % of profits acquired and assurance of employment through contractual commitments) for workers, would most likely lead to higher retention, and would mean a decrease in gaps in employment, higher production and resulting profits. Ultimately the angle that ACF would take is that all such activities are supportive of increase in production, which is good for the economy which is good for Sindh and good for Pakistan. However the debate of whether ACF is/should be a rights based organisaton goes beyond the scope of this report.
During the evaluation ACF staff raised the concern of the risks that engaging in such activities could pose for ACF’s presence in the region. It is not suggested that ACF should engage in a campaign on land rights (such as Oxfam) but awareness-raising within the framework of the programme. Furthermore, by taking such an approach in collaboration with other NGOs (local and international) as well as UN agencies, ACF could be more coordinated and effective in pushing for policy change. Also, agencies with a longer presence in the area, and greater knowledge of individual government institutions, could have informed an advocacy approach. Oxfam and WWF have been working on issues of irrigation and farmers rights in Sindh therefore it would make sense for ACF collaborate with them. The Agricultural Forum of Pakistan, is such a platform, that ACF did not engage with during the programme.
Finally, as suggested in terms of dealing with governments with the programme components that were designed, such strategic level advocacy should not be a bonus of the programme manage, but the FSL Coordinator or even the Country Director, because it is these profiles that are more suited negotiating and pushing change at this level.
Learning from Mid-Term Evaluation
In March 2012, one year before the end of the programme, a mid-term evaluation was conducted. The aim of this evaluation was to assess the programme and inform the remaining year of the programme. The recommendations from this evaluation however had little bearing on the final year of activities. This is partly due to the weak nature of some of the recommendations, but also to the fact there were no systems put in place to insure the inclusion of the recommendations. This could have taken the form of an allocation of each recommendation to one particular member of staff and also a defined timescale for the implementation. Also it was particularly unfortunate that the FSL PM was absent during the evaluation. Involvement in the evaluation would have enabled the recommendations to be developed in collaboration with the FSL PM and facilitate their incorporation into the programme.
Page 32 of 58
ACF World Bank Programme Final Evaluation
CONCLUSIONS AND RECOMMENDATIONS
1. Interaction with government. ACF did not have an effective, assessment based strategy for dealing with government bodies. Conduct a thorough assessment of all related government bodies, covering their capacity, their interest and their priorities. Such assessments could have included power mapping, SWOT analysis and risk mapping. Any development programme has to work with authorities no matter how difficult it is. This is what ACF has done for a while with nutrition programmes.
2. Advocacy. Government support for improving the lives of coastal farmers that should be enhanced. Management of water resources, giving sharecroppers a better deal (to land owners), farmer/co-operative support could be brought higher up the government list of priorities. Formulation and implementation of an advocacy approach would be appropriate in order to make this space for the programme activities and gain buy-in (and continued support) for them within the government.
3. World Bank. This was a pilot programme, which could suggest that there is an intention to fund further activities once approaches had been tried and tested. Considering the lack of engagement from WB the indication is that they are not concerned, however ACF would not know, as they ‘admitted defeat’ early on in regard to the relationship with WB. WB should be called to account for their lack of engagement with the programme and ACF should also question why this was. ACF had a solid set-up and was addressing a clear need with the potential for large-scale impact within the mandate of ACF and WB. Therefore ACF should have maintained a more persistent approach with WB with the intention of pursing more long-term sustained funding.
4. Irrigation. Water management solutions are an essential element of the requirements for increasing agricultural production. ACF should explore community based water management solutions, efficient irrigation mechanisms and maintenance/renovation of irrigation infrastructure.
5. Planning. Considering this is a pilot project with a strong research component. The period for research and assessment should have been much more extensive, giving the time to fully assess the status quo (deeper market analysis), test various approaches (SDUs, seeds), and then monitor their use. Time bound goals and research objectives should have been outlined.
6. Marketing Assessment. The marketing component was not based on a thorough assessment of not only the markets, but also of the socio-economic position of farmers. ACF should conduct an in depth anthropological study to enable a better understanding of the social dynamics affecting sharecroppers and farmers (after all these 2 need to be addressed with 2 different approaches). By having a marketing specialist from the beginning and beginning the market assessment from the beginning this would have enabled trialling in at least one agricultural cycle and also allowed for usual obstacles such as flooding and cyclones.
Page 33 of 58
ACF World Bank Programme Final Evaluation
7. Credit. The farmer’s reliance on the wholesalers for the provision of credit is keeping them in a cycle of debt. In order to cultivate what is sometimes only a small proportion of land available they need to borrow for inputs, and also for other emergencies (weddings, health). By providing an alternative fairer source of credit, farmers would be able to invest in greater production (utilising more of the land available), and then continue to increase profit margins year-on-year. Drawing on experience elsewhere discussions with Tameer Bank did start but were too late. Had marketing activities started sooner, and this challenge been identified earlier on, a partnership with Tameer Bank would have been possible.
8. Integration. Integration between the WaSH and FSL activities was practically non- existent. Firstly, integration should have been better defined at programme design stage, and in planning. Secondly, ACF leadership should have compelled the WaSH and FSL teams to meet regularly to cross-check activities and explore avenues for increased impact and efficiency gains by working together. This should have been instigated at capital level and managed at base level (FieldCo for example).
9. Monitoring. There were a number of monitoring activities that took place but these were ad-hoc and not uniform across villages. A monitoring and evaluation plan should be put in place before the start of activities. This should be linked to a clear theory of change analysis with time bound and defined expect outcomes. Furthermore this should be the case across all villages and should be adapted depending on specific needs. Clear systems should be put in place to follow-up evaluation recommendations. In an ideal world all programme staff would be sufficiently training in M&E but a more realistic approach would be to have an M&E specialist who provides such support to each programme.
10. Emergency Response. A clear protocol for the programme team in the advent of emergency response should have been outlined from the outset of the programme. This includes contractually obliging programme staff to shift activities to the response. This would be integrated into a wider EPRP (Emergency Preparedness and Response Plan).
11. Co-operatives. The co-operatives structures provided a great base for community mobilisation and for ACF to influence the communities. Farmers drew wide ranging benefits from the co-operatives. Co-operative activities should extend to emergency preparedness and response activities as well.
Page 34 of 58
ACF World Bank Programme Final Evaluation
ANNEXES
Annex A: Logical Framework Analysis Annex B: Terms of Reference Annex C: Evaluation Itinerary and Meetings Annex D: Bibliography Annex E: DAC based ratings table Annex F: Field Questionnaire Annex G: Best Practice
Page 35 of 58
Annex A: Logical Framework of Analysis
Title of the Improving Living Conditions in a Changing Environment – Thatta Coastal Farmers Action Principal To improve the living conditions in coastal Thatta of approximately 6,500 small farmer (farmers with 1-10 cultivatable acres of land) and tenant (indentured Objective farmers to landlords) households by increasing household income and decreasing household expenditure and burdens. Intervention Logic Objectively Verifiable Indicators Sources of Verification Risks and Assumptions
R1: Increase autonomy/self • 400 agricultural households receive Baseline Reports Natural disasters such as Results sufficiency on agricultural certified saline tolerant seeds and other Monitoring/Impact Reports floods, cyclones and droughts inputs inputs Post-Market Reports have been known to strike • 400 agricultural households receive Final Livelihood Assessment Thatta coastal areas on an improved agricultural techniques trainings Reports annual or bi-annual basis; and on-farm natural resource management which could adversely affect • 20 farming communities receive seed the project outputs. saving containers. This project will compete with • 400 households practice seed saving the established system for techniques and improved on-farm natural money lending for agricultural resource management • inputs. Decrease livelihood expenditures by 5 to 10 percent R2: Reinforce agricultural
wholesale markets • 3 to 5 Farmer Organizations received
marketing cooperative training
• 6 wholesale markets are organized at harvest time with transportation support • Sales increased at least by 10% for the R3: Renew access to boreholes farmers turned saline • 400 households use low-cost desalination units to treat saline groundwater R4: Increase knowledge and • 400 households attend hygiene education analysis in the livelihood sector sessions ACF World Bank Programme Final Evaluation
for Thatta district coastal areas • At least 10% decrease in healthcare costs for 400 households. • 2 livelihood surveys are conducted YES - BASELINE and ENDLINE • Surveys are presented and disseminated
As per result 1 Activities 1.1 Mobilize 400 farming households for introduction of saline tolerant seeds and seed saving techniques in cooperation with Pakistan Agricultural Research Council 1.2 Conduct soil analysis within 2 Union Councils to identify appropriate certified saline tolerant seeds 1.3 Distribute certified saline tolerant seeds and required inputs to 400 households and conduct an Environmental Management Plan (EMP) to identify/mitigate any potential adverse environmental impact 1.4 Integrate local Agricultural Department and 3-5 Farmer Organizations in improved agricultural techniques trainings and on-farm natural resource management (e.g. seed saving techniques and mesquite harvesting) 1.5 Distribute vouchers for community seed saving containers to 20 farming communities
As per result 2 2.1 Establish 6 wholesale markets within targeted zones within 3 years 2.2 Mobilize 3-5 Farmer Organizations for cooperative marketing of agricultural products. 2.3 Mobilize 4-6 wholesale buyers to participate in wholesale market. 2.4 Transport agricultural products to major transportation lines 6 times
As per result 3 3.1 Select 400 interested households that live near a saline borehole. 3.2 Rehabilitate all non-functioning boreholes located near the selected households. 3.3 Select/design the most appropriate low-cost desalination unit for each household. 3.4 Distribute the selected units to each household. 3.5 Train households in operation and maintenance of the units. 3.6 Setup/Train a water committee for each borehole. 3.7 Conduct hygiene education sessions for all households 3.8 Identify local businesses to manufacture/sell these units.
As per result 4 4.1 Conduct livelihood surveys at the beginning and the end of the project 4.2 Presentation of the results to the main actors involved in the area 4.3 Dissemination of the reports
Page 37 of 58
Annex B: Terms of Reference
TERMS OF REFERENCE
For the Final Evaluation of ACF’s Programme
Improving Living Conditions for Coastal Farmers in a Changing Environment, Thatta District, Sindh Province, Pakistan
Programme Funded by Japanese Social Development Fund and the World Bank
Contract Reference [TF093387]
ACF World Bank Programme Final Evaluation
February 2013
1. CONTRACTUAL DETAILS OF THE EVALUATION
1.1. Key Evaluation Dates
Expected Start Date: 1st March End Date: TBC Submission of Draft TBC Report Submission of Final TBC Report
1.2. Language of the Evaluation
Language Requirements for the English Evaluation: Language of the Report: English
1.3. Workplan & Timetable
Activities Working Days Briefing HQ (teleconference) + preparation 3 Travel to mission 1 Briefing mission, review of documents, preparation of fieldwork + evaluation 2 framework. Detailed methodology to be proposed to ACF prior to commencing fieldwork. (Islamabad) Field work (Sindh) 9 Collection of secondary information in capital + meeting with WB + JDSF 3 Data analysis and preparation of the draft report, common editing of the report 5 with ACF Pakistan Debriefing and presentation of preliminary findings in-country on the basis of 1 the draft report Travel back from mission 1 HQ debriefing with desk officer 1 Finalization of the report on the basis of Field, HQ and ACF-UK Feedback 3
1.4. Budget Details/Conditions
N/A
2. DETAILS OF THE PROGRAMME
Page 39 of 58
ACF World Bank Programme Final Evaluation
Name of the Improving Living Conditions for Coastal Farmers in a Changing Programme: Environment Location: Keti Bandar and Kharo Chan Talukas of District Thatta, Sindh Province Starting Date: 3/11/2009 End Date: 3/11/2013
Page 40 of 58
ACF World Bank Programme Final Evaluation
2.1. Map of Programme Area
Page 41 of 58
ACF World Bank Programme Final Evaluation
2.2. Programme Overview High investments in inappropriate agricultural inputs, poor returns on agricultural activities and high health care costs from the incidence of waterborne diseases have created low standards of living for coastal farmers in district Thatta. As a result many small farmers and tenants are dependent on loans not only to procure agricultural inputs to sustain their livelihood activities but also to meet daily needs. Although many coastal areas of Pakistan face similar challenges, the salinity problem in Thatta (with 40% of the total farmlands affected) is particularly severe. Factors underlying the chronic vulnerability in the area include: Irreversible salinization of agricultural areas High dependency on inappropriate and untested seeds for the area (seeds supply is coming from the international markets. Government of Pakistan certified seeds are not available) Nonexistence of physical or formal wholesale market for agricultural produce (sometimes resulting in price differences of 250%) Lack of access to clean drinking water With the support of the Japanese Social Development Fund and the World Bank, ACF is working in coastal Thatta since March 2009 with the overall objective of improving the living conditions of 6,500 small farmer households (farmers with 1-10 cultivatable acres of land) and tenant (indentured farmers to landlords) by increasing household income and decreasing household expenditure and burdens.
2.3. General Objective
To improve the living conditions in coastal Thatta of approximately 6,500 small farmer (farmers with 1-10 cultivatable acres of land) and tenant (indentured farmers to landlords) households by increasing household income and decreasing household expenditure and burdens.
2.4. Specific Objectives/Results
Result 1: Increase autonomy/self-sufficiency on agricultural inputs Mobilize 400 farming households for introduction of saline tolerant seeds and seed saving techniques in cooperation with Pakistan Agricultural Research Council Conduct soil analysis within 2 Union Councils to identify appropriate certified saline tolerant seeds Distribute certified saline tolerant seeds and required inputs to 400 households and conduct an Environmental Management Plan (EMP) to identify/mitigate any potential adverse environmental impact Integrate local Agricultural Department and 3-5 Farmer Organizations in improved agricultural techniques trainings and on-farm natural resource management (e.g. seed saving techniques and mesquite harvesting) Distribute vouchers for community seed saving containers to 20 farming communities
Result 2: Reinforce agricultural wholesale markets Establish 6 wholesale markets within targeted zones within 3 years Mobilize 3-5 Farmer Organizations for cooperative marketing of agricultural products. Mobilize 4-6 wholesale buyers to participate in wholesale market. Transport agricultural products to major transportation lines 6 times
Page 42 of 58
ACF World Bank Programme Final Evaluation
Result 3: Renew access to boreholes turned saline Select 400 interested households that live near a saline borehole. Rehabilitate all non-functioning boreholes located near the selected households. Select/design the most appropriate low-cost desalination unit for each household. Distribute the selected units to each household. Train households in operation and maintenance of the units. Setup/Train a water committee for each borehole. Conduct hygiene education sessions for all households Identify local businesses to manufacture/sell these units.
Result 4: Increase knowledge and analysis in the livelihood sector for Thatta district coastal areas Conduct livelihood surveys at the beginning and the end of the project Presentation of the results to the main actors involved in the area Dissemination of the reports
3. AIM OF THE EVALUATION
3.1. Target User(s) of the Evaluation
ACF FSL Technical Advisors, ELA Unit Implementing HQ Desk Officer, FSL Technical Advisor Field Level Country Director, Deputy Country Director, MEAL Manager, FSL Program Coordinator, FSL Program Manager Other GoP, PARC, WB, local communities, other stakeholders operating in coastal Sindh area,
3.2. Overall Objective of the Evaluation
This 4-year development program sees a change from ACFs other more emergency- orientated programs. The evaluation should assess the programme systems in place and evaluate whether they were conducive to achieving high impact, and identify the key lessons learned, detailing the process, structural and/or technical issues that led to major strengths and weaknesses. This evaluation is an important process in identifying ACF’s strengthens and weaknesses, to take forward for future such programmes (for example the upcoming EU grant).
3.3. Specific Objectives of the Evaluation
Assess the efficiency, effectiveness and appropriateness of the project systems in place in order to achieve the given objects. Examine how these systems adapted to the challenges brought on by the floods in 2010 and 2011. Draw out examples of how ACF did and did not successfully adapt to these
Page 43 of 58
ACF World Bank Programme Final Evaluation
shocks, and how we might better prepare for future such shocks. Assess the impact and sustainability (including resilience of households and communities, community participation and exit-strategy) of the program. Assess how the program responded to the mid-term evaluation conducted in February- March 2012. Provide concrete recommendations on how to capitalize on experience during the programme (e.g. replication or scale-up).
3.4. Scope of the Evaluation
Questions relating to efficiency, effectiveness and appropriateness of the project systems + adaption of the program to shocks. Was the internal coordination and communication efficient? Did the program reach the intended results and objectives as presented in the LFA (or subsequent amendments)? If not, what are the factors and constraints for the risk of non-achievement? What hampered the achievement of the targets? Did ACF sufficiently anticipate the constraints? Were there adequate systems in place to monitor the performance of the program and if so did this data have a positive impact on program performance? Did ACF respond and adapt the program to ensure the achievement of the program results and objectives in light of the 2010 and 2011 floods? Was information on the changes sufficiently passed onto the communities and authorities? How did ACF deal with these shocks and could ACF have better responded?
Questions relating to impact contribution and sustainability of the program. What evidence (or indications) are there that the action has contributed to achieving the project objective? Did the authorities and communities understand, accept and welcome ACF program? Was there appropriate community participation throughout the program to ensure continuation of activities after? Were the activities adequately coordinated with other development and local actors, in particular the Pakistan Agricultural Research Council (PARC) and the Pakistan Nuclear Institute of Agriculture and Biology (NIAB)? What were the main constraints faced with regard to coherence and coordination, how were they managed and how could ACF have improved on their management? Were the activities adequately integrated between ACF WaSH and FSL departments at each of the following stages of the program: Assessment of needs Project proposal Targeting of communities and beneficiaries Implementation of activities Monitoring Has ACF sufficiently planned and executed an exit strategy that would facilitate the continuation of programme activities? Do national development actors (local NGOs, local authorities etc.) have the information to duplicate the project to other areas that face similar challenges based on ACF’s experience in Keti Bunder and Garho?
Page 44 of 58
ACF World Bank Programme Final Evaluation
Questions related to Component B
Marketing Activities Is it likely that there will be collective market efforts from the farmers after the duration of the project? Has a market process been established and been reinforced for three years thereby creating a habit for selling future harvests? Have target communities/groups have been able to establish backward and forward market linkages to their advantage?
Technical Capacity Do farmers have the technical capabilities to identify appropriate seeds and manage agricultural activities; such as fertilizer and pesticide applications? Do farmers have the technical capabilities to save seeds in order to lower annual expenditures on agricultural inputs? Do farmers have the capacity to better manage natural resources on the farm (such as capitalizing on mesquite trees and maintaining drainage channels) Do communities and local authorities have a guideline on livelihoods profiles and programming recommendations for reference and advocate tools?
Learning from mid-term evaluation How did ACF respond to recommendations made in the mid-term evaluation? Address each recommendation. What systems were in place to facilitate incorporation of recommendations?
3.4. Evaluation Criteria
ACF subscribes to the Development Assistance Committee (DAC) criteria for evaluation: Impact, Sustainability, Coherence, Coverage, Relevance / Appropriateness, Effectiveness and Efficiency. ACF also promotes systematic analysis of the monitoring system and cross cutting issues (gender, HIV/AIDS etc). Whereas not all external evaluations are expected to use DAC criteria to structure the report nor should they necessarily cover all of the criteria, but where appropriate the DAC standards should be used in data analysis and reporting. In all cases, the evaluator will be expected to use the following table to rank the performance of the overall intervention using the DAC criteria. The table should be included as an Annex to the report.
Criteria Rating Rationale (1 low, 5 high) 1 2 3 4 5 Impact Sustainability Coherence Coverage Relevance/Appropriatenes s Effectiveness Efficiency
Page 45 of 58
ACF World Bank Programme Final Evaluation
3.5. Best Practices
The evaluation is expected to provide one (1) key example of Best Practice from the project/programme. This example should relate to the technical area of intervention, either in terms of processes or systems, and should be potentially applicable to other contexts where ACF operates. This example of Best Practice should be presented as an Annex to the report.
3.6. Evaluation Outputs
The result of this evaluation should be presented in a written report and through several oral presentations: One on the mission (to Head of Mission and relevant technical staff) One at HQ (in person or via teleconference).
3.7. Methodology
3.7.1. Preparation
Review of project documents (proposal, logical framework, donor reports, activity and monitoring reports, assessment reports, capitalisation documents, budget follow-up, etc.)
3.7.2. Briefing
Prior to the evaluation taking place, the evaluator is expected to attend a briefing at HQ level, and at field level with the Head of Mission and/or the relevant technical focal point. Briefings by telephone must be agreed in advance.
3.7.3. Field activities
Consultants are expected to collect an appropriate range of data. This includes (but not limited to): Direct information: Primary data collection using standard and participatory evaluation methods (e.g. semi structured interviews, FGD and observation) in households and communities. These will involve key informant interviews and FGDs in the field, and also interviews with staff in both Thatta and Islamabad. Other project stakeholders including, but not limited to, local authorities, World Bank/JDSF staff, other INGOs and local NGOs should also be consulted. Indirect information: Interviews with local representatives; interviews with project staff expatriate and national staff; meeting with local and provincial authorities (NIAB, PARC, Department of Agriculture), groups of beneficiaries, humanitarian agencies, donor representatives and other stakeholders. For indirect data collection, standard and participatory evaluation methods are expected to be used (HH interviews and FGDs with beneficiaries, non-beneficiaries, key informants – health workers, teachers and leaders). Secondary information analysis: including analysis of project monitoring data or of any other relevant statistical data related to saline soils agriculture in the Indus River Delta.
3.7.4. Report
Page 46 of 58
ACF World Bank Programme Final Evaluation
The report shall follow the following format. Cover Page Table of Contents Executive Summary: must be a standalone summary, describing the programme, main findings of the evaluation, and conclusions and recommendations. This will be no more than 2 pages in length. Main Body: The main body of the report shall elaborate the points listed in the Executive Summary. It will include references to the methodology used for the evaluation and the context of the action. In particular, for each key conclusion there should be a corresponding recommendation. Recommendations should be as realistic, operational and pragmatic as possible; that is, they should take careful account of the circumstances currently prevailing in the context of the action, and of the resources available to implement it both locally and in the commissioning HQ. Annexes: Listed and correctly numbered. Format for the main body of the report is: Background Information Methodology Findings & Discussions Conclusions Recommendations Annex I (Best Practice) Annex II (DAC-based Rating Table)
The report should be submitted in the language specified in the ToR. The report should not be longer than 30 pages excluding annexes. The draft report should be submitted no later than 10 calendar days after departure from the field. The final report will be submitted no later than the end date of the consultancy contract. Annexes to the report will be accepted in the working language of the country and programme subject to the evaluation.
3.7.5. Debriefing & Learning Workshop
The evaluator should facilitate a learning workshop: To present the draft report and the findings of the evaluation to the ACF Pakistan Team and other stakeholders. To gather feedback on the findings and build consensus on recommendations. To develop action-oriented workshop statements on lessons learned and proposed improvements for the future.
3.7.6. Debriefing with ACF HQ
The evaluator should provide a debriefing with ACF HQ in New York on her/his draft report, and on the main findings, conclusions and recommendations of the evaluation. Relevant comments should be incorporated in the final report.
6. RESOURCES AND DOCUMENTATION The following documentation will be made available: Project proposal Budget Quarterly reports
Page 47 of 58
ACF World Bank Programme Final Evaluation
Baseline Livelihood Survey Report Baseline Conventional Farmer Practices Monitoring Reports Market Survey Report Marketing Strategy Any other relevant document SPHINX software for data analysis
4. PROFILE OF THE EVALUATOR Significant field experience in the evaluation of humanitarian / development projects for major donors. Relevant degree / equivalent experience related to the evaluation to be undertaken Significant experience in coordination, design, implementation, monitoring and evaluation of programmes Good communications skills and experience of workshop facilitation Ability to write clear and useful reports (may be required to produce examples of previous work) Fluent in English
Page 48 of 58
• • Understanding of donor requirements • Ability to manage the available time and resources and to work to tight deadlines • Independence from the parties involved
5. RIGHTS The ownership of the draft and final documentation belong to the agency and the funding donor exclusively. The document, or publication related to it, will not be shared with anybody except ACF before the delivery by ACF of the final document to the donor.
ACF is to be the main addressee of the evaluation and its results might impact on both operational and technical strategies. This being said, ACF is likely to share the results of the evaluation with the following groups: • Donor(s) • Governmental partners • Various co-ordination bodies
Intellectual Property Rights All documentation related to the Assignment (whether or not in the course of the evaluator’s duties) shall remain the sole and exclusive property of the Charity.
ACF World Bank Programme Final Evaluation
Annex C: Itinerary and List of Meetings (March 2013)
Monday 4th Tuesday 5th Wednesday 6th Thursday 7th Friday 8th Saturday 9th Sunday 10th UK UK Islamabad Islamabad Islamabad Thatta Town office + GH Preparation Preparation Travel Preparation Finalisation of Finalisation of evaluaiton evaluaiton Meeting with FSL Meet with Erin methodologie and methodologie and PM Alabdeen (CD) tools tools Meet with Zach (WasHCo) 11am 3pm Adan HoB Definition of work TMK - Security plan and meetings Briefing
Speak to Muriel (HQ FSL Advisor) Speak to charmaine (HQ Desk Officer) Meet Duwi - AdminCo 4pm
Monday 11th Tuesday 12th Wednesday 13th Thursday 14th Friday 15th Saturday 16th Sunday 17th Thatta Town Thatta Town Thatta Town Thatta Town Thatta Town Thatta Town Thatta Town meet Waheed (DPM- Depart Thatta Depart Thatta 10 am leave ISL marketing component) 0800 0800
12pm arrive meet Mohsin Khan (Tech Arr Yaqoob Arrive Ayoob Depart Thatta Karachi Research Sup) Memon 1100 Gujrio 1000 0800 Data analysis Data analysis
50
ACF World Bank Programme Final Evaluation
Arr Ali Bux Dep. Yaqoob Depart Ayoob Murghor (Co-op) 2pm Thatta Memom 1300 Gujrio 1230 1000 meet Waheed Arr. Harshim (DPM- marketing Sholani (Co-op) Arrive Suleman Dep. Ali Bux component) 1400 Mallah 1315 Murghor 1230 meet Moseen Arr Yaqoob Khan (Tech Dep. Harshim Depart Suleman Murghor (Co-op) Research Sup) Sholani 1600 Mallah 1600 1000 Arr. Thatta Base Dep. Yaqoob 1800 Arrive Thatta 1800 Murghor 1230 Arr. Thatta Base 1700
Monday 18th Tuesday 19th Wednesday 20th Thursday 21st Friday 22nd Saturday 23rd Sunday 24th Thatta- Karachi - Thatta Town Thatta Town Thatta Islamabad Islamabad Islamabad Islamabad Meet with Thatta-Hydrabad Alabdeen FSL PM Data anaylsis and Data anaylsis and (2hrs) 1000-1400 report writing report writing meet SAWFCO, Gahro, (Noor Mohammed Bozdar + Ayaz Charchar) 1000 meet Deputy Meet District Co- Director of meet WWF, Gahro, operative Officer Agricultural (Mohummad Abbasi), 1100 (Thatta) - Allah Marketting, Fazal meet NRSP, Meet with Faisal Junio Mirjat 1000- Junejo, 1100 Thatta, (Aza Ali (Deputry Admin) 1100 Kaka) TBC 1600
Meet Yameen Memon + Abdul Qadir Arbab meet Deputy (conduted mid- Commissioner term evalaution + (Thatta) - Meet Shazad some trainings) Arr. Shambhu Khan Murghor Mohammed Nawaz (PQA Manafer) 1100-1300 1300 TBC 1630
51
ACF World Bank Programme Final Evaluation
meet NRSP, Meet Amir Thatta, (Kahlid Dep. Shambhu Khan Hydrabad- Thatta (Deputy FSL Jawar) 1500 Murghor 1430 (2hrs) Coord) (1700)
meet Khuda Bukhsh Beharani Dept. of Social Welfare, (meet at Speak to Paola house) Thatta (Former FSL 1700 Coord) 1800 Arr. Ali Shah 1500 leave Thatta 1200 (skype)
Leave Karachi Meet Chris Dep. Ali Shah 1630 1600 (Deputy CD) 2000
Monday 25th Tuesday 26th Wednesday 27th Thursday 28th Friday 29th Saturday 30th Sunday 31st Islamabad Islamabad Islamabad UK UK UK UK
Data analysis and Data analysis and report Depart ISL at report writing writing 0405 Meet with Mike (LogCo) 1200 Meet with World Bank 0830
Presentation/Debriefing with Arrive London ACF ISL Team 1600-1730 1225
Interviews/FGDs per village
Yaquoob Memon – FGD with 10 farmers (5x land owners (less than 8 acres) 5x sharecroppers) Yaquoob Memon – SSI village leader and lead farmer (both land owners) Haji Hashim Sholani – FGD 6 x landowners (less than 8 acres) Haji Hashim Sholani – SSI with lead farmer Ayoob Gujiro – FGD with 10 farmers (6 of them co-op members) Suleman Mullah – FGD 7xfarmers including lead farmer 52
ACF World Bank Programme Final Evaluation
Suleman Mullah – SSI with lead farmer Ali Bux Murghor – FGD with co-op General secretary, lead farmer, co-op president and 1 other farmer. Ali Bux Mughor – SSI with Shafi Muhammad Mughor (chairman of co-operative) Yaqoob Murghor – FGD with lead farmer, general secretary and one other farmer Yaqoob Murghor – SSI with lead farmer Sayed Aetbar Ali Shah – FGD with lead farmer and 22 other farmers Shambu Khan Murghor – FGD with 5x farmers and lead farmer
53
ACF World Bank Programme Final Evaluation
Annex D: Bibliography
General Programme Documents • Japanese Social Development Fund – Grant Proposal • Socioeconomic Study and Proposal for Livelihood Improvements: Badin and Thatta Districts, April 25, 2005 • Pakistan Floods – Report on Impact to Ongoing Activities, December 2010 • ACF Pakistan External Evaluation of the programme “Reducing vulnerability with DRM in KPK” • ACF PK 2013-2015 Country Strategy draft 13012013 v4 • ACF PKI1A - JSDF Budget Realignment - Cover Letter 14SEP2011 • ACF PKI1A - JSDF Budget Realignment - Financial Data 14SEP2011 • Annex 5- ACF PK DRR Assessment Report_KPK_SINDH_FINAL • WB Beneficiary Database • PKI1A Flood Impact Report_final_ NTC • WB - Budget modification approval • jsdf proposal and comments • Logical Framework Analysis • ACF Pakistan Sindh Mid-Term Evaluation Final Report
FSL Documents • ACF _Pakistan_JSDF_Endline_Final_Report -Yameen _ Alab+ Aamir 17 March • Monthly and Quaterly Monitoring Reports • Business Plans of three cooperatives-Prepared during the External Training for Cooperatives & Leader Farmers • Cooperative By-laws-cooperative Ali Bux Murghor • ACF External -Market Training Report April 2011 • Draft for the manual Post Harvest Management • KAP Survey internal evaluation Report-final • Training Workshop manual on Cooperative Marketing • Training Workshop manual on Leadership • Training Training Workshop manual on Record Keeping-REVISED • Paddy Marketing Activity Report 2012_final • Wheat Marketing Activity Report 2012 Final • Wholesale Market Strategy Paper +comments+Alab • Marketing Manual on formation of Marketing Cooperatives in Thatta • METHODOLOGY FOR SOIL TESTING, DEMO,TRAINING last draft • Marketing Survey: A Baseline Survey of Agriculture Marketing System in Keti Bander & Karochan Talukas of District Thatta, Sindh, Pakistan • MoU between Agricultural Extension Department and ACF (File name: MoU-FSLWB- 1) • ACF AHH_Pakistan_JSDF_Endline_Draft_report_MA_JF • AHH Pakistan - FSL Baseline Assessment Thatta District, Pakistan final OCT 2009 • APR 201201 -FSL- JSDF January 2012 final indicator list xls • APR 201207-FSL-JSDF February 2013 Alab
54
ACF World Bank Programme Final Evaluation
• FSL Workshop-Presentation FSL WB Thatta January 2013 • Paddy Cultivation Research Report Ali + Alab • Paddy Research report-2012 • Strategy for paddy research (2011)- final • Strategy for paddy research (2012) final draft • Annexure 4-List of trainings organized for farmer organizations & farmers • Bio Pesticide Neem Spray Training Manual • List of Trainings • VEGETABLE SEED PRODUCTION TECHNOLOGY • FSL Exit Strategy WB Program+A+PV+FSL team to be validated
WaSH Docments • ACF PAK solar desal research report rev3 • Nivarna Survery Report • Monthly and Quarterly Reports • WASH Thatta Survey August 09 - Results - 09-11-23 • WASH world bank PDM impact assessment report v1 • Renew access to boreholes turned saline: Solar desalination research report, March 2012 • WASH Thatta Survey August 09
55
ACF World Bank Programme Final Evaluation
Annex E: Best Practice
Title of Best Practice Identification of lead farmers in each village as a focal point for communication and trainings.
Innovative Features & Key The system of lead farmers was a strong point of the Characteristics training component. Lead farmers were identified in each village as the focal point for trainings. They were identified on the basis of their willingness and also their ability to effectively transfer messages to other farmers in the village. ACF used these focal points to organise training sessions within the villages and also acted as the main liaison with other village lead farmers. The community mobilisation of this programme was a strong element, and the system of lead farmers was a main factor in facilitating this.
Practical/Specific Document the system of identification, the Recommendations for Roll Out characteristics of the individual identified, and also what systems/guidelines are in place for the lead farmer.
56
ACF World Bank Programme Final Evaluation
Annex F: DAC Ratings Table
Criteria Rating Rationale (1 low, 5 high) 1 2 3 4 5 Impact X It could be concluded that the activities around improved agricultural techniques and the provision of SDUs contributed to good impact as they addressed well assessed and evidence based problems. However the marketing activities had little to no impact as they were not based on a sound assessments and a coherent strategy. Sustainability X Sustainability was high in terms of the co- operatives and the trainings, as ACF made concerted efforts to ensure that knowledge remained in the community and co-operative activities (in terms of knowledge sharing) continued by producing and disseminating guidelines and training manuals. The sustainability of the SDUs was weaker as the cost of them remained to high for beneficiaries to purchase additional units. The outstanding deficiency in terms of accountability was ACFs ineffectiveness in involving local authorities in its activities. Coherence x The FSL and WaSH components were coherent in so far as they addressed the same problems but integration between the 2 was poor. The programme was coherent with ACF’s recent (2013-2015) strategy and ACF’s international strategy. Coverage x Coverage exceeded the programme objective but the targeting of the 23 villages should be questioned as they were not necessarily the most needy in the area. Relevance/Appropriatenes X The programme was appropriate to the needs s of the farmers, and addressed a WB needs assessment well. Salination is a serious problem for farmers, and the programme did well to address that challenge. Effectiveness X Ultimately the programme was on the whole effective as the WaSH results and the training results were met. The marketing component was not effective as it was not based on suitable assessments and the strategy was
57
ACF World Bank Programme Final Evaluation
therefore misguided, resulting in the programme falling short in setting up markets. Efficiency x The programme was marred by inefficiencies, in terms of budget issues and HR turnover. Ultimately most programme objectives were met, but only in the closing months of the programme. The programme budget was reportedly mostly spent, but difficulties in making budget adjustments with WB, and procurement challenges, negatively impact efficiency.
58