Nalitabari Area Development Programme

Evaluation Terms of Reference

Nalitabari Area Development Programme Programme number: BGD-11 - 00459 Phase (Summative: Interim) Evaluation: 1st Phase (FY 2007 to FY 2011)

January 2011

World Vision Bangladesh

Published: Nalitabari ADP, January 2011©World Vision Bangladesh

Phase Evaluation TOR of Nalitabari ADP: FY’11 Table of Contents Table of Contents...... i Acknowledgement...... ii Affirmation...... iii Glossary...... iv Introduction...... v 1. Evaluation Summary...... 1 2. Description of Programme and Projects Being Evaluated...... 1 2.1. Impact indicators for program goal, project goal and outcomes...... 2 2.2 Major Interventions under different projects...... 4 Health...... 4 Education...... 4 Economic...... 5 Sponsorship Management...... 5 Disaster Management Project...... 5 2.3. Union wise target population...... 6 2.4. Project wise target beneficiaries...... 6 3. Evaluation Target Audiences...... 6 3.1 Direct stakeholders:...... 7 3.2 Indirect stakeholders:...... 7 4. Evaluation Type...... 7 5. Evaluation Purpose and Objectives...... 7 5.1. Purpose of the Phase Evaluation...... 7 5.2 Specific objectives of the phase evaluation...... 8 6. Evaluation Methodology...... 10 7. Limitations...... 11 8. Authorities and Responsibility...... 11 8.1 Team Members and Roles...... 11 8.2. Manning Schedule...... 12 8.3. Tasks of consultant/consulting firm...... 12 8.4. Responsibility of WVB...... 13 8.5. Responsibility of Community...... 13 8.6. Profiles of the lead consultant...... 14 8.7. Phase Evaluation plan...... 14 8.8. Activity Schedule...... 14 8.9. Development of Phase Evaluation Proposal for ADP (Submission of Proposal)...... 15 8.10. Evaluation of Consulting Firm...... 15 8.11. Work location:...... 15 8.12. Reporting language...... 16 8. 13. Tentative time frame for the Phase Evaluation...... 16 8.14. Confidentiality and Copyright...... 16 9. Evaluation Team...... 16 10. Logistics...... 17 11. Products...... 17 12. Budget...... 18 13. Documents of review...... 18 14. Lessons Learned...... 19 15. Roots of communication...... 19

Phase Evaluation TOR of Nalitabari ADP: FY’11 i Acknowledgement

It is a pleasure to produce Terms of Reference (TOR) for Nalitabari Area Development Program (ADP) Phase evaluation with full support of Nalitabari ADP team and in light of LEAP. This document will help effectively conduct this evaluation, with involvement of the ADP team, WV Bangladesh Regional and National staff, the different stakeholders of the ADP, World Vision UK and the consulting firm who will be appointed to conduct the Phase Evaluation of Nalitabari ADP.

We want to extend our thanks to Nalitabari community members including leaders, partners, government officials, schoolteachers and the children who actively participated in implementing planned interventions in the last five years. We believe that the life of the target community has been positively impacted with the input of the mentioned stakeholders.

We are expressing our heartfelt thanks to those ADP colleagues who provided their assistance and advise to prepare these TOR. We are grateful to Mr. Nazrul Islam, DME Coordinator of Bogra Region, for his timely and appropriate guidelines, coordination and intellectual inputs to draft such documents. Special thanks go to members of World Vision Bangladesh Quality Ministry Division (QMD), including WV Bangladesh National DME Coordinators, for giving suggestions and extending the necessary cooperation to prepare these TOR.

Our cordial thanks to Mr. Pintu Areng, Associate Director of Bogra Region, for his kind, passionate commitment to complete the task in time, and help by ensuring effective leadership on the road to effective ADP management. We would also like to acknowledge the contribution of Mr. Theophil Hajong, Director of the Field Operations Division, who is concerned about program evaluation quality and guided us to design effective TOR. We would like to pronounce his name with honour for his dynamic leadership and caring that has contributed to implement the ADP in the target community in a peaceful manner.

We are also grateful to WVUK and its Program Officers for their inputs, directives and continued funding that helps us to implement activities without interruption.

Finally, thanks to GOD who has allowed Nalitabari ADP to implemented its planned interventions successfully since the inception of its current phase in FY 2007 and given us a good opportunity to serve the poorest and the most vulnerable people of Nalitabari community.

Buli Hagidok, ADP Manager Nalitabari ADP Date: 5 January 2011

Phase Evaluation TOR of Nalitabari ADP: FY’11 ii Affirmation

These TOR have been developed by Nalitabari ADP team on the basis of information from different stakeholders of Nalitabari Upazila. Relevant data & information that are used in the TOR are viewed as reliable and authentic.

The primary quantitative and qualitative data collected throughout the monitoring and reporting process remain the property of the communities and families described in these TOR. Information and data must be used only with their consent.

Buli Hagidok ADP Manager Nalitabari ADP

Phase Evaluation TOR of Nalitabari ADP: FY’11 iii Glossary ADP Area Development Program AIDS Acquired Immune Deficiency Syndrome CBO Community Based Organization CRS Customer Relations Services DG Development Group DGM Development Group Member DIP Detailed Implementation Plan DME Design Monitoring and Evaluation FGD Focus Group Discussion FY Fiscal Year

GAD Gender and Development GO Government Organization GOB Government of Bangladesh HH Household HIV Human Immune Deficiency Virus KII Key Informant Interview LEAP Learning through Evaluation with Accountability and Planning M&E Monitoring & Evaluation

NGO Non-Government Organization NO National Office PDD Program Design Document QMD Quality Ministry Division RDD Re-Design Document RO Regional Office (former Zonal Office) SMC School Management Committee SO Support Office SPSS Statistical Package for Social Science STI Sexually Transmitted Infection SWOT Strength Weaknesses Opportunities and Threats TDI Transformational Development Indicator TIN Tax Identification Number TOR Terms of Reference VAT Value Added Tax WATSAN Water and Sanitation WVB World Vision Bangladesh WVUK World Vision UK

Phase Evaluation TOR of Nalitabari ADP: FY’11 iv Introduction

Nalitabari Area Development Program (ADP) started in FY1999 as a non LEAP-aligned ADP covering five unions and one municipality. The foreseen lifetime of Nalitabari ADP spans from 1999 to 2013, including the first two years as Seed project. World Vision UK is acting as support office and provides financial & technical support to Nalitabari ADP.

Nalitabari ADP is in Nalitabari Sadar Upazila, 190 kilometres north east of Dhaka, the capital of Bangladesh, and at 5 hours by car from WV Bangladesh National Office in Dhaka. Nalitabari Upazila consists of 1 Municipality comprising 9 wards and 12 Unions, which in turn comprise 138 villages. The total land area of the Upazila is 327.56 sg. km . Nalitabari ADP operates in 5 unions (partially) and 1 municipality comprising 9 wards (partially) under Nalitabari Upazila of Sherpur district.

According to the new Design, Monitoring and Evaluation scheme foreseen by LEAP, in 2006 Nalitabari ADP completed its (Re)-Design Document (P/RDD) for five years (FY 2007-2011), with five projects included: Health, Education, Economic Development, Sponsorship Management & Disaster management. As per LEAP guidelines, a Baseline Survey was scheduled in the 2nd year of the current phase i.e. FY 2008 and was conducted accordingly. The baseline survey was mainly indicator- based and a key aim was to identify benchmarks against each approved indicator. After receiving approval of the P/RDD from WVUK and WVB management, Nalitabari ADP started implementing the new planned interventions for the well being of the target community. Quantitative and qualitative progress, mostly at activity and output level, are being measured on a monthly, semi annually and annual basis.

Phase Evaluation TOR of Nalitabari ADP: FY’11 v 1. Evaluation Summary

Programme Nalitabari ADP Programme Phase FY 2007 to FY-2011 Evaluation Type Phase (Summative: Interim) Evaluation Evaluation Purposes  To assess the extent to which program and project goals and outcomes have been achieved based on the current log frame, design and monitoring data.  Determine the relevance, effectiveness, efficiency, impact, sustainability and accountability of the program  Identify lessons learned, best practices and areas for improvement (translated in actionable recommendations) for the next ADP redesign and implementation phases

Primary Methodologies Indicator based data will be collected through a participatory process involving community people and all level stakeholders. Relevant tools will be HH survey, FGD, document review, KII, SWOT analysis, review of case studies/success stories. Both quantitative and qualitative methods will be used. Evaluation Start and End Start – March 2011 Dates End - June 2011 Anticipated Evaluation June 2011 Report Release Date

2. Description of Programme and Projects Being Evaluated

A description of the local context of the program area has been provided in the program design document and there have been no major changes ever since. In the light of LEAP, Nalitabari ADP prepared its PDD for its first implementation phase with a budget amount of US$ 23,21,105.

Project wise phase budget (FY-07 to FY-11)

Health Project Education Economic Sponsorship Disaster Total Budget FY Budget (US$) Project Dev. Project Project Manageme (US$) Remark Budget (US$) Budget (US$) Budget (US$) nt Project Budget (US$) FY-07 1,46,766.00 1,99,039.00 87,691.00 1,11,626.00 40,729.00 585,851.00 Including cash for work budget FY-08 1,23,597.00 1,44,933.00 85,222.00 1,24,986.00 32,361.00 5,11099.00 FY-09 1,25,000.00 1,45,000.00 79,000.00 1,30480.00 27,000.00 5,06,480.00 FY-10 92,500.00 1,00,000.00 50,000.00 1,27,675.00 22,500.00 3,92,675.00 FY-11 70,000.00 1,00,000.00 30,000.00 1,07,000.00 18,000.00 3,25000.00 Total 5,57,863.00 6,88,972.00 3,31,913.00 6,01,767.00 1,40,590.00 23,21,105.00

This phase evaluation will provide information about the relevance, efficiency, effectiveness, sustainability, impact and accountability of the program by objectively verifying the progress of projects and program interventions. It will assess progress against each indicator set at the beginning of this phase. This type of evaluation will help decision makers and programmers to review the programme design based on formulated recommendations and identified lessons learnt. It will also create an opportunity to improve and fill gaps. Positive lessons learned enable project management to

Phase Evaluation TOR of Nalitabari ADP: FY’11 1 replicate successes and promising practices and negative lessons learned help avoid making the same mistakes in future. A phase evaluation also provides public accountability and transparency to all levels of stakeholders. Nalitabari ADP management has decided to conduct a phase evaluation for all its projects and program within June 2011.

The program and project goal and outcome level indicators will be considered during this phase evaluation in addition to program sustainability and transition considerations (not included here). The project wise Goal, Outcome, Output and major interventions are indicated below:

2.1. Impact indicators for program goal, project goal and outcomes

Goal/outcome statement Measuring indicators Program goal: “Empower community capacities to sustain transformational development process leading to improved well-being of children, family and communities through health, education, and economic development” Health Project Project goal: Sustained  Reduced incidence of childhood illnesses community good health  Reduction of severe malnutrition  People adopting hygienic practices  Community adults (aged 15 – 49) are aware of HIV/AIDS infection  People receive health care services

Outcome 1.1 Good health care  Children mortality rates reduced practiced in the community  Incidence of six diseases reduced

Outcome 1.2 Incidence of  Reduce severe malnourished children aged 0 to 59 malnutrition among children and months from present rate to 15% pregnant and lactating mothers  Reduce severe malnourished pregnant mothers from reduced present rate to 10%  Reduce severe malnourished lactating mothers from present rate to 10%

Outcome 1.3 Reduced incidence  Diarrhea diseases incidence reduction from present of water borne diseases and arsenic rate to 5% poisoning in the community  Arsenic poisoning incidence reduced from present rate to 5% Outcome 1.4 Increased awareness  Increased HIV/AIDS protection knowledge from on HIV/AIDS and STI existing to 70%  Reduced percentage of STI patient from existing (To be informed within short time and PDD to be updated)

Outcome 1.5 Project Quality  Health programme effectively functioning Assurance enhanced Education Project Project Goal: Improve  Literacy rate of children increased from present rate to educational status of the target 70% community  Adult Literacy rate increased from present to 50%

Outcome 2.1 Improved quality of  Grade completion rate of secondary children of age primary, secondary and higher 12- 18 years upgraded from present to 50% education  5% Drop out rate of girls at primary level  5% Drop out rate of boys at primary level

Phase Evaluation TOR of Nalitabari ADP: FY’11 2 Goal/outcome statement Measuring indicators Out Come: 2.2. Enhance DGMs  80% DGMs can read, write and do simple arithmetic literacy rate Outcome 2.3 Enhance moral and  People’s care for others increased from Low to ethical values of target community Medium as per TDI standard

Outcome 2.4 Gender equity  Dowry decreased from existing to 20% established in the community  Early marriage decreased from existing to 20%  Increased woman participation in family decision making from 13.3% to 50%  Reduction in violence against women Outcome 2.5 Project quality  Education Program effectively functioning assurance enhanced Economic Development Project Project Goal: Enhance livelihood  60% HH able to spent on treatment by register doctor security and empowered target  90% of HH get three time full meals in a day communities purchasing capacity  76 % HH per capita income increased (from 1700 tk increased to 3000 tk BS-2001)

Outcome 3.1:Increase crop &  (5% to -6%) Crop Yield increased from existing 3 ton fruits production to 3.15 ton Rice 3%, Vegetable 2%, Fruits 1%. (DAE report)  Milk production increased from existing to 10% Outcome 3.2 Increase livestock  Poultry birds production increased from existing to production 10%  Egg production increased from existing to 10% Outcome 3.3:Enhance fish culture  Fish production increased from existing to 20 %  HH increased in cultivation of pond fish from existing to 15% Outcome 3.4 Sound environment  Increased dustbin used for waste disposal from and forest resources enhance existing to 10%  HH planted trees increased from existing to 10% Outcome: 3.5. Increase household  300 self-employment created. income from non-farm sources Outcome: 3.6. Establish marketing  Mobility to market increased existing to 20% facilities Outcome: 3.7. DGs effectively  90% DGs effectively functioning in the community functioning in the community Outcome 3.8 Project Quality  Economic program effectively functioning assurance enhanced Sponsorship Management Project Project Goal: Quality sponsorship  100% parents and community people satisfied with program fully operationalized with sponsorship program established relationship between  100% parents cooperate to have photography of sponsors and children, their children for various reports families and communities  All sponsors satisfied with the responses made by CRS staff (This indicator is not evaluation purposes but correspondence with sponsors effectively by CRS staff) Outcome 4.1. Reduced dropout  100% parents cooperate in selection and registration rate in selection and registration of of their children as sponsored per CRS standard sponsored children as per CRS  100% surveyed families followed up as per CRS standard standard

Phase Evaluation TOR of Nalitabari ADP: FY’11 3 Goal/outcome statement Measuring indicators Outcome 4.2 Effective CRS  Zero pending of CRS correspondence management implemented and  All staff involved in CRS job successfully maintained compliant with the CRS sponsor relation services standard international audit Outcome 4.3 Project Quality  Management policies and practices in place on time Assurance enhanced and demand Disaster Management Project Project Goal: Improved  80 % of target household able to cope with the hazard household resilience of disaster Outcome 5.1 Increased  90% of Household adopted at least one coping community based capacity on mechanism disaster preparedness and  95% HHs aware of disaster early warning messages management Outcome 5.2 Enhanced quick  100 % of affected targeted HH returned to normal life response to the affected communities Outcome 5.3 Project Quality  Disaster programme effectively functioning Assurance enhanced

2.2 Major Interventions under different projects

The P/RDD of Nalitabari ADP foresees the following major interventions:

Health  Conduct training for community people on WATSAN and water borne diseases.  Install safe drinking water and sanitation facilities in the target communities  Conduct training for parents and adolescent girls on nutrition, Primary Health Care (PHC)  Implement supplemental nutritional services program for mother and children  PD Hearth Program functioning  Established nutrition feeding center  Ante-natal care (ANC) and post-natal care (PNC) implementation  Arrange training on immunization  Arrange training on diarrhea management  Implement networking and collaboration with GO and NGO  Provide support to keep Expanded Program on Immunization (EPI) centre well functioning  Arrange health education for community adolescent boys and girls  Ensure curative health care service facilities  Treatment for community disabled  Assist with materials for disabled  Nutrition fair organized  Arrange training program on HIV/AIDS and STI for target group  Organize campaign program on HIV/AIDS and STI

Education  Educational support (tuition fees, exam fees, admission fees, books, stationery etc.) to student and institutes  Support educational institutions  Ensure coaching program for selected students  Teachers are trained on teaching skills  School Management Committee (SMC) members are oriented on school management  Implement Non-formal education program for child laborers

Phase Evaluation TOR of Nalitabari ADP: FY’11 4  Implement Functional Educational (Adult Education) course  Ensure post literacy and continuing education for functional education (FE) graduates  Moral development training  Establish net-working among GO and NGOs  Training/workshop on Gender and Development (GAD) and child rights implemented  Workshop/gathering on moral values implemented for children and church members for transformed communities  Vocational training for drop-out students/ unemployed youth  Sports & cultural program for students

Economic  Develop Non-farm skills of target population  Improve leadership capacity of DG/CBO  Arrange training for farmers on improved agricultural production practices  Establish demonstrations on improved crop production practices  Provide livestock training  Livestock demonstration established  Provide livestock input  Provide fish culture training  Established fish demonstration  Provide fish input  Tree plantation and forestation  Training on forestation  Provide agricultural production inputs (improved seeds, fertilizers, mini irrigation equipment, livestock, fingerlings, etc.) for farmers  Enhance collaboration / network between GO/ NGO  Leadership and management training  Arrange DG fair and agriculture fair

Sponsorship Management  Orientation and training imparted to parents, community leaders, volunteers and project staff on child recruitment and management of sponsored children  Produced Customer Relations Services (CRS) recommended reports regularly as per schedule  Community based CRS management system established for sensitizing and involving community people with their active participation in managing sponsorship activities  Ensure effective child record management and transformational engagement establishing CRS administration and management system  Form Child forums and keep them functioning  Training on child leadership  Training on child protection  Training on child trafficking  Arrange cultural program for RC and non RC  Arrange debate competition

Disaster Management Project  Arrange training for community people on disaster preparedness  Establish early warning system  Arrange training on forestation  Display risk and resource maps  Distribute relief goods among those affected by wild elephants

Phase Evaluation TOR of Nalitabari ADP: FY’11 5  Campaign and simulation on disaster management  Ensure tree plantation  Networking with GO/NGO and other relevant agencies  Response with relief and rehabilitation inputs

2.3. Union wise target population

SL Name of # of # of HHs Targeted Total Targeted # Unions/Ward Villages HHs Population Population 1 Nalitabari 17 3322 2550 14722 12750 2 Kakorkandi 14 3934 3243 15635 15342 3 Rupnarayankura 10 3711 3320 14632 16600 4 Poragaon 13 4038 3450 16940 17250 5 Ramchandrakura 14 3888 3550 16248 17750 6 Nalitabari 9 ward 6347 2228 29012 15642 Pourshavea

Note for 2.3 & 2.4: As per PDD, our ADP has 86 target/working villages, though, union wise, 68 villages in total can be counted. There are additional 503 HHs located in 18 adjacent villages which are out of our working unions. Before the project LEAP design, many registered children (RCs) were selected from there: so the 18 villages were also included as targeted area, with about 2517 target people involved as RC, RC families and assisted students. Gradually, Nalitabari ADP staff reduced the number of registered children there (e.g. 18 to 22 aged registered children) and all projects interventions in those 18 villages were conducted adopting a low profile. As a result, right now, a very minimum number of RCs and beneficiaries from those villages are included as target beneficiaries.

2.4. Project wise target beneficiaries Year wise RC (FY-07 to FY-11)

FY Male Female Total FY-2007 1328 1220 2548 FY-2008 1252 1219 2471 FY-2009 1122 1130 2252 FY-2010 949 1205 2154 FY-2011 937 1260 2197

Project Name Target Villages & Wards Beneficiaries (HH) Health Project 86 village and 9 wards 62509 Education Project 86 village and 9 wards 35646 Economic Development Project 68 village and 9 wards 36020 Sponsorship Management Project 86 village and 9 wards 17560 Disaster Management Project 68 village and 9 wards 15550

3. Evaluation Target Audiences There are several direct and indirect evaluation stakeholders/partners/program participants. Direct stakeholders are the persons who are directly involved in program activities. Indirect partners include family members of direct stakeholders and local community people in the target areas.

Phase Evaluation TOR of Nalitabari ADP: FY’11 6 3.1 Direct stakeholders: The direct target groups selected for the interventions of Health, Education, Economic Development, Disaster management and Sponsorship Projects include:  Family members of Development Groups (male, female, adolescents and children)  Family members of Non-DG poor households (male, female, adolescents and children)  Sponsored children and family members (Boys, Girls, Parents, adolescents and children)  Poor non sponsored children and family members (Boys, Girls, Parents, adolescents)  Local educational institutions, Local Government institutions, CBOs and Registered forums  Community leaders/members (School Teachers/SMC members, Volunteers, etc.)-(Male, female)  Disaster Management Committee members  Members of some committees like WATSAN, SMC etc.

3.2 Indirect stakeholders:  Family members of DGs/ CBOs and RCs  Social / Religious & community leaders  GOB representatives  NGO representatives

Both direct and indirect stakeholders will be involved and asked questions about program impact in the framework of the evaluation process. The ADP and all relevant stakeholders will use the report to assess the progress and achievements of the program/projects, to identify promising practices for future implementation, areas of improvement. This may lead to inclusion of new outputs and/or outcomes in the next design document.

4. Evaluation Type

This will be the first phase evaluation of Nalitabari ADP after LEAP alignment. The phase evaluation will be made both at project and program levels. It will be conducted to review achievements and deviations of each planned interventions of each project. The evaluation will mainly focus on assessing progress made towards achievement of results at outcome and goal levels. Based on the results and findings of the evaluation, the ADP and WVUK will take steps for further redesign of the program to address detected problems and ensure child well-being through sustainable transformational development. The ADP will then enter a 2nd phase after LEAP alignment.

The ADP progress has been regularly reported to WVUK and WVB management following LEAP guidelines. But behavioral changes, practical impact and other long term effects need to be measured to assess the sustainability of implemented interventions. This will also encourage donors, WVUK, target communities, ADP staff and WVB management to continue development efforts for the target community.

The phase evaluation should provide evidence-based information that is credible, reliable and useful, enabling the timely incorporation of findings, recommendations and lessons learned into decision- making processes towards further replication in program/projects.

5. Evaluation Purpose and Objectives

5.1. Purpose of the Phase Evaluation

The purposes of the Phase Evaluation are:

Phase Evaluation TOR of Nalitabari ADP: FY’11 7 1. Evaluate to which extent output, outcome and goal level project and program indicators have been achieved, assessing this in quantitative and qualitative terms, over time and comparing the ADP achievements with data at national and district level.

2. When evaluating impact, pay special attention to change of awareness, knowledge, attitude and practices within target communities.

3. Identify lessons learned and challenges (with possible solutions) and formulate actionable recommendations, which can be utilized to make World Vision work in the program area more effective.

5.2 Specific objectives of the phase evaluation

The ADP evaluation will focus on 6 evaluation criteria: relevance, efficiency, effectiveness, sustainability, impact and accountability. The following issues will be assessed:

5.2.1 Relevance (program and projects significance with respect to specific felt needs of the communities): - To what extent do/did the program/projects respond to priority issues of the community? - How significant to the community situation and worthwhile are/were the program/projects? Should other activities be added ? - Should some activities be re-designed ? - To which extent is ADP work aligned with current Government Plans and priorities ? - Has World Vision Bangladesh taken account of changes in needs and revised the program to meet these ? 5.2.2 Effectiveness (assessment of how far the intended results were achieved in relation to output/outcome/goal targets set in the original logical framework; and how assumptions have affected the projects achievements): - To what extent were output, outcome and goal indicators achieved ? - To which extent did the ADP work specifically contribute to the achievement of targets ? What were other contributing factors ? - Quality of services provided - Were there any changes compared to design ? Why? - Looking at project implementation, assess their level of integration (in the light of World Vision’s Integrated Focus for Programming) and if/how this could be strengthened - Identification of key strengths (such as successful innovations or new good practices) and weaknesses of the program. How can weaknesses be overcome? - Have the ADP teams done the right things in the right ways?

- 5.2.3 Impact (assessing the repercussions of the interventions in the medium and long-term at community level: - What have been the effects of the project /program interventions on target people and the community?To which extent are changes in the community specifically attributable to the program/projects and what are other contributing factors? - - What contributions has the ADP made to the improvement of the socio-economic status and to boosting income earning opportunities for the target population? - Extent to which the program had a positive impact on the more disadvantaged groups (e.g. women, differently-able, Ethnic communities….)

Phase Evaluation TOR of Nalitabari ADP: FY’11 8 - What was the unintended (positive or adverse) impact of the program ? - Which factors have hindered positive impact ? How can they be overcome ? Stakeholder should disaggregate impact statements by the consulting firm. 5.2.4 Efficiency of planning and implementation (in term of cost, speed and quality with which inputs/resources/means have been converted into desired products/outputs): - Was financial spending in line with plan? - Have project outputs been achieved at reasonable cost? - Which approaches would lead to the same or increased results in a more efficient way ? - Was there sufficient funding for each project ? - Were the Project Management & ADP management structure and its staffing appropriate in relation to the tasks executed? - Assess the quality of the working relationships with stakeholders/partners

5.2.5 Potential for sustainability, replication and multiplying effects (in terms of likelihood of continuation of the stream of benefits produced by the projects after their end): - Will the program/projects contribute to lasting benefits? - To what extent will the project impact contribute to WV’s transformational development? - To what extent are community members leading the development process ? What is required to improve this ? - If World Vision was to stop funding now, what would happen? - What support is needed to ensure that benefits are sustained? - What aspects of the program could effectively be replicated? What benefits would this achieve? The different aspects of sustainability (institutional, financial, socio-cultural, political…) should be taken into account. 5.2.6 Accountability to beneficiaries - Are communities informed of World Vision activities and plans, including future ones? - Are there mechanisms in place for community members to provide their feedback or make a complaint? - What is the degree of beneficiary participation in the program, including in decision – making? - Were all interventions culturally appropriate?

5.2.7 Additional specific issues of questions: In addition to the above, the evaluation team will specifically assess the following issues also: 5.2.7.1. Cross-cutting themes: the evaluation will assess to which extent the following 5 cross cutting themes have been addressed in actual implementation: disability, environment, gender, peace building and conflict-resolution and protection (including child protection). Examples of issues to address: - To what extent do men and women have equal access to program benefits? - To what extent do differently able people have access to program benefits? - Have program activities had any harmful effect on the environment? - Have program activities ensured that children are not be put at risk? - Have program activities promoted understanding between different groups, or created tension? 5.2.7.2 Lessons learned: the evaluation should also highlight key lessons learnt through the period of the program and indicate how they can be used to strengthen ADP implementation in the future.

Phase Evaluation TOR of Nalitabari ADP: FY’11 9 6. Evaluation Methodology

The evaluation will be participatory with a focus on learning, success and action; it will be led by an external evaluation consultant/consulting firm. This evaluation should use both quantitative and qualitative methods for data/information collection. The ADP staff will provide background information to design the survey sample. To conduct the evaluation, baseline survey techniques need to be followed or any other appropriate method, which is more relevant and compliant with the method, used for the ADP baseline. In the past, World Vision used two-stage cluster sampling design for the evaluation of various ADPs due to its principle of simplicity, low cost and ease of operation. Besides, the well known Demographic and Health Survey in Bangladesh and elsewhere to draw a nationally representative sample, and by UNICEF follow a similar methodology for its multiple indicator cluster survey. The target communities will be made aware of the evaluation’s objectives in order to facilitate their maximum participation. The voice of target communities, including women, children, will be strongly listened to and reflected upon. Attention will be paid to the viewpoints of community minorities and vulnerable groups (e.g. elderly, differently able, Aborigenous groups…). As a part of organizational policy and LEAP guidelines, World Vision Support Offices, collaborative partner CBOs/NGOs, local govt. representatives, other stakeholders and the community need to be involved in the phase evaluation to ensure active participation of all partners throughout the process. The phase evaluation team members will visit the ADP locations and conduct several qualitative working sessions with stakeholders as a part of the phase evaluation. Evaluation will be undertaken for each of projects/interventions following the goals and outcomes set for them. Some of the methods for evaluation will done by selected consulting farm which can meet our evaluation purpose

Document review (Secondary data): This refers to gathering and analysis of pre-evaluation information (Design Document, annual work plans, annual reports, half-yearly reports, other records, baseline survey report, Annual monitoring report etc). The review should be limited to the critical information that the evaluation team needs to fulfill the evaluation tasks.

Onsite visual inspection: For an experienced observer, this is an excellent way to become familiar with program locations and realities. This method does not stand alone, and other assessment methodologies should be used in conjunction. Onsite visual tasks include: - To observe people’s physical condition and activities; ask questions. - To visit homes or shelters, water sources, schools, compartment offices. - To observe the daily lives of HH (use women as interviewers) and disadvantaged groups. - To make sketches, take photographs, or use videos, photos, video footage, and even sketches as communication tools to convey the reality of the situations.

Key informant interview: This technique can include interviews with individuals or groups. Loosely structured interviews will be conducted with informed stakeholders in GoB agencies, NGOs, CBOs, local leaders, representatives from community groups, project facilitators and local media sources. The focus should be on obtaining factual information that is crosschecked with other sources.

FGD or qualitative methods of investigation: The focus group discussions will be conducted with community members, including influential members, to measure programme/project impact and prospects of sustainability of services rendered through short and long-term projects and interventions. FGDs should involve 8-12 carefully chosen participants grouped to discuss one or more issues concerning them all. The facilitator will use a discussion guide, a record keeper who will record comments and observations.

Questionnaires and Surveys: This method allows a surveyor to gain information from a large number of people in a structured way according to specific questions, often in ways that allow statistical analysis. These can range from being very simple to complex, which makes them useful in both demographic/development/empowerment/relief and rehabilitation contexts. They can

Phase Evaluation TOR of Nalitabari ADP: FY’11 10 include structured sets of closed questions (yes/no or multiple choice questions) or they can include open-ended questions such as semi-structured interviews. Fixed-choice or fixed response questionnaires are good for gathering data that needs to be analyzed statistically, while open- ended or free-response questions can be particularly good for determining people’s feelings and attitude.

Sampling: Sampling allows surveyors to make a generalization about an entire population on the basis of characteristics of a sub-set or sample of the population. Attributes of a population are estimated through interviews or surveys with a representative section or sample of the population.

Case study: This method is meant to give a snapshot of a particular family /individual /institution’s situation over a period of time and can be used for assessment, baseline monitoring and evaluation purposes. It documents the life story or sequence of events over time related to a person, location, HH or organization in order to obtain insight into people’s impact and to learn about people’s experience, dreams, and understanding of the context and human factors behind summarized data collected through other means. The team should agree beforehand as to how the subject will be selected. Questionnaires, semi-structured interview questions, and other relevant methodologies will need to be employed.

SWOT Analysis (Strength, Weaknesses, Opportunities and Threats): This method is used to determine strengths, weaknesses, opportunities & threats in relation to a project or group, how such a situation can change over time. This method is very useful when qualitatively assessing, for example, the services provided by the project, the relationship between stakeholders and the organizations of implementing partners, groups and the project team itself.

7. Limitations

In the context of Nalitabari ADP working area, budget foreseen for the evaluation, the socio- economic conditions of the target community, including local government, the following limitations may need to be faced during the evaluation:  In the time of cultivation & harvesting many people will be absent from home.  Budget may not be allow to select quality consulting firm  Migration in another place in low profile  Languages spoken by ethnic groups in few villages

8. Authorities and Responsibility

8.1 Team Members and Roles

The ADP management wishes to have the Phase Evaluation conducted by a renowned consulting firm having experience of similar type of studies. The Consulting firm should lead the Phase Evaluation involving staff from World Vision Bangladesh, target community and local implementing partners (GOB agencies, CBOs/NGOs). If required, the firm can hire extra personnel primarily in consultation with the concerned firm and nurturing dialogue with the ADP management for quality control, data analysis and other purposes mentioned in the proposal. The consultant chosen should ensure presence of women staff and adequate gender expertise to capture the findings of female respondents.

The Phase Evaluation should have two teams: the core team and the field enumeration team. Both teams will be formed by the consulting firm. The core team will be formed by the Lead consultant and subject based consultant. The overall process will be conducted by the core team and members of the core team will be physically present in the field during data collection. WVB Regional DME Coordinator, National DME Coordinator, Associate Director and ADP team will be members of the core team and will assist the consulting firm to complete the assignment in time and produce a more

Phase Evaluation TOR of Nalitabari ADP: FY’11 11 informative report. The field team will comprise at least 20 enumerators, supervisors and trainers. Both teams will work as per advice of WVB and guidance of the consulting firm.

The field enumeration team will be responsible for data collection from the field using various tools. The firm will select the field enumerators from Nalitabari ADP community in consultation with the ADP Manager/Team. The consulting firm will also hire supervisors who will supervise the enumerators’ work. They will be trained, including on survey data collection. Survey related arrangements will be attended to by the consulting firm and related costs will be borne by it. No extra payment will be made to the firm except the contract amount.

8.2. Manning Schedule

The consultant/consulting firm will develop and submit a manning schedule as a part of survey proposal. The manning schedule should describe the estimated duration of personnel deployment for the whole evaluation period and the time-span for each of the specialists and support staffs including volunteers (if any). It should be presented in line with the estimated budget provision and in the form shown below: MANNING SCHEDULE Sl# Position/Name/Nos W1 W2 W3 W4 W5 W6 ------W14 W15 W16 1. Team Leader 2. Consultant 1/2/3 3. Position-4/5 4. 5. 6. 7. Volunteers/10 8. 9. 10. Positio-10/2

8.3. Tasks of consultant/consulting firm

1. Review the ADP Design Document (project logical framework, Detailed Implementation Plan- DIP, Monitoring & Evaluation Plan, Performance Indicator Tracking Table and other relevant documents) 2. Ensure participation of both WVB technical and programming staff members mentioned above in this evaluation process. 3. Develop survey methodology and sampling procedures required for the survey in response to ADP demands and in consultation with ADP management, Regional DME Coordinator and National DME Coordinator/Member of Quality Ministry Division (QMD), WVB. 4. Develop questionnaires and other tools incorporating appropriate questions for information items and include proxy indicators so that complex outcome indicators (e.g. socio-economic indicators) can be derived from them. Ensure having indicators disaggregated by gender, age and ethnicity and where appropriate also by religion, poverty status and other vulnerabilities, e.g. widows, orphans or people with disabilities etc. 5. Translation of survey tools in local language i.e. Bengali where necessary. 6. Conduct pre-testing of questionnaire in ADP working area and update the same with appropriate questions. 7. Finalize the sample size in consultation with ADP management and technical persons in the Regional and National Office of WVB. 8. Form a team for Phase Evaluation involving ADP staff, different stakeholders at community level and Regional & National Office of WVB.

Phase Evaluation TOR of Nalitabari ADP: FY’11 12 9. Organize a comprehensive training session to orient the team members on tools and methodology. A practical session can be included for data collection for the enumerators for collection of quality data and minimizing ambiguities of questions. Maximize quality of data to be collected by training enumerators and supervisors, and developing/complying with procedures of data collection from the field. Detailed guidelines will be developed to make questionnaires understandable. 10. Develop guidelines for quantitative and qualitative survey data collection. 11. Collect data using finalized tools from planned primary and secondary sources. 12. Develop data entry, error-checking modules and data analysis program in MS Access or standard format. 13. Store data into database using data entry program developed. 14. Design dummy tables and share with Regional DME Coordinator/ADP before finalization, and generate summary formats for report writing. 15. Use both quantitative data analysis using statistical techniques (SPSS and EPI) and qualitative data analysis in calculating progress at outcome and goal level indicators. 16. Write report using error free data, survey observations, findings and analysis obtained from other analytical computer packages. 17. Present evaluation findings and send draft report for comments to ADP, Regional Office and National Office along with raw data (SPSS, EPI) for further checking. 18. Arrange report sharing sessions and present finding to ADP staffs, Community, and WV regional/National Office staffs. Incorporate review feedback and present revised draft report to ADP including community and National office. 19. Finalize report after incorporating feedback. 20. Submit the final report (a set of five copies) to ADP management for final approval accompanied by a soft copy of report and all primary tables and databases based on which the report has been produced. 21. Submit final data (MS Access, SPSS, EPI) to ADP within 10 days just after submission of final report. 22. Submit all collected questionnaires and other relevant documents, which have been collected from ADP/RO/NO, WVB to ADP within 60 days after submission of final report.

8.4. Responsibility of WVB 1. Choose consulting firm 2. Provide relevant documents 3. Give information about the ADP working area, community, beneficiary etc. 4. Follow up phase evaluation process to collect quality data as per indicators 5. Review and give feedback on tools and draft reports 6. Coordinate with support office, partners & other stakeholders and incorporate their feedback 7. Ensure payments based on agreed terms

8.5. Responsibility of Community 1. Assist the survey team in providing information 2. Assist in the preparation of field work 3. Assist in resolving conflict/problems (if any) 4. Assist in identifying HH/recognize area 5. Respond appropriately 6. Extend cooperation in data collection 7. Attend meetings/seminars/workshops/sharing sessions 8. Cooperation extended without pay (volunteer services) 9. Provide input on what the findings mean to them 10. Help develop practical recommendations

Phase Evaluation TOR of Nalitabari ADP: FY’11 13 8.6. Profiles of the lead consultant

The Lead consultant should preferably have-  An advanced university degree (PhD preferred) in development studies/social science/public management/Statistics/other relevant disciplines, with proven experience in the development context of Bangladesh. Experience in working with mixed culture i.e. Bangali, Ethnic communities will be an added value - Design and management of evaluation processes, including multiple stakeholders - Sound experience in evaluation of program/projects, including complex ones - Experience in combining qualitative and quantitative evaluation methods - Previous experience in leading at least 5 evaluations/surveys as team leader - Experience in interview design and implementation - Qualitative and quantitative data collection and analysis - Project/program/policy planning, monitoring and management - Sustainability issues of development programs - Experience in evaluation of WVB programs or other World Vision Programs/Projects - Understanding of World Vision Area Development Programme (ADP) development philosophy

Personal skills - Strong analytical report writing (English) skills - Leadership, team work and conflict resolution skills - Capability to bring together diverse stakeholders - Report writing in both Bengali and English language

8.7. Phase Evaluation plan

The Phase Evaluation proposal should be accompanied by a detailed Phase Evaluation plan following the format below or better ones. The plan must be based on focus area (goal and outcome of projects/ program) and assumptions as per logical frameworks.

Phase Evaluation Plan

Phase OVI or line Survey Data Location of Means of Time needed Evaluation focus of inquire tools source & Data analysis quantity collection Goal: As per Design Document Outcome 1: Do Outcome 2: Do Outcome 3: Do Outcome 4: Do

8.8. Activity Schedule

Detail activity schedule is expected in the proposal using the following matrix.

Sl# Activity W W2 W3 W4 W5 W6 … …. …. W10 W11 W12 Type 1 .. . . 1 Activity –1 2 Activity –2

Phase Evaluation TOR of Nalitabari ADP: FY’11 14 3 Activity 3 4 5 6 7 8 9 10 Activity 10

8.9. Development of Phase Evaluation Proposal for ADP (Submission of Proposal)

The consultant/consulting firm should produce a proposal for the Phase Evaluation based on these TOR and submit the same to the Nalitabari ADP Office or Bogra Regional Office of World Vision Bangladesh (where asked) within stipulated timeframe. The evaluation proposal should contain two separate envelopes: a. Technical proposal and b. Financial proposal  Technical proposal that should demonstrate the consultant’s knowledge and understanding of the World Vision ADP development approach, general and detailed methodology that the firm is proposing for the phase evaluation, methods and procedures of data collection as deemed relevant for the survey and certification of the consultants and key personnel in favour of the firm.  The financial proposal should specify: - Head-wise cost-estimate; - Salary/honorarium of professional/experts and other support staff including social costs (VAT, tax etc); - Cost of data collection with alternate scenarios-(a) cost of mobilizing survey enumerators by WVB and (b) cost mobilizing by the consulting firm; - Cost of stationeries and report production of draft and final in 5(five) copies.

8.10. Evaluation of Consulting Firm

Among the submitted proposals only the technical proposals will be opened first at the time of evaluation. The financial proposals will remain sealed until technical proposals are evaluated. The technical quality of the proposals will be evaluated on the basis of two major score points:

 Understanding the survey work i.e. (a) understanding the objectives, (b) quality of methodology, (c) innovativeness, (d) work programme, (e) comments on TOR and (f) presentation of proposal.  Quality of firm, its logistics and human resources i.e. (a) experience of firm and similar and different types of studies conducted for national and international agencies, (b) experience of the Evaluation Consultant, (c) technical supporting staffs and logistics facilities, (d) experience of the firm/consultant with World Vision Bangladesh/any child focused organization like Save the Children/Plan International/etc. The firm should provide relevant and appropriate evidence in support of their experience/deeds and information provided for qualifying themselves.

8.11. Work location:

Nalitabari ADP command areas are situated in Nalitabari Upazila, Sherpur District, Bangladesh.

Phase Evaluation TOR of Nalitabari ADP: FY’11 15 8.12. Reporting language: English/ the consulting firm should translate in Bengali during sharing with community people.

8. 13. Tentative time frame for the Phase Evaluation

The whole process of Phase Evaluation should be completed within a period of 90 days (including holidays). An extended period (max. 15 days) may be allowed on justification of extension placed by the consultant/consulting firm. The timeframe will be validated from date of contract signed for the work.

8.14. Confidentiality and Copyright

All papers shared with the consulting firm are confidential to World Vision Bangladesh and should not be used outside of World Vision Bangladesh without permission. Information received by the Consulting firm from Nalitabari ADP and World Vision Bangladesh and ADP working area should be treated as confidential. The Phase Evaluation report will be owned by Nalitabari ADP, World Vision Bangladesh and disseminated to authorities as the organization sees fit.

9. Evaluation Team

An advisory team will work to make the phase evaluation a success. The roles of responsibility of advisory body are indicated below:

Sl.no Description of Advisory Responsibilities Committee 01 Team Leader - Prepare Evaluation Plan in collaboration with ADP, (Outside WVB/Chief National Office and Evaluation Team consultant of evaluation) - Review documents - Form Core and advisory team and conduct meetings with them - Finalize appropriate sample size and methodology - Develop evaluation tools - Ensure appropriate data collection and analysis process - Contribute to solving problems (if they arise) - Present findings to different level of stakeholders - Write the evaluation report (detailed in section 8.3) 02 Subject based consultant - Review documents (Outside WVB) - Develop and finalize evaluation tools - Pre-test of questionnaire - Ensure training for data collector - Supervise data collection - Data analysis and report writing (detailed in section 8.3) 03 Regional DME Coordinator, - Ensure timely feedback on evaluation tools WVB - Supervise data collection - Ensure feedback on draft tabulation plan - Ensure feedback on draft report - Ensure relevant documents - Maintain close liaison with Firm /consultant /NO /ADP - Attend meetings/sessions/workshops

Phase Evaluation TOR of Nalitabari ADP: FY’11 16 Sl.no Description of Advisory Responsibilities Committee 04 Director Quality Ministry - Ensure timely feedback on tools Division/National Design, - Ensure feedback on draft report Monitoring & Evaluation - Attend meetings/dialogue/workshops Coordinator from World Vision Bangladesh 05 National Sponsorship - Ensure timely feedback on tools Coordinator/Regional CIM - Ensure feedback on draft report Coordinator - Attend meetings/dialogue/workshops 06 Director Field Operations - Ensure timely feedback on any queries/ensure Division /Regional appropriate working condition Associate Director, WVB - Ensure feedback on draft report - Maintain close liaison with Firm /consultant /NO /ADP /SO - Attend meetings/dialogue/workshops 07 Respective ADP Manager, - Ensure relevant documents WVB - Prepare field visits - Supervise data collection - Contribute to solving problems (if any arise) - Contribute to presenting findings to different level of stakeholders 08 Program Officer, Support - Provide info to Evaluation TOR Country - WVUK and WVB will validate the evaluation TOR - Ensure feedback on draft report - Provide technical advice as per ADP/WVB request - Advice from support office will be honored and incorporated in the phase evaluation as and when emerging during the evaluation.

10. Logistics

All relevant logistics (accommodation, transportation, etc.) will be managed by the consulting firm. In case of WV staff, logistical arrangements will be borne by World Vision. Necessary services such as translators, interpreters, drivers (for firm/consultant), data processor, facilitators, computer, printer etc. associated with the phase evaluation will be managed by the consultant/consulting firm. The ADP may allow the survey firm/consultant to arrange sharing/discussion sessions in an ADP hall room/regional/national office conference room based on availability.

11. Products

Though the product of this evaluation process will be the “Final Report on Phase Evaluation of Nalitabari ADP”, there will be some other products like the database for the quantitative survey, FGD records and findings and the primary and secondary tables. These products will be valuable records and evidence for any programs to be planned in the near future. Specifically, the evaluation products should be presented or preserved in the form of: - The final report (both in hard & soft format) - The database of quantitative survey (MS Access, SPSS and EPI) - The consolidated records and findings of FGD as qualitative data - Summary primary and secondary tables/matrices used in report (both in hard and soft formats) - The presentations and summary of recommendations, conclusions and lessons learned used during sharing and debriefing with stakeholders.

Phase Evaluation TOR of Nalitabari ADP: FY’11 17 - Evidences and records of debriefings and feedback obtained as outcome and used for incorporation and updating the final report.

Report format The Programme Evaluation Report will follow the below format:  Executive Summary  Acronyms  Introduction  Programme/Project Background  Evaluation objectives and methodology  Observations and findings including tables/graph  Evaluation of Main Program Components (taking into consideration issues to address mentioned in these TOR) - Output, outcome and goal achievement for each project and the program - Relevance - Effectiveness - Efficiency - Impact - Sustainability - Accountability - Cross cutting themes - Lessons learned  Programming opportunities  Limitations  Conclusions and Recommendations  Others if any  Annexes including table presentation of all data collected and summary of qualitative data by project

Paper and font Size: A4 size paper and Times new Roman 12 size are required.

12. Budget

The proposed budget will be based on outcomes of the evaluation offers by ADP management and purchase committee, as per allocation of fund set-aside in the budget document. Required budget for phase evaluation is available in FY 2011 plan. Consulting firm should have VAT registration number and TIN number. VAT and Tax will be deducted as per Government rules.

Mode of Payment

All payment should be in account pay cheque and vat/tax should be deducted at source. The consulting firm will have the opportunity to take 30 percent advance after signing contact, 30 percent after submission of draft report and draft summary finding sharing, and another 40 percent to be given after World Vision receiving the Final Report with necessary documents.

13. Documents of review

The following documents might be useful for review during the various phases of phase evaluation process: - ADP Design document - Semi-annual and Annual program Management Reports - Annual Monitoring Reports - Baseline Survey report

Phase Evaluation TOR of Nalitabari ADP: FY’11 18 - TDI report - WVB policy documents - Project audit report - M&E monthly reports and Annual work plans

14. Lessons Learned

 At the end of the evaluation process, WVB and WVUK will specifically discuss how the evaluation findings will be integrated into ADP redesign and proceed with this task.

 World Vision Bangladesh will organize a dissemination/lessons learnt workshop in Nalitabari or another location to disseminate the final findings/recommendations of the evaluation report to relevant staff and foster reflection around those.

 Also, the evaluation findings will be communicated to project beneficiaries and community representatives in a way that respects their dignity and security. These will be informed of evaluation results and will be actively involved in ADP redesign,

15. Roots of communication

In all respects, the ADP Manager should be the contact person for the survey work. The consultant/consulting firm will have the opportunity to contact the Regional Associate Director in case of emergencies.

Contact Person Buli Hagidok Pintu Areng ADP Manager Associate Director, Bogra Region, WVB Nalitabari ADP Rai Bahadur Street (Near Noor Mosjid) Araiani Bazar, Joleswaritala, Bogra Nalitabari, Sherpur Phone(+88-051) 69641; Mobile: (+88) 01730331787 Mobile: (+88) 01714045711 Email: [email protected]

Phase Evaluation TOR of Nalitabari ADP: FY’11 19