M&E Manual Guide for Ifad Funded Projects in Vietnam

M&E Manual Guide for IFAD Funded Projects in Vietnam

Managing for Impact in Rural Development

M&E Manual Guide for IFAD Funded Projects in Vietnam

1 M&E Manual Guide for IFAD Funded Projects in Vietnam Table of Contents

LIST OF TABLES...... 7 LIST OF FIGURES...... 7 LIST OF BOXES...... 8 ABBREVIATION...... 9

Chapter I. How to set up M&E system 1. Basic M&E concepts...... 15 1.1. M&E Terms...... 15 1.2. Purpose and objectives of IFAD funded projects’ M&E...... 16 1.3. Result based management...... 17 1.4. Participatory M&E...... 18 1.5. IFAD M&E Methodology...... 19 2. M&E system...... 20 2.1. What is M&E system...... 20 2.2. Setting up the M&E system...... 23 3. M&E planning...... 26 3.1. The structure of the monitoring plan...... 26 3.2. The structure of the evaluation plan...... 27 3.3. M&E and project logical framework...... 29 3.4. Detailing M&E plan based on project appraisal report...... 31 3.5. Using the M&E Matrix for detailed planning...... 32 3.6. Indicative contents for an M&E Plan...... 33 4. Data collection methods...... 36 4.1. Quantitative and qualitative methods...... 36 4.2. Collecting secondary data and information...... 37 4.3. Survey...... 38 4.4. Key informant interview...... 39 4.5. Focus group discussion...... 39 4.6. Observation method...... 39 4.7. Annual supervision mission...... 39 5. Gender and M&E...... 39 5.1. The importance of gender in M&E...... 39 5.2. What will a gender sensitive Monitoring system monitor?...... 40 5.3. How do we do gender and M&E...... 41 6. M&E implementation responsibilities...... 41

3 M&E Manual Guide for IFAD Funded Projects in Vietnam

6.1. Senior M&E Officer in the PCU...... 41 6.2. Knowledge Management Officer...... 42 6.3. District level...... 42 6.4. Commune level...... 42

Chapter II. Measuring and reporting RIMS first and second level results 1. Overview...... 45 2. Why to measure and report the first and second level results?...... 46 3. Collecting information for measuring and reporting first and second level results...... 46 3.1. Selecting first and second level results...... 47 3.2. Measuring and reporting first level results...... 48 3.3. Measuring and reporting second level results...... 52

Chapter III. How to conduct annual outcome survey 1. Introduction...... 59 1.1. What is the annual outcome survey and why to do it?...... 59 1.2. How and when to conduct the survey?...... 59 2. How to prepare and conduct an annual outcome survey?...... 60 2.1. Step 1 – Fine-tuning the standard questionnaire...... 61 2.2. Step 2 - Selecting the sample...... 63 2.3. Step 3 – Enumerators’ training and field testing of the questionnaire...... 64 2.4. Step 4 – Logistical planning...... 68 2.5. Step 5 – Data collection...... 69 2.6. Step 6 – Data entry...... 70 2.7. Step 7 – Data analysis...... 71 2.8. Step 8 – Report writing...... 71

Chapter IV. How to conduct RIMS and RIMS+ baseline and end- line survey 1. What is the RIMS and RIMS+ survey and why to do it?...... 75 2. How and when to conduct the survey...... 76 3. How to prepare and conduct a RIMS and/or RIMS+ survey...... 76 3.1. Step 1 – Develop the questionnaire and interview guide...... 76 3.2. Sample selection...... 80 3.3. Step 3 – Enumerators’ training and field testing of the questionnaire...... 83 3.4. Step 4 – Logistical planning...... 87 3.5. Step 5 – Data collection...... 89 3.6. Step 6 – Data entry...... 90 3.7. Step 7 – Data analysis...... 91 3.8. Step 8 – Report writing...... 91

4 Table of Contents

Chapter V. Sampling methodology 1. Definitions...... 95 1.1. Population...... 95 1.2. Sample...... 95 1.3. Sampling frame...... 96 1.4. Sample size...... 96 1.5. Sampling method...... 96 2. Sampling method in projects funded by IFAD...... 98 2.1. The first stage: Select village...... 98 2.2. The second stage: Select household...... 100 3. Stand-by households...... 103 4. Respondent selection...... 103

Chapter VI. How to conduct key informant interview 1. Definition of key informant interview...... 107 2. Classification of key informant interview...... 107 2.1. Unstructured interview...... 107 2.2. Semi-structured interview...... 108 3. Steps conducting key informant interviews...... 108 3.1. Indentify interview topics...... 109 3.2. Identify key informants...... 109 3.3. Develop questions...... 109 3.4. Design key informant interview guide...... 110 3.5. Pre-test and revise the questions or interview guide...... 110 3.6. Conduct the interview...... 111 3.7. Analyse and write report...... 112

Chapter VII. How to conduct focus group discussion 1. Focus Group Discussion...... 115 2. Steps to facilitate a focus group...... 116 2.1. Identify topics and information needed...... 116 2.2. Specify information needs for each topic...... 116 2.3. Prepare the discussion guide...... 117 2.4. Select participants...... 118 2.5. Pre-test and revise the discussion guide...... 118 2.6. Conduct focus group discussion...... 119 2.7. Analyze information and write report...... 122

5 M&E Manual Guide for IFAD Funded Projects in Vietnam

Chapter VIII. How to entry data using CSPro 1. What is CSPro?...... 127 2. Data Entry application...... 127 3. Accessing Data that has been saved...... 129 4. Modifying data and adding new cases...... 130 5. Compare data tool...... 130 6. Merging data (Concatenate Data Tool)...... 134 7. Exporting Data Tool...... 137 8. Some notes on developing database using CSPro...... 140

Chapter IX. How to analysis data by SPSS 1. Opening an existing SPSS data set...... 143 2. Set up variable name and code variable...... 143 3. Data analysis...... 145 3.1. Frequency...... 145 3.2. Calculate the mean (Descriptive)...... 147 3.3. Crosstab (use for ordinal and nominal variables)...... 150 3.4. Compare the means of variables...... 152 3.5. Dealing with missing values...... 153 3.6. Dealing with outliners: Only applied to quantitative variables...... 153 3.7. Notes...... 157 Annexes...... 159

6 Table of Contents LIST OF TABLES

Table 1.1 Comparison of Monitoring and Evaluation ...... 15 Table 1.2 Result Chain...... 18 Table 1.3 Distinguish M &E and participatory M & E...... 19 Table 1. 4 Key M&E tasks...... 22 Table 1.5 Example of Critical Reflection Schedule for a project...... 25 Table 1.6 Questions to guide the detailed planning of M&E conditions and capacities...... 26 Table 1. 7 Example for determining the monitoring method of group discussion...... 27 Table 1.8 Illustration of the structure of an evaluation plan ...... 28 Table 1.9 Example of log frame ...... 30 Table 1.10 Task needed when detailing the M&E plan based on a project appraisal report...... 31 Table 1.11 Contents of the M&E matrix...... 32 Table 1.12 Indicative contents for an M&E operational Plan...... 34 Table 1.13 Examples of multi-purpose M&E methods...... 36 Table 1.14 Comparison of quantitative and qualitative methods...... 37 Table 2.1. Example of reporting first level...... 52 Table 2.2. Example of reporting first level...... 52 Table 2.3 Rating Scale for second-level results...... 52 Table 2.4. List of performance questions for each factor...... 53 Table 2.5 A framework of factors and guiding performance questions ...... 55 Table 4. 1 Suggestion of additional information/questions of RIMS+ questionnaire...... 78 Table 5. 1 The sampling frame of TNSP in Tuyen Quang ...... 79

LIST OF FIGURES Figure 1.1 The M&E system and how it links to the project strategy and operations...... 21 Figure 3.1 An overview of the Annual Outcome Survey design...... 60 Figure 5. 1 The use of control group ...... 96

7 M&E Manual Guide for IFAD Funded Projects in Vietnam LIST OF BOXES

Box 1.1 Steps to set up the M&E system...... 23 Box 1.2 The basic performance questions per level of the objective hierarchy...... 24 Box 1.3 Examples of performance questions...... 24 Box 1.4 Example for gathering information on food security changes...... 33 Box 2.1 Results Chain Terminology...... 46 Box 3. 1 Example for the necessary of Focus Group Discussion...... 60 Box 3.2 Some points should be kept in mind when designing additional questions...... 62 Box 3.3 Questions to be answered by pilot...... 62 Box 3.4 The duties of enumerators...... 65 Box 3.5 The duties of the enumerator team leader...... 65 Box 3.6 The duties of supervisors...... 66 Box 4.1 Requirements for a good questionnaire...... 77 Box 4.2 Some points should be kept when designing additional questions...... 77 Box 4.3. Questions to be answered by pilot...... 80 Box 4. 4 The duties of enumerators...... 83 Box 4.5 The duties of the enumerator team leader...... 84 Box 4.6 The duties of supervisors...... 84 Box 6. 1 Examples of topics for key informant interview...... 109 Box 6.2 Example questions for the topic “Community participation in local planning” ...... 110 Box 6.3. Tips to have a good key informant interview...... 112 Box 7. 1 Example of topics for focus group discussion...... 116 Box 7.2. Example for questions designed for Focus group discussion suggested by IFAD...... 117 Box 7.3. Hint identify participants for group discussions with IFAD projects ...... 118 Box 7. 4. Example of Focus group discussion introduction ...... 121 Box 7.5. Some suggestions to have a good focus group discussion...... 122

8 Table of Contents ABBREVIATION

AWPB Annual Work Plan and Budget CDB Commune Development Board CDF Community Development Fund CIG Common Interest Group CSPro Name of Data Entry Software DASU District Agricultural Support FG Focus Group FGD Focus Group Discussion GIS Geographical Information System GPS Global Positioning System IFAD International Fund for Agricultural Development KI Key Informant KII Key Informant Interview M&E Monitoring &Evaluation MTR Mid-Term Review NA No Available PCU Provincial Coordination Unit PMU Provincial Management Unit RBM Result Based Management RDS Respondent-driven sampling RIMS Results and impacts measuring system SEDP Socio-Economic Development Plan SI Sampling Interval SPSS Name of statistical analysis software SWOT Strength Weakness Opportunities and Threats TNSP Tam Nong Support Project TOR Term of Reference VDP Village Development Plan VDP Village Development Plan VND Vietnam Dong

9 M&E Manual Guide for IFAD Funded Projects in Vietnam

10 M&E Manual Guide for IFAD Funded Projects in Vietnam Introduction about the Manual Guide

has been working for and with the poorest people in Viet Nam, including ethnic minorities, IFAD small-scale farmers and households headed by women. At present, IFAD is working at ten provinces in Viet Nam (09 projects), and some other projects will be implemented soon. All IFAD-funded projects, no matter what their specific focus is (e.g. natural resources, micro-enterprises value chains, microfinance) are expected to bring positive changes in the beneficiaries’ lives in terms of food security, incomes and livelihoods. To address these objectives, the projects should be managed and implemented in a proper result-oriented way. .

In order to assess its progress against implementation plans (monitoring), and to assess the outcomes and impact it has achieved (evaluation), the project’s monitoring and evaluation (M&E) system is extremely important. The purpose of monitoring and evaluation is to obtain accurate information that will be used by decision-makers in the project to improve and ensure proper use of resources as well as to maximize the impact achieved from these resources. In 2003, IFAD adopted a Results and Impact Management System (RIMS), which is a comprehensive system to monitor and document the outputs, outcomes and impacts of IFAD-funded projects. In addition, the last two years have seen the quasi- experimental approach for rigorous impact evaluation being integrated into the IFAD RIMS (commonly referred to as RIMS+).

However, there is a fact that most of IFAD project management staff in Viet Nam have limited exposure to IFAD M&E system, including the RIMS and RIMS+ (while those have some experience are moving around). Therefore, building and improving capacity for IFAD project’s management staff is extremely important. The management staff needs to be equipped with knowledge and practices on project management and M&E, especially they have to familiar with the RIMS/RIMS+, in order to organize and manage resources so the project is completed within defined scope, quality, time and cost constraints.

Development and Policy Research Center (DEPOCEN) is assigned by IFAD Vietnam to prepare a Manu- al Guide for IFAD funded projects in Viet Nam, to equipped the IFAD partners in Vietnam with knowledge and practices on M&E.

Therefore, this Manual Guide is prepare mainly for the two major audiences:

n Project managers: the people responsible for managing the various aspects of project implementation. This includes the project director, managers of project components and the responsible managers of partner or contracting organizations who are implementing a specific element of the project;

n M&E staff: the staff of a project or implementing partners and contractors who have responsibilities for setting up and/or implementing M&E systems.

Besides above major audience, the Manual Guide is also useful for the consultants, who providing external assistance on project M&E and information management, and anyone in IFAD and cooperating institution staff, who is providing guidance, supervision or support to the project.

The ideas in this Manual Guide are not a mandatory M&E system with which all projects must comply. The Manual Guide describes what is considered-and has proven to be-good practice in project M&E. The development of this Manual Guide is mainly based on series of IFAD technical guidance/paper, including

11 M&E Manual Guide for IFAD Funded Projects in Vietnam

“Managing for Impact in Rural Development – A Guide for Project M&E”, “Result and Impact Management System – Practical Guidance for Impact Survey”, “Result and Impact Management System – First and Second Level Results Handbook”, Technical Guidelines No.1 to No.5 and other literatures about M&E in general. Structure of the Manual Guide

The Manual Guide has nine chapters. The first Chapter provides basic M&E concepts and guidance on how to set up M&E system, gender and M&E, and a brief about method to collect data for the purpose of M&E.

Chapter II to Chapter VII are about selected methods and tools to collect data for M&E, e.g. Measur- ing and Reporting first and second level of results (Chapter II), How to conduct annual outcome survey (Chapter III), How to conduct RIMS and RIMS+ baseline and end-line survey (Chapter IV), Sampling methodology (Chapter V), How to conduct key informant interview (Chapter VI), and How to conduct focus group discussion (Chapter VII).

Chapter VIII and Chapter IX provide guideline to use the two software CSPro and SPSS for data entry and analysis.

The Annexes provide number of template questionnaires and forms developed by IFAD and DEPOCEN to facilitate the M&E system of IFAD projects in Viet Nam.

12 How to set up M&E system

Chapter I. How to set up M&E system

13 M&E Manual Guide for IFAD Funded Projects in Vietnam

14 How to set up M&E system

Chapter I. How to set up M&E system

1. Basic M&E concepts

1.1. M&E Terms

Monitoring is a continuous management function that aims primarily at providing programme managers and key stakeholders with regular feedback and early indications of progress or lack thereof in the achievement of intended results. Monitoring tracks the actual performance against what was planned or expected according to pre-determined standards. It generally involves collecting and analysing data on programme processes and results, and recommending corrective measures.

Evaluation is a time-bound exercise that attempts to assess systematically and objectively the relevance, performance and success, or the lack thereof, of ongoing and completed programmes. Evaluation is undertaken selectively to (i) Answer specific questions to guide decision-makers and/or programme managers, and (ii) Provide information on whether underlying theories and assumptions used in programme development were valid, what worked and what did not work and why. Evaluation commonly aims to determine the relevance, validity of design, efficiency, effectiveness, impact and sustainability of a programme.

Table 1.1. Comparison of Monitoring and Evaluation

Monitoring Evaluation Periodic: at important milestones such as the mid-term of Continuous. programme implementation; at the end or a substantial period after programme conclusion. Keeps track; oversight; analyses and documents In-depth analysis; Compares planned with actual achievements. progress.

Focuses on inputs, activities, outputs, Focuses on outputs in relation to inputs; results in relation to implementation processes, continued relevance, cost; processes used to achieve results; overall relevance; likely results at outcome level. impact; and sustainability.

Answers what activities were implemented and Answers why and how results were achieved. Contributes to results achieved. building theories and models for change. Alerts managers to problems and provides options Provides managers with strategy and policy options. for corrective actions.

Internal and/or external analysis by programme managers, Self-assessment by programme managers, supervisors, community stakeholders, donors, and/or external supervisors, community stakeholders, and donors. evaluators.

The Relationship between Monitoring and Evaluation

Monitoring and evaluation are intimately related. Both are necessary management tools to inform decision-making and demonstrate accountability. Evaluation is not a substitute for monitoring nor is monitoring a substitute for evaluation. Both use the same steps however, they produce different kinds of information. Systematically generated monitoring data is essential for successful evaluations.

15 M&E Manual Guide for IFAD Funded Projects in Vietnam

1.2. Purpose and objectives of IFAD funded projects’ M&E

Purpose of M&E

M&E is an important management function to track progress and support the decision-making process. It aim to to meet the requirements of a number of donors, but also of partners and targeted communities that work directly with your organization, who benefit most from the communicating progress through the M&E system. Through a thorough review of work progress, your organization can design programs and activities more effectively and ensure greater benefits to the project target group.

For IFAD funded projects, all projects gather information to assess progress against implementation plans (monitoring), and all projects gather information to assess the outcomes and impact it has achieved (evaluation). Good projects collect accurate information and produce timely reports. Great projects collect accurate information, and not only produce good reports, but also use the information to make decisions that improve efficiencies and enhance impact. The purpose of monitoring and evaluation is to obtain accurate information that will be used by decision-makers in the project to improve and ensure proper use of resources as well as to maximize the results and benefits achieved from these resources.

Objectives of M&E

The M&E system in the IFAD funded projects are expected to achieve the following four objectives:

- Monitor and Manage Project Progress. Information will be obtained to measure the progress of the project against planned activities and production of outputs. This will also include assessing the quality of the activities and outputs toward achieving the desired outcomes. The information will be used by decision-makers to adjust plans by expanding activities that produce results at a reason- able cost, modifying activities that need to produce greater results or be implemented efficiently and discontinuing activities that are producing unsatisfactory results.

- - Assess Project Outcomes and Impact. Project M&E systems will obtain information to enable decision-makers to assess the short- and medium-term effects of project interventions (outcomes), towards achieving impact on targeted impact groups (development objective) and in making contributions toward higher level impact (project goal). Outcome-level results assessment will focus on changes in behaviour or systems as a result of project outputs. At the objective-level, the M&E systems will obtain information to assess the likely long-term impact of the project on the lives of targeted impact groups. At the goal-level, the project M&E systems will provide information to assess the overall change, from all combined factors, toward meeting the goals of the program. The information obtained will be used by decision-makers to make modifications to the project strategy, reallocating resources, as required.

- Capture and Disseminate Lessons Learned and Good Practices. The project will generate useful lessons learned, both positive and negative, on implementation and impact as well develop good practices for achieving impact. The project M&E systems will systematically capture these and ensure that they are not only well documented, but widely disseminated within the project as well as outside of the project. In addition, good practices and lessons learned from outside the project will be channeled into the project to be adapted and tested.

- Build Local Capacities for M&E. Finally, the project M&E systems have a specific objective to build capacities among stakeholders to be able to undertake effective monitoring and evaluation using cutting edge ideas and technologies. The project will produce clear guidelines for gathering,

16 How to set up M&E system

recording, analyzing and presenting information tailored to stakeholder needs, and for sharing such information across departments at different levels and with community and other stakeholders.

1.3. Result based management

Results Based Management (RBM) is a management strategy by which an organization ensures that its processes, products and services contribute to the achievement of desired results (outputs, outcomes & impacts). RBM rests on stakeholder participation, on clearly defined accountability for results, and on effective communication that ensures constraints to achieving objectives are resolved in good time. It also requires monitoring of progress towards results and reporting on performance/feedback which is carefully reviewed and used to further improve the design or implementation of the programme.

The purpose and benefits of results-based monitoring

• It supports achievement of positive results and objectives;

• It facilitates the identification of negative results and risks and in this way allows introducing measures to correct them;

• It clarifies tasks and responsibilities and establishes feedback and discourse mechanisms between all stakeholders;

• It creates a well structured and transparent basis for decisions, based on real data and information;

• It facilitates communication about achieved results and therefore:

- Improves communication with the contractor;

- Facilitates co-financing.

• It facilitates understanding and presentation of contributions to the Millennium Development Goals.

RBM involves three steps

1) Performance planning (e.g. project design or preparation of the annual work plan and budget): use Project Logframe; Cost Tables; Results-Oriented Annual Work Plan.

2) Performance measurement (e.g. through periodic M&E or ad hoc studies): using M&E Tools

3) Performance evaluation and reporting: use Supervision reports; Annual Progress Reports; Mid- Term and Completion review reports.

Project management need to facilitate or undertake the regular collection, analysis and evaluation of information on results, in order to ensure that resources are used in the best way for expected objectives to be achieved, at all levels.

The Results’ Chain concept

Results’ Chain is the core of RBM, it consists of series of results: result level 1– outputs, result level 2 –outcomes and result level 3 – impact (intended or unintended, positive or negative) of a development intervention. The outputs, outcomes and impact are described in Table 1.2.

17 M&E Manual Guide for IFAD Funded Projects in Vietnam

Table 1.2 Result Chain

Results’ level Definition Example

Level 3 Positive and negative, long-term effects produced by a development intervention, Positive impact: increased farmers’ income (Impact) directly or indirectly, intended or unintended Increased capacity by small farmers to grow Level 2 The likely (or achieved) short-term and high-value crops; (Outcomes) medium-term effects of an intervention Increased production of high value crops in project area

Level 1 The products and services produced (through Activity of project: organized training course activities) by the project and provided to direct à result: Farmers trained in the production (Outputs) beneficiaries of high-value crops Financial, material and human resources used Input Human, material, technical for implementing activities

Process to achieve the results

The transformation of inputs into outputs, through activities, is quasi automatic (e.g. if a training course is organized, then the output “people trained” is achieved). Outputs do not get automatically translated into the expected outcomes or impact. For example, it may not be sufficient to teach farmers how to grow high-value crops and unless they also have access to improved seeds, they will not necessarily be able to grow high-value crops. External factors cause a positive or negative impact on the expected results. Although project management may be able to control certain factors, some other external factors will be largely beyond project management’s control, e.g. natural disasters, epidemics….

1.4. Participatory M&E

Participatory M&E (PM&E) advocates stakeholder engagement and inputs into the planning and implementation stages of a project, as well as during evaluations. This has the benefits of:

- Identifying opportunities for better targeting and design of activities to met local needs and prefer- ences;

- Encouraging local ownership and support to project activities by developing project-related stakeholder relationships that are more focused on equal partnerships;

- Gaining the support of project gatekeepers, such as government staff, that may be need to provide access to some regions, organisations, and people;

- Empowering and building the capacity of project beneficiaries and local implementing partners;

- Assessing benefits to project beneficiaries, as they themselves perceive these benefits;

- Directing interventions in a way that maximizes local benefits, particularly direct benefits to the target group.

Core principles of PM&E

- Primary stakeholders are active participants – not just sources of information;

- A focus on building the capacity of local people to analyze, reflect and take action;

18 How to set up M&E system

- There are opportunities for joint learning of stakeholders at various levels;

- It aims to create greater stakeholder commitment to take corrective actions to help themselves.

Participatory Project Management and PM&E need a different mindset than traditional approaches to project management do. Project staff need to be prepared to regularly share their knowledge and information openly with key project actors.

Table 1.3. Distinguish M &E and participatory M & E

M&E Participatory M&E

• Aims at making a judgment on the programme for • Is a process of individual and collective learning and capacity accountability purposes rather than empowering development through which people become more aware and programme stakeholders conscious of their strengths and weaknesses, their wider social realities, and their visions and perspectives of development • Strives for “scientific” objectivity of M&E findings outcomes. This learning process creates conditions conducive to thereby distancing the external evaluator(s) from change and action stakeholders • Emphasises varying degrees of participation (from low to high) of • Tends to emphasise the needs for information of different types of stakeholders in initiating, defining the parameters programme funders and policy makers rather than for, and conducting M&E programme implementers and people affected by the programme • Is a social process of negotiation between people’s different needs, expectations and worldviews. It is a highly political • Focuses on measurement of success according process which addresses issues of equity, power and social to predetermined indicators transformation • Is a flexible process, continuously evolving and adapting to the programme specific circumstances and needs

Types of Stakeholders could participate in M&E

- The community whose situation the programme seeks to change

- Project Field Staff who implement activities

- Programme Managers who oversee programme implementation

- Funders and other Decision-Makers who decide the course of action related to the programme

- Supporters, critics and other stakeholders who influence the programme environment..

1.5. IFAD M&E Methodology

The proposed M&E system is oriented around a three-level framework within which to monitor implementation progress toward producing proposed outputs and assessing the impact of these outputs in inducing behavioral or systemic change and impact on targeted impact groups and wider development goals.

i. Output Monitoring: The first level refers to the regular and continuous monitoring of project activities, project processes, and the outputs produced as a result.

ii. Outcome Assessment: The second level relates to the monitoring and assessment of behavioral or systemic change that is being induced by the outputs produced by the project.

iii. Impact Assessment: Finally, the third level refers to the assessment of changes occurring in

19 M&E Manual Guide for IFAD Funded Projects in Vietnam

the quality of life on targeted impact groups as a result of the behavioral or systemic changes caused by the project. This impact includes both the impact on specific targeted impact groups as reflected in the project’s development objective, as well as the contributions that the project is making towards the achievement of its overall goal.

The M&E system within the IFAD funded projects is a results-based system which will focus not just on how well the activities of the project are being executed, but also on answering the question “have the goals and objectives of the project been achieved; and how is this supported by evidence?”. Rather than focusing on how much output is being generated by the project, the monitoring and evaluation system will use survey data, qualitative information and self-assessments by participants to measure behavioural and systematic changes, as well as how these changes are affecting the lives of targeted beneficiaries.

In 2004, IFAD introduced a methodology for tracking the impact of IFAD assisted projects called the Results and Impact Management System (RIMS). RIMS is a comprehensive system for measuring and reporting on the results and impact of IFAD supported country programmes. RIMS consists of a standardized list of common indicators that can be applied across all IFAD projects and programmes. The list includes 1st level indicators (of outputs), 2nd level indicators (of outcomes), and 3rd level indicators of (impact). See Annex 1 for the 1st and 2nd level indicators.

Within the system, four RIMS standardized indicators are mandatory for all projects:

i. Number of households with improvement in household assets ownership index (anchor impact level indicator);

ii. Reduction in the prevalence of child malnutrition (anchor impact level indicator);

iii. Number of households that have improved food security (compulsory outcome level indicator);

iv. Number of persons receiving project services (direct beneficiaries - compulsory output level indicator).

v. In addition to these mandatory indicators, other performance indicators at the different levels have been identified during the project design phases as shown in the project Logical Frame- work. 2. M&E system

2.1. What is M&E system

The M&E system is the set of planning, information gathering and synthesis, reflection and reporting processes, along with the necessary supporting conditions and capacities required for M&E to make a valuable contribution to planning, decision-making and learning. Key project stakeholders need to develop the different elements of the system together if they are all to use the M&E outputs to improve implementation.

The project strategy (the plan for what will be achieved and how it will be achieved is the starting point for project implementation and setting up the M&E system. The strategy is the basis for working out the project operations required to efficiently and effectively achieve project objectives. The completion of project activities leads to a series of actual outputs, outcomes and impacts. A core function of M&E is to comparethe actual outputs, outcomes and impacts with what was planned in the project strategy, and understanding the differences in order to identify necessary changes in strategy and operations to ensure that project purpose is achieved..

20 How to set up M&E system

The M&E system consists of four interlinked parts: (i) setting up the M&E system; (ii) implementing the M&E system means gathering and managing information; (iii) involve project stakeholders in reflecting critically; and (iv) communicating and reporting the results of M&E. Ultimately the results from M&E – both the communication processes and the information- will feed into results-based planning processes and improve the project strategy and operations.

For M&E to work as a tool for managing for impact, project management and the M&E staff need to be clear on how to identify, agree upon and follow up on project improvements. If this process for guiding changes is not in place, even a very good M&E system will not have much value for the project.

Figure 1.1. The M&E system and how it links to the project strategy and operations

21 M&E Manual Guide for IFAD Funded Projects in Vietnam

Table 1.4. Key M&E tasks

Establish the scope and purpose of the M&E system Indicate key performance questions and indicators, plus associated monitoring mechanisms Identify organizational arrangements for M&E Early design phase (formulation and appraisal) Develop terms of reference for M&E staff Indicate the process for how M&E is to be established during start-up Establish an indicative M&E budget Document the above in the M&E framework Revise performance questions, indicators and monitoring mechanisms after reviewing the project strategy

Start-up prior to loan effectiveness Organize training with staff and partners likely to be involved in M&E Initiate a project implementation manual with key staff Prepare a project implementation manual with key staff Review project design in relation to M&E with key stakeholders Develop a detailed M&E plan, taking into consideration existing mechanisms Start-up after loan effectiveness with partners Put in place necessary conditions and capacities for M&E to be implemented Ensure information needs for management are met Coordinate information gathering and management Facilitate informal information gathering and communication Support regular review meetings and processes with all implementers Main implementation Prepare for supervision missions Prepare for and facilitate the annual project review Conduct focused studies on emerging questions Communicate results to stakeholders Prepare annual progress reports Collate information for the mid-term review Facilitate the internal review process to prepare for the external review process Mid-term review Help respond to MTR feedback Adjust the M&E system. As necessary Assess what the implementers can do to sustain impact and sustain M&E after closing down-and implement these ideas Phasing-out and completion Hold workshops and do field studies with key stakeholders to assess impacts Identify lessons learned for the next phase and/or other projects

22 How to set up M&E system

2.2. Setting up the M&E system

Setting up an M&E system involves six steps that need to be dealtwith twice – generally at initial design and in detail at start-up- constructed around the project results chain:

Box 1.1. Steps to set up the M&E system

1. Establishing the purpose and scope – Why do we need M&E and how comprehensive should our M&E system be? 2. Identifying performance questions, information needs and indicators – What do we need to know to monitor and evaluate the project in order to manage it well? 3. Planning information gathering and organising – How will the required information be gathered and organised? 4. Planning critical reflection processes and events – How will we make sense of the information gathered and use it to make improvements? 5. Planning for quality communication and reporting – What, how and to whom do we want to communicate in terms of our project activities and processes? 6. Planning for the necessary conditions and capacities – what is needed to ensure that the M&E system actually works?

2.2.1. Establishing the purpose and scope

Clear definition of the purpose and scope of the intended M&E system helps when deciding on issues such as budget levels, number of indicators to track, type of communication needed and so forth. The appraisal report will include a brief M&E purpose statement but you will need to revisit this question at start-up with representatives of implementing partners and primary stakeholders.

When formulating the purpose at appraisal or revisiting it during start-up, ask yourself the following question:

• What are the main reasons to set up and implement M&E, for us – as implementing partners and primary stakeholders – and for other key stakeholders?

Define the scope of the M&E system by asking:

• What level of funding is potentially available?

• What level of participation in M&E by primary stakeholders and partner organizations is desirable and feasible?

• How detailed does the M&E information have to be, either in terms of quantitative or qualitative data?

• What sort of baseline study is desirable and feasible?

• What are the current M&E capacities among primary stakeholders and partners organizations, and how will this affect the desired levels of M&E? 2.2.2. Identifying performance questions, information needs and indicators

A performance question helps focus your information-seeking and information analysis processes on what is necessary in order to know if the project is performing as planned or, if not, why not. Once you have your performance questions, you can more easily decide what information you need to track rather than what is nice to track.

23 M&E Manual Guide for IFAD Funded Projects in Vietnam

Working with performance questions to guide indicator analysis will give you more integrated and meaningful picture of overall project achievements. Answering these questions requires descriptive analysis and quantitative information. Starting by identifying performance question make it easier to recognise which specific indicators are really necessary. Sometimes a performance question can be answered directly with a simple quantitative indicator. However, very often the question can only be answered with a range of qualitative and quantitative information.

Box 1.2. The basic performance questions per level of the objective hierarchy

n Activities – What have we actually done? n Outputs – What have we delivered as a result of project activities (e.g., number of people trained)? n Outcomes (results) – What has been achieved as a result of the outputs (e.g., extent to which those trained are effectively using new skills)? n Impacts – What has been achieved as a result of the outcomes (e.g., to what extent are NGOs more effective)? What contribution is being made to the goal? Are there any unanticipated positive or negative impacts? n Lessons – What has been learned from the project that can contribute to improved project implementation or to building relevant fields of knowledge?

Box 1.3. Examples of performance questions

Performance questions are not just about what has been achieved. They also ask why there is success or failure and what has been learned to improve future action. Examples of performance questions include: n How has the purchasing power of target households as compared to non-target groups? What external factors have influenced any changes? n To what extent are target households better able to meet their housing, education and health needs than non- target households? n How have the diversity, production and productivity of agriculture in the target area changed as a result of project activities and as a result of external factors?

2.2.3. Planning for information gathering and organising

For each information need or indicator, you must establish how the information will be collected and organized. Not only will each indicator require choosing a different method, but for each indicator or information need you will usually present several options. There are many methods – qualitative and quantitative and individual versus group-based.

Particularly critical at the moment of method selection is knowing who will be involved in collecting, compiling and analyzing. The more that the intended users of the methods can involved in selecting or developing the methods, the more chance there is that they will understand them and use them correctly. If methods are selected by someone not using them, then training users in the method will be essential.

Gathering data is one thing. But each bit of information also needs to be collated, perhaps summarised and certainly analysed by the right people.

24 How to set up M&E system

2.2.4. Planning critical reflection processes and events

Critical reflection can occur formally and informally. Formally, it can be facilitated during project meetings, workshops with partners and primary stakeholders or as part of external evaluation. Informally, it can occur in ongoing discussions between project stakeholders.

Table 1.5 Example of Critical Reflection Schedule for a project

Note: This is a project-centric list. Primary stakeholders will have their own schedule of events as will each of the implementing partners, The art of participatory M&E is to link these different schedules for shared learning.

Critical Reflection Processes and Purpose and Description Whom to involve Timing Events Participatory review Update the situation analysis, revise Representatives of Three-day workshop of project strategy problems/visions, adjust objective intended primary at start-up hierarchy and assumptions stakeholders, staff or participant organization, all project staff, facilitator Development of Assess different information needs, Representatives of Four or five full-day M&E plan with take stock of who is already doing what, intended primary meetings during the stakeholders agree on priority information areas, refine stakeholders, staff or first six months of the questions/indicators, decide on methods, participant organization, project agree on responsibilities all project staff, facilitator Quarterly progress Discussion of key successes and problems Senior staff of PMU and One-day meeting reviews by PMU partner organization every three months staff Field visits Firsthand look at what is happening in the Field staff, supervisors of Weekly visit for field field, informal chats about how activities field staff, project director staff, monthly for the are being implemented project director Annual project Summary of key successes and problems, Representatives of Once a year review ideas for changing project activities/ intended primary outputs and assumptions, review of stakeholders, staff of implications for the project log frame, implementing partners, identification of lessons learned about all project staff, facilitator project implementation, M&E system adjustment

Periodic review Focused discussion about the strategy and Key stakeholders Once a year in the workshops of key operations of key components to adjust the of the project first two years; after project components objective hierarchy, solve problems and component: intended that, once every two identify lessons learned primary stakeholders, years implementing partners, field and senior project staff Preparation Explain the mission purpose, agree on Small group of One month prior the for supervision what the project and stakeholders would primary stakeholders supervision mission missions like to get out of the mission, identify who representatives, senior needs to prepare what before the mission, staff of the participant organize the logistics organization, senior project staff

25 M&E Manual Guide for IFAD Funded Projects in Vietnam

2.2.5. Planning for quality communication and reporting

M&E –related findings have many potential audiences. When reporting on progress with the AWPB, you will direct yourself to funding agencies, steering committees, cooperating institutions and implementing partners. Primary stakeholders have a right to knowing overall how the project is progressing and they deserve the opportunity to react to initial findings. Funding agencies and managers need information on impact, while all implementing partners need to understand problems in order to find solutions. Two sets of M&E findings will need to be communicated.

First, it is good practice to discuss draft M&E findings with implementing partners and primary stakeholders in order to get feedback on accuracy, reach joint conclusions and agree on next steps. Once the M&E findings are agreed upon, these can be communicated to funding agencies, cooperating institutions, government departments and other projects. This second set of final findings will fulfill accountability needs but can also serve for advocacy purposes. 2.2.6. Planning for the necessary conditions and capacities

Usually, the appraisal report has indicate budget for M&E and describe how M&E should or could be organized. But getting the M&E system working means thinking of appropriate incentives, ensuring you have the right and enough human capacity at hand and thinking about ways of storing and sharing information. Table below lists key questions that need answers when detailing M&E to make it operational.

Table 1.6. Questions to guide the detailed planning of M&E conditions and capacities

Conditions and Questions to Guide Detailed Planning Capacities

Human capacity for M&E - What are the existing M&E capacities? - What training will be necessary? - What consultancy support will be required? Incentives for - Are M&E responsibilities included in job descriptions and terms of reference? implementing M&E - How will reflection and learning among staff, partners and the intended primary stakeholders be encouraged? Organizational structures - Will there be an M&E unit or will M&E be spread among all parties? If there is a unit, how many people will it have and where will it be located, under whose authority? - How closely connected will M&E staff be with project management? Management information - What information must be stored and accessible, when, how and for whom? systems (MIS) - What level of computerization is required and appropriate? - What expertise will be required to set up the information management system? Financial resources - Is there a separate M&E budget and have sufficient resources been allocated? - Has the staffing allocation for the project taken into account time for all relevant staff to undertake M&E activities? 3. M&E planning

3.1. The structure of the monitoring plan

The monitoring plan specifies 5 issues: (i) what are the monitored subjects?; (ii) When is the monitoring conducted?; (iii) How is the monitoring conducted?; (iv) Who carry out the monitoring?; and (v) monitoring cost.

26 How to set up M&E system

(i) What are the monitored subjects?

Important factors in determining the monitored subjects, including:

- - Main subjects that receive information;

- Questions;

- Indicators (for changes);

- Information on the implementation;

- Unexpected changes;

- General problems;

- Initial evaluation on the effectiveness of the results.

(ii) When is the monitoring conducted?

- After identifying the monitoring objectives, indentify the timeline for the monitoring. It is necessary to determine a timeline suitable with the timeframe of the whole project.

(iii) How is the monitoring conducted?

- Group discussion to determine the monitoring method, example:

Table 1.7. Example for determining the monitoring method of group discussion

How to measure the Monitored issues Indicators How to report the results indicators Build models for crop Number of participants Report of project officials on Were the objectives production for project (households) the number of participants achieved? households Number of households that Questionnaires Adjustment measures apply the model (if objectives were not achieved)?

(iv) Who conduct the monitoring?

Who conduct the monitoring? In the monitoring plan, it is necessary to present the name and position of the monitoring official. This presentation has two effects, (i) link the monitoring official with specific activities and hold him/her responsible for his/her activities; and (ii) the monitoring official knows what, how and when to do.

(v) Monitoring cost

Whether there are expenditures for the monitoring. If there are, estimate the monitoring cost, via monitoring activities and supports, such as means, for best results.

3.2. The structure of the evaluation plan

The structure of an evaluation plan is similar to that of a monitoring plan. Accordingly, the evaluation is also based on an established plan. An evaluation plan specifies 5 issues: (i) The evaluation objectives; (ii) Evalu- ation methods; (iii) Organization of the evaluation; (iv) Evaluation cost; and (v) Evaluation information-use plan.

27 M&E Manual Guide for IFAD Funded Projects in Vietnam

The key point of an evaluation plan is to identify what are evaluated and how to do it. Generally, an evaluation plan contains the purpose and the evaluated subjects, questions, indicators to measure results, data- collecting methods, work plan and resources to implement the evaluation.

The evaluation plan must identify explicitly what are evaluated, such as a project, an organization, a sector or a programme. As demonstrated above, an evaluation plan includes information like contexts, objectives, specific activities, costs, expected impacts and results.

Table 1.8. Illustration of the structure of an evaluation plan

Requirement Content

Objectives, evaluation content What are evaluated? – What we need to know? Base on the project design – the Objectives section; or the Project Evaluation Plan – Evaluation Objectives section; or the Term of Reference to identify the evaluation objectives and content.

Evaluation method Present the evaluation methods used to collect information in order to specify the evaluation objectives and content.

Organize the evaluation Who will do what? What is his/her role? What are the roles of the partners, the relevant stakeholders and the evaluation team?

Evaluation cost Full estimated expenditure for each evaluation activity.

Evaluation information-use plan Structure of the evaluation report. Expected purposes of information. Plan to use and share information.

The five key questions

Followings are the five standard types of evaluation questions:

1. Relevance – Was/Is the project a good idea given the situation needing improvement? Does it deal with target group priorities? Why or why not?

2. Effectiveness – Have the planned purpose and component objectives, outputs, and activities been achieved? Why or why not? Is the intervention logic correct? Why or why not?

3. Efficiency – were inputs (resources and time) used in the best possible way to achieve outcomes? Why or why not? What could we do differently to improve implementation, thereby maximizing impact, at an acceptable and sustainable cost?

4. Impact – To what extent has the project contributed towards its longer-term goals? Why or why not? What unanticipated positive or negative consequences did the project have? Why did they arise? To what extent has the project contributed towards poverty reduction (or other long-term goals)? Why or why not? What unanticipated positive or negative consequences did the project have? Why did they arise?

5. Sustainability – Will there be continued positive impacts as a result of the project once it has finished? Why or why not?

The M&E operations will focus on the questions of “effectiveness” and “efficiency”. More strategic reflec- tions, like during annual reviews and supervision mission, will look at the questions of “relevant”, “impact” and “sustainability”. Some projects are also asked to prove their cost-effectiveness.

28 How to set up M&E system

3.3. M&E and project logical framework

The logical framework (or logframe for short) is a tool to present objectives into specific and measureable indicators. In addition, the logframe specifies how to address those objectives as well as assumptions to achieve them. This tool also clarifies the relation between the objectives of the project and the activities in the project design. A specific log frame can be described as follows:

 Long-term objectives: impacts, purposes, expected long-term results – contributions of the project.

 Medium-term objectives: outcomes- these are intermediate resultsshort-term and medium-term benefits that the project aims to secure. Outcomes are a result of local changes in people’s behaviors and in the performance of institutional services. They are the change effected in actors’ behaviours and capabilities (i.e. in actors’skills, attitude, knowledge, practices), and the change in performance (efficiency and effectiveness) of local systems (e.g learning systems, extension, banking systems), that occur as a result of outputs delivered by the project. Outcomes are generally measured in terms of what the actors within the sphere of the project have done or are now prepared/able to do. Their achievement increases the likelihood of contributing to the overall (longer-term) goal.

 Short-term objectives: quick outputs and results – products or services that are directly provided by the project. Outputs are generally measured in terms of what project teams have done or delivered.

 Indicators: To monitor and evaluate the results of a project/programme, measure impacts, results and outputs as expected to be built. These measuring methods are usually called indicators.

 Criteria: Once the project objectives and relevant indicators are built, project management officials and M&E officials need to identify annual project evaluation method or criteria on the basis of final expected project’s results. If the progress criteria, final output criteria and result criteria have been identified in the log frame in the drafting of the project’s plan, M&E officials still have to evaluate their appropriateness in the operational environment. In case the final output criteria are agreed by the sponsor but the annual progress criteria have not been established, the project officials and M&E officials have to use completion criteria of the project to determine suitable annual criteria. Practical criteria can be built by dividing indicators into different levels of completion.

 Risk: All projects face a certain kind of risks that hamper the achievement of objectives and/or expected results. Thus, these risks need to be identified, monitored and evaluated closely by M&E officials to propose necessary responses to reduce the impacts of any internal or external risk. Regarding internal risks, their impacts lie within the project. Internal risks may be delays in signing contract that affect important decisions, delays in disbursement for stakeholders’ operation, or change in number of staff, etc. This kind of risks need to be controlled in order to achieve the objectives and expected results in a timely manner. On the other hand, external risks lie outside the project but still affect its operational environment. Thus, they are often out of the control of the project. However, they still need to be monitored, evaluated and controlled right after they are recognized as potential threats to the project.

 Risk management and risk planning: need to be built, especially for projects in unstable environment or potentially unstable environment. In those projects, specific actions should be planned in preparation for worse conditions. Evaluating the level of risks is part of this process. Project

29 M&E Manual Guide for IFAD Funded Projects in Vietnam

officials, particularly M&E officials, should be required to notify the Project Management Board on the unsolved risks for timely responses.

 Assumption: External factors (e.g. events, conditions or decisions) may affect the progress or the success of a project/programme. The initial assumptions are critical requirements for the success of a project/programme. Risk assumptions are conditions that may threaten the implementation of a project/programme.

Table 1.9. Example of log frame

Project description Indicator Indicator verification Risk

Objectives • Support the • Number of CIGs • CIGs’ work reports. • People are still establishment of established in project hesitated/shy to CIGs of farmers on area. take part crop production and animal husbandry.

Purposes • Share working • Number of CIGs • Reports of the Policy • People are still experiences for offered loans Bank. hesitated/shy to households. take part • Number of training • Reports on the • CIGs receive courses organized progress of the • Limited fund- supports from the project managing capacity • Household’s income project: financial and of the people (in comparison that • Annual survey on the technical supports and the Project previously) project results. Management Board. • Improve income

Outcomes • CIGs have access • Number of CIGs • Reports of the Policy • Limited fund- to loans and other offered loans Bank. managing capacity financial services of the people • Number of CIGs • Surveys and the Project • Farmers have that apply farming Management Board. access to knowledge techniques and information • People are not used on improved crop to new cultivation husbandry practices customs.

Outputs • Establish CIGs in • Number of established • CIGs’ work reports. • People are still project area. CIGs in project area. hesitated/shy to take part

Activities • Offering loans to • Volume of CIG loan • Reports of the Policy • Limited fund- CIGs disbursements; and Bank. managing capacity repayment rate of the people • Provide farming- • Quarterly reports and the Project technical training to • Number of farmers of the Project Management Board. CIGs attending training Management Board.

Inputs • Finance • Quarterly • Project information • Limited fund- disbursement management system managing capacity • Human resources of the people • Quarterly participation • Reports on the • Training equipment and the Project of the members of the progress of the Management Board. Project Management project Project. • Number of teaching materials prepared for distribution

30 How to set up M&E system

3.4. Detailing M&E plan based on project appraisal report

A good appraisal report will include an indicative M&E framework that provides enough detail about these questions to enable budgeting and allocation of technical expertise, give funding agencies an overview of how M&E will be undertaken, and guide project and partner staff during start-up. But this will only be indicative and needs to be adjusted and detailed further during the start-up phase. Below table presents the tasks for each step in taking the outline of an M&E system from an appraisal report and designing the details that make it operational.

Table 1.10. Task needed when detailing the M&E plan based on a project appraisal report

Outputs in Project Task during Project Start-up to Develop a Detailed M&E Design Steps Appraisal Report (the M&E Plan M&E Framework) 1. Establishing the • Broadly defined • Review the purpose and scope with key stakeholders. purpose and scope purpose and scope of M&E in the project context 2. Identifying • List of indicative key • Assess the information needs and interests of all key performance questions and indicators stakeholders. question, indicators for the goal, purpose • Precisely define all questions, indicator and information and information and output levels needs for all levels of the objective hierarchy. needs. • Check each bit of information for relevance and end-use. 3. Planning information • Generally described • Plan information gathering and organizing in detail (who gathering and information gathering will do use which method to gather/synthesize what organizing and organizing methods information, how often and when, where, with whom, with to enable resource what expected information product). allocation • Check the technical and resource feasibility of information needs, indicators and methods. • Develop formats for data collection and synthesis

4. Planning for quality • Broad description of key • Make a precise list of all the audiences, what information they communication and audiences and types of need, when they need it and in which format. reporting information that should • Define what is to be done with the information – simply send be communicated to it, provide a discussion for analysis, seek relevant feedback them to enable resource for verification, etc. allocation • Make a comprehensive schedule for information production, showing who is to do what by when in order to have the information ready on time. 1. Planning critical • General outline of key • Precisely detail which methods/approaches are to be used, reflection processes processes and events with which stakeholder groups and for what purpose. and events • Identify who is responsible for which reflective events 1. Planning for the • Indicatives staffing • Come to a precise definition of: the number of M&E staff, necessary conditions levels and types, their responsibilities and their linkages, incentive needed to and capacities clear description of make M&E work, organizational relationships between key organizational structure M&E stakeholders, the type of information management of M&E, indicative system to be established and a detailed budget. budget

31 M&E Manual Guide for IFAD Funded Projects in Vietnam

3.5. Using the M&E Matrix for detailed planning

To make M&E operational you need much more detail, which can be summarized in the “M&E matrix”. Here we will briefly outline the M&E matrix, looking at each column in turn.

Table 1.11 Contents of the M&E matrix

Performance Information Baseline Data-gathering Required forms, Analysis, reporting, questions needs and information methods, planning, training, feedback and indicators requirements frequency and data management, change processes status and responsibilities expertise, resources and responsibilities responsibilities and responsibilities

Step 1. Identifying Performance Questions

Rather than starting with indicators, first identify performance questions. This helps you focus your information gathering on what you will really use for understanding and improving project performance. Iden- tifying performance questions (indicators and selection methods) will be iterative: make an initial choice, assess its feasibility, accept and use it or reject it and find the next option.

Step 2. Identifying Information Needs and Indicators

Using your performance questions, you can more easily identify useful indicators and other information need for which you will need to collect data. Only data that help answer your performance questions are necessary. This helps avoid collecting information that is difficult to use to guide the project strategy and operations. .

Step 3. Knowing what Baseline Information You Need

Many baseline studies suffer from information overload and lack of use. When deciding whether you need to collect baseline data for a particular performance question, ask yourself if you need to compare information to be able to answer the question. If not, or if information already exists, then you will not need to collect baseline data.

Step 4. Selecting Which Data Collection Methods to Use, by Whom and How Often

Once you have decided what information is needed and what indicators will be use, you need to decide which methods will be used for gathering the data. You have many options: methods that are more qualitative or more quantitative, more or less participatory, and more or less resource intensive. Each will provide information of varying degrees of accuracy and reliability. Deciding which methods to use requires balancing these different factors. When you examine the consequences of a particular performance questions or indicator, you may need to change it if it is impractical or too expensive. This includes looking at who will be using the method and how often it will be applied. For example, if you have no existing capacity to use your preferred method, you need to plan training – or choose another method if you have no resources for this. Frequency of collection also needs to be established. This will vary per question and indicators. If one critical indicator needs to be collected often, then you may need to reduce the frequency of another less important indicator or delete it altogether.

32 How to set up M&E system

Box 1.4. Example for gathering information on food security changes

For gathering information on food security changes you could consider three different methods (1) a detailed household survey conducted by independent researchers, (2) a participatory assessment process where women household members do their own monitoring and discuss their findings, or (3) focus-group meetings to discuss changes that specific social groups have experienced. The first method would the most resource intensive but may have yield the most quantifiable outputs. If well facilitated, the second method can also yield precise results but at a lower cost than the first method and perhaps with interesting discussions from which new project by village women. The third method would yield the least precise and least quantitative information but would be the least resource intensive. Before embarking on resource intensive data- collection exercises, carefully consider whether a simpler method would yield sufficient information of good enough quality for your purpose.

Step 5. Identifying the Necessary Practical Support for Information Gathering

For a method to lead to the information you require, you will need to organize the conditions to make it work. These are often forgotten in the focus on identification of indicators but are critical to success. For each method, consider if and how you need to: • Develop forms to record data; • Develop forms, filling systems and databases for collating and storing information; • Train staff, partners or community members who will be involved; • Check and validate data; • Organize external M&E or research expertise that may be needed; • Agree on responsibilities for different tasks; • Ensure everyone has sufficient financial resources and equipment.

Step 6. Organizing Analysis, Feedback and Change

In the rush to get out and start collecting data, many M&E units pay insufficient attention to the process of using the information for analysis and directing changes in the project.

To make sure that data will be used – and not just collected – think about how you will organize the analysis of information for each performance question. Sometimes a performance question cannot be answered without prior analysis of several bits of information. Who will do it? When will it happen? Also consider what form information should be in so that it can be used by different stakeholders. For example, will it be useful to present information visually, in graphs or maps? Or do you need to organize several community meetings to get more feedback on the initial analysis of the information?

Most importantly, consider how the generated information can be used to check progress and make improvements as the projects proceeds.

3.6. Indicative contents for an M&E Plan

The operational plan for project M&E provides the conceptual and, above all, practical basis for planning, monitoring and evaluation within the project. The document also defined the main concepts related to M&E, the redesigned log frame matrix, the adjustment of information collection systems and the database of primary stakeholders.

33 M&E Manual Guide for IFAD Funded Projects in Vietnam

The M&E operational plan will be the reference point for stakeholders throughout the project life. So it needs to be comprehensive enough, at macro level, to provide a clear picture of the overall project intentions and how the M&E system will serve this. At the micro level, it must give fine detail on schedules, responsibilities, budgets and so on, which will help guide the drawing-up of AWPBs..

Table 1.12. Indicative contents for an M&E operational Plan

Topic Description

• Purpose and • Project overview and objectives, rationale and justification for the design of the M&E system scope • How the M&E system will support project management and meet the reporting requirements and information needs of different stakeholders. • Summary of overall experience of M&E undertaken with key stakeholders • Discussion of extent of participation, balance between qualitative/quantitative approach, resource intensiveness and the intended poverty focus of the M&E system

• Approach Overview of how stakeholders will be involved, what learning-oriented approaches will be used and, in general terms, what information gathering and analyzing methods will be used; for example, the extent of use of participatory approaches, geographic information systems, computer-based information systems or baselines surveys • Revised logical Precise definition of all performance questions, indicators and information needs for all levels of framework, the objective hierarchy: plus • Assessment of the information needs and interests of all key stakeholders performance questions, • Assessment of the indicators of exogenous factors and assumptions (e.g., climate, prices, indicators, outbreak of pests and disease, economic situation, policy environment) information • Assessment of the information needs and indicators for relevance and end-use and for technical needs and and resource feasibility sources • Selection of performance questions and indicators • Management • Purpose of the management information system information • Organization of information gathering and synthesis: system and reporting - For each expected information product – who, what, when and where - Schedule of information production – who what, when, to whom, for what purpose - How computerized networks and manual archiving systems are expected to function, with/to whom, for which data - Outline of data storage needs • Expected reporting outputs, for example: - Informal communication and feedback channels - Report flows – deadlines and frequencies - AWPBs – outline of the AWPB format, including output/activity plans and budgets, consolidated budgets, a training plan, a procurement plan, a contracted services plan - (Bi-) Annual progress reports for the project as a whole and each component, village-based reviews - (Bi-) Annual financial reports - Recurrent supervision missions

34 How to set up M&E system

Topic Description

• The M&E Precise definition of methods to be used with different stakeholder groups for two core purposes: work plan 1. M&E resources, activities and implementation for effective project operations: (processes and events) • Project resources: transport use, allowances, register for assets, register of services/technical and timing of assistance activities • Project activities: training (workshops, study tours, etc.), construction (technical or social infrastructure), scheme organization, trials and demonstrations, credit lines, etc. • Other monitoring activities 2. M&E of outcomes and impact for guiding the project strategy, for example: • Proposed surveys: baseline/household, component, staff • Participatory annual assessment and planning workshops • Other annual evaluation and beneficiary assessments, reviews, and planning sessions • Midterm review and project completion report • Feasibility of methods in terms of technology and resources • M&E work plan schedule: integrated schedule of key events and reporting/decision making moments • Critical event agenda

• Establishing M&E organization: conditions and • Necessary institutional and stakeholder linkages for M&E capacities • Existence (or not) of a specific M&E unit and how it relates to the project structure and hierarchy of authority Human resource needs: • Number, capacities and responsibilities of different stakeholders in M&E, including project staff and primary stakeholders • Training needs of stakeholders and staff Resource needs: • Vehicles and equipment • Technical assistance • The M&E Detailed budget allocation budget • Appendices • Original and revised log frames • List of proposed indicators • Outline formats for data collection, annual and biannual schedule of activities, etc. • Outline formats for preparing: quarterly, biannual and annual reports, a summary of main project achievements; status reports on project inputs and resources, project outputs and results, evaluation studies – summary of findings and recommendations. • Baseline survey questionnaire • Staff job descriptions and details of allowances • Technical Assistance terms of reference • M&E work plan • Detailed budget of M&E

35 M&E Manual Guide for IFAD Funded Projects in Vietnam

4. Data collection methods

There are many methods commonly used for M&E. They have been grouped as follows:

 Sampling methods;

 Core M&E methods (such as stakeholders analysis and questionnaires);

 Discussion methods for groups (such as brainstorming and role plays);

 Methods for spatially-distributed information (such as maps and transects);

 Methods for time-based patterns of change (such as diaries and photographs);

 Methods for analysing relationships and linkages (such as impact flow diagrams and problem trees);

 Methods for ranking and prioritizing (such as matrices).

Before selecting your methods, first consider three important aspects:

 Quantitative versus qualitative methods;

 Individual versus group-based methods;

 The extent to which a method can be participatory.

Table 1.13. Examples of multi-purpose M&E methods

Qualitative data Quantitative data

Methods for groups Case studies, brainstorming, focus groups, Nominal group technique, maps, transects, SWOT, drama and role plays, maps, transects, historical trends/timelines, seasonal GIS, historical trends/timelines, seasonal calendars, flow diagrams, matrix scoring calendars, rich pictures, visioning, flow diagrams, and ranking well-being ranking

Methods for individuals Semi-structured interviews, case studies, maps, Biophysical measurements, structured transects, diaries, historical trends/timelines, questionnaires, maps, transects, GIS, seasonal calendars, flow diagrams diaries, flow diagrams

4.1. Quantitative and qualitative methods

Quantitative methods

Quantitative research refers to the systematic empirical investigation of quantitative properties and phenomena and their relationships and that quantitative methods are research techniques that are used to gather quantitative data — information dealing with numbers and anything that is measurable. Quantita- tive methods answer questions such as: What? How much? How many? When?. These methods are useful to provide numerical and quantified results that can be aggregated.

Qualitative methods

Qualitative methods aims to acquire an in-depth understanding of human behaviours and the reasons that govern such behaviours. Qualitative methods are used to investigate the why and how, not just the what, where, when.

36 How to set up M&E system

Table 1.14. Comparison of quantitative and qualitative methods

Quantitative Methods Qualitative Methods

Use To numerically measure “who, what, when, To qualitatively analyse “how and why”. where, how much, how many, how often”.

Examples Standardized interviews; surveys using Free and guided interviews (including focus closed-ended questions; observation. group); surveys using open-ended questions; observation; interpretation of documents.

Strengths . Provide quantitative, accurate and . Useful when planning a programme precise “hard data” to prove that certain concerned with social change; problems exist ; . Provide a thorough understanding of . Can test statistical relationships between programme/project context in order to interpret a problem and apparent causes; quantitative data; . Can provide a broad view of a whole . Provide insights into attitudes, beliefs, motives population; and behaviours of a small sample population (families, communities); . Enable comparisons; . Establish baseline information which can . Establish baseline information which can be used for evaluating qualitative outcomes be used for evaluating impact. (changes in knowledge, attitudes, behaviours, institutional processes etc.); . Useful in case of money and time constraints; . Useful for getting feed-back from stakeholders.

Both methods could be used in combination (such as focus group discussion and key informant interview methods could be combined with a quantitative survey...).

4.2. Collecting secondary data and information

Compilation of data from project staff/agency records or service providers’ records

This is recommended for activity (therefore output) monitoring of projects. This can provide information concerning the whole beneficiaries’ universe, offer basic performance information. At the same time, it’s essential for accountability purposes (e.g. show what the project budget was spent for)..

Compilation of data from beneficiaries’ records:

This is the best when groups of beneficiaries are formed (if project deals with individual households, this will be too complicated). This method is good for outcome and impact monitoring (e.g. if beneficiary groups are requested to keep records/information on agricultural/dairy production; volumes of sales; incomes, etc.). Through it, Whole or part of beneficiaries’ universe can be monitored. This can inform on qualitative aspects as well and be good tool for impact monitoring… but at a cost. In addition, it demands in terms of human resources and project MIS.

4.3. Survey

Survey is an effective method to capture information on a small sample that represents for the beneficiaries. This method can replace the method, in which the entire group of beneficiaries are monitored, allowing monitoring of the results and impact of the project on the beneficiaries. To achieve this method

37 M&E Manual Guide for IFAD Funded Projects in Vietnam should be used many times and need strict requirements for sampling skills and the principles of investigator management. During the project implementation, the project should carry out following survey: (1) Baseline survey; (2) Annual outcome survey; (3) End-line survey.

4.3.1. Baseline survey

The baseline survey will involve mainly the collection of quantitative data against the project’s outcome, objective and goal levels. This will include the collection of both secondary data where appropriate and primary data where secondary data sources cannot provide sufficient information. Secondary data sources include the General Statistics Office and sources from within the project’s stakeholders, including the Ministry of Planning and Investment and the Ministry of Agriculture and Rural Development. The quantitative survey will be population based – meaning that the data will be collected not only from targeted direct beneficiary households but also from non-beneficiaries in order to accurately gauge the impact of the project, including the ripple effects of the project. Information emerging from the baseline survey will be analysed by stakeholders at the province and district level in each province in an analysis workshop to determine whether and how the baseline findings should influence project implementation or strategy.

4.3.2. Annual Outcome Survey:

Annual surveys will be conducted by the project to obtain quantitative information on indicators at the outcome and objective levels. These annual surveys are basically a condensed replication of the baseline, mid-term and end-line surveys to provide information for decision-makers in the event that a strategic change needs to be made before an evaluation occurs. They are also a critical capacity building tool for building M&E capacities of project staff for obtaining and using M&E information..

4.3.3. End-line survey:

Near the end of the final year of the project, a final evaluation will be conducted to assess the impact that has been achieved by the project as well as to identify lessons learned and good practices generated by the project. As with the mid-term evaluation, the final evaluation will include both a quantitative end-line survey to collect data against the project’s outcome, objective and goal level indicators, as well as collection of qualitative information to understand how and why impact has been achieved. The end-line quantitative survey will be population-based, and the data will be collected, analyzed and made available before the qualitative information gathering phase of the evaluation begins. As with the mid-term evaluation, the qualitative evaluation will be conducted by a team of external evaluators to determine why and how changes reflected in the quantitative data comparisons between baseline and end-line have occurred. This team will also look for other impact that may have been achieved and not captured by the quantitative survey. The evaluation team will also use information generated by the final social audit to inform their findings and identification of lessons earned and good practices..

4.4. Key informant interview

A key informant interview is a dialogue between researchers and people who can provide detailed information (hereinafter called ‘interviewee’) in order to understand interviewees’ lives, experiences and perception on particular issue. This is qualitative method. This is the best method in the case of sensitive information, the respondents do not want to disclose information which they answer with others. This method is also very effective in finding solutions, suggestions, or new ideas. How to conduct a key informant interview is presented at Chapter VI.

38 How to set up M&E system

4.5. Focus group discussion

Focus group discussion (also known as intensive group discussion, focus group interview; hereinafter referred to as “focus group”) aims to obtain qualitative information on attitude, perception, opinion and idea from a group of people who have in-depth understanding about specific topic.

This method is used to collect detailed qualitative information from groups of 8-12 people with common characteristics and allow a deep understanding about the behavior, attitudes, feelings, perceptions and perspectives of the beneficiary. Group discussions are often used to assess the needs, collect opinion from many people to evaluate the same problem, or discuss on the controversial content, or collect new ideas.

4.6. Observation method

In addition to project reporting, project staff from the PCU and the district supporting office will be responsible for conducting periodic field visits to observe project activities. They will document their observations in concise field trip reports that will also describe their plans for addressing any problems or opportunities they have observed. These field trip reports will be attached to quarterly reports in the project reporting system so that they can be considered by decision-makers at various levels. 4.7. Annual supervision mission

Annual supervision missions will be carried out by IFAD to review the yearly progress towards the objectives and to help with the development of an annual plan. The supervision missions will use a qualitative approach but will also review the annual survey results and other qualitative and quantitative data collected and reported by the project. The supervision mission will check the quality of results presented in the annual survey report and other monitoring reports. 5. Gender and M&E

5.1. The importance of gender in M&E

If M&E does not capture such gender differences, it could lead to following problems:

 Can have negative impact on women;

 Not be able to achieve the desired outcome because we are not addressing the constraints of women and men appropriately;

 Can make either men or women unsatisfied with the project, since they feel that their needs are ignored;

 Might overlook gender differences in vulnerability and might underestimate the impact of external pressure on women and men;

 Can lose the opportunity to empower women (improve their confidence, self esteem, build capacity of leadership and self-organization) because we are not able to identify and augment support to the trends that might be leading to their empowerment.

Therefore, integrating a gender dimension into M&E will help:

 Integrating gender into M&E system will help to measure the extent to which a project has ad- dressed the different needs of men and women, and has made an impact on their lives and overall social and economic well-being

39 M&E Manual Guide for IFAD Funded Projects in Vietnam

 It will also improve project performance during implementation, allow for midterm corrections, and make it possible to derive lessons for future projects.

5.2. What will a gender sensitive Monitoring system monitor?

Gender-sensitive M&E will monitor the project’s impact on gender relations and women’s empowerment. Gender division of labor - Who is more responsible for working in the household, women or men? - When the project have been implemented, has male shared the job with female or female has to work more? Gender differences in access and control over resources (e.g. Income, employment, land, social services) - Who controls income in the household? - Who participated in job training more, female or male? And have the gender difference in applying the knowledge to household economics. - Who has written on the land use certificate more? And the gender difference in target of using land? - The gender difference in receiving social service: influence into female’s health and ability to access information. Gender differences in information and knowledge - Are there gender difference in accessing the same information (about amount of information and how to access) - The difference in amount of information made opportunities economic different between female and male Decision making patterns in the household and community, - Who in the household has the decision power? (compare with the contribution of female and male in total income: whether person contributing the most in total income has the decision power?) - The participation of female and male in activities of community (the voice and respectfulness opinions in community activities) Women and men’s attitude and self-confidence The gender difference in aware in project (object, target and activities), then evaluate gender difference in participating project activities. Women and men’s attitude and self-confidence:

The difference between female and male about self-confidence in all activities ( the participation and responsibility )

Gender differences in vulnerability and coping strategy

E.g. Differences in dealing problems and in adjusting to external shock.

5.3. How do we do gender and M&E

Gender and M&E should not only collect data, but:

40 How to set up M&E system

- Analyze the reasons for these changes and gender differences;

- Develop appropriate and feasible interventions;

- Make concrete pathways towards implementation of the recommended intervention.

How do we do gender and M&E in evaluating impact

- Basic principle for gender and M&E is to disaggregate all data by sex;

- Disaggregating by head of household is not enough, since if we interview only the head of household (who are often men) à Select both women and men respondents during sampling;

- Designing gender-based questions for survey, focus group discussion and in-depth interview.

6. M&E implementation responsibilities

In each province, the PCU will have the overall responsibility for the implementation and the coordination of the M&E activities. An M&E Unit composed of a Senior M&E Officer and a Knowledge Management Officer will be created within each PCU that will take the primary responsibility to meet the objectives described above. These staff will develop and operationalize the project’s monitoring and evaluation system for monitoring the progress of the project. They will also coordinate evaluation activities for assessing impact. The Knowledge Management Officer will also assume the responsibility to identify, document, and disseminate good practices and lessons learned that emerge from the project as well as to identify, obtain, adapt and distribute relevant best practices and lessons learned that have been generated by other IFAD projects in the country or other organizations. At the District-level, an M&E Officer within the District Agri-business Support Unit will implement M&E activities at the District-level and provide support to the Commune Development boards for gathering accurate M&E information. These M&E staff will be the focus of capacity building for M&E within the project.

In the first year of the project, the M&E staff will focus on establishing a functional M&E system for the project. This would include reviewing the results framework and indicators with decision-makers to ensure that they are still relevant, refining the M&E plan, assessing staff training needs on M&E, organizing training on M&E at all levels (Provincial, District and Commune), coordinating the baseline survey, designing reporting forms, developing databases, coordinating the social audit baseline, refining the social audit tools and process, and coordinating analysis of baseline data.

From the second year, the M&E staff will focus on timely compilation of monitoring information for decision-makers and reporting and conducting annual surveys. In the third year of the project, the M&E staff will coordinate the social audit and mid-term evaluations. 6.1. Senior M&E Officer in the PCU

In particular, the Senior M&E Officer in the PCU will be responsible for:

 Setting up the M&E system, including developing/adapting formats and defining processes as per the M&E framework;

 Providing guidance and direction to district/commune officers and implementing partners for effective implementation of M&E activities and regular reporting;

 Compiling quarterly and annual project progress reports;

 Organizing and coordinating the baseline survey, annual surveys, mid-term evaluation survey and

41 M&E Manual Guide for IFAD Funded Projects in Vietnam

final impact survey;

 Coordinating the communication of M&E results and information sharing with project staff, stakeholders, and implementing partners,

 Assisting DARD in developing a monitoring system including the forms and implementation (TQP); and

 Facilitating a participatory review of M&E outputs and the use of M&E information for sound decision-making aimed at improving project efficiency and effectiveness.

6.2. Knowledge Management Officer

The PCU also has a Knowledge Management Officer who is responsible for:

 Managing M&E databases generated by the project;

 Coordinating the identification and documentation of good practices and lessons learned being generated by the project;

 Ensuring that reports and other documents produced by the project are circulated to decision- makers, including the project steering committee, in a timely fashion;

 Managing a circulation list of stakeholders and other organizations and ensuring that documents produced by the project are disseminated as appropriate to parties listed;

 Responding to requests from decision-makers to obtain information on good practices and lessons learned from other projects or organizations.

6.3. District level

At the district level, the Monitoring and Evaluation Officer within the District Technical Support Unit will be responsible for:

 Collecting commune reports and aggregating them in district reports;

 Providing data and information on project activities undertaken at the district level;

 Supporting and assisting community facilitators in the implementation of M&E activities at the commune level;

 Assisting the PCU M&E unit in conducting annual surveys and the social audit; and

 Collecting information from the district line agencies to measure institutional indicators.

6.4. Commune level

Commune level data will be collected by the CDB, who will also engage project beneficiaries and other stakeholders in conducting commune-level participatory monitoring and evaluation activities and will provide support to survey teams carrying out the annual surveys, social audits and impact evaluation surveys. Project implementing partners, including government agencies and community based organizations, will play a crucial role in supporting the project M&E by providing relevant data on the interventions undertaken and the results achieved.

42 Measuring and reporting RIMS first and second level results

Chapter II.

Measuring and reporting RIMS first and second level results

43 M&E Manual Guide for IFAD Funded Projects in Vietnam

44 Measuring and reporting RIMS first and second level results

Chapter II. Measuring and reporting RIMS first and second level results

1. Overview

RIMS (Results and impacts measuring system) is a system for measuring and reporting results and impacts achieved by the projects. The system looks at three levels of results:

First-level results

First-level results correspond to the project activities and outputs - measure financial and physical progress; they are mostly quantitative and are reported on an annual basis. These indicators are measures of results at either the activity, or output levels of the logical framework (logframe). Project outputs can be measured through simple quantitative indicators. For example, the indicator “Number of people trained in livestock production” provides the necessary information for assessing the output of a training program during a given period: namely how many people have been trained. However, this indicator does not provide information on whether the training succeeded in transferring knowledge to the participants on a given topic (animal feeding), or whether it contributed to improved production technologies. These changes correspond to the subsequent step of the result-chain, namely project outcomes.

Second-level results

Second-level results: correspond to project outcomes - measure improved functionality or behavioral change, are more qualitative and normally take longer to realize. This level corresponds to either the output, or objective level of the project logframe. These correspond to ‘what happen next” to households, individuals, groups, communities, or institutions after the implementation of project activities and achievement of outputs. Measuring outcomes means analyzing changes in the behaviors of households and individuals, changes in the performance of groups and institutions, etc. These changes are very context specific and depend on the characteristics and objectives of the project. Various methods can be employed for measuring these changes, including studies, participatory approaches, questionnaire-based surveys, focus group discussions, etc.

Second-level results are included in the RIMS framework in the form of assessments. These look at the extent to which a given project activity was successful in reaching specific outcomes — assessment of effectiveness — and at the extent to which the benefits are likely to be sustainable after the end of project support —assessment of sustainability. RIMS second-level indicators can also prove useful to assess whether second-level results are being achieved.

Third -level results

Third-level results correspond to project impact - measure the combined effects of the first and second level results, are quantitative (e.g., households reporting increased assets) and are measured at three points during project life (benchmark, mid-term and completion). These refer to the goal or objective level of the project logframe.

This Chapter will provide guideline to measure and report the first and second level results. Measuring the third-level results by RIMS and RIMS+ survey will be presented at Chapter IV.

45 M&E Manual Guide for IFAD Funded Projects in Vietnam

Box 2.1. Results Chain Terminology

 Inputs: the financial, human and material resources necessary to produce the intended outputs of a project.  Activities: actions taken or work performed in a project to produce a specific output by using inputs such as funds, technical assistance and other types of resources.  Outputs: tangible immediate results that are produced through the implementation of activities.  Outcomes: short-term and medium-term effects of an intervention’s outputs.  Impact: long term positive and negative effects produced by a development intervention.

2. Why to measure and report the first and second level results?

The data gathered should be used to critically reflect on the extent to which expected results have been achieved. The findings of the analysis should be used to define corrective actions and make decision on improving the project strategy. The information can provide a useful basis for analysis of project performance and contribute to decision-making. Comparing targets of RIMS first-level results with actual achievement may lead to an analysis of factors that positively or negatively affected project implementation. The analysis of RIMS second-level results provides more in-depth information on the changes occurred at the level of beneficiaries, households, communities or institutions.

For example, after a training program on livestock veterinary practices, the small number of trainees adopting the recommended techniques may highlight problems with the training method, use of inappropriate language or inaccessible locations. Taking into consideration these findings, the project management may decide that in the future, training will take a different form or that another training service provider will be recruited.

3. Collecting information for measuring and reporting first and second level results

In order to assess whether first-level results have been achieved, a suitable system for gathering and managing information on project outputs needs to be included in the M&E activities. This may require the identification of information exchange systems between the project management and service providers or other stakeholders.

For example, in order to access information on the number of households receiving the project-sponsored vaccination services, a system of data exchange between project management and the veterinary or animal health centres responsible for the vaccination program should be established. Data on the number of active borrowers should be collected from the microfinance institutions or the banks participating in the project lending program.

In order to assess RIMS second-level results, studies, participatory and/or questionnaire-based data collection methods can be employed. For example, for assessing whether the intended objectives of a training campaign were reached, data can be collected from questionnaires, interviews with beneficiaries, focus group discussions, etc.

Information can also be collected from existing sources. For example, in order to assess whether the project has contributed to improve the financial viability of participating microfinance institutions or banks, the reports prepared by these institutions may be used.

46 Measuring and reporting RIMS first and second level results

3.1. Selecting first and second level results

The RIMS first and second level results have been grouped in the following categories: - Natural resources (Land and water) - Agricultural technologies and production - Rural financial services - Markets - Rural enterprise development and employment - Policy and community programming - Social infrastructure - Total outreach

Selection of first level results

The selection of first-level indicators (from the menu of available RIMS indicators) is made on the basis of specific project characteristics and does not depend on how the indicators are grouped. Only those RIMS indicators that are relevant to the project should be selected and reported.

The indicators should be grouped according to project components. As many indicators as appropriate should be chosen to describe the outputs achieved by the project during the period under review.

For example, if a component includes rehabilitation of rural infrastructure combined with initiatives of social mobilization, useful indicators can be found under the “Natural resources”, “Social development” and “Policy and community development” categories.

Relevant RIMS first-level indicators are often used in the project Logframe at the level of activities and outputs. Nevertheless, it is recommended that the entire set of indicators be carefully screened in order to determine those relevant to the project.

Every project is required to report the indicator showing the number of people and of households that received project services during the period under review. If relevant, the number of groups or communities receiving project services can also be reported.

Selection of second level results

As for the first-level, RIMS second-level results should be identified on the basis of the project characteristics and only those relevant to the project should be reported. Some RIMS second-level results can be traced back to first-level results.

For example, if the project reports the number of groups managing infrastructures formed/strengthened, the assessment of the likelihood that these groups will be sustainable should be provided. Similarly, if the project provides data on the number of people trained in post-production technologies, the outcome in terms of improved access to market opportunities should be assessed.

Other results cannot be traced back to a unique first level result. Several first-level results may contribute to one second-level result. For example, the improved performance of financial institutions may be the effect of training initiatives for staff at financial institutions, training of government regulatory staff; and strengthening of savings and credit groups.

The Logframe can be used to guide the choice of the relevant RIMS second-level results.

47 M&E Manual Guide for IFAD Funded Projects in Vietnam

3.2. Measuring and reporting first level results

3.2.1. Measuring first level results

Tip 1: For indicators about rehabilitation of infrastructure and facilities

 Indicators: The RIMS first-level indicators look at the following types of infrastructure: Irrigation schemes, livestock water points, rainwater harvesting systems, Fishponds, marketing, processing and storage facilities, roads, others.

 Measurement: all indicators listed above are based on the simple counting of the number of infrastructure/facilities that have been constructed and/or rehabilitated by the project during the period under review. The number should be calculated on the basis of the infrastructure/facilities where construction/rehabilitation works have been fully completed during the period under review.

 Methods: Initiatives where construction works are ongoing at the time of reporting should not be considered. Every construction/rehabilitation initiative should be reported in the period when it is concluded. Also, AWPB target should refer to the number of initiatives expected to be concluded during the period under review.

 Data collection: Beneficiaries and local authorities can be consulted for collecting updated information on the actual status of construction/rehabilitation works. Useful data can also be gathered from the technical documents related to the construction or rehabilitation process. Information contained in the payments and financial records of the Project Management Unit should also be used.

Tip 2: For indicators about training

 Indicators: RIMS first-level indicators provide information on the number of people (disaggregated by sex and ethnic, poor and non-poor) trained by topic: infrastructure management, natural resources management, crop production practices and technologies, livestock practices and technologies, fish production practices and technologies, post-production, processing and marketing, business and entrepreneurship, non-agricultural IGAs, vocational training, community management, etc.

 Methods: The number of people trained should be calculated regardless how the training was extended. Hence, the number of people trained (disaggregated by sex and ethnic, poor and non- poor) through study tours during a given period should be added to those trained in farmers’ field schools and seminars during the same period.

If training is extended to the same person over two years, the person participating in the training should be counted in both years. The cumulative figure should however show that only one person was trained although the project extended its capacity building efforts over a longer period of time.

 Planning: Planning target values of these indicators implies estimating the number of people (disaggregated by sex and ethnic, poor and non-poor) expected to participate in a given training events. Target figures should be estimated when budget decisions are taken. For example, the amount allocated for a training event may be established on the basis of the expected number of training participants. Hence, if 200 people are expected to participate at a cost of 100.000vnd per person, a budget of LCY 20.000.000vnd is allocated for the training. The figure of 200 people should be used as AWPB target.

 The comparison of expected and actual number of training participants allows assessing the relevance of the training topics. For example, if only 5 people participated in a training event out of 50 people expected, this may indicate low interest towards the training topic.

48 Measuring and reporting RIMS first and second level results

 Classification: The number of people trained should be reported on the basis of the classification and definition provided in each of the above indicators. For example, 100 people participated in a training campaign on farming practices that covered both crops and livestock. This initiative should be reported under either indicator of crops training or livestock training, but not both. Select the indicator based on the time spent per subject during the training.

A similar problem can be faced when the same group of people participate in different training initiatives. For example, during a given period, the same 100 farmers participated in two training initiatives: one on livestock practice and the other on crop technologies. In this case, project results should be reported separately as two training events took place. The project in fact actually trained 100 farmers on crop production and 100 on livestock practices. However, when the total number of ‘individuals receiving project services is calculated, the figure of 100 farmers should be used.

 Data collection: The choice of the most suitable method for collecting data on the number of people trained depends on how the training is implemented. If the training is executed by an NGO or a training institute, a system for information exchange should be established. For example, a short report may be prepared by the training institute at the end of each training session. This may provide information on the number of participants (sex disaggregated), and any other socio-economic characteristics required by the M&E system.

Tip 3: For indicators about group formation/strengthening

 Indicators: RIMS first-level indicators look at the number of groups formed/strengthened by the project. The following types of groups are considered: groups in charge of infrastructure management; groups involved in natural resources management; savings and credit groups; marketing groups; community groups, common interest groups.

Groups can receive support in different ways: technical, managerial or financial assistance can be provided. Activities for strengthening the managerial and operational capacity of groups include: direct on-the-job training, mobilisation of technical assistance, distribution of materials, participation in study tours, organisation of seminar and training events, etc.

For each type of group formed/strengthened, the number of people in these groups (sex disaggregated) and the number of these groups with women in leadership positions should be reported. The number of groups with women in leadership positions should not be higher than the total number of groups formed/strengthened during a given period.

 Planning: Annual target results often only deal with the number of groups expected to receive project support. Planning the number of people (sex disaggregated) in these groups and the proportion of groups with women in leadership positions can be more difficult. By using an average figure for group membership (number of people) and an estimate of the average male/female ratio per group, figures on group membership (sex disaggregated) can be extrapolated.

For example, savings and credit groups are formed on average by 10 members, 75% of whom are women. However, experience in also shows that few groups would have women in leadership positions (only about 20%). If project activities for PY2 expect to involve 100 savings and credit groups, planned figures for number of people in these groups would be: 1000 total people (100 groups x 10 members per group) or 750 female and 250 male members. Twenty savings and credit groups (20% of 100) are expected to have women in leadership positions.

49 M&E Manual Guide for IFAD Funded Projects in Vietnam

 Methods: If project support to a group is extended over more than one year, the group should be counted in all years for which it receives support. The cumulative figures should show, however, that only one group has been strengthened, even though support was extended over a longer period of time. Groups newly formed or strengthened by the project should be counted from the first year of project support.

Difficulties can be faced when the number of people ‘participating’ in the group is calculated. For example, a farmers group can have a restricted number of members but its activities involve a much broader number of people. A narrow definition of participation is suggested. Hence, only the members of the group should be counted.

Monitoring the participation of people in the groups formed/strengthened is relevant for assessing the likelihood that the groups would be sustainable at the end of project support (see Tip 6). Meet- ing with female members can help to identify factors that determine (in a positive or negative way) their involvement in project activities..

Tip 4: Access to services

 Indicators: The RIMS first-level indicators include various indicators aimed at providing information on how many people, groups or enterprises access the services facilitated by the project. These are: People accessing facilitated advisory services; enterprises accessing facilitated non-financial services; enterprises accessing facilitated financial services; People accessing development funds. If access to financial services is considered: Active borrowers; Voluntary savers.

 Planning: Planning target results implies estimating the demand for a given service within a given period of time. In the case of the rural finance indicators, ‘active borrowers’ and ‘voluntary savers’ offer a snapshot of the value of gross loan portfolio as at a certain date. For this reason, these are not applicable during the project planning phase (AWPB or appraisal). Indicators such as the number of loans expected to be disbursed during a given project implementation period are more appropriate when expected results are planned.

 Data collection: In order to access data on the number of people, groups or enterprises accessing the services, a system of information exchange should be established between project management and the institution, agency or enterprise that actually provides the service. For example, the service provider and the project management may agree on a reporting form that should be submitted in a regular manner showing the number of people accessing the service (sex disaggregated) and any other socio-economic characteristics required by the project M&E system.

In many cases, the information needed by project M&E system will be routinely collected by the service provider for its own purposes. Additional information required by the project may prove useful for the service provider. For example, the information collected through project M&E system may be useful to the service provider to better understand the characteristics of its clients, their satisfaction with respect to the service received, etc. If data are already routinely collected by the service providers, these should be used.

3.2.2. Reporting first level result

First level results are reporting annually. The first RIMS report should be submitted at the end of the calendar year of effectiveness. An Excel workbook should be developed for ease of reporting. The reporting of RIMS first level figures includes two sections.

50 Measuring and reporting RIMS first and second level results

First section

The first section refers to the annual expected result (reported in the Annual Working Plan and Budget (AWPB) for a given indicator, its actual value for the period under review and how this compares with the expected result.

 AWPB Target: This is the expected achievement of the first-level result during the period under review: e.g. the number of people expected to participate in project-sponsored training initiatives, the km of roads expected to be fully constructed/rehabilitated during the period, etc . The AWPB is the main information source for this figure.

If AWPB figures are not available, the cell should report “NA”. If the initiatives are implemented on a demand-driven basis, the cell should report “DD”. [Use of NA and DD is however discouraged.]

 Actual Result: This corresponds to the results actually achieved during the period under review (during PY1, PY2): e.g. the number of people that participated in the training initiatives implemented within this period of time, the kilometers of fully completed construction/rehabilitation roads, etc.

If an indicator is relevant for the project but no progress has been achieved during the period under review, the RIMS indicator form should show “0” (zero). All indicators should be reported in the measurement unit listed under the indicator chart (number, kilometers, hectares, male/female, etc.)

 Percent of AWPB: This is calculated by dividing the actual by the planned results in the AWPB.

Second section

The second section looks at the cumulative value of each indicator and how this compares with the expected (target) results reported in the Logframe or in the Appraisal documents.

 Appraisal Targets: This is the expected achievement of a given variable throughout all project years (from PY1 to Pin). The appraisal target should be taken from the project appraisal documents, cost tables or from the Logframe.

If target figures are not available in the Logframe or appraisal document, the cell should report “NA”. Lack of planned figures in the Logframe crucially hinders the possibility to use results information for decision making. For this reason, revision of the Logframe should be considered in order to improve its results- orientation and make it more useful for assessing project performance.

 Cumulative: The cumulative value is the sum of the values reported in all the previous years added to the value of current year. For example, in PY1, the project constructed 20 market facilities (Table 2.1). In PY1, the cumulative and actual values of the indicators are identical. In PY2, the number of market facilities constructed is 10. The cumulative value for PY2 corresponds to the number of market facilities constructed in PY1 plus the number constructed in PY2 (Table 2.2). The cumulative value for PY3 is the sum of market facilities constructed in PY1, PY2 and PY3, etc. Actual values for previous years are entered on the RIMS form to verify the cumulative value.

 % Appraisal: This is calculated by dividing the actual cumulative result by the target established at appraisal..

51 M&E Manual Guide for IFAD Funded Projects in Vietnam

Table 2.1. Example of reporting first level

Indicator Unit PY 1 Cumulative Actual % AWPB Appraisal Actual % Appraisal Target Market facilities Number 30 20 66% 50 20 40% constructed/ rehabilitated

Table 2.2. Example of reporting first level

Indicator Unit PY 1 PY2 Cumulative AWPB Actual % AWPB Actual % Appraisal Actual % Target AWPB Target AWPB Target Appraisal Market facilities số 30 20 66% 10 10 100 50 30 60% constructed/ lương rehabilitated

3.3. Measuring and reporting second level results

3.3.1. Measuring second level result

A new measurement approach for second-level results has been introduced. The achievement of second- level results is measured by a rating scale. A score of 1 corresponds to highly unsatisfactory effectiveness or very weak sustainability. By contrast, a score of 6 corresponds to highly satisfactory effectiveness or very strong sustainability.

Table 2.3. Rating Scale for second-level results

Score Effectiveness Assessment Sustainability Assessment

1 Highly Unsatisfactory. The intended results are Very Weak. None of the supporting factors are in highly unlikely to be achieved. No further resources place. Sustainability is very unlikely. should be committed until a new approach is devised. Consideration should be given to cancelling component/output. 2 Unsatisfactory. The intended results have not been Weak. Hardly any of the supporting factors are in achieved. Major corrections need to be introduced. place. Sustainability is unlikely. 3 Moderately Unsatisfactory. The intended results have Modest. Some of the supporting factors are been achieved to a limited extent. in place but they are not sufficient to ensure sustainability. Sustainability is unlikely. Corrections need to be introduced to improve performance. 4 Moderately Satisfactory. The intended results have Moderate. Some supporting factors are in place been partly achieved. Modifications should be but additional support is needed to ensure introduced to improve performance. sustainability. 5 Satisfactory. The intended results have been Strong. The most important supporting factors are achieved. in place. Sustainability is likely. 6 Highly Satisfactory. Intended results have been Very Strong. All supporting factors are in place surpassed. The implementation approach can be that will ensure sustainability. Sustainability is very considered as a best practice. likely.

52 Measuring and reporting RIMS first and second level results

The rating should be made on the basis of the project-specific data and information. Projects choose the methods for measuring second-level results that is most suitable to the local context and project circumstances. The methodology for assessing second-level results is therefore flexible and may consist of a mix of qualitative and quantitative methodologies.

The project M&E system is the primary source of the information used to determine the rating. RIMS second-level indicators provide useful data in making the assessment. Survey and other quantitative methods can be used as well as qualitative methodologies such as direct observation, focus group interviews, mapping exercises or case histories. Official statistics and reports prepared by government, donors or NGOs can also be used as evidence.

The annual stakeholders’ workshop may be a valuable opportunity to gather different points of view on project performance; ratings may be established or validated at such events. Each stakeholder brings different interpretations that can ultimately enrich critical reflection. A wide and participatory assessment would contribute to learning and enhance the overall reliability of the assessments.

Measuring tips

Tip 5: For measuring the sustainability of Infrastructure

 Methods: Various factors may hinder the possibility for beneficiaries to access the benefits of infrastructure construction/rehabilitation initiatives implemented by the project. Table 2.4 provides a list of factors that should be taken into consideration in assessing whether the infrastructure is likely to be sustainable after the end of project support. A list of guiding performance questions is also provided. The list should be expanded to take into account the characteristics and the context in which the project operates. It is important to take into consideration the linkages existing among the various supporting factors, e.g., technical factors (such as maintenance) depend on the financial viability of the groups or organisations in charge of infrastructure management.

Table 2.4. List of performance questions for each factor

Aspect Factor Performance question and data

Financial

Users association Ability to collect user fees Are mechanisms in place? Do members pay regularly? Are funds for maintenance available?

National/local Ability to finance Shortfall between actual and budgeted expenditures? authorities Willingness to finance Likely government priorities

Institutional

Users Association Sustainability Frequency of conflict; exit rate Will the association continue to exist without external project finance?

Capacity Are the skills required for infrastructure management in place? Are members adequately trained for maintenance? Are equipment for maintenance and management available?

53 M&E Manual Guide for IFAD Funded Projects in Vietnam

Aspect Factor Performance question and data

National/local Capacity Are the skills required for infrastructure management and/or authorities supervision in place?

Existence Is infrastructure management/supervision under a special unit? Will it continue to exist?

Technical

Design Soundness Are the structure sound? Do they have structural problems? Where high-quality materials used for construction?

Operation and Ability Do those responsible have the required skills for operation maintenance and maintenance?

Environmental Are the environmental consequences undermining the sustainability of project benefit? Is the location at risk of erosion?

Social

Use Are people using the infrastructure?

Participation Are beneficiaries involved in maintenance and management?

 Collection: Data relevant for formulating this assessment can be gathered from various sources: institutions or groups managing the infrastructure can provide important insights on the factors affecting sustainability. It is however important to keep in mind that these may have a vested interest in under (or over) estimating performance figures, threats, risks, etc. The assessment may also be based on studies undertaken by engineers, institutional specialists, etc.

Tip 6 For measuring the sustainability of groups

 Methods: In order to assess whether the groups formed/strengthened by the project are likely to be operational after the end of project assistance, a set of indicators providing information on the performance and functioning of the groups (or a sample of them) should be identified.

 Indicators may include: number of meeting held, number of participants, number of people exit- ing from the group, number of intra-group conflicts can be monitored, financial selfsufficiency.

Quantitative indicators can be enriched with qualitative considerations related to management capabilities, support from partner institutions, etc. Baseline data may provide information on the status of the groups before project support. Monitoring of the groups (by project staff or contracted service providers) should show whether changes have occurred and how these changes affect the likelihood that a given group will be sustainable after the end of project assistance. For example, in a given group, a decrease in the number of meetings held may be interpreted as an indication that the members are not interested in the activities of the group. The second approach does not require the regular monitoring of performance indicators. The sustainability of the groups can be assessed during participatory initiatives with group members and stakeholders.

54 Measuring and reporting RIMS first and second level results

Table below provides a framework of factors and guiding performance questions that can be used for formulating this assessment.

Table 2.5. A framework of factors and guiding performance questions

Aspect Factor Performance Questions and Data

Internal Social support Number of members that do not renew membership

Number of new members

Number of intra-group conflicts

Number of meeting held

Financial Financial self-sufficiency

Capacity to collect fees

External Support from partner institutions Are partner institutions willing to support groups

3.3.2. Reporting RIMS Second-Level Results

The achievement of second-level results is reported on the basis of a rating scale from 1 to 6. A score of 1 corresponds to highly unsatisfactory effectiveness or very weak sustainability. A rating of 6 corresponds to highly satisfactory effectiveness or very strong sustainability. IFAD encourages projects to honestly assess second-level results taking into consideration all available information. A low score does not mean that the project management is not committed to achieve development objectives. Negative results should be used to identify corrective actions and therefore increase the likelihood that development objectives will be achieved.

The rating should be reported under the column of “actual” for the period under review. No other value or number should be shown in the RIMS form for second-level results. The section on cumulative and appraisal values does not apply to RIMS second-level results.

When the RIMS form is submitted (normally an Excel workbook), supporting evidence used for the assessments should be included in an additional worksheet or separate text document(s). Clear indications of the methodology and source should be provided. The supporting evidence should be in summary form. Where studies or surveys have been carried, it is not necessary to submit the entire study/survey but clearly reference these as source material. The evidence may include RIMS second level indicators which should be reported as evidence under the relevant second level result.

RIMS second-level results should be provided at the time of Mid-term Review (MTR). The ratings of RIMS second-level results should be updated every year on the basis of the new data and evidence available.

55 M&E Manual Guide for IFAD Funded Projects in Vietnam

56 How to conduct annual outcome survey

Chapter III.

How to conduct annual outcome survey

57 M&E Manual Guide for IFAD Funded Projects in Vietnam

58 How to conduct annual outcome survey

Chapter III. How to conduct annual outcome survey

1. Introduction

1.1. What is the annual outcome survey and why to do it?

Whereas it would be ideal if all IFAD-funded projects would be able to regularly collect and analyse outcome and impact information from their entire beneficiary population (for example through participatory M&E or census-based surveys), one also needs to recognize that this is, in most cases, highly challenging.

While IFAD has developed a standard methodology for impact measurement2, these impact surveys are not providing the type of results’ information that can allow Project Management Teams to take timely, corrective action during the course of project implementation. Such impact surveys are indeed primarily intended to document project impact at completion.

In an effort to shift the focus from impact documentation at completion to outcome measurement during project implementation, IFAD is now encouraging all its projects in the Asia and the Pacific region to survey annually a small sample of beneficiaries in order to:

(i) measure more regularly the positive or negative changes/outcomes taking place at the household level;

(ii) provide early evidence of project success or failure;

(iii) provide timely performance information so that corrective actions may be taken if required;

(iv) assess targeting efficiency.

The Annual Outcome Survey (AOS) is therefore a simple household survey that will be undertaken annually by project staff and will cover a small sample of 200 households selected randomly in project areas (project beneficiaries) and 200 households selected randomly in non-project areas (non-beneficiaries, to be used as a comparison group).

Overall, the exercise is expected to take no more than 3 months and can be implemented by project staff and extension officers, with or without external support.

1.2. How and when to conduct the survey?

Annual outcome surveys should be conducted every year starting from the 2nd year of implementation.

The survey should be implemented in conjunction with qualitative assessments that will complement the household-level information, providing information on “why” and “how” some outcomes were or were not achieved. This means that in each village, in addition to the household interviews, the team should conduct a certain number of focus group (FG) discussions and key informant (KI) interviews (see Chapter VI and Chapter VII for more details on qualitative assessments).

2 Namely the Results and Impact Monitoring System (RIMS) which includes the organization of mandatory, standard impact surveys called RIMS impact surveys - See IFAD website: http://www.ifad.org/operations/rims/index.htm)

59 M&E Manual Guide for IFAD Funded Projects in Vietnam

The figure below provides an overview of the AOS design. The figure shows that the field data, for the treatment group, the collection teams will go to 20 villages, and in each village they will conduct 10 household interviews as well as focus groups and/or key informant interviews. The number of qualitative interviews to conduct will be decided by the project (number in the figure below given as examples). For the control group, it is necessary to collect the quantitative data by the survey only, it is not necessary to conduct FGDs and KIIs.

Figure 3.1 An overview of the Annual Outcome Survey design

In some cases (e.g. the first year you will conduct an annual outcome survey), it is recommended to conduct Focus Group Discussions and Key Informant Interviews after the survey is completed and results analysed. Hence, it could be that certain initial findings will prompt you to want to better understand certain reported problems.

Box 3. 1. Example for the necessary of Focus Group Discussion

For example, if survey results show that 95% of farmers are not satisfied with project services, and since the standard questionnaire does not include open-ended questions such as “why are you satisfied/dissatisfied?”, you may want to organize Focus Groups Discussions with beneficiary farmers in order to understand the reasons of their dissatisfaction.

2. How to prepare and conduct an annual outcome survey?

To conduct the Annual Outcome Survey, the following steps are undertaken:

1) Designing questionnaire, interview guide and pilot;

2) Sampling;

3) Recruiting enumerators, team leaders and supervisors; training;

4) Logistical planning, preparation for data collection;

5) Data collection;

6) Data entry;

60 How to conduct annual outcome survey

7) Data analysis;

8) Writing report

Some activities may not necessarily be implemented chronologically, they can be conducted in parallel with others. For example, sampling, questionnaire design, enumerators recruit can be performed simultaneously; data entry form could be designed right after questionnaire is finalized. In the following parts, the detail for each step will be presented.

2.1. Step 1 - Fine-tuning the standard questionnaire

2.1.1. Questionnaire design

A standard questionnaire is provided by IFAD and presented in Annex 2. Some parts of the questionnaire (see text highlighted in yellow) must be adapted by the project staff according to the specificity of project interventions. All sections of the questionnaire can either be used as they are, or they can be modified according to the specific information needs of the project. The administration of the questionnaire should not take more than 30 minutes per household. .

Requirement for a good questionnaire

The questionnaire plays a very important role in every survey. A good questionnaire is one of the factors contributing to the success of a survey. A good questionnaire matches following requirements:

- The questionnaire can collect sufficient information for assessing outcomes (and impact) of project;

- Collected information must present project’s objectives and indicators;

- Questionnaire has appropriate length, it has to cover all aspects as desired but the time for interview should not be too long;

- The questions must be easy to ask and answer: try to avoid misunderstanding the questions otherwise it can lead to incorrect answers;

- Questionnaire has appropriate structure and clear format

Steps to design questionnaire

Step 1: Study carefully project documents to understand the project’s objectives, activities, the logical framework and indicators for monitoring and evaluation.

Step 2: Identify the indicators which can be collected through the Annual Outcome survey. Make a list of indicators that the Annual Outcome survey needs to assess.

Step 3: Study the template questionnaire of IFAD: identify the sections and questions that can be adapted directly to the project; determine which questions need to be adjusted and added. It is good idea to find and learn from similar available survey of other projects or the questions that were used in relevant surveys, such as the baseline survey (See the template questionnaire of IFAD in Annex 2).

Step 4: Develop new questions to collect information for the indicators. Gender issues should be taken into account when designing the questions.

61 M&E Manual Guide for IFAD Funded Projects in Vietnam

Box 3.2. Some points should be kept in mind when designing additional questions

 Each question should have a specific purpose; avoid questions that overlap or duplicate each other or questions with an unclear purpose  The question must not give any hints, bias or lead respondents to any answers  Have clear instruction for answering the questions or skipping questions.  Wording with local language, avoid using acronyms or jargon that are multi-meaning, unclear or abbreviated

Step 5: Structuring the questions into sections in a logic order. The questions are divided into sections, each of which focuses on a single theme on project’s activity or objective. Questions should be numbered in each section for the convenience of coding and data entry.

Step 6: Questionnaire presentation: Each section should be designed to fit on one page or less. Avoid breaking a question in 2 pages, in a necessary case, move both the questions and the answers to a new page; text size and style should be easy to read, instructions for skipping questions should be highlighted and placed in a visible place, usually to the right of the question.

2.1.2. Develop an interview guide

Interview guide provides detailed instructions to help enumerators understand: (i) purpose of each question; (ii) how to raise the question; (iii) how to record the answer; and (iv) how to deal with different situation raised during the interview. Interview guide is provided to every enumerator during the training course and for the survey. When developing the interview guide, it is necessary to take into account the following issues:

Content of the guide: In the introduction of the interview guide, it is essential to indicate general rules for interviewing and recording the answers. In the next parts, the interview guide explains in detail the content, purpose of each question; provides definitions, solutions to special cases and gives example if necessary. Interview guide must be specific and detailed.

Presentation: Interview guide must be clearly displayed on the content of each question and each session. In special case, it is essential to clear annotation (bold, italic, underline) so the enumerator can remember easily.

2.1.3. Pilot

The purpose of pilot is to assess the feasibility of the survey, consider the issues in the questionnaire need to be revised. Sample size for pilot in this case is about 5-10 households. Households chosen to take part in the pilot should be involved in all activities mentioned in the questionnaire so that all content can be tested.

Box 3.3 Questions to be answered by pilot

1. Does each question measure what it is supposed to measure? 2) Are all the words understood? 3) Do all respondents interpret the question in the same way? 4) Are all response choices appropriate? 5) Is the range of response choices actually used? 6) Do respondents correctly follow directions? 7) Does it create a positive impression that motivates people to respond? 8) How long does it take to complete? 9) Does it collect the information you want?

62 How to conduct annual outcome survey

2.2. Step 2 - Selecting the sample

Sampling is one of the most important steps in the Annual Outcome Survey. Sampling directly affects to the result of survey. If the sample is representative, the survey will provide reliable information about results and impacts of project; if the sample is not representative, the survey’s result can not reflect correctly the project’s results.

Sample size for Annual Outcome Survey is 200 households in the project area and 200 households in the non-project area (including 20 clusters; 10 households at each selected cluster). The sample can be replicated every year for the Annual Outcome survey.

2.2.1. Sampling methodology

The sampling method applied to IFAD projects is two-stages sampling method.

Stage 1: Selects clusters (20 villages in the project area and 20 villages in the non-project area)

Select villages in the project area

Step 1: Establish the sampling frame - List all villages of project communes.

Step 2: Calculate the sampling interval (SI), using the formula:

SI = Total number of villages/20

(In which, SI is ‘Sampling interval’; ‘20’ is number of villages to be selected.)

Step 3: Select a random number (R) between 1 and SI.

To select a random number, you can use the function in Excel ‘Randbetween’ (type “=randbetween (1, SI)”) in a cell. The other way, we can type “=RAND()*SI”, in which SI is the sampling interval.

Step 4: Select villages

The Rth village from the sampling frame (corresponding to the random number –R) is the first village selected. Add the sampling interval–SI to select the 2nd village. Add again SI to select the 3rd village. Con- tinue until you reach 20 villages..

Select village in the non-project areas

Step 1: Select 10 non-project communes

The control group (in the non-project) should have similar socio-economic conditions as the treatment group. To select these communes, list all non-project communes with information on the poverty and percentage of ethnic minority (the two criteria used to select treatment communes). Choose 10 communes which have most similar conditions to the project communes.

Step 2: Select 20 villages from 10 selected communes

The sampling methodology used to select the villages for control group is applied to choose the villages for the treatment group. Select randomly 20 villages from 10 selected communes.

List all villages of 10 selected communes and replicate steps presented above to select 20 villages.

63 M&E Manual Guide for IFAD Funded Projects in Vietnam

Stage 2: Select the households (200 households for the treatment group and 200 households for the control group)

Select 10 households from each selected village (20 villages in the treatment group and 20 villages in the control group) for the Annual Outcome Survey. It is recommended to select 05 stand-by households in case the households on the formal list are unavailable for interview or absent from home.

There are several ways to select the households (applied to both treatment group and control group) as followings.

(i) Systematic Random Sampling (list of all households in the selected villages are available in softcopies): identify the SI and then using function “randbetween” in Excel.

(ii) Random Sampling (list of all households in the selected villages are available): using function “data analysis” in Excel to randomly select 20 numbers with name of household’s head (softcopies of list are available) or select the 20 numbers only then match with the name and order (number) of the households in the hardcopy lists.

(iii) Random Sampling (list of all households in the selected villages are not available): Identifying the approximate centre of the village. A pencil or bottle is spun to select a random walking direction (also called a transect line). Count the number of households encountered along the transect line between the centre and the perimeter of the cluster (end of the village). This number is divided by 10 (total number of households needed) to determine the interval at which households will be selected in the transect line. When the households located in the transect line less than the number desired, all the households will be interviewed. In this case, come back to the village center for a second random walking direction and repeat the process.

2.2.2. Stand-by households

During the survey, the households selected in advance may be not available for interview or they refuse to take part in the survey. In this case, it is necessary to have stand-by households to replace. In the An- nual Outcome Survey, the number of stand-by households should be equal or half of needed households (05-10 households). The stand-by households are selected by using the similar method.

2.2.3. Respondent selection - gender balance

In order to ensure the gender balance in the survey, also to collect information from both gender groups, it requires at least 30% interviewees are women or men in each cluster. Therefore, it is better to specify the interviewee from the household in advance. The local supported staffs (commune staff and village staff) play a very important role in specifying respondent and informing households about the interview.

2.3. Step 3 - Enumerators’ training and field testing of the questionnaire

2.3.1. Recruiting enumerators, team leaders and supervisors

In the Annual Outcome Survey, the role of enumerators, team leaders and supervisors are very important. Ideally, each group has 01 supervisor, 01 team leader and 05-06 enumerators. Besides, it should have female and male enumerators in each group to ensure gender balance and facilitate the interviews.

64 How to conduct annual outcome survey

Enumerators

Enumerators can be project staff at different level (provincial, district or commune level). Enumerators must be trained carefully and be competent in interviewing. In addition, survey teams working in areas with ethnic minorities should each have enumerator(s) from the relevant ethnic minorities. It is recommended that female enumerators will conduct interview with female respondents and male enumerators will conduct interview with male respondents.

Box 3.4. The duties of enumerators

 Coordinate with other members during fieldwork to ensure the survey is running properly and on schedule;  Complete all parts of the questionnaire, and checking the filled-in questionnaire to make sure each the answer to each question has been recorded fully, clearly and correctly;  Re-interview if requested by the supervisor;  Fully take part in the evaluation sessions with the supervisors and ask them for advice if any problems occur.

Survey team leader

The survey team leader should be M&E officer at the provincial or district level with knowledge of the survey area. The team leader is responsible for planning, assigning work, ensuring the survey progress smoothly and the safety of team members.

Box 3.5. The duties of the enumerator team leader

 Guarantee the best preparation for the field test: organize and schedule interviews, contact the area that is to be surveyed, prepare the questionnaire, prepare other logistics and especially ensuring the safety of enumerators during the fieldwork.  Delegate assignments for the enumerators: the assignment of work needs to be suitable for each enumerator.  Coordinate all activities of the team, examining the quality of collected questionnaires.  Keep closely in touch with supervisor when carrying out the field work.  Set interview goals every day for the enumerator team (based on experience and local conditions of the project area).  Provide logistics for the survey.  Monitor the progress of work, reporting to supervisor at the end of the day.

Supervisors

Supervision is very important activity which heavily contributes to the success of a survey. Ideally, supervisors should not be project staff but external resources to ensure the objectivity and accuracy of the survey. In the absence of sufficient financial resources to hire external supervisors, the M&E officers from PCU should supervise the survey. Supervisors are responsible for ensuring the quality of the information obtained by the enumerators and the survey overall. The supervisors report to the project manager about the survey process.

65 M&E Manual Guide for IFAD Funded Projects in Vietnam

Box 3.6. The duties of supervisors

 Supervise the interviewing process: the supervisor will observe interviews, ensure the interviews run smoothly and record common mistakes made by enumerators to avoid mistakes in next interviews.  Guiding, answering and explaining to enumerators in case of any misunderstanding or lack of interview skills  Ensure the progress of work: keep in touch with and give daily updates to enumerators and use the monitoring progress map to keep track of the progression of work as well as to make sure that the daily amount of work done is consistent with initial estimates.  Check all answer sheets: check responded information, ensure that there is no missing in the questionnaire, check for mistakes made in writing the answer code and questionnaires. If the questionnaire has any vague information, the supervisor needs to require enumerators to provide more details or to re-interview. The mistakes made in the questionnaire must be recorded in the check answer sheets record. (see Annex 9).  Review and learn by experiences everyday as well as disseminating the schedule and plan for the following day.  Be responsible for retaining all questionnaires and handing them over to the Survey Team Leader for data entry.

2.3.2. Training

Before conducting the survey, all enumerators, team leaders and supervisors should be trained carefully about the survey and necessary skills as well as their responsibilities.

The training will cover following issues

(i) The purposes of the Annual Outcome Survey and steps to conduct;

(ii) Structure of the questionnaire; the purpose and content of each question;

(iii) Interview skills, how to raise the questions and how to record the answers;

(iv) Sampling method;

(v) The role and responsibility of enumerator, team leader, supervisor;

(vi) How to enter and analyze data.

Training materials:

(i) Lectures;

(ii) Questionnaire;

(iii) Interview guide;

(iv) Data entry form and software for entering and analyzing data;

(v) Guidance for data entry and analysis.

The tentative duration of the training is six days, including one day for the enumerators to practice at the field. After the training, enumerators and other members will have a deep understanding about the process, method and tools of the survey.

2.3.3. Field-test

The field-test will be conducted after all enumerators participate in questionnaire training. This can be conducted in one or two villages.

The purposes of field-test

66 How to conduct annual outcome survey

(i) Help enumerators to familiarize themselves with the questionnaire and practice the interview skills they have learnt;

(ii) Help supervisors and team leaders practice their tasks such as: planning, arranging and assigning enumerators to interview households, managing the survey and checking the information collected in the questionnaires;

(iii) Identify mistakes in the questionnaire for finalization;

(iv) Identify unexpected situations in practice to share with all enumerators and supervisors for resolution.

Preparation for field-test

- A list of households to be surveyed should be prepared in advance; if apply the sampling method of walking randomly, decision on which direction to go, the distance between interviewed households should be discussed and agreed before going to the field.

- Planning and inform local officials about the field-test in order that they help to prepare the list of households for interview. The notification should be completed at least one day before the field test for a good preparation. The list of households to be interviewed should be provided to the enumerators, team leaders and supervisors.

- Villages selected for the field-test may be in the sample list, but ideally, they should not be in the sample as if there is any significant adjustment of the questionnaire then it is not necessary to come back for re-interview. The villages selected for the field-test should have similar characteristics to those villages that will be surveyed.

- It is necessary to have a plan for enumerators allocation. All enumerators will be divided into pairs, each pair will be assigned to interview the specific number of households.

- Transportation, time and gathering place should be informed and agreed with all enumerators. Re- garding transportation, enumerators may use their personal vehicles by arranged bus to the field. Hiding bus will be costly but it is safer and avoid time-wasting. Regarding time, all enumerators should be informed about the time to meet before and after completing the interview.. All interview test should be completed in the morning. Regarding gathering place, the gathering place can be at village center, head of village house or the Commune People’s Committee office. The gathering place should be the most convenient place for all enumerators.

- Prepare all necessary documents including the list of surveyed households, questionnaires, manual guides, notebooks, pens, receipts, travel warrant.

Practice interview

- Each pair of enumerators will be assigned to interview 01-02 households. The enumerators will take it in turns interviewing each household, and during the interview process enumerators will take notes of any mistakes in the questionnaire or unexpected issues.

- Team leader will be responsible for assigning interviewees and keeping in contact with the enumerators and the supervisors in the field test in order to solve any potential problems (for example, a change of sample because some households are unavailable, or refuse to participate in the survey).

- The supervisor will supervise some pairs f the enumerator. After completing supervision of the first pair, the supervisor can come to the other pair. While observing the interview, supervisors should

67 M&E Manual Guide for IFAD Funded Projects in Vietnam

take notes about errors in the questionnaire or interview mistakes of the enumerators.

- When finishing the interviews, the enumerators should clean the questionnaires and handover them to the team leader, who will be responsible to review the questionnaires before giving them to the supervisor for checking. Questionnaires collected from the field test will be used for data entry and analysis training course.

Experience sharing and lessons learnt after the field-test

After the field-test, all enumerators, team leaders and supervisors will return to the class to sharing interview experience and lessons learnt. The questionnaire and interview guide will also be finalized.

2.4. Step 4 - Logistical planning

One of the factors that make a survey successful is good logistics. Here are the logistical preparations for the survey.

Scheduling the survey

The survey schedule must take into account the available resources (finance, human) and the production, lifestyle, culture and customs of surveyed community. The survey should avoid harvest times or holi- days. Also, it is necessary to consider the topography and weather conditions of the survey areas. If the enumerators are few, or the households are in remote areas, difficult and take time to get there, the time for completing survey could be longer. In addition, it may be necessary to have plan for accommodation for enumerators to stay at the village until completing the survey.

Local assistant staff

The PCU should inform officially the local government about the survey in advance. The local government should provide support to facilitate the survey, such as making list of households, inform households about the survey and take the enumerators to the households, and do the translation in case of necessary.

Village Head

Team leader should contact and provide a list of survey households to the Village Head so that he/she will help to inform the households in the list staying at home for interview. This will help to increase the response rate by increasing the chance that the representative of the survey households will be at home to participate in interviews. The Village Head could also be a guide (take the enumerator to where the survey household located), translator for the interview.

Local Guide

Because surveys are often conducted in remote areas where households are far apart, it is useful for the survey team to have somebody to take them to the selected households. These guides could be the village head, deputy village, a chairwoman of Women Union or a person assigned by the village head, who knows the about village area and residents. The number of guides depends on the areas, the number of enumerators and funding for the guides. They will help allocate the households into groups (e.g. on same rout) to facilitate the travel between households of enumerators.

68 How to conduct annual outcome survey

Interpreter

Most of projects are focus providing support to poor ethnic minority households and in many cases, these people might not about to speak influent Vietnamese. In this case, interpreters are necessary and should be taken into account when planning the survey. The interpreter could be local officials or enumerators who can speak the interviewees language.

Transportation

Project areas usually are remote and mountainous places. Therefore, the transportation should be take into account when the survey is planned. The enumerators could use their own vehicles or could be provided transportation by the project.

No matter what type of transportation is used, it should be in a good condition and budgeted. If enumerators travel by themselves, petrol cost could be calculated according to the distance to each area.

Survey Materials

Each enumerator needs to be equipped with a full package of questionnaire, interview manual guide, the list of households for survey, pen and notebook. In addition, it is necessary to prepare other documents such as travel warrants; allowance and receipts for the guide or interpreter; gifts/money for the respondents and receipt; answer check sheet and the progress monitoring form.

Budget estimation

Estimating the budget is very important, it helps to allocate money for each specific activity. All items should be taken into account when make a estimation of budget, such as allowance for enumerators and supporters, transportation, gifts for respondents, candies for kids, etc.

2.5. Step 5 - Data collection

Survey on a large scale is a main assignment of whole survey because this task decides the quality of survey. Besides, supervision also plays a very important role in the success of the Annual Outcome Survey.

Data collection process

All enumerators will be provided questionnaires, manual guide for interview, household lists, receipts for households and enumerators, travel warrants.

Each project supporting unit at district level might be in charge of conducting the survey at their locality. The project supporting unit at district should keep in mind the following points:

- Each district will schedule the survey based on the time, survey areas, number of enumerators. The plan should be consistent with the overall survey plan of the whole project. The team leader will be in charge of the progress of the survey at his/her locality.

- The safety for enumerators should be taken into account, such as: transportation, meals, accommodation.

- Every day, all completed questionnaires must be handed over to the team leader. Team leader will check the number of questionnaires collected and the quality of the completed questionnaires before handing them to the supervisor. After careful checking, supervisor will give back the questionnaires to the team leaders. The team leader will keep the checked questionnaires for entering data.

69 M&E Manual Guide for IFAD Funded Projects in Vietnam

Supervision

As mentioned above, a successful survey depends heavily on the supervising. This is one of the most important activity that determine the quality of survey. Supervisors and team leaders are responsible for the quality and the progress of the survey. Each data collection team must be accompanied by a supervisor during the whole data collection process.

2.6. Step 6 - Data entry

2.6.1. Data entry form design

The data entry form can be designed right after the questionnaire is finalized. Many software can be used to enter data, however it is recommended that CSPro should be used for data entry because of its strong advantages. CSPro is a user-friendly mechanism for double-checking and entry error detection. CSPro is fully designed as a questionnaire that facilitates the process of data entry as well as minimizes simple mistakes during the process such as missing, mismatching, or misreading data. Furthermore, CSPro allows different users to combine input data and so permits the simultaneous cooperation and participation of these different users; additionally, could be able to cooperate simultaneously; it can also find mistakes automatically by using a double-check command.

Data from CSPro can be easily transferred to other statistical software such as Excel, SPSS or STATA for further processing and analysis.

2.6.2. Data entry

Step 1. Data cleaning

All questionnaires will be checked and cleaned by the supervisors and data entry staff. If there is any missing or unclear information, the enumerators may be requested to add or clarify. In case of necessary, the enumerator may be requested to come back for re-interview. After the questionnaire is cleaned or clarified by the enumerator, the supervisor has to re-check the questionnaire before handing over to data entry staff.

Step 2. Entering data

The questionnaire received by data entry officials is cleaned questionnaire. Each data entry staff will be responsible for a number of questionnaires assigned by the team leader. All questionnaires will be entered twice by two different staff.

Data entry officials are responsible for entering data into the computer, therefore, in addition to understand the software used for data entry, they should meet the following requirements:

- Understand the content of questionnaire and its structure and logic;

- Adhere to time schedule laid out by the team leader/supervisor;

- Preserve all questionnaires received from team leader/supervisor.

The data entry process also should be supervised under a team leader or supervisor.

Step 3. Compare Data

There should be two data entry officials inputting a same set of questionnaire. This is to compare and reduce subjective errors and CSPro allows to do this action.

70 How to conduct annual outcome survey

Step 4. Concatenate database

In case that each district is in charge of entering their data, then it is necessary to concatenate the data from all surveyed districts to be a unique database. So this step could be easily undertaken in CSPro, using the Concatenate data function in Tools.

Step 5. Export to SPSS/STATA/EXCEL

After completing the data entry process using CSPro, the database could be exported for analysing by professional statistic software, such as SPSS, STATA or EXCEL, upon the analysist.

2.7. Step 7 - Data analysis

Entered data will be exported into an analyzing software. When analyzing data, the following requirements should be met:

- Ensure comparison between the treatment group and the control group; between male-headed households and female-headed households; between male respondents and female respondents; between ethnic minority households and Kinh household;

- Analyze the correlation between relevant issues;

- Comparison with the result of previous survey;

- The analyzing indicators must link to the project targets.

For data analyzing, it is recommended using SPSS because of its simpleness. The results will be inputs for report writing.

2.8. Step 8 - Report writing

The Annual Outcome Survey report is mainly based on analyzed data in the survey. However, the report also use the information obtained from KII and FGD to have in-depth analysis and explanation for the findings of the survey (see the Chapter VI on Key informant interview and Chapter VII on Focus group discussion).

The report must address following requirements:

In terms of content

- The report should cover the project’s objectives, analyzing the identified indicators;

- The report should clarify project activities which have been implemented;

- The report should emphasis the different results of the project, the comparison between the treatment group and the control group, as well as different targeted groups of the project (e.g. women, poor households, or ethnic minorities);

- The emphasis should be on the meaning and implications of the data analy¬sis for the project, not on the methodological aspects;

- Ensure the clarity and coherence of the structure, content, vocabulary and logic between para- graphs and sections.

71 M&E Manual Guide for IFAD Funded Projects in Vietnam

In terms of presentation

- In addition to the main content, the report should also have a list of acronyms, a table of contents, and list of annexes. Table of contents and list of acronyms should be at the beginning for reference, and the annex should be at the end of the report;

- The structure of the sections should be clear and logic;

- Surveys results should be presented first and foremost through charts/graphs, and the accompanying text should only highlight the most important findings from the charts.

- When preparing charts/graphs, be careful with the use of colours, as the charts/graphs will not be readable if the report is printed in black and white (not everyone has a colour printer!). Instead of using colours, use different “textures” for the charts.

Before drafting the report, it is necessary to design a detail report outline. From the outline, a full report will be developed (see suggested outline for the Annual Outcome Evaluation report in Annex 4).

After a draft report is prepared, the PCU’s director board will organize a workshop to share the findings and collect comments on the draft report from project staff and relevant stakeholders. After that the report will be finalized and ready for sharing at a wider community.

72 How to conduct RIMS and RIMS+ baseline and end-line survey

Chapter IV.

How to conduct RIMS and RIMS+ baseline and end-line survey

73 M&E Manual Guide for IFAD Funded Projects in Vietnam

74 How to conduct RIMS and RIMS+ baseline and end-line survey

Chapter IV. How to conduct RIMS and RIMS+ baseline and end-line survey

1. What is the RIMS and RIMS+ survey and why to do it?

RIMS (Result and Impacts Measuring System) is a comprehensive system for measuring, aggregating, and reporting on the results and impact of IFAD supported country programmes and projects. To measure the impact of IFAD support, all projects funded need to conduct a RIMS impact survey from the beginning to the end of the project based on the IFAD standards. The RIMS survey has two basic parts:

(a) The household assets survey: this forms the basis of the Household Asset Index which cap- tures the relative wealth of survey respondents.

(b) The child malnutrition survey: Based on standard methodology (measurement of children’s height and weight), this survey measures three main variables: chronic malnutrition (stunting relative to height-for-age); acute malnutrition (wasting or weight-for-height); and proportion of children underweight (weight-for-age).

However, while international research has shown that change in the levels of chronic malnutrition is a practical indicator of the impact of poverty reduction programmes and that any project activities that increase household income have the potential to decrease chronic malnutrition, recent IFAD experience in the Asia and Pacific region also shows that there may be external events that may adversely impact acute and chronic malnutrition (for example if the RIMS completion survey is undertaken the same year of a severe drought). If this happens, wrong conclusions could be derived as per the lack of a positive project impact.

Moreover, most IFAD-funded projects are multi-sectorial interventions which cover a wide range of sub- sectors (natural resources management; agricultural development; irrigation; infrastructure; livestock; microfinance; community development; etc. ), it is felt that the scope of the standard RIMS impact surveys may be too limited to reflect the variety of impacts that IFAD-funded projects may have at the household or community level.

It is therefore recommended that, while conducting the standard RIMS impact survey, projects shall also conduct a RIMS+ impact survey.

The RIMS+ is a set of additional questions that can be tailored to the specificity of individual project interventions. In doing so, it allows the collection of a more coherent and comprehensive set of data to document the diverse impacts of projects. Therefore, implementation of the RIMS+ survey will help the project executive board have more detailed data, closer to the results as well as analysis of the gender factors affecting on the project.

In addition to the households in the project area participating in the survey, in RIMS+ survey, the households in the non-project area have chance to take part in the survey. The participation of households in non-project area will help evaluation on the impact of projects on the project area is better.

75 M&E Manual Guide for IFAD Funded Projects in Vietnam

2. How and when to conduct the survey

RIMS and/or RIMS+ survey is implemented at the beginning (as baseline survey) and the end of the project (end-line survey) corresponding to the IFAD’s requirement. For on-going IFAD funded projects in Viet Nam, there are two options for conducting RIMS+ survey:

- For the projects already conducted RIMS baseline survey: in this case, when the projects complete, it is necessary to conduct RIMS+ end-line survey (using RIMS questionnaire for baseline survey and RIMS+ questionnaire for end-line survey);

- The projects have not yet conduct baseline survey: conduct RIMS+ baseline survey and RIMS+ end-line survey (using same questionnaire for the two surveys).

In the questionnaire used in the RIMS+ survey, in addition to the RIMS standard questions, there are new parts and questions. For additional questions, interview time will be extended by 30-45 minutes.

The detailed steps from preparation to implementation of RIMS and RIMS+ baseline and end-line survey will be introduced in the next parts.

3. How to prepare and conduct a RIMS and/or RIMS+ survey

To conduct a RIMS and/or RIMS+ survey, the following steps are undertaken:

1) Develop the questionnaire and interview guide;

2) Sample selection;

3) Enumerators’ training and field testing of the questionnaire;

4) Logistical planning;

5) Data collection;

6) Data entry;

7) Data analysis;

8) Report writing.

Some activities may not necessarily be implemented chronologically; they can be conducted in parallel with others. For example, sampling, questionnaire design, enumerators’ training can be performed simultaneously; data entry form could be designed right after questionnaire is finalized. In the following parts, the detail for each step will be presented.

3.1. Step 1 - Develop the questionnaire and interview guide

3.1.1. Questionnaire design

IFAD has designed a template standard questionnaire for the RIMS survey that categorized by different, however, the information in this template questionnaire has limitation in reflecting fully the project impacts. Given the limitation of the RIMS questionnaire, IFAD also provide some modifiable modules of questions for the RIMS+ questionnaire. Therefore it is recommended that more time and resources are spent on adapting the template questionnaires and designing new contents to fit with each project as well as local conditions.

76 How to conduct RIMS and RIMS+ baseline and end-line survey

Box 4.1. Requirements for a good questionnaire

 The questionnaire can collect sufficient information for assessing outcomes (and impact) of project;  Collected information must present project’s objectives and indicators;  Questionnaire has appropriate length, it has to cover all aspects as desired, but the time for interview should not be too long;  The questions must be easy to ask and answer: try to avoid misunderstanding the questions otherwise it can lead to incorrect answers;  Questionnaire has appropriate structure and clear format.

Steps to design questionnaire

Step 1: Study carefully project documents to understand the project’s objectives, activities, the logical framework, and indicators for monitoring and evaluation.

Step 2: Identify the indicators which can be collected through the survey. Make a list of indicators that the survey needs to assess.

Step 3: Study the template questionnaire of IFAD: identify the sections and questions that can be adapted directly to the project; determine which questions need to be adjusted and added. It is good idea to find and learn from similar available survey of other projects or the questions that were used in relevant surveys, such as the annual outcome survey (See RIMS+ questionnaire of IFAD in Annex 5).

Step 4: Develop new questions to collect information for the indicators. Gender issues should be taken into account when designing the questions.

Box 4.2. Some points should be kept when designing additional questions

 Each question should have a specific purpose; avoid questions that overlap or duplicate each other or questions with an unclear purpose;  The question must not give any hints, bias or lead respondents to any answers;  Have clear instruction for answering the questions or skipping questions;  Wording with local language, avoid using acronyms or jargon that are multi-meaning, unclear or abbreviated.

The following is suggestions on the structure and content of the RIMS+ questionnaire.

77 M&E Manual Guide for IFAD Funded Projects in Vietnam

Table 4.1. Suggestion of additional information/questions of RIMS+ questionnaire

Category Standard RIMS RIMS+

- Information on the province, commune, village, and Cover page N/A households interviewed. - Name of enumerator, supervisor, and survey date.

- Ethnicity of household head and members A. General information - Household members: - Access to school of children (gender sensitive) about household age, sex, literacy - Participation in common groups/unions

- Floor material - Roof material

B. Household - Number of bedrooms - House ownership certificate and licensee (gender sensitive) characteristics - Sources of drinking water - Toilet type

C. Asset ownership - Assets - Other tools for agriculture farming - Fuel for cooking - Tools for tilling the land

D. Land No information - Farmland size; Land use certificate (gender sensitive)

E. Food security Period of hungry season - Food shortage resolution (gender sensitive) - Daily nutrition (gender sensitive)

F. Crop production Few information - Farm size - Yield and volume of sales - Selling price - Income from each crop - Input costs - Division of labor (gender sensitive)

G. Livestock No information - Number of livestock - Volume of sales - Input costs - Income from each livestock - Division of labor (gender sensitive)

H. Market assess No information - The highest income product - Selling location and time of travel to this location - Seller and buyer - Sources of market information - Distance from house to selling places

78 How to conduct RIMS and RIMS+ baseline and end-line survey

Category Standard RIMS RIMS+

I. No information - Extension training: participation, attendance times, applicability (gender sensitive) - Vocational training: participation, types of vocational training, income changes before and after participating (gender sensitive)

J. Non farming No information - Non-farm Income activities - Identifying household’s main income earner - Primary source and secondary source of income - Financial management (gender sensitive)

K. Credit assess No information - General loans: borrower, the person responsible for paying, source of loan, use of loan (gender sensitive) - Loan supported by project: borrower, the person responsible for paying, purpose and use of loan, the effectiveness of loans (gender sensitive)

K. Socio-economic No information - Participating in developing socio-economic development plan development plan and infrastructure plan (gender sensitive) and infrastructure - Disseminating community information building plan - Satisfaction with local infrastructure

M. Natural disaster and No information - Frequency of disasters vulnerability - Risks, and vulnerability - Resilience (capacity to withstand or recover from disasters)

N. Gender equality No information - Changing awareness of family violence - Role of women in decision making process in the family and community

O. Anthropometry - Age (in months) No new information - Height and weight of children under 5

3.1.2. Develop an interview guide

79 M&E Manual Guide for IFAD Funded Projects in Vietnam of local events and check sheet for anthropometric values. These two sheets help reduce error values in anthropometry section (the calendar of local events and check sheet for anthropometry values will be presented in Annex 6 and 7).

3.1.3. Pilot

The purpose of pilot is to assess the feasibility of the survey, consider the issues in the questionnaire need to be revised. Sample size for pilot in RIMS and RIMS+ survey is about 10-15 households. House- holds chosen to take part in the pilot should be involved in all activities mentioned in the questionnaire so that all content can be tested.

Box 4.3. Questions to be answered by pilot

1) 1) Does each question measure what it is supposed to measure? 2) Are all the words understood? 3) Do all respondents interpret the question in the same way? 4) Are all response choices appropriate? 5) Is the range of response choices actually used? 6) Do respondents correctly follow directions? 7) Does it create a positive impression that motivates people to respond? 8) How long does it take to complete? 9) Does it collect the information you want?

3.2. Sample selection

Selecting the sample directly affects to the result of survey. If the sample is representative, the survey will provide reliable information about results and impacts of project; if the sample is not representative, the survey’s result can not reflect correctly the project’s results.

In terms of sample size, the standardized RIMS survey is conducted with 900 randomly selected households in the project area (including 30 clusters; 30 households at each selected cluster) without the control group; in the RIMS+ survey, in addition to 900 households in the project area (including 30 clusters; 30 households at each selected cluster), at least 300 households in the non-project area (including 10 clusters; 30 households at each selected cluster) will be surveyed. In theory and ideally, the sample size of the control should be larger than the treatment. However, due to limited resources, the sample size for control group in this case is smaller than the treatment group

In terms of methodology, RIMS and RIMS+ survey usually apply two-stage sampling method. This method ensures that each household has an equal chance of being selected for the survey. The first stage is the selection of the cluster, and the second is the random selection of the households within clusters selected in the first stage.

80 How to conduct RIMS and RIMS+ baseline and end-line survey

3.2.1. Sampling methodology

 Stage 1: Select villages (30 villages in the project area and 10 villages in the non-project area)

Select villages in the project area

Step 1: Establish the sampling frame - List all villages of project communes.

Step 2: Calculate the sampling interval (SI), using the formula:

SI = Total population/30

(In which, SI is “Sampling interval”; “population” is total number of households in the all villages, “30” is number of villages to be selected.)

Step 3: Select a random number (R) between 1 and SI.

Step 4: Select villages

The first village corresponds to the village that has an accumulative population equal to or greater than the random number.

The second village is the village that has an accumulative population equal to or greater than the random number, plus the SI.

The process is continued until there are 30 villages selected.

Select villages in the non-project areas

Step 1: Select 10 non-project communes

The control group (in the non-project area) should have similar socio-economic conditions as the treatment group. To select these communes, list all non-project communes with information on the poverty and percentage of ethnic minority (the two criteria used to select treatment communes). Choose 10 communes which have most similar conditions to the project communes.

Step 2: Select 10 villages from 10 selected communes

The sampling methodology used to select the villages for the treatment group is applied to choose the villages for the control group. List all villages of 10 selected communes and replicate steps presented above to select 10 villages.

 Stage 2: Select the households (900 households for the treatment group and 300 households for the control group)

Select 30 households from each selected village (30 villages in the treatment group and 10 villages in the control group). In addition to the official households it is recommended to select 10-15 stand-by households in case the households on the official list are unavailable for interview or absent from home.

There are several options to select the households (applied to both treatment group and control group) as follows.

81 M&E Manual Guide for IFAD Funded Projects in Vietnam

Option 1: Systematic Random Sampling

This method requires that the list of all households in the selected villages is available in softcopies. The following steps are undertaken:

Step 1: List all households of each village in excel

Step 2: Calculate sampling interval

SI = total number of households/selected households

(Round up if the decimal figure is more than or equal to 0.5. Round down if the decimal figure is less than 0.5)

Step 3: Select a random number (R)

Use function ‘Rand’ in Excel. Type (= rand()*SI) in a cell

Step 4: Select households

Rth household in the list is the first selected household.

Add the sampling interval –SI to select the 2nd household

Similarly, the order number of the 30th household is identified by adding the order number of the 29th household and the sampling interval (SI).

We have the complete list of 30 households participating in the survey.

Option 2: Random Sampling (when the list of all households in the selected villages are available)

Using function “data analysis” in Excel to randomly select 30 numbers with name of household’s head (softcopies of list are available) or select the 30 numbers only then match with the name and order (number) of the households in the hardcopy lists.

Option 3: Random Sampling (when the list of all households in the selected villages are not available)

This method is conducted as follows:

- When survey team goes to the village, identify the approximate centre of the village. A pencil or bottle is spun to select a random walking direction (also called a transect line).

- Count the number of households encountered along the transect line between the centre and the perimeter of the cluster (end of the village). This number is divided by total number of households needed to determine the sampling interval (SI).

- Select a random number between 1 and total number of households. The random number corresponds to the first interviewed household.

E.g.: the random number is 6 (R=6)

- The second household is far from the first one SI households

E.g.: the first interviewed household is the 6th household on the transect line, SI=4  the second interviewed household is the 10th household on the transect line.

82 How to conduct RIMS and RIMS+ baseline and end-line survey

- Continue the process until the required number of households has been interviewed.

- When the households located in the transect line less than the number desired, all the households will be interviewed. Then, the team comes back to the village center for a second random walking direction and repeats the process.

3.2.2. Stand-by households

During the survey, the households selected in advance may be not available for interview or they refuse to take part in the survey. In this case, it is necessary to have stand-by households to replace. In the RIMS and RIMS+ survey, the number of stand-by households should be 10-15 households. The stand-by households are selected by using the similar method applied in official household selection.

3.2.3. Respondent selection - gender balance

3.3.1. Recruiting enumerators, team leaders and supervisors

The role of enumerators, team leaders, and supervisors are very important in any survey. Ideally, in the RIMS and/or RIMS+ survey, each group has 01 supervisor, 01 team leader and 05-06 enumerators. Be- sides, it should have female and male enumerators in each group to ensure gender balance and facilitate the interviews. Enumerators

Enumerators can be project staff at different level (provincial, district or commune level). Enumerators must be trained carefully and be competent in interviewing. In addition, survey teams working in areas with ethnic minorities should each have enumerator(s) from the relevant ethnic minorities. It is recommended that female enumerators will conduct interview with female respondents and male enumerators will conduct interview with male respondents.

Box 4.4. The duties of enumerators

Survey team leader

83 M&E Manual Guide for IFAD Funded Projects in Vietnam

Box 4.5. The duties of the enumerator team leader

 Guarantee the best preparation for the field test: organize and schedule interviews, contact the area that is to be surveyed, prepare the questionnaire, prepare other logistics and especially ensuring the safety of enumerators during the fieldwork;  Delegate assignments for the enumerators: the assignment of work needs to be suitable for each enumerator;  Coordinate all activities of the team, examining the quality of collected questionnaires;  Keep closely in touch with supervisor when carrying out the field work;  Set interview goals every day for the enumerator team (based on experience and local conditions of the project area);  Provide logistics for the survey;  Monitor the progress of work, reporting to supervisor at the end of the day.

Supervisors

Supervision is very important activity which heavily contributes to the success of a survey. Ideally, supervisors should not be project staff but external resources to ensure the objectivity and accuracy of the survey. In the absence of sufficient financial resources to hire external supervisors, the M&E officers from PCU should supervise the survey. Supervisors are responsible for ensuring the quality of the information obtained by the enumerators and the survey overall. The supervisors report to the project manager about the survey process.

Box 4.6. The duties of supervisors

 Supervise the interviewing process: the supervisor will observe interviews, ensure the interviews run smoothly and eligible children have been weighted and measured, and record common mistakes made by enumerators to avoid mistakes in next interviews;  Guiding, answering and explaining to enumerators in case of any misunderstanding or lack of interview skills;  Ensure the progress of work: keep in touch with and give daily updates to enumerators and use the monitoring progress map to keep track of the progression of work as well as to make sure that the daily amount of work done is consistent with initial estimates;  Check all answer sheets: check responded information, ensure that there is no missing in the questionnaire, check for mistakes made in writing the answer code and check value written in the anthropometry section. If the questionnaire has any vague information, the supervisor needs to require enumerators to provide more details or to re-interview. The mistakes made in the questionnaire must be recorded in the check answer sheets record;  Review and learn by experiences everyday as well as disseminating the schedule and plan for the following day;  Be responsible for retaining all questionnaires and handing them over to the survey team leader for data entry.

3.3.2. Training a. The training

Before conducting the survey, all enumerators, team leaders, and supervisors should be trained carefully about the survey and necessary skills as well as their responsibilities.

The training will cover following issues:

(i) The purposes of the RIMS and RIMS+ survey and steps to conduct;

(ii) Structure of the questionnaire; the purpose and content of each question;

(iii) Interview skills, how to raise the questions and how to record the answers;

84 How to conduct RIMS and RIMS+ baseline and end-line survey

(iv) Sampling method;

(v) How to use GPS machine and measure children’s weight and height;

(vi) The role and responsibility of enumerator, team leader, supervisor;

(vii) How to enter and analyze data

Training materials and equipments:

(i) Lectures;

(ii) Questionnaire;

(iii) Interview guide;

(iv) Global positioning system (GPS) (if necessary);

(v) Weight and height scales for anthropometry;

(vi) Data entry form and software for entering and analyzing data;

(vii) Guidance for data entry and analysis.

The training should take at least six days including the 4-day survey training (includes one day for field- test) and 2-days data training. b. Measure children’s weight and height

Object: All children are under 5 years old will be measure weight and height.

Procedure:

Child standing height measurement is applied with children above 2 years old.

- Place the measuring board on a hard, flat surface against a wall. Make sure the measuring board is stable;

- Take off the child’s shoes and hat;

- Bring the child to the measuring board;

- Ask the child to look straight ahead, make sure the child’s line of sight is parallel to the ground;

- When the child’s position is correct, lower the headpiece on top of the child’s head, read and record the measurement on the questionnaire;

- Check the recorded measurement on the questionnaire for accuracy and legibility.

Child length measurement is applied with children under 2 years old:

- Place the measuring board on a hard, flat surface against a wall. Make sure the measuring board is stable;

- Place the child lie on the board, the child should be looking straight up so that the child’s line is perpendicular to the board;

- Make sure the child is lying flat at the center of the board;

85 M&E Manual Guide for IFAD Funded Projects in Vietnam

- Place the child’s knees and feet in the correct position. Make sure the child’s legs are straight;

- When the child’s position is correct, read and record the measurement on the questionnaire;

- Check the recorded measurement on the questionnaire for accuracy and legibility.

Weighing a child who is able to stand:

- Place the scale on a hard, flat surface;

- Ask the child to stand on the scale;

- Record the weight of the child.

Weighing an infant or young child by an Adult

- Place the scale on a hard, flat surface;

- Ask the mother or caretaker to stand on the scale;

- Record or remember the weight of the mother or caretaker;

- Hand the baby to the mother, the weight of baby and mother will be displayed;

- Record the baby’s weight. c. Field-test

The field-test will be conducted after all enumerators participate in questionnaire training. This can be conducted in one or two villages.

The purposes of field-test

 Help enumerators to familiarize themselves with the questionnaire, scale and measures, and practice the interview skills they have learnt;

 Help supervisors and team leaders practice their tasks such as: planning, arranging and assigning enumerators to interview households, managing the survey and checking the information collected in the questionnaires;

 Identify mistakes in the questionnaire for finalization;

 Identify unexpected situations in practice to share with all enumerators and supervisors for resolution.

Preparation for field-test

 A list of households to be surveyed should be prepared in advance;

 Planning and inform local officials about the field-test in order that they help to prepare the list of households for interview. The notification should be completed at least one day before the field test for a good preparation. The list of households to be interviewed should be provided to the enumerators, team leaders, and supervisors.

 Villages selected for the field-test may be in the sample list, but ideally, they should not be in the sample as if there is any significant adjustment of the questionnaire then it is not necessary to come back for re-interview. The villages selected for the field-test should have similar characteristics to those villages that will be surveyed.

86 How to conduct RIMS and RIMS+ baseline and end-line survey

 It is necessary to have a plan for enumerator allocation in advance. All enumerators will be divided into groups; each pair will be assigned to interview the specific number of households.

 Transportation, time, and gathering place should be informed and agreed with all enumerators. Regarding transportation, enumerators may use their personal vehicles by arranged bus to the field. Hiding bus will be costly but it is safer and avoid time-wasting. Regarding time, all enumerators should be informed about the time to meet before and after completing the interview. All interview tests should be completed in the morning. Regarding gathering place, the gathering place can be at village center, head of village house or the Commune People’s Committee office. The gathering place should be the most convenient place for all enumerators.

 Prepare all necessary documents including the list of surveyed households, questionnaires, manual guides, notebooks, pens, receipts, travel warrant, weight and height scales.

Practice interview

 Each pair of enumerators will be assigned to interview some households. The enumerators will take it in turns interviewing each household, and during the interview process enumerators will take notes of any mistakes in the questionnaire or unexpected issues.

 Team leader will be responsible for assigning interviewees and keeping in contact with the enumerators and the supervisors in the field test in order to solve any potential problems (for example, a change of sample because some households are unavailable, or refuse to participate in the survey).

 The supervisor will supervise some pairs the enumerator. After completing supervision of the first pair, the supervisor can supervise the other pair. While observing the interview, supervisors should take notes about errors in the questionnaire or interview mistakes of the enumerators.

 When finishing the interviews, the enumerators should clean the questionnaires and handover them to the team leader, who will be responsible to review the questionnaires before giving them to the supervisor for checking. Questionnaires collected from the field test will be used for data entry and analysis training course.

Experience sharing and lessons learnt after the field-test

3.4. Step 4 - Logistical planning

One of the factors that make a survey successful is good logistics. Here are the logistical preparations for the RIMS and RIMS+ survey.

Scheduling the survey

87 M&E Manual Guide for IFAD Funded Projects in Vietnam

Local assistant staff

Village Head

Local Guide

Interpreter

Transportation

Project areas usually are remote and mountainous places. Therefore, the transportation should be taken into account when the survey is planned. The enumerators could use their own vehicles or could be provided transportation by the project.

Survey Materials

Each enumerator needs to be equipped with a full package of questionnaire, interview manual guide, the list of households for survey, pen, and notebook. In addition, it is necessary to prepare other documents such as travel warrants; allowance and receipts for the guide or interpreter; gifts/money for the respondents and receipt; answer check sheet and the progress monitoring form.

88 How to conduct RIMS and RIMS+ baseline and end-line survey

Weight and height scales

In order to assess the indicators of child malnutrition, it is necessary to measure the weight and height (length) of children less than 5 years old. As a result, enumerators must carry measuring equipment that meet IFAD’s quality requirements. Because most survey sites are remote and travelling is often difficult, the equipment should be compact but still of a high quality. It would be optimal if each pair of enumerators is equipped with one set of equipment each, but if not there should be arrangements to share among the group.

Budget estimation

3.5. Step 5 - Data collection

Survey on a large scale is a main assignment of whole survey because this task decides the quality of survey. Besides, supervision also plays a very important role in the success of the RIMS and RIMS+ survey.

Data collection process

All enumerators will be provided questionnaires, manual guide for interview, household lists, receipts for households and enumerators, travel warrants. Besides, the enumerators being responsible for interviewing the households having children will be provided scale, measures and GPS machine (if necessary).

Each project supporting unit at district level might be in charge of conducting the survey at their localities. The project supporting unit at district should keep in mind the following points:

- Each district will schedule the survey based on the time, survey areas, and a number of enumerators. The plan should be consistent with the overall survey plan of the whole project. The team leader will be in charge of the progress of the survey at his/her locality.

- The safety for enumerators should be taken into account, such as: transportation, meals, accommodation.

Supervision

As mentioned above, a successful survey depends heavily on the supervising. This is one of the most important activities that determine the quality of survey. Supervisors and team leaders are responsible for the quality and the progress of the survey. In order to do these tasks supervisors and team leaders have responsibilities for supervising the interview, checking questionnaires completed by enumerators, and give daily updates to enumerators and use the monitoring progress map to keep track of the progression of work. Each data collection team must be accompanied by a supervisor during the whole data collection process.

89 M&E Manual Guide for IFAD Funded Projects in Vietnam

3.6. Step 6 - Data entry

3.6.1. Data cleaning

All questionnaires will be checked and cleaned by the supervisors and data entry staffs. If there is any missing or unclear information, the enumerators may be requested to add or clarify. In case of necessary, the enumerator may be requested to come back for re-interview. After the questionnaire is cleaned or clarified by the enumerator, the supervisor has to re-check the questionnaire before handing over to data entry staff.

3.6.2. Entering data

The questionnaires received by data entry officials must be cleaned. If the RIMS standard questionnaire is used, the RIMS software will be used to enter data.

If the RIMS+ questionnaire is used, given that the standard RIMS software cannot accommodate additional questions of RIMS+ questionnaire, you will have to create a separate data-base in order to enter and analyse answers to these additional questions. CSPro is a user-friendly mechanism for double- checking and entry error detection. CSPro is fully designed as a questionnaire that facilitates the process of data entry as well as minimizes simple mistakes during the process such as missing, mismatching, or misreading data. Furthermore, CSPro allows different users to combine input data and so permits the simultaneous cooperation and participation of these different users; additionally, could be able to cooperate simultaneously; it can also find mistakes automatically by using a double-check command. Data from CSPro can be easily transferred to other statistical software such as Excel, SPSS or STATA for further processing and analysis. Therefore, it is recommended to use CSPro for entering RIMS+ data. In this case, all 900 questionnaires of the treatment group and 300 questionnaires of control will be entered in RIMS software for the RISM standard questions. The whole parts and questions of the RIMS+ also will be entered into the data form using CSPro software.

Each data entry staff will be responsible for a number of questionnaires assigned by the team leader. All questionnaires will be entered twice by two different staffs.

Data entry officials are responsible for entering data into the computer; therefore, in addition to understand the software used for data entry, they should meet the following requirements:

- Understand the content of questionnaire and its structure and logic;

- Adhere to time schedule laid out by the team leader/supervisor;

- Preserve all questionnaires received from team leader/supervisor.

The data entry process also should be supervised under a team leader or supervisor.

Compare Data

There should be two data entry officials inputting a same set of questionnaire. This is to compare and reduce subjective errors and CSPro allows doing this action.

90 How to conduct RIMS and RIMS+ baseline and end-line survey

Concatenate database

Export to SPSS/STATA/EXCEL

After completing the data entry process using CSPro, the database could be exported for analysing by professional statistic software, such as SPSS, STATA or EXCEL, upon the analysist.

3.7. Step 7 - Data analysis

Entered data will be exported into analyzing software. When analyzing data, the following requirements should be met:

- Analyze the correlation between relevant issues;

- Comparison with the result of previous survey;

- The analyzing indicators must link to the project targets.

For data analyzing, it is recommended using SPSS because of its simpleness. The results will be inputs for report writing.

3.8. Step 8 - Report writing

After the last available data sets and results are analyzed, the next step is to write the report; this is done by using data obtained from the survey. The report will reflect the results and impacts from the project and must satisfy the following requirements:

In terms of content

- The report should cover the project’s objectives, analyzing the identified indicators;

- The report should emphasis the impacts of the project;

- The report should emphasis the comparison between the treatment group and the control group, as well as different targeted groups of the project (e.g. women, poor households, or ethnic minorities);

- The emphasis should be on the meaning and implications of the data analy¬sis for the project, not on the methodological aspects;

- Ensure the clarity and coherence of the structure, content, vocabulary and logic between para- graphs and sections.

91 M&E Manual Guide for IFAD Funded Projects in Vietnam

In terms of presentation

- The structure of the sections should be clear and logic;

- Surveys results should be presented first and foremost through charts/graphs, and the accompanying text should only highlight the most important findings from the charts.

- When preparing charts/graphs, be careful with the use of colours, as the charts/graphs will not be readable if the report is printed in Instead of using colours, use different black and white (not everyone has a colour printer!). “textures” for the charts.

Before drafting the report, it is necessary to design a detail report outline. From the outline, a full report will be developed.

After a draft report is prepared, the PCU’s director board will organize a workshop to share the findings and collect comments on the draft report from project staffs and relevant stakeholders. After that the report will be finalized and ready for sharing at a wider community.

92 Sampling methodology

Chapter V.

Sampling methodology

93 M&E Manual Guide for IFAD Funded Projects in Vietnam

94 Sampling methodology

Chapter V. Sampling methodology

1. Definitions

1.1. Population

The population includes units (elements and particular phenomenon) that need to be observed and analyzed (quantitative analysis) to find out their rules and nature in specific time and space. E.g. in a study of the impacts of a project in 2012 in Tuyen Quang province, the population consists of all households in the project area of Tuyen Quang in 2012.

1.2. Sample

A sample is a subset of a population that is randomly selected to be representative of the population. Sample selection is the selection of a specific number of units in the population that are representative of the population by using the different sampling methods.

E.g.: In order to evaluate the impacts of a project in a locality, select randomly 900 households among all beneficiaries of the project to participate in the survey.

As the population is often very large, there are many difficulties in terms of finance and human resources in conducting the survey. Therefore, sample selection will:

- Increase the feasibility of the survey;

- Save resources (finance and human);

- Reduce mistakes in the processing and analysis of data.

In RIMS impact survey, only households in the project area participate in the survey (called the treatment group). In RIMS+ impact survey, in addition to the treatment group, households in non-project area also have a chance to take part in the survey (called the control group).

The control group will represent what the treatment group would have been like if there was no project/ intervention. And instead of comparing before- after of the treatment group only, compare the difference in before-after between the treatment and control groups (see the Figure 5.1). So we could analyze what would happen to the targeted group if the intervention had not happened. Therefore, the importance is to select a good control group, which is have the same characteristics (observed and unobserved) with the treatment group, only difference is treatment.

95 M&E Manual Guide for IFAD Funded Projects in Vietnam

Figure 5.1. The use of control group

1.3. Sampling frame

Sampling frame is the list of communities/villages (clusters) in the project/ non-project area that is used to select the representative communities/villages (clusters) in the project/ non-project area. With each sampling method, the information about the number of households, poverty rate, and proportion of ethnic minorities is required.

1.4. Sample size

Sample size refers to the number of households to be included in the survey. An appropriate sample size is determined by three factors: (i) the sale of population, (ii) implementation cost and (iii) the acceptable margin of error. The larger the sample size is, the more precisely the results calculated from the samples are concluded for the population; however, it should be noted that the larger the sample size is, the higher the cost and the longer the time needed to conduct the survey are.

According to IFAD’s requirement, sample size of the treatment group is 900 households for RIMS baseline and end-line survey and 200 households for the annual outcome survey. For the control group, in theory and ideally, the sample size of the control should be larger than the treatment. However, due to limited resources, a same sample size for both groups for Annual Outcome Survey is recommended and a sample size of at least 300 households is recommended for RIMS baseline and end-line survey.

1.5. Sampling method

The important step in any survey is to identify the best sampling method. Sampling directly affects to the result of survey. The good method will select the appropriate and representative sample and one can make extrapolations from the sample to the population. The followings are popular sampling methods.

Random sampling method (also called probability sampling method)

With these methods, each member of a target population has an equal probability of selection. The types of random sampling methods include the following:

96 Sampling methodology

• Simple random sampling: Used when the overall population has no distinct characteristics and is homogeneous and when the overall population is not too widely dispersed. The random selected sample of beneficiaries will be representative of the overall the population. This method will be detailed later.

Advantages: simple and time-saving.

Disadvantages: all elements have to be listed, there might be difficulties in collect information because households are scattered in a large area, and it is not always able to choose the best representative sample.

• Stratified sampling: this method is used to increase survey precision and reduce the margin of error when the overall population of project beneficiaries includes many group or strata with very distinct, mutually exclusive socio-economic characteristics. Stratified sampling involves the process of dividing members of the population into homogenous sub-groups/strata before sampling. Once sub-groups/strata are identified, random sampling is applied within each sub-group.

Advantages: simple, time-saving and implementation cost-saving.

Disadvantages: potential systematic errors.

• Cluster sampling: this method is used when there are difficulties in geographical conditions, the population can be divided into many groups and areas. Applying this method will help reduce survey costs on finance and human resources. The population is divided into many clusters based on geographical area, and then some clusters are randomly selected. All members of selected groups will be surveyed.

Advantages: it is not necessary to list all elements in the population, reduce cost on finance and human resources.

Disadvantages: the sample cannot be the best representative.

Non-random sampling (also called non-probability sampling)

With these methods, only some members of the beneficiary population have a chance of selection, while other will be excluded a priori from participation in the survey. Information gathered through non-random samples cannot be extrapolated to the overall population of beneficiaries. there are many non-random sampling, some popular methods include the following:

• Convenience sampling (also called accidental sampling): the sample is selected conveniently or accidentally that is easy to reach or close to hand. Example: select the households are near, easy to access.

Advantages: save time and reduce cost.

Disadvantages: the representation of sample is not ensured, leading to low preciseness.

• Purposive sampling (also called judgmental sampling): Purposive sampling involves choosing the sample based on the best judgment on who would be appropriate for the study.

97 M&E Manual Guide for IFAD Funded Projects in Vietnam

Advantages: it is able to choose the appropriate sample for the study.

Disadvantages: the representation of sample is not ensured.

• Respondent-driven sampling (RDS): RDS combines “snowball sampling” (getting individuals to refer those they know, these individuals in turn refer those they know and so on) with a mathemati- cal model that weights the sample to compensate for the fact that the sample was collected in a non-random way.

Advantage: this is very useful and especially convenient method for this study of hard-to-reach groups.

Disadvantages: the sample may not be representative of the population, so conclusion cannot be validly made about the entirety of the target population.

The procedure for IFAD impact surveys employs a two-stage methodology. This method ensures that each household has an equal chance of being selected for the survey. The first stage is the selection of the village/cluster, and the second is the random selection of the households within villages/clusters selected in the first stage. The detail steps of this method will be mentioned in the next part. 2. Sampling method in projects funded by IFAD

As mentioned above, IFAD impact surveys usually apply two-stage sampling method. The detailed steps in each stage are described as follows:

2.1. The first stage: Select village

According to IFAD’s requirements, 20 villages in the project area and 20 villages in the non-project area are selected for the annual outcome survey; 30 and 10 villages respectively for both the baseline and end-line surveys. The first stage includes the following steps:

Step 1: Construct sampling frame

Sampling frame is constructed using population data of each village, it is important to make sure that the population figures are up to date and that data are available for all villages.

- List all villages of project communes;

- Calculate the cumulative population, this process is continued until the list is complete. The cumulative amount must equal the total of households of all villages.

E.g.: In “TNSP Project in Tuyen Quang”, there are 25 project communes; thus, the sampling frame is the list of all villages of 25 project communes. The sampling frame is constructed as follows:

98 Sampling methodology

Table 5.1 The sampling frame of TNSP in Tuyen Quang

Cumulative District Commune Village Population No population 1 Na Chao 121 121 2 Bản Nhùng 54 175 3 Bản Tùn 68 243 4 Bản Nuầy 80 323 5 Phiêng Quân 69 392 6 Phiêng Rào 58 450 7 Không Mây 72 522 Na Hang Nang Kha 8 Nà Noong 111 633 9 Nà Chang 74 707 10 Nà Reo 49 756 11 Nà Kham 119 875 12 Nà Khá 109 984 13 Nà Vai 82 1,066 14 Lũng Giang 52 1119 … … … … … … 233 Lũng Hoa 110 18,651 234 Hoa Lũng 81 18,732 Dại Phu 235 Thái Sơn Đông 70 18,802 236 Thái Sơn Tây 62 18,864 237 Ninh Hoà 190 19,054 238 Thanh Bình 206 19,260 239 Thanh Sơn 98 19,358 240 Đồng Chùa 103 19,461 Son Duong 241 Núi Độc 100 19,561 242 Đồng Phai 97 19,658 Hop Hoa 243 Cầu Đá 222 19,880 244 Đồng Giang 153 20,033 245 Đồng Báo 84 20,117 246 Việt Hoà 43 20,160 247 Tân Trào 206 20,366 248 Tân Dân 116 20,482 TOTAL 20,482

99 M&E Manual Guide for IFAD Funded Projects in Vietnam

Step 2: Calculate sampling interval (SI)

The sampling interval (SI) will be used systematically to assign villages (clusters) from the sampling frame. The SI is equal to the total population of the project area, divided by the number of clusters.

Total population SI = No. of clusters (Round up if the decimal figure is more than or equal to 0.5. Round down if the decimal figure is less than 0.5)

20482 E.g.: SI = = 683 30 Step 3: Select a random number (R)

The random number is used to determine the starting point for the first cluster. It should have a value between zero and the SI. To find the random number, you can use the function in Excel “Rand” (type “=rand()*SI) in a cell. We will have a random number which is a value between 1 and SI.

Step 4: Select village (cluster)

The first village (cluster) corresponds to the village that has an accumulative population equal toor greater than the random number. In the above example, Phieng Rao village (the accumulative population is 450) in Nang Kha commune, Na Hang district is the first selected village.

The second village (cluster) is the village that has an accumulative population equal to or greater than the random number, plus the SI. In the above example, the second selected village has an accumulative population equal to or greater than 1089 (=683+406); that is, the second village is Lung Giang village in Nang Kha commune, Na Hang district.

Similarly:

- The third village has an accumulative population equal to or greater than 1772 (=1089+683)

- The fourth village has an accumulative population equal to or greater than 2455 (1772+683)

…

The process is continued until there are 30 villages selected.

2.2. The second stage: Select household

The second stage of two-stage sampling method is to select household. In the annual outcome survey, 20 official households are selected in each village; in baseline and end-line surveys, 30 official households are selected in each village. Household selection can apply one of following methods based on the actual conditions.

Method 1: Systematic sampling

This method is applied when there is the list of households. Systematic sampling is conducted similarly to the sampling method applied in village selection in the above part.

100 Sampling methodology

Step 1: List all households of each village in excel Step 2: Calculate sampling interval: SI = total number of households/selected households (Round up if the decimal figure is more than or equal to 0.5. Round down if the decimal figure is less than 0.5) Step 3: Select a random number (R) Use function ‘Rand’ in Excel. Type (= rand()*SI) in a cell Step 4: Select household Rth household in the list is the first selected household. Add the sampling interval –SI to select the 2nd household Similarly, the order number of the 30th household is identified by adding the order number of the 29th household and the sampling interval (SI). We have the complete list of 30 households participating in the survey.

Method 2: Simple random sampling

This method is applied when the list of households in the computer is available. Using function ‘‘Data Analysis’’ in excel to choose households participating in the survey. The following steps should be undertaken: Step 1: Assign each household with a number in Excel Step 2: Using function “Sampling” in Excel - Click ‘‘data’’ on the toolbar, enter ‘‘Data analysis’’. The screen will display a dialog box as follows:

- Select ‘Sampling’ and click ‘OK’. The screen will display a dialog box ‘‘Sampling’’

101 M&E Manual Guide for IFAD Funded Projects in Vietnam

In the dialog box, there are following items:

+ “Input range”: is the area selected à select the area of order numbers of households.

+ “Random”: enter the number of selected data points à enter“30”; that is, the computer selects 30 random numbers corresponding to the number of households participating in the survey.

+ “Output Range”: enter the address of the outputs à select any cell.

After finishing the above three steps, click “OK”

Step 3: Complete the household list

Computer will select 30 random numbers corresponding to the order numbers of 30 households. Fill the name of the head of household corresponding to an order number.

Similarly, we will complete the list of households participating in the survey in the other villages.

Method 3: Simple random sampling without using Excel software

Option 1: Using Households list

This method is useful when survey team does not have computer in the field and have available household list. The following steps should be done:

- Assign each household in the community a number (attach head of household’s name if necessary) and write it on a pieces of paper, the number of pieces is equal to the total number of households in the village.

- Put them in a jar or bag, mix them and draw out 30 pieces of paper.

- The number drawn represent the household numbers from the list, complete the list of 30 households.

102 Sampling methodology

Option 2: Random Walk

This method is used if no accurate household list is available. The following steps should be done:

- When survey team go to the village, identify the approximate centre of the village. A pencil or bottle is spun to select a random walking direction (also called a transect line).

- Select a random number between 1 and total number of households. The random number corresponds to the first interviewed household.

Example: the random number is 6 (R=6). the first interviewed household is the 6th household on the transect line, SI=4 à the second interviewed household is the 10th household on the transect line.

- Continue the process until the required number of households has been interviewed.

3. Stand-by households

During the survey, the households selected in advance may be not available for interview. In this case, it is necessary to have stand-by households to replace. In the annual outcome Survey, the number of stand-by households should be equal or half of needed households, in the baseline or end-line impact survey, the stand-by household can be 10 or 15 households. The stand-by households are selected by using the method applied in official household selection.

4. Respondent selection

According to IFAD’s requirement, in order to ensure the gender balance in the survey, it requires at least 30% interviewees are women or men in each cluster. Therefore, it is better to specify the male interviewee or female interviewee from the household in advance, inform interviewee about the interviews and assign female or male enumerators to interview.

103 M&E Manual Guide for IFAD Funded Projects in Vietnam

104 Sampling methodology

Chapter VI.

How to conduct key informant interview

105 M&E Manual Guide for IFAD Funded Projects in Vietnam

106 How to conduct key informant interview

Chapter VI. How to conduct key informant interview

1. Definition of key informant interview

What is a key informant interview?

Key informant interviews are used in the following cases: - Research topics are new and undefined; - Being opinion polls when definitions, problems, phenomena, or subjects’ charateristics researched are uncertain; - Need more in-depth information to maximize unerstanding about research topics ; - Profound insights into meanings of phenomena, actions rather than their frequency. - To answer questions “WHY” and “HOW” adding more information for quantitative data. Advantages

- Interviewees are free to give opinions and views; - Researchers are active and flexible in conducting interviews depending on spefic actual conditions (subject, situation, time and interviewer’s competency, etc.) to make interviews on different ways. Disadvantages

- Because of diversified and diffused answers so that it is very difficult to synthesize - Because key informants are often expert on research issues causing an academic gap between interviewers and interviewees that could make what interviewers’ take-note different from interviewees’ opinons; - It is also difficult for a data clerk to process the results of key informant interviews if he/she doesn’t participate in interviews; - It requires a certain academic knowledge and interview skill of researchers. 2. Classification of key informant interview

Key informant interviews are divided into two types: unstructured interview and semi-structured interview. 2.1. Unstructured interview

That is a method independent on predefined topics or questions, researchers only need remember the theme interviews and do not need to use a list of topics in the interview. The determinant of the success of the unstructured interview is the ability to ask questions to evoke an effective way. In other words, it is the ability to stimulate the respondents to provide additional information. The interviewer who is expert in the areas of interview, good at interviewing skills will lead the interviewees following to the story, the interview topic. He/She will make an interview as a conversation rather than as a question - answer and

107 M&E Manual Guide for IFAD Funded Projects in Vietnam still obtain the necessary content. Advantages

- Researcher can actively change the order of topics depending on the circumstances of the interview and the interviewee’s answers; - An unstructured interview like a talking which makes interviewees feel comfortable and open to answer interview topics; - It is especially useful in cases of need to interview those who provide information many times, in many different contexts; - It is useful in those cases that can not use official interviewes Disadvantages

- No sample preparation is available so that each interview is an unique conversation so it is difficult to systematize the information and data analysis; - Interviewers must have knowledge, experience and skills; - It is very easy to miss the content of interview if the interviewer does not have the skills to dig deeply into research themes; - If the interviewee is not open to interview, the interviewer hardly lead story without prepared topic. 2.2. Semi-structured interview

Semi-structured interview is an interview based on a list of prepared questions or topics on a certain issue. However, the order and question asking may depend on the context and characteristics of respondents Advantages

- Using the interview guide will save time; - A list of questions will help identify the issues that need to crawl but still allow the flexibility to discuss emerging issues if necessary; - Easy to systematize and analyze the information obtained; - Helps interviewers not miss the contents to be interviewed. Disadvantages

- It takes time to research and explore to identify research topics and design appropriate questions prior to the interview Thus, based on the characteristics of the study of IFAD projects as well as pros and cons of the 02 types of key informant interviews, semi-structured interview method is appropriate and should be applied. 3. Steps conducting key informant interviews

To conduct a key informant interview, 07 following steps should be undertaken: 1) Identify the topics that need in-depth information

2) Identify the key informants

3) Develop the questions by topics 4) Prepare the interview guide

108 How to conduct key informant interview

5) Pre-test and finalize the questions or interview guide 6) Conduct the interview 7) Analyze and write report In which, the step 3, 4, and 5 are just required for semi-structured method. 3.1. Indentify interview topics The topic identification is the first important step to conducts the others. The number of topics depends on the research objectives, time and available resources. Each key informant could be interviewed for one or more certain topics.

Box 6.1. Examples of topics for key informant interview

- Community participation in local planning from assessment on participation of male and female; - Household accessibility to market information: from angle of poor household, near poor households and wealthy households; - Quality of extension services for the poor and ethnic minority; - Quality of vocational training for poor women.

3.2. Identify key informants

Key informant must be carefully selected. Key informant is a person who has in-depth understanding about the research topics and can provide detailed information and opinions on a specific subject based on his/her knowledge of that issue. They can be experts or people experienced in research theme or people got actual experiences in research objectives. To identify the informants who can provide exact and appropriate information, following method could be applied:  Identifying the relevant groups from which key informants may be drawn.  Consulting from different information resources, maybe from a focal point of the research, from available documents, materials, etc. It is necessary to ensure the diversity of the key informants in each topic (about age, ethnic, religion, position). Depending on research objectives, in IFAD projects, key informants often are commune leaders, village leaders, extension services workers, women, ethnic minorities, etc. 3.3. Develop questions

Almost questions used in key informant interview are opened-questions (no guideline). The content of the questions should be clear, transparent, and easy to understand. In addition to questions related to the topic of the interview, it should include suggest questions. The followings are common questions used in key informant interviews:

- Description question: request interviewees describe issues related to the research topic such as characters, events, places, people, etc. This type of question is often used to start interview making the audience feel confidence because of active environment. For example: Can you tell me about your family members?

- Opposition question: Interviewees compare events and discuss their significance. For example: Compared to 05 years ago, the main crops of the family changed? Why have those changes?

109 M&E Manual Guide for IFAD Funded Projects in Vietnam

- Question on perspective /values: study interviewees’ process of thinking and analysis, what they think about certain people, issues, or events. For example: what can the group that the family/family members participate in contribute in the development of production? - Question on knowledge: find out what information interviewees really have and their views on such information. For example: What do you know about local projects? How can you know/get that information? - Question on feeling: study what interviewees see, hear and feel. Through this, the interviewees describe the impact that they are objects. For example: How do you think about the project to contribute to the development of the locality?

- Biographical question: find out some personal characteristics of interviewees. For example: In the past, what was your job?

Box 6.2. Example questions for the topic “Community participation in local planning”

1) How did the Commune Socio-Economic Development Plan (SEDP) in this year make? 2) Which activities in SEDP did you propose? 3) Which activities discussed in Village Development Plan (VDP) were not developed in SEDP? 4) In your opinion, why were these activities not developed in SEDP? 5) What do you want to propose in accessing Community Development Fund (CDF) resources? 6) How does Commune Development Board (CDB) decide? 7) How are you satisfied with SEDP process? 8) What is your suggestion for change of SEDP in the next year?

3.4. Design key informant interview guide

After the questions designed, the next step is to prepare the interview guide. In case there are many people taking charge of key informant interview, this task is special necessary. If key informant interview following a variety of topics, interview guide will be developed under each theme, so the different topics will be differences in the contents of the interview guide. The interview guide need include the following contents: - General information: purpose, subject and interview content; - Themes, content frame and questions for each theme; - Principles, requirements towards researchers when conducting interviews; - Detail guide of how to open and close interview, how to question and encourage interviewees to provide information. For a good guide, it takes time and depends on researchers’ experience in the certain research problems. Sometimes guidelines must be prepared and rewrite many times and after each interview also need to complete to adapt to the new circumstances and subjects. 3.5. Pre-test and revise the questions or interview guide

Similarly to the quantitative survey, pre-test should be conducted to check the suitability and accuracy of the questions before finalizing the questions and interview guide. The pilot of key informant interview could be conducted with 02 – 03 key informants for each topics.

110 How to conduct key informant interview

3.6. Conduct the interview

After completion of preparatory work, a key informant interview will be conducted. To conduct the interview, following points should be taken into account: - Place to interview the informant: depending on the content, object to find an appropriate place for interview; avoid interviewing with the presence of a third person, especially presence of whom causing interference. - Time to conduct interview: The interview should not take more than two hours (the most proper duration is between 01h-01h30’). - Record the answer: It should consult the key informant about using record equipment. In the case of participants agree with the recording, it is still important to take note the content. Note to observe and record the changing emotions of the interviewee (silence, laughing, uncomfortability, incoop- eration, etc.) and find out the cause of these attitude changes. Conducting a key informant interview

How to open a key informant interview? - Confirm with the respondents of the anonymous of the interview. Explain why their comments or observations on a certain topic are important for the study; - Ask permission for recording the interview (if possible) and take notes during the interview; - Always be honest, straightforward and really concerned on what interviewees mention. Let interviewees lead the interview - Ensure that interviewees understand the question; - Ensure that interviewees are able to provide all necessary information that the study needs; - Ensure that they are willing to spend time and effort on the interview; - Attempt to create a friendly communication, safety, sincerity and sympathy; - But do not let the interview become a regular dialogue to avoid rambling, the point. - Rule: Putting certain questions and then let interviewees free. Don’t interrupt them to provide the information that they believe is important. Applying polling technique - Silence: keep quiet to wait for interviewee saying, can come with a nod and waiting eyes; - Repeat: repeat the last sentence interviewee said object and ask them to continue;

- Motivation: encourage interviewee by expression of sympathy, or “yes”, “right”;

- Ask a long question: make interviewee answer more and more impression;

- Guide: do not drive interviewee’s reply by giving questions like “Do you think that ...”. It is better to ask, “What do you think about ...”; - If the interviewee is talkative or digress from the topic: use tip to interrupt them but does not make they displeased;

- Confirm: show that you have taken a certain information about the topic of the interview that makes the interviewee more open and less anxious for information disclosure.

111 M&E Manual Guide for IFAD Funded Projects in Vietnam

How to close the interview? - Keep good relations with the interviewee for subsequent interviews; - Be grateful attitude and appreciate the information that respondents had provided; - Ask for contact information of respondents: personal phone number, email address (if unavailable);

- Able to end the interview earlier than planned or requested for lengthening interviews (if necessary).

Box 6.3. Tips to have a good key informant interview

Improve interview skill  Practice as much as possible;  Rehearse with others under supervision/help of an experienced person;  Rehearse interview with your interested topics with someone who are really expert in Recorder usage  Prepare a good recorder with battery;  Always try before interview;  Always note-taking for cases of unavailable recording. 3.7. Analyse and write report

The data collected from note-taking and transcript is qualitative information, so information summary after interview is very important. Ideally, there is one person in charge of analysis the information collected and writing the report. How- ever, in order to do so, all interviewers should prepare and provide him/her their transcript of the interview. In this case, the outline and content of the transcript should be agreed among the interviewers and report writer. Followings are steps to gather and analyze information collected from key informant interview: - After each interview, interviewer should prepare an interview summary in form of transcript, including the answers for each question, and assessment of interviewer about the attitude of interviewees; - Summering information by topics: for each topic, prepare a short summary (2-3 pages) to present most important findings, also to emphases on agreed opinions and opposite views; . - Writing report: The report will analysis all information collected through key informant interviews. Depends on the purpose, each topic could be used to illustrate the findings from quantitative survey, to tell stories of best practices, or to develop to be a case study.

Structure of a story with information for a key informant interview Headline Standfirst/Lead paragraph Photo Summary Caption Story (most important part) What happened? What did the project do? So what? What are the results? Examples Testimonial/quotes form participants Background (less important part) Details about the project, components, supporting data Contact Name(s) for further information, job title, address, phone (optional)

112 How to conduct focus group discussion

Chapter VII.

How to conduct focus group discussion

113 M&E Manual Guide for IFAD Funded Projects in Vietnam

114 How to conduct focus group discussion

Chapter VII. How to conduct focus group discussion

1. Focus Group Discussion

What is a focus group discussion?

When use focus group?

Group discussions are often used to assess the needs, collect opinion from many people to evaluate the same problem, or discuss on the controversial content, or collect new ideas.

Advantages

- Provide a significant amount of information quickly and less expensive than individual interviews;

- Very useful in understanding the concepts, attitudes and behavior of the community;

- Support for determining the appropriate questions for key informant interview.

Disadvantages

- Facilitation of a focus group requires considerable skills – both in moderating the group and in adequately recording the responses;

- The flexible format makes it susceptible to facilitator bias, which can undermine the validity and reliability of findings;

- Discussions can be sidetracked or dominated by a few vocal individuals;

- The information can be more difficult to analyze than ones from key informant interview;

- The number of concerned issues in the focus group may be less than that one of key informant interview.

2. Steps to facilitate a focus group

To conduct the focus group discussion, following steps should be undertaken:

1) Identify topics and information needed for each topic

2) Prepare guide for discussion

3) Pre-test and revise the discussion guide

4) Select participants

5) Conduct FGD

6) Analyze results and write report

115 M&E Manual Guide for IFAD Funded Projects in Vietnam

2.1. Identify topics and information needed a. Identify topics

The first step to prepare a FGD is to identify topics and specific information needs for each topic (for example: the information you need in order to know whether a particular outcome was achieved and why/why not). To serve the annual assessment results, focus group discussions can be made after the results of the quantitative survey to which may collect information to explain and clarify the survey results obtained. In this case, the discussion topics will be determined after survey completion and results of the preliminary analysis. On the other hand, it can also determine the topics of discussion groups before the survey and interviews with key informant to detect issues that need to learn, study, gather more qualitative and quantitative information

Box 7.1. Example of topics for focus group discussion

- Community participation in local social and economic development planning; - Market information accessibility, market linkages; - The effect of the cropping technical support that the project has made to the poor, ethnic minorities - Poor Household accessibility to financial services; - The effectiveness of vocational training and job consultancy for women; Advantages and disadvantages in the formation and development of interest groups; The effect of participation in interest groups / cooperatives.

2.2. Specify information needs for each topic

After identifying the topic of focus group, the next task is to determine the information to be collected for that subject. To do so it is necessary to design appropriate questions. Similar to key informant interview, Almost questions in focus group discussion are opened-questions, such as “why?” and “how?”

Twelve is the maximum number of questions for any one group. Ten is better, and eight is ideal.

Focus group participants won’t have a chance to see the questions they are being asked. So, to make sure they understand and can fully respond to the questions posed, questions should be:

 Short and to the point;

 Focused on one dimension each;

 Unambiguously worded;

 Open-ended or sentence completion types;

 Non-threatening or embarrassing;

 Worded in a way that they cannot be answered with a simple “yes” or “no” answer (use “why” and “how” instead).

116 How to conduct focus group discussion

Box 7.2. Example for questions designed for Focus group discussion suggested by IFAD

Here are questions designed for focus group discussions suggested by IFAD under the title “Production of agricultural technology”; participants were farmers, the objective of the discussion is (i) assess the quality of training and extension services; (ii) evaluate the impact of training courses on agricultural / livestock production:

• What kind of training or extension services have you recently received by the project? Why have you participated in?

• In your family, who often came to training courses? How often did women in your family participate in those trainings? When women participate in the training courses mentioned above, who would take over housework? Was place held training convenient? Was training time convenient? How good of attitude were the extension faculty / staff?

• Had the training topics / extension met your demand? Could you impact on selecting of the training topics / extension services?

• Could you apply the new technology? If not, why not? If yes, what was the result? Could new technologies help increase your agricultural / livestock productivity?

• Did you have sufficient access to the necessary production inputs? If not, why didn’t you access? How could the project help?

There are three types of focus group questions:

(i) Engagement questions: introduce participants to and make them comfortable with the topic of discussion (ii) Exploration questions: get to the meat of the discussion (iii) Exit question: check to see if anything was missed in the discussion 2.3. Prepare the discussion guide

After specifying the topics and information needs for each topic, the next step is to prepare the discussion guide. The discussion guide should be developed for each specific topic. In the discussion guide, you can leave blank space under each question to record the opinions of participants. The discussion guide includes the following contents: - General information: purpose, subject and discussion content; - Themes, content frame and questions for each theme; - Participants and number of participants; - Principles, requirements towards facilitator and assistant when conducting discussion;

- Detail guide of how to open, facilitate and close the discussion; how to question and encourage participants raise their voices.

For a good guide, it takes time and depends on researchers’ experience in the certain research problems. Sometimes guidelines must be prepared and rewrite many times and after each interview also need to complete to adapt to the new circumstances and subjects.

117 M&E Manual Guide for IFAD Funded Projects in Vietnam

2.4. Select participants

Each discussion should not have more than twelve people; ideally, there are 8-12 people taking part in and they must be the objects that share a certain number of characteristics consistent with the theme of the discussion; for example, the same education, same age, same gender, etc.

Focus groups participants can be recruited in any one of a number of ways. Some of the most popular include:

- Nomination – Key individuals nominate people they think would make good participants. Nominees are familiar with the topic, known for their ability to respectfully share their opinions, and willing to volunteer about 2 hours of their time.

- Random selection – If participants will come from a large but defined group (e.g. a cooperative group) with many eager participants, names can be randomly drawn from a hat until the desired number of verified participants is achieved.

- All members of the same group – Sometimes an already existing group serves as an ideal pool from which to invite participants (e.g. common interest group, women saving and credit group).

- Same role/job title – Depending on the topic, the pool might be defined by position, title or condition (e.g. commune officer, extension staff, project officers).

- Volunteers – When selection criteria are broad, participants can be recruited with flyers and news- paper ads.

Box 7.3. Hint identify participants for group discussions with IFAD projects

For IFAD projects, it should organize a group discussion on the same topic with the participation of different groups. For example on the theme “The participation of the community in designing local social and economic development plan”, it can invite the following groups: - Women; - Poor households; - Rich households; - Ethnic minority households; - Kinh households. Discussion should be held separately for each group to avoid the more vulnerable groups (the poor, minorities, etc.) be overwhelmed by other groups and therefore do not have the opportunity to raise their opinions.

2.5. Pre-test and revise the discussion guide

Pre-test is an important step in designing interview guide in general and designing discussion in particular. For group discussion, it is recommended to conduct a pre-test with a certain discussion topic. In the process of testing, it notes to check the suitability and accuracy of the questions and terms used in the questions; questions whether could collected information for the purpose of research or not; questions whether express coordinator’s subjective opinion or not; questions whether make suggestion discuss or not; question whether are easy to answer or not; whether question sequence is consistent or not; whether participants are appropriate or not; whether skills of facilitator are good enough, etc. If it found any inappropriate content and discussion topics, it should be timely adjusted.

118 How to conduct focus group discussion

2.6. Conduct focus group discussion

The topic of discussion will be conducted with different groups. Group discussion is just conducted after a well-preparation. In addition to good preparation, the discussion facilitator and the assistant play a very important role in the discussion. The followings will present some issues to be considered and the responsibilities of facilitators and assistants when carrying focus group discussions.

 Issues to be taken into account when carrying FGD

Location

Depending on the content, participants to find the appropriate place. Generally it should carry out in a separate place where participants feel comfortable to discuss and raise opinions, and try to avoid the participation of unexpected persons. Participants are recommended to sit in a circle so that all can see each other and discuss easily.

Time

The ideal amount of time to set aside for a focus group is anywhere from 45 to 90 minutes. Beyond that most groups are not productive and it becomes an imposition on participant time.

Discussion issues

It should not let the participants know about detailed content of the discussion but only if general content. Only when we start the discussion, the coordinator will give specific details about the study to participants, to avoid the case of agreement among participants before the group discussions.

 The responsibilities of facilitators and assistants

The responsibilities of facilitators

The facilitator can be an M&E staff at provincial level or the leader of an enumerator team, or other project staff. The facilitator is responsible for:

- Directly instructing and facilitating the discussion;

- Raising the questions in a managed time;

- Encouraging the participants to take part in the discussion;

- Explaining all questions from the participants.

The ideal focus group moderator has the following traits:

- Can listen attentively with sensitivity and empathy

- Is able to listen and think at the same time

- Believes that all group participants have something to offer no matter what their education, experience, or background

- Has adequate knowledge of the topic

- Can keep personal views and ego out of the facilitation

119 M&E Manual Guide for IFAD Funded Projects in Vietnam

- Is someone the group can relate to but also give authority to (e.g. a female moderator is most appropriate for a group of all women discussing gender equality in the family)

- Can appropriately manage challenging group dynamics

The responsibilities of assistants

The assistant can be a M&E staff at provincial level or project staffs at district level. The assistant is responsible for:

- Run a tape recorder during the session

- Take notes

- Note/record body language or other subtle but relevant clues

- Allow the moderator to do all the talking during the group

 Conducting group discussion

At the beginning both moderator and assistant moderator are expected to welcome participants, offer them food, help them make their name tents, and direct them in completing pre-group paperwork. The facilitator will introduce the names of participants in the discussion, or guide the participants of the discussion to introduce the name, age, location. Assistant should record this information, can number participants in order. The facilitator states the topics and then invites guests to discuss; participants are called name or number previously associated.

It may be important to collect demographic information from participants if age, gender, or other attributes are important for correlation with focus group findings. Design a short half page form that requires no more than two or three minutes to complete. Administer it before the focus group begins.

Once consent forms and demographic surveys are collected and reviewed for completeness, the ques- tioning begins. The moderator uses a prepared script to welcome participants, remind them of the purpose of the group and also sets ground rules.

120 How to conduct focus group discussion

Box 7.4. Example of Focus group discussion introduction

FOCUS GROUP INTRODUCTION WELCOME

Thanks for agreeing to be part of the focus group. We appreciate your willingness to

participate.

INTRODUCTIONS

Moderator; assistant moderator

PURPOSE OF FOCUS GROUPS

We have been asked by ______to conduct the focus groups.

The reason we are having these focus groups is to find out______.

We need your input and want you to share your honest and open thoughts with us.

GROUND RULES

1. WE WANT YOU TO DO THE TALKING.

We would like everyone to participate.

I may call on you if I haven’t heard from you in a while.

2. THERE ARE NO RIGHT OR WRONG ANSWERS

Every person’s experiences and opinions are important.

Speak up whether you agree or disagree.

We want to hear a wide range of opinions.

3. WHAT IS SAID IN THIS ROOM STAYS HERE

We want folks to feel comfortable sharing when sensitive issues come up.

4. WE WILL BE TAPE RECORDING THE GROUP

We want to capture everything you have to say.

We don’t identify anyone by name in our report. You will remain anonymous.

Before asking the first focus group question, an icebreaker can be inserted to increase comfort and level the playing field.

The focus group moderator has a responsibility to adequately cover all prepared questions within the time allotted. The facilitator should not make their assessment on discussion. Because the moderator holds a position of authority and perceived influence, s/he must remain neutral, refraining from nod- ding/raising eyebrows, agreeing/disagreeing, or praising/denigrating any comment made. Make sure everyone involved is to express an opinion, avoid discussion is dominated by the opinions of some participants.

121 M&E Manual Guide for IFAD Funded Projects in Vietnam

Box 7.5. Some suggestions to have a good focus group discussion

To get participant fully explain their answers: 􀂃 “Can you talk about that more?” 􀂃 “Help me understand what you mean” 􀂃 “Can you give an example?”

Strategies to deal with challenging participants: 􀂃 Self-appointed experts: “Thank you. What do other people think?” 􀂃 The dominator: “Let’s have some other comments.” 􀂃 The rambler: Stop eye contact; look at your watch; jump in at their inhale. 􀂃 The shy participant: Make eye contact; call on them; smile at them. 􀂃 The participant who talks very quietly: Ask them to repeat their response more loudly.

When the focus group is complete the moderator thanks all participants and distributes the honorarium (incentive).

Immediately after all participants leave, the moderator and assistant moderator debrief while the recorder is still running and label all tapes and notes with the date, time (if more than one group per day), and name of the group.

2.7. Analyze information and write report

In order for all participant comments to be understandable and useful, they must be boiled down to essential information using a systematic and verifiable process. Begin by transcribing all focus group tapes and inserting notes into transcribed material where appropriate. Clean up transcripts by stripping off nonessential words. Simultaneously assign each participant comment/quote a separate line on the page as well as each new thought or idea therein. Label each line with the participant and group number, e.g. a comment from participant 6 in group 2 would be assigned the number 2.6. Each line is then entered into an Excel database as follows: COMPILE

1) Use a separate Excel data base spreadsheet for each group. 2) Within each spreadsheet, use one sheet per question. 3) Label three columns on each sheet. - One column for coding - One column for the participant ID# - One column for responses 4) Enter each separate response or idea on a separate line with participant ID attached. The coding column is filled in during the next phase - analysis.

ANALYZE

122 How to conduct focus group discussion

1) When all comments have been entered, look for common categories or themes across the entries for each question.

2) Several people to participate in this process.

3) Once consensus has been achieved regarding the best categories for organizing the data, assign a number or letter to each category.

4) Then assign the number/letter of the category that best fits to each entry on the sheet.

5) Use the Excel ‘Sort’ function to group entries by the categories you have assigned to them.

6) If some entries seem inconsistent for their category, consider re-categorizing or adding another category. It may also be apparent that one or more categories can be collapsed.

7) Arrange categories from those with the largest number of entries to those with the smallest.

8) Repeat for each group.

SYNTHESIZE

1) Identify category and sub-category heading titles.

2) Write a short paragraph summarizing findings for each sub-category possibly noting similarities and differences across groups.

3) Add powerful quotes to each sub-section

As an alternative to the spreadsheet method described here, you can also use a manual approach to analyzing focus group data. Make a copy of the transcript after it has been cleaned and labeled. Working on a large table, cut entries into separate strips and run a glue stick over the back. Categorize by sticking entries onto separate sheets of paper labeled with broad headings. Re-categorize as indicated until you are satisfied with your groupings. Enter into a Word document.

Once focus group findings are organized in the synthesized format they are ready for presentation. If a more formal report is required, findings can be written up in a narrative format that includes an executive summary, background section, methods used, major findings, conclusions, and recommendations.

Another way to make a report of FGD is following these steps:

- Make a short summary of main points for each questions.

- Review all the answers, take notes of the points most frequently mentioned and divide the responses into those “for” or “against” a certain issue, or by degree of enthusiasm about an issue.

- Take out any important quote to emphasis certain points and make the analysis more lively

- Make a report from the summary of all FGD of all topics, or for each topics and used as illustra- tions for the survey findings report.

123 M&E Manual Guide for IFAD Funded Projects in Vietnam

124 How to conduct focus group discussion

Chapter VIII.

How to entry data using CSPro

125 M&E Manual Guide for IFAD Funded Projects in Vietnam

126 How to entry data using CSPro

Chapter VIII. How to entry data using CSPro

1. What is CSPro?

The Census and Survey Processing System (CSPro) is a free software package for entering, editing, tabulating, and disseminating data from censuses and other surveys. It was designed by the United States Census Bureau, Macro International, and Serpro, S.A.

CSPro is a user-friendly mechanism for double-checking and entry error detection. CSPro is fully designed as a questionnaire that facilitates the process of data entry as well as minimizes simple mistakes during the process such as missing, mismatching, or misreading data. Furthermore, CSPro allows different users to combine input data and so permits the simultaneous cooperation and participation of these different users; additionally, could be able to cooperate simultaneously; it can also find mistakes automatically by using a double-check command. Data from CSPro can be easily transferred to other statistical software such as Excel, SPSS or STATA for further processing and analysis. Therefore, it is recommended to use CSPro for entering RIMS+ data.

This Chapter will focus on the components: “Data entry application” and “Data Dictionary”.

 The “Data Entry Application” is where data entry occurs. Each survey is saved as a separate “case”. Here, you can search for specific surveys or answers within a survey in addition to performing basic data entry. You can also modify previously entered cases.

 The “Data Dictionary” is used to compare the contents of two data files to identify the differences, concatenate (join end-to-end or merge) two or more data files, tabulate frequencies to check for outliers, and export data to files can be imported into spreadsheets or databases. 2. Data Entry application

This module allows you to enter data into the template data entry form. CSPro automatically save the data entered into the template. To enter the survey data, follow these steps:

Step 1. Double click on the CSPro Data Entry Application file in the folder in which you save your template. It is the file that has a computer icon next to it.

Step 2. The “Select data file” window will open. To create a new file for data entry (called a “case” in CSPro), navigate to the folder in which you want to store the file and type in a name for the file. Then click “Open”.

Step 3. When you are asked if you want to “Create the file?”, click “Yes”. The data entry screen will open.

127 M&E Manual Guide for IFAD Funded Projects in Vietnam

Step 4. There are four modes of operation within Data Entry

a. Add – Entering new case (a case corresponds to a questionnaire. Each questionnaire has a distinct code).

b. Modify – Modifying previously entered cases.

c. Pause – Temporarily stop adding or modifying cases.

d. Stop – Stop adding or modifying cases. In this mode, the data entry form is not visible.

Step 5. Colors are used to indicate the status of each field.

a. White – The cursor or mouse has not moved onto that field during data entry.

b. Green – The cursor or mouse has passed through that field for data entry input. All fields in a survey must be green.

c. Yellow – The cursor or mouse has passed through that field for data entry input and you have moved backwards over the field.

d. Gray – The field is protected and data entry cannot occur for this field.

Step 6. To enter data, click the first empty field and start typing. To move to the next field, use the “Enter” or “Tab” key.

Step 7. When you reach the last field on the first pate and press “Enter” or “Tab”, you are immediately brought to the next page of the survey. When you finish answering the last question of the entire survey, pressing “Enter” or “Tab” brings you to the next case. To navigate between pages within a case, use the “Page Up” and “Page Down” keyboard keys.

Step 8. If you have to verify a response and wish to come back to a survey later, you can “partially save” a case. To save a partially entered survey you must at minimum enter the survey number. To temporarily stop data entry, select “Stop” from the “Mode” menu, or click Stop on the toolbar. The “Stop Adding” window will open; click “Partial Save.”

Step 9. To resume data entry, select “Add” from the “Mode” menu, or click Add on the toolbar to add a new entry.

Step 10. When data entry is complete, from the “Mode” menu, select “Stop” or click Stop on the toolbar and close the window. Your changes are saved automatically.

IMPORTANT: As data is entered, completed fields turn green. Yellow fields indicate that the person entering data has moved backwards over it or has skipped over the field during data entry.

128 How to entry data using CSPro

In order to change a yellow field to green, you should not just click the field and correct the value. You should go to the first field in the survey and click Enter or Tab until the yellow field is reached, then change the value and continue clicking Enter or Tab until the end of the survey is reached. Some users have encountered errors correcting individual values without going through the survey from beginning to end.

3. Accessing Data that has been saved

When you want to access a saved input data file (add new cases or make changes), you must first open the Data Entry Application (the one with the computer icon), then open the input data file (with Type “File”) in the dialogue box that opens up.

Double-click the CSPro icon on the desktop of your computer, Start CSPro window will appear. Choose “Open an existing application” and click OK.

Choose the folder containing the Data Entry Template of your project. Choose the file with icon , then click Open.

129 M&E Manual Guide for IFAD Funded Projects in Vietnam

4. Modifying data and adding new cases

Step 1. Open the Data Entry Application.

Step 2. Select the Input Data File containing the data you wish to modify.

Step 3. Double click on the survey number you wish to modify or click the survey number once and from the “Mode” menu, select “Modify”.

Step 4. The “Survey Number” field will be automatically selected. Press “Tab” or “Enter” to advance to the field you wish to modify. DO NOT mouse click on the field – you must use the “Tab” or “Enter” key to move from field to field. While you may be able to click and modify, doing so may cause errors in the program’s operation – it is a quirk of CSPro.

Step 5. Make the edit and then press the “Enter” or “Tab” key until reaching the end of the survey. This step is crucial to ensuring that changes are recorded properly.

Step 6. To add a new case to an existing Input Data File, open the file and from the “Mode” menu, select “Add Case.”

5. Compare data tool

Before analyzing your data, you should go through a few quality checks. The first one involves comparing data. In order to reduce data entry error, it is recommended that each survey be entered by TWO different data processors. While individuals can make errors easily (e.g., finger slips on the keyboard, entering “222” instead of “22”) it is unlikely that two people will make the same error. CSPro includes a “Compare Data” tool that allows you to compare the contents of two data files that contain the same cases (identified by Household Code) and locate the differences.

The same cases (surveys) are entered twice using the Data Entry Application, preferably by two different data processors, and saved to two separate data files.

For example, data entry person Huong might save her data file as “Baseline_Lang Quan_ Huong_ 100_150”, and Thang would save his as “Baseline_ Lang Quan_Thang_ 100_150” (100-150 indicates that they both entered surveys numbered 100 through 150 into their data file.) To identify differences between Huong and Thang’s surveys, follow the following steps.

Step 1: From the Menu Toolbar (CSPro 4.1 window) select the “Tools” menu near the top of the screen, then “Compare data”.

130 How to entry data using CSPro

Step 2: In the pop-up window, select the dictionary file (.dcf) with the icon and click open.

Step 3: In the “CSDiff” window, select the variables you wish to compare. If you want to compare all variables, click the box next to the dictionary icon to select all.

Step 4: Click on the stoplight icon in the toolbar. The “Run Compare data” window will appear on the screen. Click on the grey box to the right of “Input File”

131 M&E Manual Guide for IFAD Funded Projects in Vietnam

Step 5: In the Input File, select the first data file (e.g. Lang Quan) and click Open.

Should be the file in type of “File”

132 How to entry data using CSPro

Step 6. Click on the grey box to the right of “Reference File”, and select the second data file (e.g. Lang Quan 2).

Step 7: Click “Compare Input to Reference” and “Compare in indexed order” and click OK.

133 M&E Manual Guide for IFAD Funded Projects in Vietnam

If the two data files are not the same, the differences will appear in the “Text Viewer” Window.

To save the output that shows the differences, select “File” at the top of the screen. In the “File” menu select ‘Save As.” Name and save the file in the folder with the Template. In order to facilitate mistake cor- rection, you can copy this to Microsoft Word, switch font words to Vn.Time.

Follow the instructions for modifying the data to correct any errors you find. If there is any doubt as to what a correct value should be, go back to the paper survey and verify what is written.

When you close the program and are asked if you want to save changes, click ‘Yes’.

Once you have corrected the data, rename5 one set of files to indicate it has gone through the “Compare Data” check.

6. Merging data (Concatenate Data Tool)

If different surveys are entered into two different data files, they will need to be merged. The “Concat- enate Data” tool is used to perform this function.

Step 1: Open the Data Entry Form. It is the file with the book icon next to it.

Step 2: From Menu Tool- bar, select Tools near the top of the screen. Select Concatenate Data.

134 How to entry data using CSPro

The “CSConcat” window opens

Step 3. The Output file is the file that receives the results of the concatenation (the file that contains the merged data). Press the Browse button and name an output file into which separate databases will be merged and click OK.

Step 4: Click Add. The “Select Files to Concate” window opens. Double click the first file you wish to merge. E.g. The first data is Lang Quan_Hung and then Select Add to select more file to concate.

SelectChọn “Add” Add

135 M&E Manual Guide for IFAD Funded Projects in Vietnam

Select file in type Ch of “File”ọn “File”

Step 5: Click “Add” again. Double click on the second file you wish to merge. The order of the list of files to concatenate is the order of concatenation. To change the order of concatenation you need to rearrange the file names in the list. You can rearrange the names by drag and drop or sorting.

Step 6: When you see the files you wish to merge listed, click “Run”. When the merging is completed, a box will say “Concatenate completed”.

136 How to entry data using CSPro

7. Exporting Data Tool

The “Export Data” Tool is a tool that you may find useful in analyzing your survey data. The tool allows you to export data from a CSPro input data file to tab or comma delimited text files that can be imported into spreadsheets or databases. For easier import into spreadsheets and databases, household and individual level data should be exported separately.

To export household level data to SPSS, complete the following steps.

Step 1: Open Data Entry Form.

Step 2: Select the Tools menu near the top of the screen, then “Export data”.

Step 3. In the file menu, select the dictionary file (*.dcf), then click Open.

137 M&E Manual Guide for IFAD Funded Projects in Vietnam

Step 4: In the “CSExport” window, click every box you wish to export the data.

Step 5. In the “CSExport” window, click “One File”, “All in One Record” and “SPSS”

Step 6. Click on the stoplight icon in the toolbar. A window will appear on the screen. In the file menu, select the data file you want to export and click “Open”. Note: Select the file in type of “File”.

138 How to entry data using CSPro

Step 7: Window “Specify Names of Exported Files” will appear.

Select Ok to create 2 files: Exported.dat and Exported.sps.

As default, CSPro will store the 2 files in the folder that contains the Data Entry Form.

Step 8: Open SPSS to see the exported files.

- Open SPSS by double-clicking the icon SPSS on the Desktop. Cancel the pop-up window.

- On the Menu toolbar, select File à Open à Syntax; select file Exported.sps.

139 M&E Manual Guide for IFAD Funded Projects in Vietnam

Step 9: After opening the file Exported.sps, select Run à All.

Step 10: in the SPSS window (with 2 tabs: Data View and Variable view), save the data by select File à Save.

8. Some notes on developing database using CSPro

 Make sure that your computer is installed CSPro.

 Turn off Vietnamese keyboard (e.g. Vietkey or Unikey).

 When saving the file or folder which data is entered in, use name in Vietnamese without mark and do not change the name of the file or folder created, as well as the data drive.

 CSPro automatically saves when a questionnaire is en- Chú ý: không được chỉnh sửa bắt kì phần nào trong phần Dicts. Mặc tered completely. If you exit Noteđịnh: Any là đang changes ở dạng madeForms nhưto the hình Dicts vẽ. N ếtabu đã may kích causevào Dicts erro thìrs to the the form when the entry does Form.đóng If l ạDicti và sau is đóopened mở lại. by accident, close the Dict then re-open the not complete, a questionnaire file without saving changes. will not saved by CSPro and you will have to enter it again.

Data management

 Create a separate folder for each district with the name of district. So the survey is conducted in how many districts, there will be the same folders. For example: Annual Outcome Survey Project in Tuyen Quang is conducted at 06 districts, so there is 06 folders created, such as Yen Son, Son Duong…  In the folder of each district, create two small folders with the following structure: project area_ name of district and non-project area_name of district.  E.g. project area_Yen Son; non-project area_Yen Son.  Once you have database of all districts, provincial officers are responsible for the data file of whole survey. This also includes two folders: project area and non-project area.

140 How to entry data using CSPro

Chapter IX.

How to analysis data by SPSS

141 M&E Manual Guide for IFAD Funded Projects in Vietnam

142 How to entry data using CSPro

Chapter IX. How to analysis data by SPSS

SPSS (Statistical Package for the Social Sciences) is a statistical analysis and data management software package. SPSS can take data from almost any type of file and use them to generate tabulated reports, charts, and plots of distributions and trends, descriptive statistics, and conduct complex statistical analyses.

As recommended, the data is entered by CSPro software and after completing, it will be exported to SPSS for statistical analysis. This Chapter will provide guidance on how to use SPSS for analysing data of IFAD projects in Viet Nam. 1. Opening an existing SPSS data set

As recommended, the data is entered by CSPro software and after completing, it will be exported to SPSS for statistical analysis. To open the data set, follow these steps:

Step 1. Open SPSS by double-clicking the icon SPSS on the Desktop. Cancel the pop-up window.

On the Menu toolbar, select Open from the File menu. This will open the Open File dialog box.

Step 2. From the Files of type drop-down list, select .sav.

Step 3. From the Look in drop-down list, select the appropriate drive where the file is located.

Step 4. In the File name box, find the name of the file to be opened and Click Open. 2. Set up variable name and code variable

Defining variables name

Variables are defined one at a time using the Define Variable dialog box. This box assigns data definition information to variables. To access the Define Variable dialog box, double-click on the top of a column where the word var appears or select Define Variable from the Data menu.

143 M&E Manual Guide for IFAD Funded Projects in Vietnam

Variable Name

This field describes the name of the variable being defined. To change the name, place the cursor in this field and type the name. The variable name must begin with a letter of the alphabet and cannot exceed 8 characters. Spaces are not allowed within the variable name. Each variable name must be unique.

Type of variable

This field describes the type of variable that is being defined. To change this field, click on the Type… button. This will open the Define Variable Type: dialog box. Select the appropriate type of data. When done, click on the Continue button.

144 How to analysis data by SPSS

Variable Label

There are two types of variable labels:

i) Variable Label: A name for the variable that can be up to 120 characters long and can include spaces (which variable names cannot). If a variable label is entered, the label will be printed on charts and reports instead of the name, making them easier to understand.

ii) Value Label: Provides a key for translating numeric data. To change the variable label, click on the Labels… button. This will open the Define Labels: dialog box. Enter the appropriate information into the fields. When done, click on the Continue button.

3. Data analysis

3.1. Frequency

This command is only applied to a qualitative variable.

Example: Analyze to know that in the total of the surveyed project households, how many male/female- headed households and what is the percentage.

Solution:

- Open the database file in SPSS: Data View.

- From the Toolbar, click Analyze, select Descriptive Statistic and then select Frequencies.

145 M&E Manual Guide for IFAD Funded Projects in Vietnam

- In Frequencies menu: select the variables related, e.g. Sex of household and sex of respondent, click OK

146 How to analysis data by SPSS

Results: Sex of household Cumulative Frequency Percent Valid Percent Percent Valid Male 11 55.0 57.9 57.9 Female 8 40.0 42.1 100.0 Total 19 95.0 100.0 Missing System 1 5.0 Total 20 100.0 Sex of respondent Cumulative Frequency Percent Valid Percent Percent Valid Male 10 50.0 50.0 50.0 Female 10 50.0 50.0 100.0 Total 20 100.0 100.0 Reading the data

- Cumulative %: the percentage increased by successive addition.

- Percent: including missing values.

- Valid %: excluding missing values.

- Look at the table of gender of head of households, the last row (total) of the frequency column presents the total number of observations (number of row, number of questionnaire) is 20, one among them is missing. We can infer that the data entry staff did not enter the data or the interviewer forgot to record the answer. Since the answer for this question is compulsory, we have to check the hard questionnaire to find out the information.

3.2. Calculate the mean (Descriptive)

This command is often used for quantitative variables and used only for one variable. The purpose is to find the mean, maximum, minimum, and standard deviation of the variable.

Note:

- Before calculating the mean of a variable, it is necessary to Dealing with outliners (if any)

- Mean of SPSS software does not take into account missing values.

Examples:

- The average distance from the home to the nearest brick/concrete/asphalt road.

- The average area of agricultural land per household?

147 M&E Manual Guide for IFAD Funded Projects in Vietnam

Solution:

- From the Toolbar, click Analyze, select Descriptive Statistic and next select Descriptive

- In Descriptive menu: select the variables that you wish to have the description. Click OK

Results

N Minimum Maximum Mean Std. Deviation A9_1 Duong gach be tong rai 20 -9000 4000 2040.00 2861.063 nhua gan nhat C2_1. Dat nong nghiep 20 1080 4500 2804.00 1410.242 Valid N (listwise) 20

Reading the data:

Look at the above table, the minimum of the first line is suspected because the distance is always greater than or equal to zero, so “-9000” is invalid value and needs to be edited.

Two cases occurred:

+ Case 1: This invalid value (-9000) is due to the mistake of data entry officer or investigator. There- fore, it is necessary to check the original hard copy of the questionnaire.

+ Case 2: In the case of the respondents did not estimate the distance from their home to the nearest brick/concrete/asphalt road (value -9999), these values will be considered as missing ones. To adjust, select Variable View, click to None in the Missing column (A9_1 and C2_1).

148 How to analysis data by SPSS

Click to icon “…”, Missing Value window will appear, enter “-9000”, “-9999” à “-9999” and “-9000” will be considered as missing values.

Then click OK:

After editing database and entering missing values, the results table will be re-produced (as below):

N Minimum Maximum Mean Std. Deviation A9_1 Duong gach be tong, rai nhua 19 1000 4000 2621.05 1229.99 gan nhat C2_1. Dat nong nghiep 20 1080 4500 2804 1410.24 Valid N (listwise) 19

149 M&E Manual Guide for IFAD Funded Projects in Vietnam

N: Total number of households answered.

Std. Deviation: Standard deviation.

Looking at the first row of the table, there are 19 people answered the question A9_1 (one missing): in which the distance from the household to the nearest asphalt/concrete road (in the research) ranges 1-4 km, and the average distance is 2.62 km.

In the second row, there are 20 households participated in the survey, the average area of agricultural land for cultivation per household is 2,804 m2, in which the lowest area is 1,080 m2 and the largest area is 4,500m2.

Note: After calculating the mean, in case of removal of outliner, it is necessary to restore this: Data à Select Cases à Select All cases.

3.3. Crosstab (use for ordinal and nominal variables)

This command is used for two qualitative variables: ordinal variables and nominal variables.

Example: To find out that sex of household affects the person named on the land use certificate or not, do the following:

• Running the correlation between two variables: sex of household (Gender_CH) and person named on forestry land use certificate (C4_2).

Solution:

- From the Toolbar, click Analyze, select Descriptive Statistic and then select Crosstab

- At Crosstabs Menu: select variables that you wish to analyse, e.g. Gioi tinh_CH for “Row” and C4_2 for “Column”

Click Cells

150 How to analysis data by SPSS

- Edit data %: Click “Cells” (at the bottom of table in SPSS 13.0 and at right corner in SPSS 16.0).

Then Click : Row, Column and Total (as below picture + click Total).

- Select Continue and click OK you will have the results as below:

Gioi tinh cua chu ho * C4_2. Lam nghiep Crosstabulation C4_2. Lam nghiep Ca vo va Total Vo Chong Khac chong Gioi tinh Nam Count 4 8 0 4 16 cua chu % within Gioi tinh cua chu ho 25.0% 50.0% .0% 25.0% 100.0% ho % within C4_2. Lam nghiep 100.0% 100.0% .0% 100.0% 80.0% % of Total 20.0% 40.0% .0% 20.0% 80.0% N÷ Count 0 0 4 0 4 % within Gioi tinh cua chu ho .0% .0% 100.0% .0% 100.0% % within C4_2. Lam nghiep .0% .0% 100.0% .0% 20.0% % of Total .0% .0% 20.0% .0% 20.0% Total Count 4 8 4 4 20 % within Gioi tinh cua chu ho 20.0% 40.0% 20.0% 20.0% 100.0% % within C4_2. Lam nghiep 100.0% 100.0% 100.0% 100.0% 100.0% % of Total 20.0% 40.0% 20.0% 20.0% 100.0%

151 M&E Manual Guide for IFAD Funded Projects in Vietnam

Reading the data table

Option 1: If we want to describe data by row.

- Example: Among 16 male heads of households who answer the question “Whose name is written in the land use certificate of forest land?” the answer is “wife” in 25% of households (4/16*100) and 50% of households have the husband’s name in the land use certificate. Look at the next row, a total of 4 households, whose heads are women, answer the above question, and all of them (1005) have both the wife and husband’s names in the land use certificate for forest land.

Option 2: If we want to describe data by column, the results are read similarly to that by row. It should be noted that the results must be practical.

3.4. Compare the means of variables

Compare Mean command compares the means of a quantitative variable with a qualitative variable (nominal or ordinal). The continuous variable is an independent variable (Dependent list) and the independent variable will a qualitative variable (Independent list). For example, we have various plants (spring paddy, seasonal paddy, corn, peanuts, etc) and want to understand the average costs of plants or compare the average costs of plants.

Example:

- The cost in agriculture (for spring paddy, seasonal paddy, peanuts, oranges, etc.) (D12): Identify the average cost for each type of crop?

- Identify the average cost per unit area (D12/D4) by each agricultural crop?

Solution:

1. The cost in agriculture (for spring paddy, seasonal paddy, peanuts, oranges, etc.) (D12): Identify the average cost for each type of crop?

- From the Toolbar, click Analyze and select Compare Means, then select Means

- In Mean menu:

 Dependent list: select the dependent variable), e.g. D12 (continuous variable)

 Independent list: select the independent variable, e.g. D2 (nominal variable)

 Press OK.

152 How to analysis data by SPSS

Results

Report

D12. Chi phi trong 12 thang gan day

D2. Gia dinh co trong loai cay sau trong 12 thang gan day? Mean N Std. Deviation

Lua xuan 1819200 20 705898.875

Lua mua 2200000 12 968597.862

Ngo 410000 4 0

Read the data

Mean: The average value

N: Number of households grew the corresponding trees.

Std. Deviation: The standard deviation.

The results table shows the average cost of each crop but it is not able to compare them to each other. Because the number of observations (N) is very different and very small, it’s not enough to compare.

- There are 20 households that grow spring paddy and their average cost for it is 1,819,200 dong.

- There are 12 households that grow seasonal paddy and their average cost for it is 2,200,000 dong.

- There are 04 households that grow corns and the average cost for it is 410,000 dong.

Note: Because this is the database of 20 households obtained from the survey, the number of “observations” is very small. The statistical results with such a small number of observations are meaningless; the statistical results are only meaningful in the case of 30 or more observations.

3.5. Dealing with missing values

Missing values: in the questionnaires, some questions include take “-9000” and “-9999” as answers. These values have absolutely no statistical values and only serve as coded values. Thus, when calculating the mean and minimum, if these values are not solved, SPSS will produce incorrect results. It is necessary to deal with these values.

3.6. Dealing with outliners: Only applied to quantitative variables

- Outliners are too small or too large values that have huge differences from other values. Outlin- ers have huge effects on the mean, causing great deviation to the actual mean; thus, the outliners should be removed from the command. Two following steps are taken:

• Step 1: Detect outliners: Analyze à Descriptive Statistics à Frequencies

153 M&E Manual Guide for IFAD Funded Projects in Vietnam

The following screen will appear, select variables to check for outliners:

E.g.: Area of forest land Variable.

Select Stasc

154 How to analysis data by SPSS

 Output screen: Look at Valid column. For example, there is an extreme value: 500.000m2.

An outliner value

Step 2: Dealing with outliners

 First, open the questionnaire to check mistakes of data entry officers. If it’s a mistake of data entry officer à Edit. In the case data entry officer did not make the mistake, this value is true. However, it needs being removed to avoid affecting the mean.

 Dealing with outliners: Choose Data à Select cases à if… and enter the condition.

155 M&E Manual Guide for IFAD Funded Projects in Vietnam

After finishing this command, clickContinue . Thus, SPSS will understand: All of the next calculations will be only applied for the households that have forest land areas < 500.000m2.

Note: After analyzing the forest land area, it is necessary to restore the households that have forest land areas > = 500.000 m2 for the next analysis.

Dataà Select casesà choose All cases à OK.

156 How to analysis data by SPSS

Choose All case s

3.7. Notes

Before analyzing the data, it’s necessary to identify research questions or what need to be analyzed, and if they have practical value. The following matters need to be noticed when analyzing data:

 Always identify if a variable is qualitative (dummy variables, nominal variables and ordinal variables) or quantitative, based on that choose appropriate commands.

 Always analyze two distinct groups: project group and non-project group (to compare the values of all indicators of two groups in the report).

 In each of the two groups (project and non-project) always analyze two groups: female-headed households and male-headed households.

 The statistical results only make senses if the answerers are above 30 (e.g., if there are only 20 households participating in Interest Group, finding how many percent of these 20 households have increased tea-leaf outputs is meaningless). This is a common statistic principle.

Display results in Vietnamese

Option 1

This is a complicate method: In Output window (result), double-click on the table that needs to be displayed in Vietnamese, then right-click on the table, choose Table properties à Cell Format (Follow the order: in “area” choose each section such as Row labels, Column labels…, in the text in the left, choose .vntime or .vnArial – see below picture).

157 M&E Manual Guide for IFAD Funded Projects in Vietnam

Option 2

Copy all in word and choose all tables by using Ctrl + A, go to Home/Font (Word 2007) and select .vnTime or .vnArial. These are fonts of the TCVN3 code.

158 Annexes

ANNEXES

Annex 1. List of 1st and 2nd level indicators...... 160

Annex 2. IFAD template questionnaire for the Annual Outcome Survey...... 164

Annex 3. The steps to conduct Annual Outcome Evaluation...... 175

Annex 4. Example of Outline for Annual Outcome Survey Report...... 176

Annex 5. IFAD Template RIMS+ questionnaire...... 177

Annex 6. Calendar of local event to estimate the Age of a Child...... 190

Annex 7. Check sheet for anthropometric Values...... 192

Annex 8. Time plan for the survey...... 193

Annex 9. The progress form...... 194

Annex 10. Answer check sheet...... 195

159 M&E Manual Guide for IFAD Funded Projects in Vietnam

Annex 1. List of 1st and 2nd level indicators

First level results Second level results

Natural resources (land and water)

1.1.1 People trained in infrastructure management

1.1.2 Groups managing infrastructure formed and/or 2.1.1 Likelihood of sustainability of the groups strengthened managing infrastructure formed and/or strengthened 1.1.3 People in groups managing infrastructure - Number of groups operational/functional 1.1.4 Groups managing infrastructure with women in leadership positions

1.1.5 Land under irrigation schemes constructed or 2.1.2 Effectiveness of productive infrastructure rehabilitated - Percentage of delivered vs. required water 1.1.6 Livestock water points constructed or rehabilitated - Number of farmers with secure access to water

1.1.7 Rainwater harvesting systems constructed or - Incremental hectares of crop grown rehabilitated 2.1.3 Likelihood of sustainability of productive infrastructure 1.1.8 Fish ponds constructed or rehabilitated - Number of functioning infrastructure - Number of farmers with secure access to water resources - Number of fishers with secure access to resource base - Number of fishing pond operational after three years

1.1.9 People trained in NRM

1.1.10 Groups involved in NRM formed/strengthened 2.1.4 Likelihood of sustainability of the NRM groups formed and/or strengthened 1.1.11 People in NRM groups - Number of groups operational/functional 1.1.12 NRM groups with women in leadership positions

1.1.13 Environmental management plan formulated 2.1.5 Effectiveness of NRM and conservation programs 1.1.14 Land under improved management practices - Hectares of land improved through soil/water conservation methods

1.1.15 Other productive infrastructure constructed/ rehabilitated

Agricultural technologies and production

1.2.1 Staff of service providers trained 2.2.1 Effectiveness: Improved performance of service providers -Operational self-sufficiency

160 Annexes

First level results Second level results

1.2.2 People trained in crop production practices and 2.2.2 Effectiveness: Improved agricultural and technologies livestock production

1.2.3 People trained in livestock production practices and - Number of farmers reporting production/yield technologies increase - Number of farmers adopting recommended 1.2.4 People trained in fish production practices and technologies technologies - Number of small farmers reporting increased herd 1.2.5 People accessing facilitated advisory services size - Number of fishers adopting recommended 1.2.6 Households receiving animals from distribution and/or technologies restocking 2.2.3 New Likelihood of sustainability of the 1.2.7 Households receiving facilitated animals health services agri/livestock production groups formed and/or strengthened 1.2.8 Crop/livestock production groups formed/strengthened - Number of groups operational/functional 1.2.9 People in Crop/ livestock production groups

1.2.10 Crop/ livestock production groups with women in leadership positions

Rural financial services

1.3.1 Savings and credit groups formed and/or strengthened 2.3.1 Likelihood of sustainability of the credit/saving groups formed/strengthened 1.3.2 People in savings and credit groups formed/ strengthened - Number of groups operational/functional

1.3.3 Savings and credit groups with women in leadership positions

1.3.4 Financial institutions participating in the project 2.3.2 Effectiveness: Improved access of the poor to financial services

1.3.5 Staff of financial institutions trained 2.3.3 Sustainability: Improved performance of the financial institutions 1.3.6 Voluntary savers - Portfolio at risk 1.3.7 Value of voluntary savings - Operational self-sufficiency 1.3.8 Active borrowers (individuals) - Active borrowers/personnel - Operating expenses ratio 1.3.9 Active borrowers (enterprises)

1.3.10 Value of gross loan portfolio (individuals)

1.3.11 Value of gross loan portfolio (enterprises)

1.3.12 People trained in financial service

Markets

1.4.1 People trained in post-production, processing and 2.4.1 Effectiveness: producers benefiting from marketing improved markets access - Number of farmers using purchased inputs - Number of fishers using purchased inputs

161 M&E Manual Guide for IFAD Funded Projects in Vietnam

First level results Second level results

1.4.2 Roads constructed/rehabilitated 2.4.2 Likelihood of sustainability of the roads constructed/rehabilitated - Number of functioning infrastructure

1.4.3 Market facilities constructed and/or rehabilitated 2.4.3 Likelihood of sustainability of market, storage, processing facilities - Number of functioning market, storage, processing facilities

1.4.4 Marketing groups formed and/or strengthened 2.4.4 Likelihood of sustainability of the marketing groups formed and/or strengthened 1.4.5 People in marketing groups - Number of groups operational/functional 1.4.6 Marketing groups with women in leadership positions

1.4.7 Processing facilities constructed and/or rehabilitated

1.4.8 Storage facilities constructed and/or Rehabilitated

Enterprise development and employment

1.5.1 People trained in Income Generating Activities 2.5.1 Effectiveness: creation of employment opportunities 1.5.2 People receiving vocational training - Number of jobs generated by small and medium 1.5.3People trained in business and entrepreneurship enterprises

1.5.4 Enterprises accessing facilitated non-financial services 2.5.2 Likelihood of sustainability of enterprises - Number of enterprises operating after three years 1.5.5 Enterprises accessing facilitated financial services

Policy and community programming

1.6.1 Government officials/trained 2.6.1 Effectiveness: promotion of pro-poor policies and institutions - Number of pro-poor legislation and regulations enforced at the local or central level - Number of households with long-term tenure security of natural resources - Number of enabling policies promulgated

1.6.2 People trained in community management topics 2.6.2 Effectiveness: community development

1.6.3 Community workers and volunteers trained - Number of community action plans included in local government plans 1.6.4 Community groups formed/strengthened - Number of community project implemented

1.6.5 People in community groups formed/strengthen 2.6.3 Likelihood of sustainability of the community groups formed and/or strengthened 1.6.6 Community groups with women in leadership positions - Number of groups operational/functional 1.6.7 Village/Community plans formulated

1.6.8 People accessing development funds

162 Annexes

First level results Second level results

1.6.9 Apex organisations formed/strengthened 2.6.4 Likelihood of sustainability of the apex organization

Social Infrastructure

1.7.1 Schools constructed/rehabilitated 2.7.1 Effectiveness of social infrastructure

1.7.2 Health centres constructed/rehabilitated - Number of households served by wells 2.7.2 Likelihood of sustainability of social 1.7.3 Drinking water systems constructed/rehabilitated infrastructure 1.7.4 Other infrastructure/facilities constructed and/or - Number of functioning infrastructure, school, health rehabilitated centres - Number of community projects functional

1.7.5 Groups managing social infrastructure formed and/or 2.7.3 Likelihood of sustainability of the groups strengthened managing social infrastructure formed and/or strengthened 1.7.6 People in groups managing social - Number of groups operational/functional Infrastructure

1.7.7 Groups managing social infrastructure with women in leadership positions

Total Outreach

1.8.1 Individuals receiving project services

1.8.2 Households receiving project services

1.8.3 Groups receiving project services

1.8.4 Communities receiving project services

163 M&E Manual Guide for IFAD Funded Projects in Vietnam

Annex 2. IFAD template questionnaire for the Annual Outcome Survey

ANNUAL OUTCOME SURVEY

I Date (DD/MM/YYYY) : /______/______/_____/

II Name of the enumerator: ______

A – HOUSEHOLD IDENTIFICATION

A.1 Village ______District or Province ______

A.2 Name of respondent : ______

A. Male A.2.a What is the gender (sex) of the respondent? B. Female

A.3 Name of the head of household : ______

A. Male (à Go to A.5) What is the gender (sex) of the household head? B. Female A.4 A.4.1. Is there a relative who takes care (financially) of your A. Yes, a relative is taking care of us household’s needs? B. No, no-one is helping us

What is the size of your household/family (e.g. nb of persons living A.5 /_____/_____/ persons under the same roof)?

/_____/ children How many children (under 16 years), adults and elderly persons are A.6 /_____/ adults there in your household/family? /_____/ elderly

A. Well off

In terms of income group, in which of the following categories would B. Average A.7 you consider that your households belongs? C. Poor D. Very poor

Are you a project beneficiary of the [Insert name of the project] A. Yes A.8 project? (e.g. involved in project activities)? B. No (à Go to questionnaire B)

IF YES TO A.7: CONTINUE WITH THE QUESTIONNAIRE A BELOW IF NO TO A.7: USE QUESTIONNAIRE B FOR NON-BENEFICIARIES

164 Annexes

QUESTIONNAIRE A - FOR BENEFICIARIES

B – PARTICIPATION IN PROJECT ACTIVITIES

B.1 Since when is your household involved in project activities (year)? /______/ [year]

Over the past 12 months, have you (or a member of your household) participated in the following project B.2 activities?

[Adapt the response options below to the context of your project]

a - [Insert name of activity a] A. Yes B. No

b - [Insert name of activity b] A. Yes B. No

c - [Insert name of activity c] A. Yes B. No

d - [Insert name of activity d] A. Yes B. No

e - [Insert name of activity e] A. Yes B. No

f - [Insert name of activity f] A. Yes B. No

g - [Insert name of activity g …] A. Yes B. No

h - [Insert name of activity h …] A. Yes B. No

i - [ continue list of activities…] A. Yes B. No

Of the activities in which you have participated, which ones do you find were B.3 /_____/ ; /_____/ ; /_____/ the most useful? [àUp to three responses only] [Use same activity Code as above]

Of the activities in which you have participated, which ones do you find were /_____/ ; /_____/ ; /_____/ B.4 the least useful? [à Up to three responses only] [Use same activity Code as above]

A. Frequently (e.g. more than 2 times/year) How often do you have contact with project staff (extension workers; B. Occasionally (e.g. at least one B.5 facilitators, etc.)? time/year) C. Rarely (e.g. less than one time/year)

A. Highly satisfied B. Moderately satisfied B.6 Overall, how satisfied are you with project services? C. Moderately unsatisfied D. Highly dissatisfied

165 M&E Manual Guide for IFAD Funded Projects in Vietnam

C - LIVELIHOODS

A. Yes C.1 Do you have a source of cash income? B. No (à Go to C.7)

Respondent with cash incomes

What is the main source of income of your household? /_____/ C.2 [Use codes below and write corresponding code on the right] [One answer only]

[Adapt the response options below to the context of your project] A. Agriculture and sales of crops G. Salaries, wages (employees) B. Fishing and sales of fish H. Handicraft C. Livestock and sales of animals I. Remittances D. Natural resources J. Begging, assistance E. Petty trading K. Other (specify: F. Unskilled labour

Do you have a secondary source of income? A. Yes B. No (à Go to C.4)

C.3 C.3.1 - What is this secondary source of income? /_____/ [Use same codes as above ] [One answer only]

A. More than sufficient

How sufficient is your income relative to the needs of your B. Sufficient C.4 household? C. Barely sufficient D. Insufficient

As compared to previous year, was there a change in your income level, and if so, was it due to your C.5 participation in project activities?

[Select only one possibility below]: Is this due to your participation in project activities?

Income is higher Yes b. No

Same income level NA NA

Income is lower Yes b. No

Respondents with no cash income

Yes C.6 Are you a subsistence-based farmer? No (à Go to D.1)

A. More than sufficient

How sufficient is your production relative to the needs of your B. Sufficient C.7 family? C. Barely sufficient D. Insufficient

166 Annexes

D – FOOD SECURITY Over the last 12 months, was there any period(s) during which A. Yes you were not able to provide 3 meals per day for all members D.1 of your family (food shortage period)? B. No (à Go to D.2)

D.1.1 - For how many weeks in total, over the last 12 months? /______/ weeks

As compared to last year, was there a change in your ability to provide 3 meals per day, and if so, is this D.2 change due to your participation in project activities?

Is this due to your participation in project [Select only one possibility below]: activities? Some improvement Yes b. No Same NA NA Situation in worse Yes b. No

E - AGRICULTURAL PRODUCTION AND IRRIGATION A. Yes, own consumption only B. Yes, sales only Do you cultivate land, and if so, for what purpose? C. Yes, consumption and sale E.1 D. No (àGo to E.5) E.1.1 - Compared to previous year, did your agricultural production increase? (e.g. you have harvested bigger A. Yes B. No (à Go to E.2) quantities) A. Small B. Medium E.1.1.1 - How would you quantify this increase? C. Large

E.1.1.2 – Did the project help you achieve this? A. Yes B. No

During the past 12 months, did the project train you in the following agricultural production technologies, and if so, did you adopt this technology?

Were trained in this [Insert the technology name] Did you try it? Did you adopt it? technology? Technology a A. Yes B. No A. Yes B. No A. Yes B. No E.2 Technology b A. Yes B. No A. Yes B. No A. Yes B. No Technology c A. Yes B. No A. Yes B. No A. Yes B. No Technology d A. Yes B. No A. Yes B. No A. Yes B. No Technology e A. Yes B. No A. Yes B. No A. Yes B. No Technology f A. Yes B. No A. Yes B. No A. Yes B. No Technology g A. Yes B. No A. Yes B. No A. Yes B. No Compared to previous year, did your crop production area A. Yes E.3 expand? B. No (à Go to E.4) (e.g. you are now cultivating a larger field or more fields) A. Small F.3.1 - How would you quantify this expansion? B. Medium C. Large

F.3.2 – Did the project help you achieve this? A. Yes B. No A. Yes E.4 Do you use irrigation system(s)? B. No ( à Go to E.5)

167 M&E Manual Guide for IFAD Funded Projects in Vietnam

F.4.1 – Compared to previous year, did you expand the crop A. Yes area that is irrigated/under irrigation? B. No (à Go to E.5) A. Small F.4.1.1 - How would you quantify this expansion? B. Medium C. Large F.4.1.2 – Did the project help you expand this irrigation area? A. Yes B. No A. Yes Do you own livestock (e.g. cows, sheep, goats, porcs)? E.5 B. No (à Go to F.1) E.5.1- Compared to previous year, do you now own more A. Yes livestock? B. No (à Go to F.1) A. Small E.5.1.1 - How would you quantify this increase? B. Medium C. Large E.5.1.2 – Did the project help you increase your livestock A. Yes B. No size? F – ACCESS TO MARKETS Do you get an income from the sale of your agricultural A. Yes B. No (à Go to G.1) production? F.1 A. Income has increased F.1.1 - Compared to previous year, was there a change in the B. Income did not change income generated from the sale of your agricultural production? C. Income has decreased Compared to previous year, is it easier for you to sell your F.2 A. Yes B. No (à Go to G.1) agriculture production (or access markets)? F.1.1 – If so, is it thanks to the project? A. Yes B. No

G – ACCESS TO RURAL FINANCIAL SERVICE Did you (or any household member) borrow money over the G.1 A. Yes B. No à Go to H.1) last 12 months? A. Informal (friends, NGOs, Credit group) G.1.1 – What was the source of credit? B. Formal (bank/micro-finance institution)

G.1.2 - How much did you borrow (total last 12 months)? /______/ [Write the amount in the local currency] [Currency]: ______

A. Yes G.1.3 – Have you repaid the amount borrowed? B. Not yet, but soon C. No, cannot repay A. Consumption (food, clothes, ceremonies, etc.) B. Income generating activities (tools; G.1.4 - What did you - or will you - use the money for? equipment, etc.) [One answer: only the main use should be reported] C. Other investments (housing improvement, land acquisition; etc.) D. Education E. Health As compared to previous year, do you have a better access A. Yes B. No to financial services? G.2 G2.1 - Would you say that this is due to the project ? A. Yes B. No (à Go to H.1)

168 Annexes

H - ENTERPRISE DEVELOPMENT AND EMPLOYMENT Do you or any household member have a non-farm enterprise? A. Yes B. No (􀂃 Go to H.2) [Provide definition of non-farm enterprise] H.1 A. No employee

H.1.1 – In addition to yourself, how many employees do you B. One to two employees have? C. Three to five employees D. More than five employees

A. Yes, establish ( Go to I.1) H.1.2 - Did the project help your household establish or expand B. Yes , expand ( Go to I.1) your enterprise/business? C. No ( Go to I.1)

Did the project help you find a job or improve your employment H.2 A. Yes B. No conditions?

I – ACCESS TO NATURAL RESOURCES

I.1 Do you have access to community fish ponds? A. Yes B. No (à Go to I.2)

I.1.1 – Is this access regulated? A. Yes B. No

I.1.2 – Has your access to the fish ponds improved since last year? A. Yes B. No

I.1.3 – Compared to previous year, did you catch a larger quantity of A. Yes B. No (à Go to I.2) fishes?

J.1.3.1 – Did the project help you achieve this? A. Yes B. No

I.2 Do you have access to marine fisheries? A. Yes B. No (à Go to I.3)

I.2.1 – Is the access regulated? A. Yes B. No

I.2.2 – Has your access to marine fisheries improved since last year? A. Yes B. No

I.2.3 – Compared to previous year, did you catch a larger quantity of A. Yes B. No (à Go to I.3) fishes?

I.2.2.1 – Did the project help you achieve this? A. Yes B. No

I.3 Do you have access to a community forest? A. Yes B. No (àGo to I.4)

I.3.1 – Is this access regulated? A. Yes B. No

I.3.2 – Do you consider that your access to the forest has improved as A. Yes B. No compared to last year?

I.3.2.1 – Did the project help you achieve this? A. Yes B. No

I.4 Do you have access to community pasture land? A. Yes B. No (à Go to J)

169 M&E Manual Guide for IFAD Funded Projects in Vietnam

I.4.1 – Is this access regulated? A. Yes B. No

I.4.2 – Do you consider that your access to community pasture land has A. Yes B. No improved as compared to last year?

I.4.3 – Compared to previous year, did the productivity of the community A. Yes B. No (à Go to J) pasture land increase?

I.4.3.1 – If yes, was it thanks to project activities? A. Yes B. No

J - WOMEN EMPOWERMENT

Who, in your household, decides how the household’s income is spent on the following items?

Husband decides alone Wife decides alone Joint decision

a. House construction A B C

b. House repair A B C

c. House furniture/equipment A B C J.1 d. Purchase of land A B C

e. Purchase of farm tools/inputs A B C

f. Purchase of new livestock A B C

g. Purchase of food A B C

h. Children education A B C

J.2 Do you need the permission of your husband to visit the following places?

Husband’s permission required?

a. The village where you were born A. Yes B. No

b. The local market/bazar A. Yes B. No

c. Church/temple/place of worship A. Yes B. No

d. Friends and family living nearby A. Yes B. No

e. Friends and family living far A. Yes B. No

f. Village meetings A. Yes B. No

170 Annexes

QUESTIONNAIRE B - FOR NON-BENEFICIARIES

C - LIVELIHOODS

A. Yes (à Go to C.2) C.1 Do you have a source of cash income? B. No (à Go to C.7)

Respondent with cash incomes

What is the main source of income of your household? /_____/ C.2 [Use codes below and write corresponding code ] [One answer only]

[Adapt the response options below to the context of your project] F. Unskilled labour A. Agriculture and sales of crops G. Salaries, wages (employees) B. Fishing and sales of fish H. Handicraft C. Livestock and sales of animals I. Remittances D. Natural resources J. Begging, assistance E. Petty trading K. Other (specify:

Do you have a secondary source of income? A. Yes B. No (à Go to C.4) C.3 C.3.1 - What is this secondary source of income? /______/ [Use same codes as above ] [One answer only]

A. More than sufficient

How sufficient is your income relative to the needs of your B. Sufficient C.4 household? C. Barely sufficient D. Insufficient

A. Income is higher As compared to previous year, was there a change in your income C.5 B. Same income level level? C. Income is lower

Respondent with no cash income

C.6 Are you a subsistence-based farmer? Yes No (à Go to D.1)

A. More than sufficient

How sufficient is your production relative to the needs of your B. Sufficient C.7 family? C. Barely sufficient D. Insufficient

A. Production is higher As compared to previous year, was there a change in your C.8 B. Same production level production level? C. Production is lower

171 M&E Manual Guide for IFAD Funded Projects in Vietnam

D – FOOD SECURITY

Over the last 12 months, was there any period(s) during which A. Yes you were not able to provide 3 meals per day for all members D.1 of your family (food shortage period)? B. No (à Go to D.2)

D.1.1 - For how many weeks in total, over the last 12 months? /______/ weeks

A. Some improvement As compared to previous year, was there a change to your ability D.2 B. Same situation to provide 3 meals per day to all members of your family? C. Situation is worse

E - AGRICULTURAL PRODUCTION AND IRRIGATION A. Yes, own consumption only B. Yes, sales only Do you cultivate land, and if so, for what purpose? C. Yes, consumption and sale E.1 D. No (à Go to E.5) E.1.1 - Compared to previous year, did your agricultural production increase? (e.g. you have harvested bigger A. Yes B. No (à Go to E.2) quantities) A. Small E.1.1.1 - How would you quantify this increase? B. Medium C. Large

Compared to previous year, did your crop production area A. Yes E.3 expand? (e.g. you are now cultivating a larger field or more fields) B. No (à Go to E.4) A. Small F.3.1 - How would you quantify this expansion? B. Medium C. Large A. Yes E.4 Do you use irrigation system(s)? B. No ( à Go to E.5)

F.4.1 – Compared to previous year, did you expand the crop area A. Yes that is irrigated/under irrigation? B. No (à Go to E.5) A. Small F.4.1.1 - How would you quantify this expansion? B. Medium C. Large A. Yes Do you own livestock (e.g. cows, sheep, goats, porcs)? E.5 B. No (à Go to F.1)

E.5.1- Compared to previous year, do you now own more A. Yes livestock? B. No (à Go to F.1) A. Small E.5.1.1 - How would you quantify this increase? B. Medium C. Large

172 Annexes

F – ACCESS TO MARKETS

Do you get an income from sales of agricultural production? A. Yes B. No (à Go to G.1) F.1 A. Income has increased F.1.1 - Compared to previous year, was there a change in the B. Income did not change income generated from the sale of your agricultural production? C. Income has decreased Compared to previous year, is it easier for you to sell your F.2 A. Yes B. No agriculture production (or access markets)?

G – ACCESS TO RURAL FINANCIAL SERVICES Did you (or any household member) borrow money over G.1 A. Yes B. No (à Go to H.1) the last 12 months? A. Informal (friends, NGOs, Credit group) G.1.1 – What was the source of credit? B. Formal (bank/micro-finance institution)

G.1.2 - How much did you borrow (total last 12 months)? /______/ [Write the amount in the local currency] [Currency]: ______

As compared to previous year, do you have a better A. Yes B. No G.2 access to financial services?

H - ENTERPRISE DEVELOPMENT AND EMPLOYMENT Do you or any household member have a non-farm A. Yes B. No (à Go to H.2) H.1 enterprise? [Provide definition of non-farm enterprise] A. No employee

H.1.1 – In addition to yourself, how many employees do you B. One to two employees have? C. Three to five employees D. More than five employees A. Yes, establish (à Go to I.1) H.1.2 - Did the project help your household establish or B. Yes , expand (à Go to I.1) expand your enterprise/business? C. No (à Go to I.1)

173 M&E Manual Guide for IFAD Funded Projects in Vietnam

I – ACCESS TO NATURAL RESOURCES I.1 Do you have access to community fish ponds? A. Yes B. No (à Go to I.2) I.1.1 – Is this access regulated? A. Yes B. No I.1.2 – Has your access to the fish ponds improved since last year? A. Yes B. No I.1.3 – Compared to previous year, did you catch a larger quantity of A. Yes B. No (à Go to I.2) fishes? I.2 Do you have access to marine fisheries? A. Yes B. No (à Go to I.3) I.2.1 – Is the access regulated? A. Yes B. No I.2.2 – Has your access to marine fisheries improved since last year? A. Yes B. No I.2.3 – Compared to previous year, did you catch a larger quantity of A. Yes B. No (à Go to I.3) fishes? I.3 Do you have access to a community forest? A. Yes B. No (àGo to J.4) I.3.1 – Is this access regulated? A. Yes B. No I.3.2 – Do you consider that your access to the forest has improved as A. Yes B. No compared to last year? I.4 Do you have access to community pasture land? A. Yes B. No (à Go to J) I.4.1 – Is this access regulated? A. Yes B. No I.4.2 – Do you consider that your access to community pasture land A. Yes B. No has improved as compared to last year? I.4.3 – Compared to previous year, did the productivity of the pasture A. Yes B. No land increase?

J - WOMEN EMPOWERMENT

J.1 Who, in your household, decides how the household’s income is spent on the following items?:

Husband decides alone Wife decides alone Joint decision a. House construction A B C b. House repair A B C c. House furniture/equipment A B C d. Purchase of land A B C e. Purchase of farm tools/inputs A B C f. Purchase of new livestock A B C g. Purchase of food A B C h. Children education A B C J.2 Do you need the permission of your husband to visit the following places? Husband’s permission required? a. The village where you were born A. Yes B. No b. The local market/bazar A. Yes B. No c. Church/temple/place of worship A. Yes B. No d. Friends and family living nearby A. Yes B. No e. Friends and family living far A. Yes B. No f. Village meeting A. Yes B. No

174 Annexes

Annex 3. The steps to conduct Annual Outcome Survey

Scheduled Step Activities Comments Responsibility time Step 1 Planning 1-2 days Depends on available sources M&E officer (finance, human) Step 2 Design tools to collect 10-15 days Design questionnaire, interview M&E officer and officers information and pilot guide and discussion guide taking charge of each project components

Step 3 Sampling 3-5 days Depends on the update and M & E officer. Project staffs availability of sampling frame make the sampling list (list of villages and communes). Step 4 Select enumerator, 2 days In case the number of project staff M & E officer and project team leader, are smaller than the necessary coordinator facilitator, assistant number to participate in the survey, and supervisor you can mobilize staffs from other departments

Step 5 Training enumerators, 5-6 days Training enumerators, supervisors in M & E officer and supervisors the questionnaire and the work to be independent consultants (if done. It is necessary to have detailed available) interview guide. Step 6 Field-test 1 day Field-test is conducted similar to the M&E officer and enumerator extensive survey Lesson learnt Step 7 Logistics for extensive 2-3 days Prepare documents and organization M & E officer and project survey coordinator Step 8 Extensive survey 10-13 days Depends on the number of Enumerator, team leader enumerators, logistics, the time the and supervisor survey is conducted and sample size Step 9 Design entry form 5-10 days Entry form can be designed after M&E officers having final questionnaire for extensive survey. Step 10 Data entry and 7-12 days Use designed entry form to enter Project staffs at district and analysis data communal level take charge of entering data Use SPSS software to analyze data M&E staffs and district project staffs are responsible for analyzing data Step 11 Conduct key 7-10 days Documents of KII and FGD can be M&E staffs and district informant interview prepared after having preliminary project staffs and focus group findings from the survey. discussion Step 12 Report writing 15-20 days Survey report M&E officer Key informant interview report and focus group discussion report Step 13 Communicate and 1-3 days Project coordinators, M & share report E officers and knowledge management officers (KM)

175 M&E Manual Guide for IFAD Funded Projects in Vietnam

Annex 4. Example of Outline for Annual Outcome Survey Report

LIST OF TABLES LIST OF FIGURES ABBREVIATION EXECUTIVE SUMMARY PART 1 - INTRODUCTION 1.1 Introduction of the project:

1.1.1 Socio-economic features of province 1.1.2 Project targets; components and activities of the project 1.1.3 Impacted objects 1.2 Introduction of the Annual Outcome Evaluation 1.2.1 Rationale of the Annual Outcome Evaluation 1.2.2 Contents in the Annual Outcome Evaluation 1.3 Methodology of the Annual Outcome Evaluation 1.3.1 Sampling method 1.3.2 Survey implementation PART 2- RESULTS OF THE ANNUAL OUTCOME EVALUATION 1.1 Description of households in the survey

1.2 Evaluation on the results and impact of the project activities

1.2.1 Educational access 1.2.2 Land access 1.2.3 Crop production 1.2.4 Livestock 1.2.5 Training 1.2.6 Market access 1.2.7 Financial service 1.2.8 Non-farm activities 1.2.9 Income management and finance 1.2.10 Socio-economic development plan building 1.2.11 Gender equality PART 3- CONCLUSION 3.1. Contribution of the project to the general development target

3.2. Poor people benefit from the project

3.3. Women benefit from the project 3.4. Ethnic minorities benefit from the project

176 Annexes

Annex 5. IFAD Template RIMS+ questionnaire

[Insert logo of

QUESTIONNAIRE ID: /___/___/___/ implementing agency]

RIMS + BASELINE SURVEY [Insert name of the project and country] I Date (DD/MM/YYYY) : /______/______/_____/ II Name of the enumerator: ______Note for enumerators:

Before starting the interview, introduce yourself, explain what the objectives of the survey are and ask the person if he/she consents to respond to the questions. If not, go to the next household.

Circle codes corresponding to the answers in the right column.

A – HOUSEHOLD IDENTIFICATION

District/Province: A.1 Village ______A.1.a ______

A.2 Name of respondent: ______A.2.a Sex of respondent:

A.3 Name of the head of household: ______

A. Male (è Go to A.5) What is the gender (sex) of the household head? B. Female A.4 A.4.1. Is there a relative who takes care (financially) of your household’s A. Yes, a relative is taking care of us needs? B. No, no-one is helping us

What is the size of your household/family (e.g. nb of persons living A.5 /_____/_____/ persons under the same roof)?

/_____/ women/girls A.6 How many women/girls and men/boys are there in your household? /_____/ men/boys

/_____/ children How many children (under 16 years), adults and elderly persons are A.7 /_____/ adults there in your household/family? /_____/ elderly

177 M&E Manual Guide for IFAD Funded Projects in Vietnam

B– ACCESS TO EDUCATION

Do you have school-aged children at home? A. Yes B. No B.1 /_____/ school-aged girls B.1.1. If so, how many? /_____/ school-aged boys

/_____/ school-aged girls B.1.2. Of these school-aged children, how many are attending school? /_____/ school-aged boys

A. There is no school nearby B. We cannot afford school fees B.1.3. In case not all are attending school, what are the main reasons? C. We need children to work D. Don’t see the value of education

C – LIVELIHOODS

A. Yes (è Go to C.3) C.1 Do you have a source of cash income? B. No (è Go to C.2 ; then to D1)

A. Practice subsistence agriculture If you don’t have cash income, how to you meet the needs of your family? B. A relative takes care of us C.2 [è Write response and go to D1] C. Other (Specify:______)

What is the main source of income of your household? C.3 /_____/ [Use codes below and write corresponding code on the right] [One answer only]

Which members of household are involved in this main activity? /_____/ and /_____/ and /_____/ C.3.a [Use codes below and write corresponding code on the right] [Up to three answers] Husband B. Wife C. Boys D. Girls

Do you have other sources of income? A. Yes B. No (è Go to C.5) C.4 C.4.1 - What are these other income sources for your /_____/ /_____/ household? [Use same codes as above ] [Max 2 answers]

Which members of household are involved in this main activity? /_____/ and /_____/ and /_____/ C.4.a [Use same codes as above ] [Up to three answers]

178 Annexes Husband Wife C.4.b Who is the main income earner in your household? Son Daughter Others

A. More than sufficient

How sufficient is your income relative to the needs of your B. Sufficient C.5 household? C. Barely sufficient D. Insufficient

In terms of income group, in which of the following categories A. Well off B. Average C.6 would you consider that your households belongs? C. Poor D. Very poor

Are you or any of your household members, a small or medium C.7 entrepreneur (e.g. you, or any member of your household, have Yes B. No (è Go to D.1) established your own business)?

Husband Wife If yes, who owns the business? Son Daughter

C.7.a Others

A. No employee

C.7.1 – In addition to yourself, how many employees do you B. One to two employees have? C. Three to five employees D. More than five employees

179 M&E Manual Guide for IFAD Funded Projects in Vietnam

D – FOOD SECURITY

Over the last 12 months, was there any period(s) during which you were not A. Yes D.1 able to provide 3 meals per day for your household (food shortage period)? B. No (è Go to D.2)

D.1.1 - For how many weeks in total, over the last 12 months? /______/ weeks

D.1.2 – In case of food shortage, what happens in your household? /_____/ and /_____/ and /_____/ [è Read possible answers below and use corresponding code on the right] [Max 3 answers]

Codes for D.1.2: C5. Both adults and children skip meals A. Adults and children just go to bed hungry D1. Women adults eat cheaper, less balanced meals B1. We reduce food rations for women adults D2. Men adults eat cheaper, less balanced meals B2. We reduce food rations for men adults D3. Boy child eat cheaper, less balanced meals B3. We reduce food rations for boys D4. Girl child eat cheaper, less balanced meals B4. We reduce food rations for girls D5. Both adults and children eat cheaper, less B5. We reduce food rations for both adults and children balanced meal C1. Women adults skip meals E1. Women go to ask/borrow food from friends/ C2. Men adults skip meals family C3. Boy child skip meals E2 Men go to ask/borrow food from friends/ family C4. Girl child skip meals F1. Women borrow money to buy food F2. Men borrow money to buy food.

D.2 During the past 12 months, how often did your household eat the following food?

A. Every day B. Once a week C. Once D. Never E. Never F. Only during a month (cannot afford) (religious production reasons) season

1. Grains or rice

2. Roots or potatoes

3. Dairy or eggs

4. Meat or fish

5. Vegetables

6. Fruits

7. Nuts or legumes

180 Annexes

E – ACCESS TO LAND

E.1 A. Yes B. No Do you have access to productive land? (è Go to E.2)

E.1.1 - What is the size of your land (in acres)? /______/ acres

A. Yes B. No E.1.2 – Is this land irrigated? C. Partly

E.1.3 – What is the size of your irrigated land? /______/ acres

E.2 Do you have property rights over this land? A. Yes B. No (è Go to E.3)

E.2.a If yes, under whose name is the land registered? A. Men B. Women

A. Very secure B. Moderately secure E.2.1 - How secure do you consider your property rights? C. Insecure D. Very insecure

A. Illegal access (squatting) B. Share-cropping arrangements E.3 What kind of access rights to you have over this land? C. Rental (< 12 months) D. Leasehold (< 5 years) E. Leasehold (> 5 years)

A. Very secure B. Moderately secure E.3.1 - How secure do you consider your access rights? C. Insecure D. Very insecure

181 M&E Manual Guide for IFAD Funded Projects in Vietnam

F - AGRICULTURAL PRODUCTION AND IRRIGATION

A. Yes, own consumption only B. Yes, sales only F.1 Do you cultivate land, and if so, for what purpose? C. Yes, consumption and sale D. No (è Go to F.5)

Do you grow the following crops, and if so, what are your annual yield levels? [Adapt the response options below to the context of your project]

Crop Grown? Yield? For consumption For sale only? For sale and F.2 only? consumption? Rice Yes No _____kg/acre/year A. Yes B. No A. Yes B. No A. Yes B. No

Wheat Yes No _____kg/acre/year A. Yes B. No A. Yes B. No A. Yes B. No

Potatoes Yes No _____kg/acre/year A. Yes B. No A. Yes B. No A. Yes B. No

Maize Yes No _____kg/acre/year A. Yes B. No A. Yes B. No A. Yes B. No

Vegetables Yes No _____kg/acre/year A. Yes B. No A. Yes B. No A. Yes B. No

Legumes Yes No _____kg/acre/year A. Yes B. No A. Yes B. No A. Yes B. No

Fruits Yes No _____kg/acre/year A. Yes B. No A. Yes B. No A. Yes B. No

Others Yes No _____kg/acre/year A. Yes B. No A. Yes B. No A. Yes B. No

What is the average sale price for each crop, and how would you estimate crop waste (e.g. proportion of F.3 production lost to animal predation, poor post-harvest handling etc.) [[Adapt the response options below to the context of your project; and specify currency]

Crop Average sale price Quantities of production lost for consumption or sales

Rice ______Currency/kg ______kg/acre

Wheat ______Currency/kg ______kg/acre

Potatoes ______Currency/kg ______kg/acre

Maize ______Currency/kg ______kg/acre

Vegetables ______Currency/kg ______kg/acre

Legumes/ pulses ______Currency/kg ______kg/acre

Fruits ______Currency/kg ______kg/acre

Others ______Currency/kg ______kg/acre

182 Annexes

A. Yes (è Go to F.4.1) F.4 During the past 12 months, have you used fertilizers? B. No (è Go to F.4.2)

A. Chemical B. Manure F.4.1 – If yes, which type of fertilizer? (-> go to F.5) C. Compost D. Mixed

A. Too expensive F.4.2 – If not, what are the reasons? B. Not available locally C. I don’t think it is useful D. I don’t know how to use fertilizers

During the past 12 months, have you received any extension A. Yes (è Go to F.5.1) F.5 services? (e.g. from the government extension staff) B. No (è Go to F.5.2)

Husband Wife F.5a If yes, who receives the service? Son Daughter Others

A. Frequently F.5.1 – If yes, how often? (-> go to G.1) B. Occasionally C. Rarely

A. Highly satisfactory B. Satisfactory F.5.2 – How satisfactory were these services? (-> go to G.1) C. Unsatisfactory D. Highly unsatisfactory

A. Services are not available B. Available, but not useful F.5.3 – If not, what are the reasons? C. Available, but too expensive D. Other (Specify: ______)

183 M&E Manual Guide for IFAD Funded Projects in Vietnam

G – LIVESTOCK AND FISHERIES

A. Yes B. No (è Go to Do you own some livestock? G.1)

G.1.1- If so, which animals do you own, how many do you own and for what purpose do you keep them?)

Animal type? Number owned? Do you sell meat or animal products (e.g. milk, eggs, animal skin)?

Meat? Milk? Others products

Dairy A. Yes B. No A. Yes B. No A. Yes B. No A. Yes B. No G.1 cattle

Beef cattle A. Yes B. No A. Yes B. No N/A A. Yes B. No

Sheep A. Yes B. No A. Yes B. No N/A A. Yes B. No

Goats A. Yes B. No A. Yes B. No A. Yes B. No A. Yes B. No

Horses A. Yes B. No A. Yes B. No N/A A. Yes B. No

Chicken A. Yes B. No A. Yes B. No N/A A. Yes B. No

Pigs A. Yes B. No A. Yes B. No A. Yes B. No A. Yes B. No

Husband Wife G.1a Who are the main animal care takers? Son Daughter Others

G.2 Are you engaged in dairy production? A. Yes B. No (è Go to G.3)

G.2.1 – If so, what is your average daily milk production? ______liter/day/lactating cow

A. No B. Yes, for sale only G.2.2 – Do you produce cheese and butter? C. Yes, for own consumption only D. Yes, both for own consumption and sale

A. Free grazing on common pasture land

G.2.3 – How to you feed your cattle? B. Free grazing on private pasture land C. Stall feeding D. Mixed (free grazing + stall feeding)

A. Yes, all year round B. Yes, most of the time G.2.4 – Are your animals well fed? C. No, not always D. No, often not

184 Annexes

G.3 During the past 12 months, have you received any animal A. Yes (è Go to G.3.1) vaccination services? (e.g. from the government extension staff) B. No (è Go to G.3.2)

A. Frequently G.3.1 – If yes, how often? (-> go to G.4) B. Occasionally C. Rarely

A. Vaccination services are not available B. Available, but not useful G.3.2– If not, what are the reasons? C. Available, but too expensive D. Other (Specify: ______)

G.4 During the past 12 months, have you received any other livestock extension services? (e.g. from the government A. Yes B. No (è Go to G.4.2) extension staff)

A. Frequently G.4.1– If yes, how often? B. Occasionally (-> go to H.1) C. Rarely

A. Vaccination services are not available B. Available, but not useful G.4.2 – If not, what are the reasons? C. Available, but too expensive D. Other

H – ACCESS TO MARKETS

Do you get an income from sales of agricultural production? A. Yes B. No (è Go to I.1) A. At the farm gate (traders come to my farm to buy my production) B. Through middlemen G.1.1 – How do you sell your production? C. Directly at the local market D. Through a Farmer Group H.1 E. Other Husband Wife G.1.2 - Who sells the production? (multiple choice) Son Daughter Others A. Easy and timely access Do you have access to information regarding market prices for H.2 B. Some access, but not easy or timely the main crops you produce and sell? C. No access

185 M&E Manual Guide for IFAD Funded Projects in Vietnam

A. Adequate storage space Do you have adequate storage space to store your production H.3 B. Inadequate storage space until market prices are higher? C. No storage space A. I usually get the best market prices B. I sometimes get good prices Do you usually get good market prices for your agricultural H.4 production? C. I almost never get good prices D. I usually don’t know about best market prices

I – ACCESS TO RURAL FINANCIAL SERVICES

Did you (or any household member) borrow money over the I.1 A. Yes B. No (è Go to J1) last 12 months? Husband Wife If yes, who borrowed money? (Under whose name is the I.1.a Son money borrowed?) Daughter Others A. Informal (friends, NGOs, credit group) I.1.1 – What was the source of credit? B. Money lender C. Formal (bank/micro-finance institution)

I.1.2 - How much did you borrow (total last 12 months)? /______/ [Write the amount in the local currency] [Currency]: ______A. Yes I.1.3 – Have you already repaid the amount borrowed? B. Not yet, but soon C. No, cannot repay A. Consumption (food, clothes, ceremonies, etc.) B. Income generating activities (tools; I.1.4 - What did you - or will you - use the money for? equipment) [One answer: only the main use should be reported] C. Other investments (housing (-> go to J.1) improvement, land acquisition; etc.) D. Education E. Health A. I didn’t need credit B. No credit sources available locally C. Interest rates were too high I.1.5 – If not, what were the reasons? D. Nobody accepted to lend me the money I needed E. For religious reasons

186 Annexes

I – ACCESS TO NATURAL RESOURCES

COMMUNITY FISH PONDS

J.1 Do you have access to community fish ponds? A. Yes B. No (è Go to J.2)

A. Very productive J.1.1 – How productive i s t h i s fi s h p o n d ? B. Somewhat productive C. Not very productive

J.1.2 – Do you have legal rights to use this fish pond? A. Yes B. No

J.1.3 – Is this activity your main source of income? A. Yes B. No

Is access to community fish pondsregulated in your J.2 A. Yes B. No community?

A. Very clear and comprehensive J.3 In your views, how clear and comprehensive is this regulation? B. Somewhat clear and comprehensive C. Unclear and not comprehensive

A. Very equitable In your views, how equitable is this regulation (e.g. do the poor J.4 B. Somewhat equitable have the same access rights as the others)? C. Not equitable

A. Regulation always enforced J.5 How strongly is this regulation enforced? B. Regulation sometimes enforced C. Regulation never enforced

A. Never Is access to the community fish ponds asource of conflict J.6 B. Sometimes within community members? C. Often

K. COMMUNITY FORESTS

K.1 Do you have access to community forests? ) A. Yes B. No (è Go to J.2)

A. Very productive J.1.1 – How productive is this forest? B. Somewhat productive C. Not very productive

J.1.2 – Do you have legal rights to use this forest? A. Yes B. No

J.1.3 – Is this activity your main source of income? A. Yes B. No

K.2 Is access to community forests regulated in your community? A. Yes B. No

A. Very clear and comprehensive K.3 In your views, how clear and comprehensive is this regulation? B. Somewhat clear and comprehensive C. Unclear and not comprehensive

187 M&E Manual Guide for IFAD Funded Projects in Vietnam

A. Very equitable In your views, how equitable is this regulation (e.g. do the poor K.4 B. Somewhat equitable have the same access rights as the others)? C. Not equitable

A. Regulation always enforced K.5 How strongly is this regulation enforced? B. Regulation sometimes enforced C. Regulation never enforced

A. Never Is access to the community forests a source of conflict within K.6 B. Sometimes community members? C. Often

L. COMMON PASTURE LAND

A. Yes B. No L.1 Do you have access to a community pasture land? (è Go to J.2)

A. Very productive L.1.1 – How productive is this pasture land? B. Somewhat productive C. Not very productive

L.1.2 – Do you have legal rights to use this pasture land? A. Yes B. No

L.1.3 – Is livestock rearing your main source of income? A. Yes B. No

Is access to community pasture lands regulated in your L.2 A. Yes B. No community?

A. Very clear and comprehensive In your views, how clear and comprehensive is this L.3 B. Somewhat clear and comprehensive regulation? C. Unclear and not comprehensive

A. Very equitable In your views, how equitable is this regulation (e.g. do the L.4 B. Somewhat equitable poor have the same access rights as the others)? C. Not equitable

A. Regulation always enforced L.5 How strongly is this regulation enforced? B. Regulation sometimes enforced C. Regulation never enforced

A. Never Is access to community pasture land a source of conflict L.6 B. Sometimes within community members? C. Often

188 Annexes

M. ARTISANAL MARINE FISHERIES A. Yes M.1 Are you involved in artisanal marine fisheries? B. No (è Go to N1) A. Abundant B. Acceptable level M.1.1 – How is the fish stock in these waters? C. Shrinking due to overfishing D. Almost depleted M.1.2 – Do you own your own fishing boat? A. Yes B. No M.1.3 – Is fishing your mainsource of income? A. Yes B. No M.2 Is access to fish and marine resources regulated? A. Yes B. No A. Very clear and comprehensive B. Somewhat clear and M.3 In your views, how clear and comprehensive is this regulation? comprehensive C. Unclear and not comprehensive A. Very equitable In your views, how equitable is this regulation (e.g. do the small M.4 B. Somewhat equitable fisher folks have the same access rights as the others)? C. Not equitable A. Regulation always enforced M.5 How strongly is this regulation enforced? B. Regulation sometimes enforced C. Regulation never enforced A. Never Is access to fish and marine resources a source of conflict within M.6 B. Sometimes community members? C. Often

N. RISKS AND VULNERABILITY Of the following natural disaster, which one has the greater probability of occurring in the next 12 months; and if it takes place, what would be the impact on your livelihoods? N.1 [Adapt the response options below by deleting disasters that can never happen in the project locality. Then, circle corresponding answer] Negative impact on your Type of natural disaster Likelihood of occurrence? livelihoods? Drought/erratic rainfalls A. High B. Medium C. Low A. High B. Medium C. Low Floods A. High B. Medium C. Low A. High B. Medium C. Low Frost/snow/blizzard/hail A. High B. Medium C. Low A. High B. Medium C. Low Typhoon/hurricane/strong wind A. High B. Medium C. Low A. High B. Medium C. Low Forest fire A. High B. Medium C. Low A. High B. Medium C. Low Earthquake/volcanic eruption A. High B. Medium C. Low A. High B. Medium C. Low Tsunami A. High B. Medium C. Low A. High B. Medium C. Low Others A. High B. Medium C. Low A. High B. Medium C. Low A. Don’t know B. Less than one month In case one of two of the above disasters would occur in the next 12 C. Less than six months months, how long do you think it would take for your household to return to a satisfactory situation? D. More than six months N2 E. More than one year F. Our household would never recover

189 M&E Manual Guide for IFAD Funded Projects in Vietnam

Annex 6. Calendar of local event to estimate the Age of a Child (Example: Dak Nong, November 2012)

Month of birth Age in months Local event 2012

November 0 October 1 September 2 August 3 July 4 June 5 May 6 April 7 March 8 February 9 January 10 2011 December 11 November 12 October 13 September 14 August 15 July 16 June 17 May 18 April 19 March 20 February 21 January 22 2010

December 23 November 24 October 25 September 26 August 27 July 28 June 29 May 30 April 31 March 32 February 33 January 34

190 Annexes

Month of birth Age in months Local event 2009 December 35 November 36 October 37 September 38 August 39 July 40 June 41 May 42 April 43 March 44 February 45 January 46 2008 December 47 November 48 October 49 September 50 August 51 July 52 June 53 May 54 April 55 March 56 February 57 January 58 2007 December 59 November Quá tuổi October Quá tuổi September Quá tuổi August Quá tuổi July Quá tuổi

191 M&E Manual Guide for IFAD Funded Projects in Vietnam

Annex 7. Check sheet for anthropometric Values

Age in Length/height(cm) Weight (Kg) month Male Female Male Female

Minimum Maximum Minimum Maximum Minimum Maximum Minimum Maximum

0-2 36 74 36 72 0.5 10 0.5 9

03-5 45 83 44 80 1 13 1 12

06-8 51 87 50 86 2 15 2 14

09-11 56 91 54 90 3 16.5 2.5 15.5

12-14 59 96 57 95 4 17.5 3 16.5

15-17 62 100 60 99 4 17.5 3 16.5

18-20 64 104 62 102 4 19.5 3.5 18.5

21-23 65 107 64 106 4.5 20.5 4 19.5

24-26 67 108 66 107 4.5 23 4.5 21.5

27-29 68 112 68 111 5 24 5 23

30-32 70 115 69 114 5 24.5 5 24.5

33-35 71 118 71 117 5 25.5 5 25.5

36-38 73 121 72 120 5 26 5 27

39-41 74 124 74 122 5 27 5 28

42-44 75 127 75 124 5 28 5.5 29

45-47 77 129.9 77 126 5 29 5.5 30

48-50 78 132 78 129 5 30 5.5 31

51-53 79 134 79 131 5 31 5.5 32

54-56 80 136 81 133 5.5 32 6 33

57-59 82 139 81 136 5.5 33 6 34.5

192 Annexes

eek 6 W

4 eek 5 W

4 eek 4 W

ve y

7 Survey

4 eek 3 W

4 eek 2 W

1 lan for t he sur A nn ex 8. T i me p lan

4 eek 1 W

Before survey leader leader Steps Logistics Planning Field-test Sampling Write report Write Design tools Extensive survey Entry form design Select enumerator, Select enumerator, supervisor and team supervisor and team Training enumerator, enumerator, Training Conduct KI and FGD Entry and analyze data Communicate and share

193 M&E Manual Guide for IFAD Funded Projects in Vietnam

0 13

0 12

0 11

0 10

9 0

8 0

7 0

6 0 August 2012

5 0

4 0

3 0

2 0

30 31 30

30 30 30 July 2012

30 29 30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 … 30 30 30 120 Total

… Chúc Chúc Hương Supervisor

… group group Control Treatment Treatment Treatment Treatment Treatment / Treatment Control group A nn ex 9. The p ro g r e ss for m hh No. 283 973 256

… District Ninh Hải Ninh Hải Ninh Hải

… Tân Hải Nhơn Hải Nhơn Hải Com name

… Com code 22867 22867 22855 TABLE ENCO D ES OF VILLAGES-COMMUNES TABLE

… Gò Thao Vill. name Vill. Mỹ Tường Mỹ Khánh Phước … 11 38 14 13 12 10 09 08 07 06 05 04 03 39 40 02 01 Vill. Vill. code

194 Annexes

Annex 10. Answer check sheet

LIST OF HOUSEHOLDS

Da Trang Village: Commune: Phuoc Tan Commune code: 22789 District: Bac Ai Village code: 10

No HH code Name of headed Enumerator Status Children Notes (contents need being edited/ household checked/added) Female Male

- F: cow bought in 2010? 1 37 Katơr Hạ 99 2 1 - G12: 96 =>2

- J4? 2 06 Pinăng Thị Gái 97 2 1 - Children’s weight and height?

- D2? - G20: unclear code 3 26 Pinăng Thị Phượng 97 2 1 - H: mismatch? + H2 – H5.1 natural fishing? + H2.2 Wood explores?

Note: Status: 1=Complete; 2=Additional

195