measuring results,measuring sharing lessons EVALUATION QUALITY ASSURANCE SYSTEM (EQAS) Guidelines For Operation Evaluations Prepared by: WFP Office of Evaluation Last updated: October 2015 Foreword The Evaluation Quality Assurance System (EQAS) is one of the building blocks for implementation of WFP’s Evaluation Policy (2008). As such, it is WFP’s Office of Evaluation’s primary means of safeguarding the international evaluation principles of: Independence: by setting standards that increase impartiality in the evaluation process and in reporting on findings; Credibility: by setting standards that ensure evaluations are evidence-based and follow transparent and systematic processes; and Utility: by building milestones into evaluation processes for timeliness and reporting standards to ensure accessibility. EQAS builds on the norms and standards of the UN Evaluation Group, the OECD-DAC Evaluation Network, related tools from the Active Learning Network for Accountability and Performance, and on the wider evaluation literature and community of practice. EQAS is a comprehensive system covering all types of evaluations: strategic, policy, country portfolio, impact, operations and synthesis evaluations. It guides all evaluations undertaken by WFP’s Office of Evaluation and its consultants and also applies to decentralised evaluations managed by Country Offices and Regional Bureaux. EQAS is a working tool for WFP’s evaluation staff and consultants covering all stages of the evaluation cycle. It is not a comprehensive handbook on evaluation and does not replace the rich range of evaluation literature. Initiated in 2007, the EQAS is subject to periodic and systematic review and update in line with the Office of Evaluation’s evolving needs and international best practice. Updates and new materials will be incorporated as they become available. Helen Wedgwood Director, Office of Evaluation TABLE OF CONTENTS 1. DEFINITION AND CRITERIA ...................................................................... 3 1.1. WFP Operations ....................................................................................................... 3 1.2. Operation Evaluation - Definition ........................................................................... 4 1.3. Operation Evaluation - Selection ............................................................................. 6 1.4. Operation Evaluation - Framework ......................................................................... 6 1.4.1. Evaluation criteria ............................................................................................... 6 1.4.2. Evaluation questions ........................................................................................ 8 2. PRINCIPLES AND ETHICAL STANDARDS ................................................ 10 2.1. Evaluation Principles ............................................................................................. 10 2.2. Code of Conduct and Ethical Standards ................................................................. 11 3. ROLES AND RESPONSIBILITIES .............................................................. 12 3.1. New Operation Evaluations model ........................................................................ 12 3.2. Office of Evaluation ............................................................................................... 13 3.3. Evaluation Company .............................................................................................. 14 3.3.1. Evaluation Manager ....................................................................................... 14 3.3.2. Evaluation Team ............................................................................................ 15 3.4. Country Office (CO) ............................................................................................... 15 3.5. Regional Bureau (RB) ............................................................................................ 16 3.6. Other internal stakeholders ................................................................................... 16 3.7. Recourse mechanisms ........................................................................................... 16 4. STEP BY STEP PROCESS GUIDE ............................................................... 16 4.1. Preparation ............................................................................................................ 18 4.2. Inception ................................................................................................................ 26 4.3. Evaluation .............................................................................................................. 33 4.4. Reporting ............................................................................................................... 38 4.5. Dissemination and Follow-up................................................................................ 46 4.6. Summary of process steps by evaluation phase..................................................... 48 5. CONTENT GUIDES AND TEMPLATES ....................................................... 50 5.1. Content guides ....................................................................................................... 51 5.1.1. Content guide for the Inception Package ....................................................... 51 5.1.2. Content guide for the Evaluation Report ....................................................... 51 5.2. Quality checklists ................................................................................................... 51 5.2.1. Inception Package Quality checklist .............................................................. 51 5.2.2. Evaluation Report Quality checklist .............................................................. 51 5.3. Evaluation Products Templates ............................................................................. 52 5.3.1. Inception Package Template .......................................................................... 52 5.3.2. Evaluation Report Template .......................................................................... 53 5.3.3. Evaluation Report Comments Matrix Template ............................................ 56 5.4. Evaluation process templates ................................................................................ 57 5.4.1. Evaluation Proposal Template ....................................................................... 57 5.4.2. Inception Package Submission Template ...................................................... 63 5.4.3. Evaluation Report Submission Template ...................................................... 64 Acronyms ......................................................................................................... 65 INTRODUCTION 1. In the context of renewed corporate emphasis on providing evidence and accountability for results under the World Food Programme’s (WFP) Strategic Plan for 2014- 2017 and organisational strengthening endeavours, WFP has committed to increase the evaluation coverage of single operations in complement to the Office of Evaluation’s (OEV) more complex evaluations of policies, strategies, country portfolios and impact of core activities. 2. Consequently, OEV has designed a new approach for Operation Evaluations (OpEv), which foresees outsourcing the management and conduct of these evaluations. Long-term agreements have been established with reputable companies for this purpose and these guidelines, which have been designed to initiate and test this approach, are primarily intended for them. Yet, in publishing them, the intention is also to provide transparency to third parties: WFP colleagues, organisations and institutions affected by WFP evaluations and users of the evaluation results. 3. These guidelines aim to clarify the quality standards for Operation Evaluations in terms of both the evaluation process and products and to provide guidance accordingly. As such, it deals primarily with conceptual and organisational matters rather than with the technicalities of the evaluation research process. The guidelines are structured as follows: Definition and criteria (chapter 1) locates operations as WFP’s standard unit of intervention; defines evaluations of single operations including their intended purpose and describes the guiding framework. Principles and ethical standards (chapter 2) presents the evaluation principles guiding WFP’s evaluation function, processes and products as well as the Code of Conduct and other ethical standards that WFP evaluation staff and associated companies and consultants should respect. Roles and responsibilities for Operation Evaluations (chapter 3) describes the new approach and model for Operation Evaluations and the ensuing roles and responsibilities of the Evaluation manager (EM) and Evaluation team as well as other stakeholders. Step by step process guide (chapter 4) covers the main evaluation process steps, with an emphasis on those that the evaluation company is responsible for. It is divided in five sections corresponding to the five evaluation phases. Each section begins with a description of the purpose of the phase and of the resulting products and presents an overview of the main tasks. The tasks are then further described in a series of sub-sections. A summary table synthesises the main steps identifying respective roles and responsibilities. Content guides and templates (chapter 5) aim to assist the evaluation teams in preparing for,