DOCSLIB.ORG

Real-Time Evaluations of Humanitarian Action an ALNAP Guide Pilot Version

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Real-Time Evaluations of Humanitarian Action an ALNAP Guide Pilot Version Real-time evaluations of humanitarian action An ALNAP Guide Pilot Version John Cosgrave Ben Ramalingam and Tony Beck Page 2 of 97 Evaluators interviewing a group from the affected population during a real- time evaluation in Mozambique. Acknowledgements ............................................................................................................................................................. 5 Table of Introduction: What is in this guide ................................................................................................................... 6 Key to information boxes .............................................................................................................................................. 8 contents Section I Guidance for Evaluation Managers ........................................................ 9 Manager Step 1 Deciding to do an RTE ......................................................................................... 10 What is an RTE? ........................................................................................................................................................ 10 What distinguishes an RTE from a regular evaluation? ............................................. 10 Triggers for starting an RTE ....................................................................................................................... 12 Who are the stakeholders in an RTE? ............................................................................................. 13 Why do an RTE? ......................................................................................................................................................... 14 What is the difference between RTEs and monitoring? ............................................. 14 Manager Step 2 Refining the standard terms of reference .................................... 15 Intervention background ................................................................................................................................. 15 Evaluation purpose ................................................................................................................................................ 16 Stakeholder involvement ................................................................................................................................ 16 Evaluation questions ........................................................................................................................................... 16 Evaluation criteria .................................................................................................................................................. 17 Other potential criteria for RTEs ........................................................................................................... 19 Recommendations and lessons .............................................................................................................. 20 Methodology ................................................................................................................................................................... 21 Work plan and schedule ................................................................................................................................... 21 Outputs ................................................................................................................................................................................ 23 Evaluation team ......................................................................................................................................................... 23 Manager Step 3 Planning and budget .............................................................................................. 24 Planning .............................................................................................................................................................................. 24 The budget ....................................................................................................................................................................... 24 Page 3 of 97 Manager Step 4 Selecting the team ................................................................................................... 26 Team size and skills ............................................................................................................................................. 26 Recruitment timeline .......................................................................................................................................... 27 Recruitment process ........................................................................................................................................... 27 Manager Step 5 Reviewing the inception report ................................................................ 28 Manager Step 6 Reviewing the report ............................................................................................. 29 Manager Step 7 The management response ........................................................................... 30 Manager Step 8 Dissemination and use ....................................................................................... 32 Section II Guidance for Evaluation Teams .............................................................. 34 Team Step 1 Planning the fieldwork .................................................................................................... 35 Team Step 2 Drafting an interview guide ...................................................................................... 37 Introduction and introductory questions .................................................................................... 37 Questions based on the terms of reference .............................................................................. 38 Closing questions and final information ...................................................................................... 38 Team Step 3 Writing the inception report ..................................................................................... 39 Team Step 4 Conducting the fieldwork ............................................................................................ 40 Record-keeping during fieldwork ......................................................................................................... 40 Interviews ......................................................................................................................................................................... 40 Surveys ................................................................................................................................................................................ 41 Documentary research ..................................................................................................................................... 41 After-action review ................................................................................................................................................ 42 Beneficiary consultation .................................................................................................................................. 42 Observation ..................................................................................................................................................................... 43 Focus group interviews ..................................................................................................................................... 43 Triangulation ................................................................................................................................................................. 44 Ethics ..................................................................................................................................................................................... 44 Team Step 5 The chain of evidence ...................................................................................................... 45 The time constraint ............................................................................................................................................... 45 Using a tool to link evidence to recommendations .......................................................... 46 Team Step 6 Recommendations ............................................................................................................... 47 Avoiding too many recommendations ............................................................................................. 47 Usable recommendations .............................................................................................................................. 48 Doing without recommendations .......................................................................................................... 49 Team Step 7 Writing the report ................................................................................................................. 50 Writing strategy ......................................................................................................................................................... 50 Report structure ........................................................................................................................................................ 50 Team Step 8 Feeding back in the field
Load more

Recommended publications
  • Review of the World Food Programme's
    Review of the World Food Programme’s Humanitarian and Development Effectiveness 2006-2011 SYNTHESIS REPORT MARCH 2012 Canadian International Development Agency (CIDA) 200 Promenade du Portage Gatineau, Québec K1A 0G4 Canada Telephone: 819-997-5006 / 1-800-230-6349 (toll-free) For the hearing- and speech-impaired: 819-953-5023 / 1-800-331-5018 (toll-free) Fax: 819-953-6088 Website: www.cida.gc.ca/evaluations-e E-mail: [email protected] © Her Majesty the Queen in Right of Canada, 2012 Cette publication est aussi disponible en français sous : Revue de l'efficacité de l'aide humanitaire et du développement du Programme alimentaire mondial Printed in Canada Review of the World Food Programme’s Humanitarian and Development Effectiveness 2006-2011 SYNTHESIS REPORT MARCH 2012 Acknowledgments CIDA‟s Evaluation Directorate wishes to thank all who contributed to this review for their valued input, their constant and generous support, and their patience. Our thanks go first to the independent team from the firm, Goss Gilroy Inc., made up of co-team leaders Ted Freeman and Sheila Dohoo Faure and analysts Danielle Hoegy, Molly McCreary and Kofi Kobia. Michelle Guertin, CIDA Evaluation Manager, substantially revised the report, bringing it to its present form. The Evaluation Directorate would also like to thank the management team of CIDA‟s International Humanitarian Assistance Directorate (Multilateral and Global Programs Branch) at Headquarters in Gatineau for its valuable support. Our thanks also go to the representatives of the WFP for their helpfulness and their useful, practical advice to the evaluators. CIDA would also like to thank the Policy and Operations Evaluation Department of the Netherlands Ministry of Foreign Affairs for its interest in this approach to assessing the development effectiveness of multilateral organizations and for its specific support of this review.
    [Show full text]
  • Shifting Mindsets
    Shifting Mindsets Creating a more flexible humanitarian response Alice Obrecht FLEXIBILITY ALNAP is a global network of NGOs, UN agencies, members of the Red Cross/ Crescent Movement, donors, academics, networks and consultants dedicated to learning how to improve the response to humanitarian crises. www.alnap.org About the authors Alice Obrecht is a Senior Research Fellow at ALNAP. Suggested citation Obrecht, A. (2019) Shifting Mindsets: Creating a more flexible humanitarian response. ALNAP Study. London: ODI/ALNAP. ISBN: 978-1-910454-94-7 © ALNAP/ODI 2019. This work is licensed under a Creative Commons Attribution-non Commercial Licence (CC BY-NC 4.0). Front cover photo credit: EU/ECHO and Save the Children. Editing by Hannah Caddick Communications management by Maria Gili Graphics by Tim Harcourt-Powell Typesetting by Alex Glynn Design by Soapbox, www.soapbox.co.uk 2 Shifting Mindsets: Creating a more flexible humanitarian response Contents Acknowledgements 5 Abbreviations and acronyms 7 Executive summary 9 About this report 13 Part I: A framework for flexibility in humanitarian action 16 Section 1: Flexibility in humanitarian action and its importance 18 1.1 Common sense but hard to do 18 1.2 What kind of flexibility are we talking about? 19 1.3 Organisational flexibility and its implications for response-level flexibility 20 1.4 Understanding the demand for humanitarian flexibility 23 1.5 The flexibility challenge within humanitarian response 25 Section 2: A framework for understanding humanitarian flexibility 27 2.1 Choosing strategies
    [Show full text]
  • Ecb Project Case Study
    ECB PROJECT CASE STUDY NGOs traditionally compete for funds; what happens when they join forces to raise money together? Catherine Gould and Katy Love September 2011 1 The big NGOs traditionally compete for funds; what happens when they join forces to raise money together? 1. Six NGOs began with $5 million in the bank; and $7 million still to find… CARE International, Catholic Relief Services (CRS), Mercy Corps, Oxfam, Save the Children, and World Vision International have come together in a unique collaboration to build field, agency, and sector level emergency preparedness and response capacity. The Emergency Capacity Building (ECB) Project aims to improve the speed, quality, and effectiveness of the humanitarian community in saving lives, improving welfare, and protecting the rights of people in emergency situations. In 2008, following a first phase of the ECB Project, the Bill and Melinda Gates Foundation accepted a proposal from the six NGOs worth $12.3 million for a further five‐year joint programme. Stepping outside their traditional grantmaking strategy to support this innovative project, the Gates Foundation provided $5 million of core funding and the agencies pledged to fundraise together the remaining $7.3 million. While CARE USA manages the contract with the Gates Foundation, the other five agencies signed a memorandum of understanding (MOU) outlining the project’s management structure and ways of working for the five‐year joint venture. This case study recounts the agencies’ attempts to together close the $7.3 million funding gap, while launching and sustaining a $12 million programme. Three years in, the ECB Project is almost fully funded, but it has not always been a smooth journey… 2.
    [Show full text]
  • More Than Just Luck: Innovation in Humanitarian Action
    More than just luck: Innovation in humanitarian action Alice Obrecht and Alexandra T. Warner HIF-ALNAP research on successful humanitarian innovation This report presents the synthesised findings from 15 case studies, undertaken by ALNAP in partnership with ELRHA’s Humanitarian Innovation Fund (HIF). It is produced as part of a broader research partnership between ALNAP and ELRHA that has sought to define and understand what successful innovation looks like in the humanitarian sector. The outputs of this research are aimed at humanitarian organisations interested in using innovative practices to improve their performance, as well as organisations outside the humanitarian sector, such as academic institutions or private companies, seeking to engage in innovation in humanitarian action. The Humanitarian Innovation Fund (HIF) supports organisations and individuals to identify, nurture, and share innovative and scalable solutions to the challenges facing effective humanitarian assistance. www.humanitarianinnovation.org ALNAP is a unique system-wide network dedicated to improving humanitarian performance through increased learning and accountability. www.alnap.org Suggested citation Obrecht, A. and T. Warner, A. (2016) ‘More than just luck: Innovation in humanitarian action’. HIF/ ALNAP Study. London: ALNAP/ODI. © ALNAP/ODI 2014. This work is licensed under a Creative Commons Attribution-Non Commercial Licence (CC BY-NC 3.0). ISBN 978-1-910454-43-5 Publication and communications managed by Alex Glynn Copy edited by Roo Griffiths Design and typesetting by Jeni Burnell and Chloé Sanguinetti MORE THAN JUST LUCK: INNOVATION IN HUMANITARIAN ACTION I Acknowledgments First and foremost the authors wish to thank the project leads for the 15 HIF-funded innovation projects that were examined in this research.
    [Show full text]
  • Best of UNICEF Research Retrospective: Documenting Impact
    Best of UNICEF Research Retrospective: Documenting impact and lessons learned Any part of this publication may be freely reproduced if accompanied by the following citation: United Nations Children’s Fund, Office of Research – Innocenti, Best of UNICEF Research Retrospective: Documenting impact and lessons learned, UNICEF Office of Research – Innocenti, Florence, 2019. The views expressed within this publication are those of the individuals interviewed and do not necessarily represent the views of UNICEF. This publication has not been edited to official publication standards and UNICEF accepts no responsibility for errors. The maps in this publication are stylized and not to scale. The maps do not reflect a position by UNICEF on the legal status of any country or territory or the delimitation of any frontiers. © United Nations Children’s Fund (UNICEF), 2019 Front cover: © UNICEF/UNI160424/Ose Written by: Scriptoria (www.scriptoria.co.uk) Report conceptualization, data collection and coordination: Jorinde van de Scheur, Alessandra Ipince, Emanuela Bianchera and Kerry Albright (UNICEF Office of Research – Innocenti) Design and layout: bounford.com Best of UNICEF Research Retrospective: Documenting impact and lessons learned UNICEF Office of Research - Innocenti The Office of Research – Innocenti is is UNICEF’s dedicated research centre. It undertakes research on emerging or current issues to inform the strategic directions, policies and programmes of UNICEF and its partners, shape global debates on child rights and development, and inform the global research and policy agenda for all children, and particularly for the most vulnerable. Office of Research – Innocenti publications are contributions to a global debate on children and may not necessarily reflect UNICEF policies or approaches.
    [Show full text]
  • Working Together in the Field for Effective Humanitarian Response 1
    WORKING TOGETHER IN THE FIELD FOR EFFECTIVE HUMANITARIAN RESPONSE 1 Working together in the field for effective humanitarian response Background paper 30th ALNAP Annual Meeting 3 –4 March, Berlin Annual Report 01 Main Pattern ALNAP is a unique system-wide network dedicated to improving the performance of humanitarian action through shared learning. www.alnap.org Have you read the ALNAP discussion starter that accompanies this study? An electronic copy of the study, the discussion starter and other related resources are available on the ALNAP website at www.alnap.org/meeting2015. Suggested citation Saavedra, L. and Knox-Clarke P. (2015). Working together in the field for effective humanitarian response. ALNAP Working Paper. London: ALNAP/ODI. © ALNAP/ODI 2015. This work is licensed under a Creative Commons Attribution- Non Commercial Licence (CC BY-NC 3.0). ISBN 978-1-910454-16-9 Publication and communications managed by Maria Gili Edited by Roo Griffiths Copy edited by Alex Potter Design and typesetting by Jeni Burnell and Maria Gili Contents Abbreviations and acronyms 4 1. Introduction 5 2. Coordination in context: are humanitarians moving closer together or further apart? 6 2.1 An increasingly atomised humanitarian system? 6 2.2 Potential benefits of working together 9 2.3 Potential challenges of working together and critiques of greater coordination 16 3. Understanding inter-organisational collaboration: different typologies 18 3.1 The spectrum of humanitarian coordination: different levels of working together 18 3.2 Factors influencing position on the spectrum 22 4. The current situation: structures for working together 26 4.1 Coordination by governments 26 4.2 The IASC humanitarian coordination system 28 4.3 NGO relations: consortiums and country networks 33 4.4 NGO partnerships 37 4.5 NGO families 40 4.6 Working with non-traditional actors 41 5.
    [Show full text]
  • Lessons Papers: a Methods Note
    Lessons Papers: A Methods Note by Neil Dillon and Leah Campbell ALNAP is a unique system-wide network dedicated to improving humanitarian performance through increased learning and accountability. www.alnap.org About the authors Neil Dillon is a Research Fellow at ALNAP Leah Campbell is a Senior Research Officer at ALNAP Acknowledgements The authors would like to thank the three main reviewers of this Methods Note: Roxani Krystalli at the Feinstein International Center, Elizabeth Parker (independent) and Karen Peachey at CaLP. Your comments and feedback throughout the drafting process were invaluable. Appreciation also goes to each of the interviewees who gave their time to discuss the issues and challenges of evidence synthesis in the humanitarian sector. Thanks are also due to the ALNAP Secretariat. To Paul Knox-Clarke and Alice Obrecht for their comments on early drafts, and to Tim Harcourt-Powell for his help in delivering the final product. All mistakes and shortcomings in the paper are the authors’ own. © ALNAP/ODI 2018. This work is licensed under a Creative Commons Attribution-non Commercial Licence (CC BY-NC 4.0). Suggested citation: Dillon, N. and Campbell, L. (2018) ALNAP Lessons Papers: A Methods Note. ALNAP Methods Note. London: ALNAP/ODI Copyediting by Deborah Eade Bibliographic editing by Renée Goulet Communications management and typesetting by Tim Harcourt-Powell ALNAP would like to acknowledge the financial support of Irish Aid in carrying out this initiative. ISBN: 978-1-910454-63-3 Contents 1. Introduction 1.1 Objectives of the Method Note 4 1.2 ALNAP Lessons Papers to date 5 1.3 The Evolving Evidence Landscape 8 2.
    [Show full text]
  • No Plan B: the Importance of Environmental Considerations in Humanitarian Contexts
    No Plan B: The Importance of Environmental Considerations in Humanitarian Contexts An Analysis of Tools, Policies, and Commitments of DEC Members Amy Johnson, Isabel Mele, Francesco Merola, Katarzyna Plewa ii BACKGROUND This report has been compiled for the Disasters Emergency Committee by a consultancy team of International Development Masters students from the London School of Economics and Political Science. iii TABLE OF CONTENTS ACRONYMS……………………………………………………………………………………….. iv LIST OF FIGURES…………………………………………………………………………………. iv EXECUTIVE SUMMARY…………………………………………………………………………… v TABLE OF RECOMMENDATIONS……………………………………………………………… vii 1. INTRODUCTION……………………………………………………………………………….. 1 1.1 INTRODUCTION 1 1.2 REPORT AIMS 2 1.3 METHODOLOGY 2 1.4 REPORT OUTLINE 3 2. WHAT’S THE PROBLEM? WHY ARE ENVIRONMENTAL CONCERNS IMPORTANT TO ADDRESS?……………………………………………………………………………….…….. 4 2.1 BENEFITS AND CHALLENGES OF ENVIRONMENTAL MAINSTREAMING 4 2.2 OVERCOMING THE CHALLENGES 5 3. WHAT’S ALREADY OUT THERE?……………………………………………………………... 6 3.1 CORE HUMANITARIAN STANDARD 6 3.2 OTHER STANDARDS AND GUIDELINES 8 3.3 ORGANISATIONAL ENVIRONMENTAL POLICIES AND COMMITMENTS 9 3.4 ORGANISATIONAL TOOLS 11 4. WHY ARE ENVIRONMENTAL, TOOLS, POLICIES AND COMMITMENTS HARD TO DEVELOP AND IMPLEMENT? …………………………………………………………….….... 15 5. WHAT MOTIVATES ORGANISATIONS TO EFFECTIVELY DEVELOP AND INCORPORATE ENVIRONMENTAL TOOLS AND POLICIES INTO THEIR WORK?............ 21 5.1 ATTITUDES AND PRIORITIES 21 5.2 MOTIVATIONS 22 6. CONCLUSIONS AND RECOMMENDATIONS: WHAT SHOULD WE DO NOW?…...… 24
    [Show full text]
  • Recent Events and Activities
    ALNAP Bulletin SEPTEMBER 2009 Welcome to the September 2009 edition of the ALNAP Bulletin, which provides regular updates to Full and Observer Members on ALNAP Secretariat activities. Since the last Bulletin, we have completed and published the 8th Review of Humanitarian Action. It contains three in-depth studies on performance assessment, impact assessment and humanitarian innovations (each chapter is covered in more detail later). It is now available online The 8th RHA was published in July 2009. and in hard copy. Recent Events and Activities All Full Members should have received their hard copies of the complete edition, stand alone copies of each of the chapters and key message sheets, the latter also March 2009 published in French and Spanish. Ben Ramalingam presented at the Save the Children Leadership Development week, on the leadership We have already received a lot of complimentary competency around ‘External Orientation of Leaders’. feedback from within and beyond the membership. His presentation covered key crisis drivers for the Please do send more feedback on all/any of the humanitarian sector, and networked approaches to chapters, positive or negative! A survey about the strategic change. format, dissemination and use of the 8th RHA is now live and readers are encouraged to take part in it. This ALNAP also made a contribution to a HFP Distance kind of feedback is essential for us to be able to make Learning Module for ECOWAS, on the topic of strategy improvements next time round. and leadership. In collaboration with the ProVention Consortium, we have also published a lessons paper on response and April 2009 recovery following urban disasters.
    [Show full text]
  • 2003. Evaluating Humanitarian Action. an ALNAP Guidance Booklet
    Evaluating Humanitarian Action AN ALNAP GUIDANCE BOOKLET Hugo Slim and Luis Enrique Eguren Draft 1 September 2003 Acknowledgements This Handbook was produced following the guidance and with the support of the ALNAP Secretariat, and in particular Kate Robertson, John Borton, John Mitchell and Sera Orzel. It has greatly benefited from the constructive comments of a Review Panel consisting of: Anne Bay Paludan, John Borton, Margie Buchanan-Smith, Andre Griekspoor, Alistair Hallam, Astri Suhrke, John Telford, Samuel Taddesse, and Peter Wiles. Tony Beck Evaluating Humanitarian Action Draft 1 ALNAP Guidance 3 Table of Contents Acronyms ........................................................................................................................................................................................................................................................................... 6 1 Purpose and Use of the Guidance Booklet ................................................................................................................................................................ 7 1.1 Background .................................................................................................................................................................................................... 7 1.2 Guidance Booklet users ...................................................................................................................................................................... 7 1.3 Organisation of the Guidance Booklet
    [Show full text]
  • Emergency Capacity Building Project: Phase II
    The Emergency Capacity Building (ECB) Project: Phase II Summary of Proposal to the Bill & Melinda Gates Foundation Submitted by CARE on behalf of the Interagency Working Group (IWG), Feb. 2008 A global project to improve the speed, quality and effectiveness of emergency preparedness and response in the humanitarian community by building capacity at the field, global organizational, and humanitarian sector levels with a focus on staff capacity, accountability and impact measurement, and disaster risk reduction. Background Disasters and emergencies are increasing in frequency, severity and complexity, affecting hundreds of millions of people and stretching the capacity of the global humanitarian system. At the same time, standards are increasingly rigorous and complex, with higher expectations for accountability and impact measurement. As a result, humanitarian relief organizations face challenges at three levels: in the field, within global agency systems and practices, and across the humanitarian sector. The Inter-agency Working Group (IWG) was formed in 2003 as a consultative group of seven of the largest operational humanitarian agencies, with decades of combined field experience in more than 100 countries. CARE International, Catholic Relief Services (CRS), International Rescue Committee (IRC), Mercy Corps, Oxfam Great Britain, Save the Children Alliance, and World Vision International (WVI) represent a critical mass in the humanitarian NGO sector. In 2005, with support from the Bill & Melinda Gates Foundation and Microsoft Corporation, the IWG launched the first phase of the Emergency Capacity Building (ECB) Project. ECB Phase I pooled the expertise of the participating agencies and their sector partners to produce more than 20 innovative tools, programs and practices to be used in the field in emergencies.
    [Show full text]
  • Emergency Capacity Building Project Humanitarian Competencies Study
    Emergency Capacity Building Project Staff Capacity Initiative April, 2006 Humanitarian Competencies Study Summary This document explores the use of competency-based human resources (HR) management systems amongst members of the Interagency Working Group on Emergency Capacity (IWG), and identifies tools useful to those developing such systems for the recruitment, management and development of humanitarian professionals. 1. Purpose and scope • Competency-based HR management practices are widely acknowledged as a ‘best practice’. Most IWG agencies have developed and implemented competency frameworks, though these are often focused on long-term posts. • Sections 2 to 4 detail the practical benefits of competency-based HR systems and consider how they can be applied specifically in the humanitarian context – including short-term roles. • Sections 5 and 6 aim to assist organizations already using competency models to develop them further and to guide future inter-agency collaboration. • Specific tools presented include: a matrix of job descriptions and levels of knowledge and responsibilities for program and HR managers; a set of generic advisory competencies; and a shortlist of competencies for use in telephone recruitment for short-term humanitarian posts. 2. Overview of competency models • Competencies represent the underlying experience, skills and behaviors required to perform effectively in a given job, role or situation. They can include characteristics, motives, skills, self-image and body of knowledge. • Core competencies are underpinning behaviors required by all humanitarian staff. • Specialist or technical competencies are based on the demands of specific jobs. • Competency models legitimize conversations on performance management and interpersonal skills, and help clarify the key "thing" that makes humanitarian workers effective.
    [Show full text]