Real-time evaluations of humanitarian action An ALNAP Guide Pilot Version John Cosgrave Ben Ramalingam and Tony Beck Page 2 of 97 Evaluators interviewing a group from the affected population during a real- time evaluation in Mozambique. Acknowledgements ............................................................................................................................................................. 5 Table of Introduction: What is in this guide ................................................................................................................... 6 Key to information boxes .............................................................................................................................................. 8 contents Section I Guidance for Evaluation Managers ........................................................ 9 Manager Step 1 Deciding to do an RTE ......................................................................................... 10 What is an RTE? ........................................................................................................................................................ 10 What distinguishes an RTE from a regular evaluation? ............................................. 10 Triggers for starting an RTE ....................................................................................................................... 12 Who are the stakeholders in an RTE? ............................................................................................. 13 Why do an RTE? ......................................................................................................................................................... 14 What is the difference between RTEs and monitoring? ............................................. 14 Manager Step 2 Refining the standard terms of reference .................................... 15 Intervention background ................................................................................................................................. 15 Evaluation purpose ................................................................................................................................................ 16 Stakeholder involvement ................................................................................................................................ 16 Evaluation questions ........................................................................................................................................... 16 Evaluation criteria .................................................................................................................................................. 17 Other potential criteria for RTEs ........................................................................................................... 19 Recommendations and lessons .............................................................................................................. 20 Methodology ................................................................................................................................................................... 21 Work plan and schedule ................................................................................................................................... 21 Outputs ................................................................................................................................................................................ 23 Evaluation team ......................................................................................................................................................... 23 Manager Step 3 Planning and budget .............................................................................................. 24 Planning .............................................................................................................................................................................. 24 The budget ....................................................................................................................................................................... 24 Page 3 of 97 Manager Step 4 Selecting the team ................................................................................................... 26 Team size and skills ............................................................................................................................................. 26 Recruitment timeline .......................................................................................................................................... 27 Recruitment process ........................................................................................................................................... 27 Manager Step 5 Reviewing the inception report ................................................................ 28 Manager Step 6 Reviewing the report ............................................................................................. 29 Manager Step 7 The management response ........................................................................... 30 Manager Step 8 Dissemination and use ....................................................................................... 32 Section II Guidance for Evaluation Teams .............................................................. 34 Team Step 1 Planning the fieldwork .................................................................................................... 35 Team Step 2 Drafting an interview guide ...................................................................................... 37 Introduction and introductory questions .................................................................................... 37 Questions based on the terms of reference .............................................................................. 38 Closing questions and final information ...................................................................................... 38 Team Step 3 Writing the inception report ..................................................................................... 39 Team Step 4 Conducting the fieldwork ............................................................................................ 40 Record-keeping during fieldwork ......................................................................................................... 40 Interviews ......................................................................................................................................................................... 40 Surveys ................................................................................................................................................................................ 41 Documentary research ..................................................................................................................................... 41 After-action review ................................................................................................................................................ 42 Beneficiary consultation .................................................................................................................................. 42 Observation ..................................................................................................................................................................... 43 Focus group interviews ..................................................................................................................................... 43 Triangulation ................................................................................................................................................................. 44 Ethics ..................................................................................................................................................................................... 44 Team Step 5 The chain of evidence ...................................................................................................... 45 The time constraint ............................................................................................................................................... 45 Using a tool to link evidence to recommendations .......................................................... 46 Team Step 6 Recommendations ............................................................................................................... 47 Avoiding too many recommendations ............................................................................................. 47 Usable recommendations .............................................................................................................................. 48 Doing without recommendations .......................................................................................................... 49 Team Step 7 Writing the report ................................................................................................................. 50 Writing strategy ......................................................................................................................................................... 50 Report structure ........................................................................................................................................................ 50 Team Step 8 Feeding back in the field