EVALUATION OF HUMANITARIAN AID BY AND FOR NGOs A guide with ideas to consider when designing your own evaluation activities

EUROPEAN COMMISSION

An independent review financed by the Directorate-General for Humanitarian Aid. The opinions expressed in the review do not necessarily reflect the policies of the European Commission.

Humanitarian Aid

2007

BACK TO TABLE OF CONTENTS Source DG ECHOphoto ofpictures: library Acknowledgements unintentionally omitted -withoursincere apologies. been possibly have them of some though annex, final the in Guide this to contributors the all list to attempted have Weplaces. other few a and Niamey Nairobi, Geneva, Brussels, in held meetings numerous the during documents essential provided readily and experience their shared time, their gave kindly who - observers and stakeholders external ECHO, DG of partners – those all to gratitude its express to wishes furthermore team consulting The actors,humanitarian whohave agreed to examine someofourdrafts. knowledgeable highly of group reference core a of comments constructive the - from Etrangères and Affaires des Ministère French the EuropeAid, ALNAP, DG of - stakeholders Sector key Evaluation few a the of insights the from benefited greatly of also has process Guide’sdesign The Assistant. support consistent the with DG ECHO, in particular Peter Cavendish, Head of produced Sector until June 2007, and Dorothea Opitz, was Guide This An independent review financedfor Humanitarian Directorate-General Aid.by the The statements expressed donot necessarilypolicies intheGuide the reflect of theEuropean Commission

Prolog Consult 3 Evaluation of Humanitarian Aid by and for NGOs

Acronyms andabbreviations Table ofContents A E D C B F Preamble (Foreword) Focusing: The Three Steps aPositiveMind-Setting: Focus onUse andImprovement Introduction Overall Diagram oftheGuide Tools F.1. Step 3: What Do We Want To Know, Actually? Step 2: The Focusing Choice Step 1: The First Two BasicQuestions Why hasDGECHOcommissioned thedevelopment ofthisGuide? What evaluation ahumanitarian isNOT What evaluation ISahumanitarian

Planning andManagingEvaluations ...... Criteria And Indicators Developing AMethodology About Deciding The Type Evaluation Of Setting The Standards ...... v. ii. viii. vii. vi. iv. iii. i. iii. ii. i.

Selecting evaluators Selecting Preparing the TOR Following up Taking approach aparticipatory Reporting Defining plan and timetable work andanalysisChoosing data tools collection The role oftheevaluation manager The proposed criteria generic Indicators The mostcommonly usedevaluation criteria ......

19 17 44 39 23 22 15 14 14 13 11 49 48 46 45 42 40 39 36 32 32 31 29 25 54 52 50 50 9

Prolog Consult 5 Evaluation of Humanitarian Aid by and for NGOs

ANNEXES G List ofContributors Bibliography Glossary Capacity andSupport G.2. F.3. G.1. F.2. Support for Evaluation andUseSupport by Activity Donors Implementing aProcessImplementing Evaluation Internal Capacity Internal Implementing anOutcomeImplementing Evaluation Efficiency Values Relevance Effectiveness i. iii. ii. iii. ii. i. iii. ii. i. iii. ii. i...... Examples ofgeneral evaluation questions Examples ofspecificevaluation questionsandindicators Examples ofgeneral indicators Examples ofspecificevaluation questionsandindicators Examples ofgeneral indicators Examples ofgeneral evaluation questions Examples ofspecificevaluation questionsandindicators Examples ofgeneral indicators Examples ofgeneral evaluation questions Examples ofspecificevaluation questionsandindicators Examples ofgeneral indicators Examples ofgeneral evaluation questions ......

75 74 82 80 76 75 75 71 71 70 70 68 67 66 66 60 59 57 57 56 79

Prolog Consult 7 Evaluation of Humanitarian Aid by and for NGOs

BACK TO TABLE OF CONTENTS Acronyms andAbbreviations WHO WFP URD UNICEF UNHCR UNEG UNDP UN TOR 3Cs SMART RTE REAM RAE QA/QC PCM OECD OCHA NGO MFA M&E LRRD LFA IFRC IDP ICRC IASC HAP-I GHD FPA DG ECHO DG AIDCO DEC DANIDA DAC CIDA CAP ALNAP AI

World HealthOrganisation World Food Programme Urgence Réhabilitation Développement (Groupe) UN Children’s Fund CommissionerUN High for Refugees UN Evaluation Group UN Development Programme United Nations Terms ofReference for development cooperation aid) andhumanitarian principles Treaty’s (Maastricht Coherence Coordination, Complementarity, Specific,Measurable, Attainable,Relevant and (indicators) Time-bound Real Time Evaluation EvaluationRapid andAssessment Method Assessment inEmergenciesRapid (UNICEF) Assurance/Quality ControlQuality Project Cycle Management Organisation for Economic Cooperation andDevelopment Office forthe Coordination ofHumanitarian Affairs(UN) OrganisationNon-Governmental ofForeignMinistry Affairs andEvaluationMonitoring Relief,Linking RehabilitationandDevelopment Logical Framework Analysis International Federation Cross oftheRed Crescent andRed DisplacedInternally Person International Committee Cross oftheRed Inter-Agency Standing Committee (UN) AccountabilityHumanitarian Partnership -International (Principles) Donorship Humanitarian Good Framework Partnership Agreement (DGECHO) European for Aid CommissionHumanitarian Directorate-General (also calledEuropeAid) European for Assistance CommissionEUExternal Directorate-General CommitteeDisasters Emergency Danish InternationalDevelopment Agency Development Assistance Committee Canadian InternationalDevelopment Agency Consolidated Appeal Process (UN) Action in Humanitarian Active Learning for Network Accountability andPerformance Appreciative Inquiry

Prolog Consult 9 Evaluation of Humanitarian Aid by and for NGOs

BACK TO TABLE OF CONTENTS A. throughout this Guide, the most important step you can take to keep evaluations as simple as evaluations keep to take can youstep important most Guide,the this throughout take place, out carrying evaluation and establishing attribution is not easy. As we emphasise Similarly, and precisely because of the complex environment in which humanitarian actions have world would definitely beaneven more miserable place withouthumanitarian workers. Guide this of authors The dangers. longstanding respect, admiration real and gratitude for what and they have witnessed unexpected in the field. or The odds, face overwhelming to often have workers Aid world. the in activities challenging most the among is aid Humanitarian actors. humanitarian of majority great the of professionalism and the integrity respect not would thatguide” “idiot’san expect not Do course. of right, was Einstein procedures. However: so, our targetprimary group are the NGOs that do not yet have firmly established evaluation in order to make evaluations as simple, affordable as possible. andIn non-obstructive doing value - should be to provide a practical overall “guidance”added an to perhaps these often and parallel - approaches, purpose major second our that felt have we confuse, or duplicate Rather than attempting to reinvent the wheel by creating yet another methodology that may as ‘methodology’a suggested intheproject’s initialtitle? than ‘Guide’rather a call to decided have we which approach, new a introduced.then regularly Whyare approaches new and updated, being are these of Many disseminated. widely been often have which objectives, and situations various to adapted existence, in already guidelines and courses methodologies, valuable of variety a are There future. their and communities their serve, we who beneficiaries the benefiting of goal ultimate the results of our actions, with the immediate objective of improving our effectiveness, andthe with about feedback obtaining in approach organised an taking than – less no and – more sensible people may become quite emotional when they hear the term! But evaluation is no There can be a lot of mystic as well as misinformation about evaluation. A number of otherwise process. learning the of importance the understood long have others, among Chinese, TheLEARN! to opportunity an foremost and first is evaluation An it? isn’taccountability, increasing for and actions humanitarian our improve to how learning for pay, to price small a is that But examined. be and examine to time a programme, a of lifetime the truth”in of “momenta not mandatory sense - is to de-dramatize evaluation. Of course, an evaluation may represent Our first major purpose in this Guide – which we would like to be considered in the advisory andcolleagues, friends Dear PREAMBLE (Foreword) "To knowthatyoudonotisalreadytosomething" "Make everythingassimplepossible,but notsimpler" (Albert Einstein) (Albert (Confucius) 11 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 12 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs The authors Happy reading – andevaluating! and updated regularly be to snapshot, a upgraded. be to meant be only can Guide this of version current the responses, and soul-searching challenges, humanitarian of evolution constant the considering that, be will Ours caveat. a include to is foreword a in tradition Finally,the the at access themany ofthe mostrelevant web sites anddocuments. boxes relevant the easy provide highlighting will hyperlinks required, Wherever bychapters. corresponding the of Guide, beginning this of parts guidance various necessary the the with throughout you provide will It below. presented is diagram overall An The Guideisdividedinto three consistent mainsections, withitskeypurposes. build up, outusefulevaluations. develop ormaintain thebasiccapacitiesto carry to agencies aid humanitarian help to is Guide this of purpose major third Accordingly,the frombeginning, thevery onhowstarting theevaluation canbeUSED. to is possible as - relevant as and - A • Our • A • of the trickiest parts!). parts!). of thetrickiest or adapted and, where relevant, ourcomments regarding strong andweak points. copied be can that focus,examples chosen the accordingto tools existing among questions that willprovide you withthemostusefulinformation). those addressing are activities evaluation that so priorities set to (how focusing and advantage) best your to evaluation use can you how mind in (bearing setting section general toolbox: devoted introduction how to to plan capacity to and maintain a focus throughout the evaluation process, evaluation the throughout focus a maintain evaluation, manage building and an and evaluation, more dissemination importantly tables BACK TO TABLE OF CONTENTS of to results chapters help you (perhaps on selecting mind- one BACK TO TABLE OF CONTENTS B. Overall diagram oftheguide (looking "outside") (looking Focus onoutcome Tool box A-for orissue: eachsector choice): (single ormultiple sectors Main 1, 2orallthree canbeselected): criteria CriteriaGeneric (notto beappliedmechanically: using results) by andfor target NGOs: (for planning andmanaging evaluations, Capacity building/learningenvironment t

&UDy
t

)FBMUI t

'PPE t

4VHHFTUJPOT
BOE
DBWFBUT t

-JOLBHFT
UP
NPOJUPSJOH XJUI
XFC
MJOLT
tool(s), andindicator(s) benchmark(s) t

3FDPNNFOEFE
FYJTUJOHöFMEUFTUFE t

&YBNQMF
PG
FWBMVBUJPO
RVFTUJPO T 7BMVFT t
3FMFWBODF t
t

&òFDUJWFOFTT Standards, types, planning, methodology, management, reporting, t

0UIFS
DPOEVDJWF
UZQFT
PG
TVQQPSU
y t

1FFST t

.FOUPSJOH t

$PBDIJOH t

5SBJOJOH
SFGSFTIFS
DPVSTFTy t

,FZ
JOUFSOBM
GVODUJPOT
JO
QMBDF (Accountability purposes, and/orlearning from levels…) operational to policy - How are we goingto usetheresults? - Why dowe want to doanevaluation? dissemination anduseofresults multiple choice): issues (singleor Cross-cutting Considering bene ts expected De ning evaluationDe ning questions t

&UDy
t

1BSUJDJQBUJPO t

$IJMESFO or combination ofboth Support by donors: Support /(0T
UP
NFBTVSF
JNQBDU
FUD
evaluations by small/mediumsized post evaluations, to balance single t

(MPCBM
KPJOU
UIFNBUJD
QFFST
NFUB FY t

%JTTFNJOBUJPO
PG
SFTVMUT
TUSBUFHJD
VTF t

)FMQ
JO
öOEJOH
FYQFSUT t

$PPSEJOBUJPO
XJUI
EPOPST
$T t

2"2$
HPPE
FOPVHI
PWFSBMM
RVBMJUZ t

'VOEJOH
BT
SFMFWBOU Generic CriterionGeneric Tool box B:
&óDJFODZ
2VBMJUZ
PG
QSPDFTT t

4VHHFTUJPOT
BOE
DBWFBUT t
-JOLBHFT
UP
BVEJU t
-JOLBHFT
UP
NPOJUPSJOH t
JOEJDBUPS T
XJUI
XFC
MJOLT

tool(s), and benchmark(s) FYJTUJOHöFMEUFTUFE
3FDPNNFOEFE t
question(s) t

&YBNQMF
PG
FWBMVBUJPO (looking "inside") (looking Focus onprocess Table 1 13 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 14 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs C. 4 3 2 What evaluation? ISahumanitarian ALNAP “normal” between example For differences situation. humanitarian a major of context the of in out carried number those and evaluations a also but similarities, many are There In addition, evaluations mustrespondIn to somekeycriteria. and - timeliness Any evaluation should activities undertaken also fulfil criteria of resourceshumanitarian are already scarce! and money, of waste a generally is Thatmore. even donate to donor generous a convince « undertaken be not should evaluation An results). intended the produce not might operation the that indicates monitoring contact forlearning the preparation or of another the similar next operation, or because beneficiary a to respond should evaluation any rule, a As resources.and budgetary interventions, as well as evaluations of these, often take place under severely limited human humanitarian that add also might one Toviolations.this, rights massivehuman and displacements services, and infrastructures of disruption access, and ones) “non-State”(except time pressure and in very fluid and fast changing contexts, with lack of security, of authorities in theUSAID Tips found be example for evaluate,can to reasons key of - course of limitative not - list A both. or outcome), on (focus external process), on (focus actor concerned the to internal either be can need reasons. This specific some for - organised adequately are these if – recipients and/or donor(s) either by requested be also could They felt. being is need specific a when a is there when partners, implementing concerned and Performance link1offers Action) inHumanitarian the following definition: Accountability for Network Learning (Active ALNAP Guide. this throughout discuss will we this basic objective – often referred to as “utilisation-focused evaluation”. to This is a theme that how learn to is improve actions the effectiveness of humanitarian what we do! of All evaluation activities evaluation should be designed around undertaking for reason main The and to summariseDGECHO’s inpromoting objectives thisdocument. evaluation, humanitarian a not, is what and is what outline to attempt will chapter This Introduction Introduction Performance andEvaluation Monitoring Tips n°11,USAID1997 Principles for Evaluation ofdevelopment Assistance, OECD/DAC, Paris 1991–seealsoLink56 Evaluating chapter Action, Humanitarian 2.2;ALNAP2006 The evaluation of humanitarian action is "a systematic and impartial examination of is"asystematicThe evaluation action andimpartial ofhumanitarian humanitarian action intended to action drawhumanitarian lessonsto improve and andpractice policy 2 niae ta hmntra porme feunl oeae ne considerable under operate frequently programmes humanitarian that indicates 4 . Link2 credibility 3 A ti Gie ugss eautos hud e ntae by initiated be should evaluations suggests, Guide this As . enhance accountability". for the sake of making evaluations making of sake the for (for example, to generate to example, (for need perceived , or learning, for opportunity maximum usefulness - which includes BACK TO TABLE OF CONTENTS », or to help to or », BACK TO TABLE OF CONTENTS 5 What evaluation ahumanitarian isNOT or external expert). More inalater willbesaidaboutevaluation types chapter. expert). or external Link staff agency (own him/herself evaluator the to and designs), research sophisticated very to at outcome), the level of sophistication of the process (fromrapid appraisalsquick-and-dirty or process at (looking focus its after), or (before,intermediate out carried is it which during programme or project the of phase the to according e.g. evaluations, to approaches and Beyond these basic principles, the evaluation literature is full of various classifications oftypes An evaluation, ornot, shouldnotbeconfused humanitarian with: Cfr footnote 3. Independence". and These thefirst two OECD/DAC reflect "Impartiality principles of Not • Seeking • an • a • Be • vulnerable groups asrelevant to theoperation; attention given to including the perspectives of males, females, children, and other with groups, beneficiary representative with consultations adequate for provide primary purpose isto purpose provide guidance for practical primary future directions; would notbeaccurate orusefulinany evaluation respect. the Otherwise, findings. misleading or biased in result inadvertently or intentionally not would that procedures use should approaches analysis and both internal and external evaluation is also possible. In all cases, the data-gathering areasforand improvement. done been had combinationwhatof A of weaknesses and strengths identifying activities, evaluation in involved directly are agency an within volunteers and staff where appropriate, most be can approach evaluation self a situations, other been In evaluated. not being operation the have of stage who any in involved evaluators use to is important, conclusions is and organisation findings the the outside of credibility the when impartial particular and objective in for information, provide to way One evaluation. the of purpose the upon depend can independence of balance appropriate The action. follow-up to the evaluation process in some way, in order to provide for buy-in and commitment Independence also needs to be balanced by the need to mean? involve this key does stakeholders what in But independent. be to needs evaluation that meaning as of evaluation, so that they are evidence based and credible. Often this is interpreted their key objective beingtotheir keyobjective improve performance. with value, added providing as and helpful as perceived be must They manner. promne o mngmn” uis a as eaie o rsucs havebeen used;or resources how examine also may audits “management” or “performance” money”,for although “value accounts, of requirements. examination involve traditionally and Audits procedures established with compliance in used been have scientific objective Adt ae nedd rmrl t cek ht rgam resources programme that check to primarily intended are Audits audit. be . Evaluations are most effective if done in a constructive a in done if effective most are Evaluations confrontational. to . Evaluations can make use of “scientific” methods, but their but methods, “scientific” of use make can Evaluations study. and reflect impartial the reactions 5 . These often are considered the defining characteristics of Eauto pann must planning Evaluation beneficiaries. 15 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 16 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 7 6 efficiency); the soundness, adequacy and application of the financial andoperational financial ofthe and application systems, procedures andinternal controls, etc. adequacy soundness, (cost use of resources the efficiency); theeconomic and the organisation of management effective the of: respect provides in i.a. audit organisation an An of management direct. the to recommendations less and feedback usually is audit and evaluation between relationship The this Guide. monitoring. on focused To optimise these linkages, DG ECHO has commissioned a contiguous methodological guide aptly outlinedby USAID help to define what should be monitored in the future. This has been complementarity very can evaluation return, In evaluation. to inputs essential are data monitoring example, For inter-relatedoften at somestages. are and process, management result-based the of elements represent all They evaluation! However, don’t in discarding be hasty monitoring or audit considerations when you plan an here.be clicking Link3 A comparative table of differences between audit, monitoring, and evaluation can be found mlctos gnrly r ls lkl t rn no rbes ih xenl audits. external and chapter F.3 Link). with problems and some aspects of the internal efficiency and performance into of an organisation (see findings glossary assess to audits management run by e.g. provided the data some utilise may evaluations Similarly, to upon likely act less and are evaluation, become generally they undertake before implications, and problems that potential studies element organisations identify evaluation one often represents serious, can of evaluation and an existence management undertaking good As the analysis. of their account of into part as take reports may sometimes Audits Performance andEvaluation Monitoring Tips n°11,USAID1997 ofKey Glossary Terms inEvaluation Based management, andResults OECD/DAC, 2002. monitoring system) butalsohow andwhy theywere achieved, to andwhat actions a • "…operating notonlywhat unitsneedto results know were achieved (viathe of allocated fundsandaid.” ofan main stakeholders the on and progress dataof extent the use operationachievementof resultsthe ongoing and ofin of management collection to inform intervals) regular indicators (at specified systematic the uses that function exercise. As defined by OECD/DAC6, monitoring is: “a continuing is: monitoring OECD/DAC6, by defined As exercise. monitoring take (viaevaluation)". to improve further performance 7 Specific cross-references will be provided where relevant in relevant where provided be cross-references will Specific 4 Link , asfollows: BACK TO TABLE OF CONTENTS Link BACK TO TABLE OF CONTENTS Why hasDGECHOcommissioned thedevelopment ofthisGuide? ubr f e msae wr dlvrd ih eakbe ossec ad euaiy in regularity, and particular: consistency remarkable with delivered were messages key of of A number number visits. field large and a workshops, interviews, with through stakeholders, extensively and consulted actors humanitarian authors the Guide, this drafting Before consideration. activities. reference humanitarian of for terms designed The specifically evaluation, for guide or methodology medium ones. and In this context small DG ECHO has some commissioned the present particular project, to in propose a common NGOs, with case the always not is this approaches, evaluation effective developed since long have partners largest ECHO’s DG of most While that purpose . the donor. leaving a than rather to need, initiative the feel they whenever actions their evaluate good to of standardpractice, a as encouraged, be should partners effectiveness. implementing and NGO particular, capacities In their strengthen further to organisations, humanitarian all of development the support to seeks ECHO DG regard, this In way. some in evaluation out carry organisations humanitarian all that important is Partnership Framework it actions, (FAFA) aid humanitarian of effectiveness a improvethe to order In agencies. UN for by linked NGOs, European-based Agreement (FPA) major for NGOs and Cross Red the DG ECHO works through including primarily UN of agencies,implementing a variety partners, for Aid (DGECHO). Humanitarian DGECHO’s are detailedinitswebsite. activities link6 evolvedalso has intoEuropeanworld. the the Commission’sin It donor Directorate-General humanitarian single largest the become rapidly has ECHO convictions. political or religion outside the Union. The aid is intended to go todirectly those in distress, irrespective of race, conflict armed or disastersnatural of victims the reliefto and assistance emergency fund to Community Humanitarian inOffice) 1992. A specific mandate was adopted European (the ECHO service, humanitarian specialised a createdCommissionEuropean the (ex-Yugoslavia,nineties early etc), the Rwanda in crises major of number a Confrontedwith It • It • It • situations andapplicablefor NGOactors. humanitarian for appropriate are these as much as 9, link guidelines evaluation by learning. levels, and improving the effectiveness of current and future actions humanitarian accountabilityall atand transparency makers,in decision aiding humanitarian the recent initiatives. Cs”) Maastricht Treaty’s criteria of coherence, coordination and complementarity (the “3 should should should , major humanitarian standards and values, and some highly relevant highly some and values, and standards humanitarian major 10, link integrate be also able ddctd ugt edn hs en nldd n h FA for FPA the in included been has heading budget dedicated A take ik 8 link link 7 and the Financial and Administrative Framework Agreement to a into increase measure ugse ta te olwn prmtr b tkn into taken be parameters following the that suggested account the of coherence timeliness a number with and of key the quality among evaluation of culture a policy recently of information documents created link 5 to be able DG such available EuropeAid as the to

17 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 18 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs This Guidewillattempt to provide responses to these variousexpectations. recommendations: receivedfrequently other few a by complemented werethe TOR to addenda helpful These above • takeinto account therealities aid, ofhumanitarian andofNGOs; • approach; provide for aparticipatory • provide • if • situations; makeitflexible andadaptableto particular • maketheGuidesimple–butnotsimplistic; • depending uponthecontext; can go into further detail; can gointo further ih frad okn apoc ta cn i i ipoig urn ad future and current improving practices. in aid can that approach looking forward a with possible, all, a “Do notimposeaone-size-fits-allapproach.” "Do notattempttoreinventthewheel!"and make make toolkit it evaluation with modular, a range so useful that of those to possible the interested organisations approaches in more BACK TO TABLE OF CONTENTS undertaking that extensive could approaches be evaluation, applied BACK TO TABLE OF CONTENTS D. approaches evaluation issocritical, andwhat we focus oninthisbriefbutcriticalchapter time.why of is wasteThat a or – aid humanitarian of use effective most the with assist greatly can that exercise useful extremely an be either can approachevaluationit,howand you Whyupon view evaluation?Depending with bother Mind-setting: apositiveMind-setting: focus onuseandimprovement! (looking "outside") (looking Focus onoutcome Tool box A-for orissue: eachsector choice): (single ormultiple sectors Main 1, 2orallthree canbeselected): criteria CriteriaGeneric (notto beappliedmechanically: using results) by andfor target NGOs: (for planningandmanaging evaluations, Capacity building/learningenvironment t

&UDy
t

)FBMUI t

'PPE t

4VHHFTUJPOT
BOE
DBWFBUT t

-JOLBHFT
UP
NPOJUPSJOH XJUI
XFC
MJOLT
tool(s), andindicator(s) benchmark(s) t

3FDPNNFOEFE
FYJTUJOHöFMEUFTUFE t

&YBNQMF
PG
FWBMVBUJPO
RVFTUJPO T 7BMVFT t
3FMFWBODF t
t

&òFDUJWFOFTT Standards, types, planning, methodology, management, reporting, t

0UIFS
DPOEVDJWF
UZQFT
PG
TVQQPSU
y t

1FFST t

.FOUPSJOH t

$PBDIJOH t

5SBJOJOH
SFGSFTIFS
DPVSTFTy t

,FZ
JOUFSOBM
GVODUJPOT
JO
QMBDF (Accountability purposes, and/orlearning from levels…) operational to policy Considering bene tsfor expected management andsta - How are we going to use the results? - Why dowe want to doanevaluation? dissemination anduseofresults multiple choice): issues (singleor Cross-cutting De ning evaluationDe ning questions t

&UDy
t

1BSUJDJQBUJPO t

$IJMESFO or combination ofboth Support by donors: Support /(0T
UP
NFBTVSF
JNQBDU
FUD
evaluations by small/mediumsized post evaluations, to balance single t

(MPCBM
KPJOU
UIFNBUJD
QFFST
NFUB FY t

%JTTFNJOBUJPO
PG
SFTVMUT
TUSBUFHJD
VTF t

)FMQ
JO
öOEJOH
FYQFSUT t

$PPSEJOBUJPO
XJUI
EPOPST
$T t

2"2$
HPPE
FOPVHI
PWFSBMM
RVBMJUZ t

'VOEJOH
BT
SFMFWBOU in which one which in mind of frame the Generic CriterionGeneric Tool box B:
&óDJFODZ
2VBMJUZ
PG
QSPDFTT t

4VHHFTUJPOT
BOE
DBWFBUT t
-JOLBHFT
UP
BVEJU t
-JOLBHFT
UP
NPOJUPSJOH t
JOEJDBUPS T
XJUI
XFC
MJOLT

tool(s), and benchmark(s) FYJTUJOHöFMEUFTUFE
3FDPNNFOEFE t
question(s) t

&YBNQMF
PG
FWBMVBUJPO (looking "inside") (looking Focus onprocess Table 2 19 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 20 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs Which benefitscanfromyou evaluations?expect What are thepotential drawbacks? project, as well as many other stakeholders interviewed by the developers of this Guide, this highlighted thefollowing potential benefits from evaluation: of developers the by interviewed stakeholders other onsetofthe many the as at well as project, partners ECHO DG with to better organised workshop A aid, beneficiaries. humanitarian benefit of and effectiveness is learning the improving from evaluation about ideas arise new can identifying that benefit major the indicated, have we As improved. canbefurther and future undertakings activities are indeed assisting beneficiaries as intended, and to identify ways in which current it still some tois formimportant undertake of evaluation activities, in order tothat ascertain information. completeYet for waiting of luxury the haverarely we actions, humanitarian of the for search to context moving appropriate fast the in Especially or case. any in exists necessary rarely very “perfect”answer,which it is Rarely informed. better are actions and decisions that so information, better provide to is evaluation of goal The comprehensive. fully or perfect, be not need approach Youritself. evaluation for goes principle same The situations. other in applied and practices these from adapted be could perhaps what and – that things are going right, it can be useful to document good practices, the reasons for these situation. Evaluation can be an invaluable tool to help in this regard. Conversely, when the amend it to is done be clear could/should any) (if what and future, the for it about learnt be could and if there is a carpet - but to try understand appear they if - carpet the under facts disturbing push to try not therefore should Agencies to andnottoviewed learn punish. positively –asopportunities be should mistakes.problems some inevitable Themake to arewe likely more the territory, one’s control. of The more innovative out we are, the more often that we break reasons,new ground in unfamiliar of variety a for expected, as happen not will or wrong go will things mistakes”.make can that work add Invariablywho “onlycould those easy”.one and Indeed, be would it that said everbeen: have“Nobody might preamble our for heading possible A and makeimprovements. very to – try taking into account experiences in answers similar situations, learn from what works well or not, perfect to is do the can we best have The situations. intractable they seemingly sometimes indeed that challenging, claim would actors aid humanitarian Few Approach evaluationasatoolthatcanassistyouinlearningndimprovement, itcanassistagenciesinimproving oftheirwork (internal thequality focus); • itcanprovide for accountability focus); to (external bothdonors andbeneficiaries • • evaluation isabout • f eut, f erig y on, n dmntae epniiiy o stakeholders; to responsibility demonstrate and doing, by learning of results, of capturing lessons learnt, addressing internal information needs, developing a culture by assessing an agency’s own performances and weaknesses, evaluation can help in and notforfindingfault,castingjudgements,orpunishment Remember:"Errarehumanumest,perseverarediabolicum!" “shaping your future”; why things may have gone wrong, what lessons BACK TO TABLE OF CONTENTS BACK TO TABLE OF CONTENTS workshop alsolisted somepotential aboutevaluation. reservations partners’approached.The is evaluation the how upon depend may which – constraints be part of a programme. There are strong benefits to evaluation, undertaking but there can also interesting most the as viewed always not is Evaluation though. straightforward, be us Let Agood situations. emergency illustrationin future can be found in the joint efforts of the interventions Tsunami Evaluation (TEC). effective Link Coalition11 more in result to order in experiences past from learned be can what about guidance provide also can It improved. be can they how and operations ongoing of effectiveness the assessing in assist can It level. Learning and improvement can take place at different levels, for example at the how evaluation can be useful and provide an opportunity tohow andto learn evaluation improve: canbe usefulandprovide anopportunity identified have you once Therefore, evaluation? a of form making some really least are at efforts without our difference that – others tell or – know possibly we can how Indeed, an be can essential of part programme evaluation activities rather than a mere – add-on or a (felt) sheer undertakings, nuisance. future or current improving and results assessing in approach, i.e. conducting evaluation focusedactivities on questions that can best assist you way. That is why your mindset, how you approach evaluation, is so important. With a positive Just like anything else, evaluation can be a negative experience if not approached in the right evaluation • evaluation canhelpavoid wasting resources oninappropriate and activities; • evaluation ofeffective isanintegral management andthePCMcycle; part • Evaluations • Evaluations • Evaluations • Budget(andtime)available too isoften limited. • Much • In • Conducting • affected inemergencies situations. orotherhumanitarian evaluation team are managed! reporting formats,reporting whichcanmultiplytheworkload. field security ispoor.field security used inthefuture. be can they that so findings, capture properly to process lack memory institutional a of be also may there projects; humanitarian for frame time short the within evaluation could bebetter usedfor operations. for resources that feeling a be can there Sometimes paperwork. reporting of lots There isalsoalackoffeedback evaluations. from somedonorsaboutperformed ...but seek–anduse feedbacksothatsuccessiveefforts particular, depends can are imposed in evaluations various some identify on often the can beevenmoreeffective. cases by carried donors skills needs donors can "Just doit!" may of be out and may the be challenging sometimes too perspectives evaluators viewed have late various in (Nike) as a are in – project long, and of emergency not requirements beneficiaries also stressful viewed to how be really contexts, as and the and helpful regarding evaluation complicated, helpful, others organisational such by especially adversely agencies. TOR as and when with and the 21 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 22 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs E. essential you for opportunity major a represents used, be can it how specifying including evaluation, can h ost Ti icue ietfig ht nomto i nee, h cn s i, and it, use can who needed, is in information what its identifying determining includes is This many evaluation onset. too any the in –or step questions important wrong most the The addressing questions. by wrong go often most Evaluations what Focusing: The three steps

to be make useful ways. focusing

sure 45&1
 45&1
 45&1
 There – (looking "outside") (looking Focus onoutcome Tool box A-for orissue: eachsector choice): (single ormultiple sectors Main 1, 2orallthree canbeselected): criteria CriteriaGeneric (notto beappliedmechanically: using results) by andfor target NGOs: (for planningandmanaging evaluations, Capacity building/learningenvironment and that t

&UDy
t

)FBMUI t

'PPE t

4VHHFTUJPOT
BOE
DBWFBUT t

-JOLBHFT
UP
NPOJUPSJOH XJUI
XFC
MJOLT
tool(s), andindicator(s) benchmark(s) t

3FDPNNFOEFE
FYJTUJOHöFMEUFTUFE t

&YBNQMF
PG
FWBMVBUJPO
RVFTUJPO T 7BMVFT t
3FMFWBODF t
t

&òFDUJWFOFTT steps Standards, types, planning, methodology, management, reporting, t

0UIFS
DPOEVDJWF
UZQFT
PG
TVQQPSU
y t

1FFST t

.FOUPSJOH t

$PBDIJOH t

5SBJOJOH
SFGSFTIFS
DPVSTFTy t

,FZ
JOUFSOBM
GVODUJPOT
JO
QMBDF is used, evaluation little (Accountability purposes, and/orlearning from levels…) operational to policy should at point least - How are we going to use the results? - Why dowe want to doanevaluation? be serves dissemination anduseofresults to multiple choice): issues (singleor Cross-cutting Considering bene ts expected De ning evaluationDe ning questions considered t

&UDy
t

1BSUJDJQBUJPO t

$IJMESFO in embark or combination ofboth some the needs upon way. before of Support by donors: Support Remember, an organisation!your Threesuccessive, /(0T
UP
NFBTVSF
JNQBDU
FUD
evaluations by small/mediumsized postevaluations, to balance single t

(MPCBM
KPJOU
UIFNBUJD
QFFST
NFUB FY t

%JTTFNJOBUJPO
PG
SFTVMUT
TUSBUFHJD
VTF t

)FMQ
JO
öOEJOH
FYQFSUT t

$PPSEJOBUJPO
XJUI
EPOPST
$T t

2"2$
HPPE
FOPVHI
PWFSBMM
RVBMJUZ t

'VOEJOH
BT
SFMFWBOU evaluation tackling Generic CriterionGeneric Tool box B:
&óDJFODZ
2VBMJUZ
PG
QSPDFTT t

4VHHFTUJPOT
BOE
DBWFBUT t
-JOLBHFT
UP
BVEJU t
-JOLBHFT
UP
NPOJUPSJOH t
JOEJDBUPS T
XJUI
XFC
MJOLT

tool(s), and benchmark(s) FYJTUJOHöFMEUFTUFE
3FDPNNFOEFE t
question(s) t

&YBNQMF
PG
FWBMVBUJPO BACK TO TABLE OF CONTENTS defining methodology (looking "inside") (looking Focus onprocess if it is not the clear from focus, focus and Table 3 how of tools. an it BACK TO TABLE OF CONTENTS 12 11 Step 1: The First Two – BasicQuestions Why to evaluate andhow to usetheresults? 10 9 8 resources” ICRC: the as such and useful actors, be must “Evaluations key of evaluation for principles guiding of statements many in evaluation community as a key aspect ofevaluationevaluation asakeyaspect community the within recognised widely is use on focus A crucial. less no is question basic second The prpit eauto qetos wih s xlie i Se 3 A hc ls o possible of list check A 3. preparedICRC been the forbyexample has whyreasons whyevaluatenot,to and Step in explained is rangeof which acorresponding questions, defining by evaluation then appropriate and 2, in Step as choice major first a among evaluate?”. The to selecting want by qualified be we to has do “why”.the It of follow-up be“what logical a only however really is “what” that should and an evaluation why”, knows into donor step “the that first think the still may agencies some as evaluation, for capacity building of approach ECHO’s DG in important particularly is The “why?”question deciding to launchanevaluation -orpropose itto adonor-shouldbe: before question basic first the that surprise a as come not therefore should It so. doing for opportunity good a or learn, to need perceived a as such reasons argued well specific, to respond should evaluation any that Guide this of chapters first the in emphasised haveWe 300 evaluators and managers establish their evaluation capacities, two very basic questions should first be tackled, an tackled, approach that hasalsobeensuggested by of theOECD/DAC theevaluation section be first should questions basic very two capacities, evaluation their establish firmly to yet have who those forHowever, only. questions key three on focus to agencies later for explain will Weexample questions. of sets corresponding with up come have actors evaluation Major most useful –andhavebe thepotential to beused? could evaluation from information where time, present the at you to important it on a small number of key questions or concerns. What are the core questions that are most One of the most important ways of keeping an evaluation doable – and useful – is by focusing "Before searchingforanswers,arewecertainthathavetherightquestions?" Preskill Sept’ andCaracelli draft on Evaluating 1997,inALNAPguidance action, Humanitarian booklet 2003 Cfrfootnote 5 Te e rfrne ad ruby h ms raal tx o eauto rmis Mcal un Patton. Quinn Michael remains: evaluation on Utilization-Focused text Evaluation: Text. TheCentury 3 readable most the arguably and – reference key The (internal) ICRC Evaluation Guidelines, 2002 April Organisation for Economic Cooperation andDevelopment, Development Assistance Committee Link13 11 . Good practice collected e.g. from a massive survey carried out among nearly nearly among out carried survey massive a from e.g. collected practice Good . Remember: wheneverythingisapriority,nothingis! how the UN Evaluation Group is recommending its partner partner its recommending is Group Evaluation UN the how Link WHY DO WE WANT TO DOANEVALUATION? 12 , stressed that attention must be focused first on “planning for . If they are not, they are a waste of time, money and money time, of waste a are they not, are they If used. (oldevaluator's motto) rd Edition. Publications, Sage 1997 10 andisincludedasabasisprerequisite 9 Link 12 Link . 8 . 23 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 24 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 13 t ipiain fr erig ugss ht rgams hud e ed conal for and foruponthisinformation. acting accountable held be should effectiveness, their about questions programmesdifficult asking and feedback seeking including learning, that suggests learning for implications its some supporting measures (chapter GLink). some supporting F.1(chapter TOR evaluation the in results of dissemination and use for moreplans regarding recommendations and addressed tools review we are where later also considerations and 3, Step These under whom. by and used, be to expected is it how or programme current There rarely is one the “right” method or approach to staff.evaluation, it all depends upon its in or purpose, stakeholders of changed up participatory made is be audience whereas the if accountability, appropriate can more be on may what operation focus on a focusing to with likely methodologies more approaches be formal may evaluation are more the users donor, primary use important the an if example, or For management used. senior be to the expected are results the how on beyond this and users intended its upon greatly depend should shaped is evaluation an How involved at variousstagesintheprocess, andthemethodsthat shouldbeused. (the “what”), be addressed to questions specific the including process, the evaluation is expected to be used, is that this should inform all of aspects the evaluation how process evaluation the of beginning the at identifying, for reason important Another oriented oraculture thinking, ofevaluation. improvement”, as quality “continuousresults- to referred often also is This do? we what of anticipated? How do we Howknow? can we improve the appropriateness and effectiveness as beneficiaries of needs the addressing we Are way? possible best the in things right the evaluative of themselves: creation ask constantlyprogramme “Are a doing weinvolvedin people the where thinking, involves use, of form important extremely but subtle, more A as accountability. to when book recent especially a exclusive, Indeed, mutually referred evaluation. to as often corresponds– comes treated it be it not beneficiaries need how learning of and and desires Accountability and programme needs the the of can to use of accomplishments form One documenting positions. fine-tuning policy include from new identifying range in can assisting use to evaluation operations chapter, As current indirect. previous and the direct in both forms, suggested of have variety we a take can use that though, remember, Do users andusesoftheevaluation”. The corresponding questionshouldbe: evaluation”,an beginningof the at use secondon and intended“identifying prioritising and for Audit. Trans-Action Publishing, 2007. Bemelmans-Videc, M-L, Lonsdale, J., and Perrin, B. HOW CAN HOW CAN WE USE THE EVALUATION? Making Making Accountability Work: Implications for Evaluation and 13 xlrn te conaiiy ocp and concept accountability the exploring BACK TO TABLE OF CONTENTS ), as well as in as well as Link), ed t be to needs who Link BACK TO TABLE OF CONTENTS Evaluation duringanoperation and focusing onprocess Step 2: The Focusing andafter, Choice –During process andoutcome seem n hs es ad ih u cuin te cn e ut hlfl o fcsn purposes. focusing for helpful quite exclusive. be mutually can they as caution, not due with certainly and sense and therefore this limitative, In should not Distinctions indicative as applied. considered be be arguably could distinctions overlapping, We have already touched upon many of these challenges, such as difficulty in access, or access, in difficulty as such challenges, these of many upon operation. touched an already have during We evaluations implementing to challenges some course of are There earlier, whenitmay betoo late to makeuseofthisinformation? doing. Why wait until afterwards, perhaps to identify what could have been done differently are you what improvements to and changes make to possible still is it while – beneficiaries of needs pressing the addressing and possible as effective as is operation your that ensure potentially offers the greatest to opportunity make use of evaluation. This can enable you to this that is operation or programme a during evaluation out carrying of benefit major The future interventions. for implications and lessons learning identify to and overalleffectiveness its assess to order in operation, the of end the at 2. and improvements; or changes appropriate make to able is operation or programme your while evaluation 1. options: suggest may table simple a As used. be to this expect you how and evaluation undertaking are you why – above 1 Step in identified reasons your to your of cycle life the in project or stage operation you whatwish to undertake atevaluation activities. deciding This of course is will be evaluation related your focusing in step key A at thewrong time. implications. relatedA action key problemthe attempting is to addresswrong the question about clear less be may ultimately – and manage properly to consuming time and difficult more are undertake, to and going get to longer costly,take more arewill invariably ground limited are there when especially wrong, resources, is go by getting too complex evaluations and complicated. Evaluations which that try to cover in too much way major a Indeed, resources, astheywould entail different evaluation planningandtools. skills, n u fcsn apoc, hs eod tp il ed s o osdr oe major some consider to us lead will step second this potential approach, focusing our In Link asked Selecting If between thetwo boxes intheoverall diagram suggestsprecisely this. exclusive,mutually be not arethey as generally quite complementary. Thearrowdouble not ‘holistic’ more a adopt to rather but etc, summative or formative outcome, or process on either you superficially) As (Step a feel we

bit approach, a distinctions the focus have 3), artificial. and well-defined just would with become the pointed to Some several choice in better turn a need of starting out, aspects, is determine look their yours! though, NOT into point components be to However, certain the which the follow for ‘what’ two the specific that the if whole sides may you

question. suggestions you want of evaluation be corresponding , there are two basic two are there 202, Link , in order to be to order in , underway still have this ‘generic interlinked. to Distinctions the deal below, questions capacity, adequately focusing’ methodology. Other, i.e. may time focusing must partly

often need (and and be

25 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 26 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 15 14 But it is important, at the least, to consider how you expect the services and products you products and services the expect you how consider to least, the at important, is it But It may be premature to address questions such as these in the early stages of an intervention. to cope asbestpossible, andto move beyond, situation? theemergency is really most required? Are there other more pressing needs? Are the tents enabling people in byparticular those most in need? Are tents, as opposed to other forms of assistance, what getting tents to an area affected by a natural and disaster. resources But its are these mobilising tents at actually beingeffective used, very be may agency your example, For situation? humanitarian are or or tolikely addressaffected realpeople emergency the of the needs by Can you be sure that your activities and outputs (products and services provided) are indeed ). oflifequality (impacts improvementsin long-term in assisting ultimatelynature, in term medium or short be may (or benefits in result to is interventions humanitarian of point only the Remember: process. on emphasis over an of dangers aware,however,the of and standards various of Guide this in later discussed as principles use make to wish may You chain). results the “outputs” in and a hl yu dniy h aporae ou fr vlain t gvn eid f time. of period programme given process or evaluation a project at your evaluation for of on extent large a to focus toappropriate be may it focusoperation, an of called) stages early the in Thus appropriate sometimes the is identify it you help as can change of theory (or logic Guide this of chapter later a In considerations. other and context, the operation, the of setting stage is Indeed, the given what of expect, toconsider realistic stage. to way sure be this one evaluation, for at be questions identifying can In impacts failure. for impacts up or yourself or outcomes outcomes long-term longer-term at of evaluation look premature to possible rarely is It table 6below Link. (input chain results the Evaluation carried out during an operation inevitably would deal with the earlier stages along a actions, humanitarian many of nature short-term the Given well. as strength a be could this but – simple very kept focusedand highly be to needs context a such in way? Evaluation possible best the provided in being are they that and beneficiaries, of evaluation, can you be sure that the services you are providing are the most critical ones to priority given to providing services rather than to evaluation. But without at least some form Or you may wishto useavariation onthisapproach. this Guide other contexts or of large scale operations. Yet this is very much possible. In a later chapter of first typical mid-termfew evaluation months than with the of an operation, in much shorter the within place take to needs generally operations programme active assist to out carried their service delivery mechanisms, between these”. theirmanagement delivery practices, andthelinkages their service instruments, policy their i.e.organisations, implementing the of dynamics internal is the it of “anevaluation A operation. an of implementation the during conducted often most performance, improve to intended evaluation An process evaluation aims at learning how to improve the organisational efficiency. According to OECD/DAC, Link, we suggest how a Real-Time Evaluation (RTE) approach has and can be used. 15 – on how implementation is proceeding (taking into account “activities” → activities activities , we will also see that articulating the intervention intervention the articulating that see also will we Link, → outputs to ensure that you are acting responsibly. acting Be areyou that ensureto Link → outcomes ) to beneficiaries, which beneficiaries, to outcomes) → impact), such as illustrated in illustrated as such impact), BACK TO TABLE OF CONTENTS formative 14 evaluation BACK TO TABLE OF CONTENTS 18 17 16 Evaluation anoperation, at theendorafter andfocusing onoutcome you canto maximisetheappropriateness andeffectiveness of your operation. what doing are you that stakeholders) other and donors, beneficiaries, include may (who an serve also can evaluation such time, same the at But intervention. particular the in involved staff and managers be would Yourprovide invaluable appropriate.and useful information. Thus the if major users of this information changes,probably still can make yet dirty”), even to “quickand perhaps (indeed simple be time would methods evaluation still is there while improvements for areas for is operation or project a during out carried evaluation of purpose major Tothe summarise, helpful ornot, andwhat elsemay berequired. small number of beneficiaries) can give you an idea if the that services you are providing are Guide this in later to as final or ex post or post ex or final as to referred (often afterwards or operation an of end the towards out carried is that Evaluation criticised for focusing overly on what they what on overly focusing for criticised provide to lead to benefits.Indeed, humanitarian and development agencies are sometimes arising. At the least, you should be thinking in these terms these in thinking be should you least, the At arising. benefits a danger of the evaluation not being used if it is not clear who can make use of what can what of use make can who clear not is it if used being not evaluation the of danger a be also can there But resulted. has what affected have may that considerations other and interventions different of interaction the consider to possible single be may a it just and from intervention, result rarely particular in outcomes Long-term operation. single a just the greatest potential for strategic use, for example inaddressing questionsthat gobeyond have may form this of Evaluations settings. different or same the in interventions similar as can potentially aidinthedesign phaseorfollow andimplementation ofthenext up, aswell increase effectiveness identified by the evaluation. But learningto lessons fromopportunities suchor evaluation concerns particular any addressing in e.g. intervention, aid to particular latethat too is indicated, have we as operation, an of end the at out carried Evaluation an of stages early the in possible be operation. might than methods robust more Itispossible of use make or negative. to bepositive could that effects, expected as well at as and modified, unintended be to needed plans and assumptions initial why at faced, challenges at have gained from the services provided services the from gained have how beneficiaries outcomes that havebenefits –the from operation, arisen the particular in Outcome remains Mapping however stillawork inprogress Link201 level. organizational or project, program, the at used be can Mapping Outcome outcomes. of achievement the to make programs development contributions the assesses and characterizes which developed, been called methodology ‘outcomea outcomes,mapping’ about more has know to interestedare who those For value oftheoperation. the about information provide to intended is evaluation summative A produced. were results planned the which to extent the determine to thereof) phase a (or operation an of end the at conducted evaluation An are to expected leadtothe plannedactivities results expected andiftheseassumptionsare realistic. larger,forbeforehandhow programmes,identifyused complextomore typically moreare evaluations ante This Guide does not get into learning, to help ensure that what is being done is appropriate, and to identify potential , even some simple methods (e.g. focus groups or interviews with a with interviews or groups focus (e.g. methods simple some even Link, summative ex ante evaluation,before which is performed a programme is implemented. Ex function accountabilityfunction 17 evaluation) can provides greater potential to look at look to potential greater provides can evaluation) 18 . It can be possible to look more comprehensivelymore look to possible be can It . , with insufficient attention to the to attention insufficient with do, , by helping to assure others assureto helping by Link, 16 . And as we discuss we as And . and results 27 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 28 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 19 evaluation process evaluations, whichare bothwidelyusedandquite self-explanatory of terms generic the Guide this to throughout using approach be will effective we - evaluating an integrate to still have who partners implementing NGO the notice A accountability –to and beneficiaries to donors. demonstratecan itself in thatorganisation, responsible a representsof signs actions, future in incorporated are findings the how demonstrating in – and questions difficult the asking addition, In changes. inevitable the for reasons the and why, and operation overall the by forused be also can well,they As effective. most made be can operations future how on focus a with function, learning Evaluation carried out at the end of an operation, as suggested above, also has an important delivered fashion. inacomplementary coordinatedor best be resourcesscantcan which waysin identify also agency,and single a of those beyond going factors other and barriers contextual consider to evaluation the for easier area.it same make the can Thisin working also agencies other with jointly this doing We suggest later that to undertake outcome evaluation such as this, you may wish to consider fromlearn itscollective experience andrepeats thesamemistakes. not does sector aid humanitarian the often too reasons, these for that suggests 253 Link ALNAP recent most the Indeed, them. of use make best Reporting on sections (see formats right the in provided not are learnt lessons the if and – learned be to and bit a this streamline Toresearchers. academic for playground favourite a become major make hs a peetd y u ohr osat ocr nt o enet h wel Otie n isd will inside and Outside wheel. the reinvent to not concern ofthisGuide, including inthediagrams. stillappearinsomeparts nevertheless constant other our by prevented humanitarian was of This accountability external or outcomes interventions with - and “inside-looking” evaluations, concerned whose focusprimary is on how your organisation is primarily working. – evaluations looking” In our constant quest for simplicity, we were tempted, at a stage,certain to use the generic terms of “outside- from it challenge and Following Up Following and Link more activity the multiplicity easily has of this understandable almost Guide of , demonstrating what has been accomplished been has whataccountabilitydemonstrating , definitions, been is ) and do not get into the hands of those who can who those of hands the into get not do and Link) to kidnapped “keep to types the it bulk simple, of from evaluation, of the our Actions Humanitarian of Review but hands “target not BACK TO TABLE OF CONTENTS criteria, simplistic”. of population” field standards practitioners and outcome 19 As .

– you some etc, may the to of BACK TO TABLE OF CONTENTS Step 3: What Do We Want To Know, Actually? 20 contexts. or settings other in used be can these that so documented be should that practices good some are there that feelvalidate. Youalso to maywant you that hypotheses Youhave may undertaken. been have that activities and strategies the of impact of appropriateness and the the strengths about learn to want you anticipated,about be can or raised, being are questions Maybe this. of limitations potential and previously, than context different a in working or approach, new a taking are you perhaps Or situations. similar future in and/or context present the in – it address to done be can what and this forreasons the identifying and clarification. For example, sometimes there is a problem, and the key question concerns identification require just and obvious be can evaluation an in explore to questions Often h atvte t b asse, f ht xcl t pirts o, cness between consensus a on, prioritise evaluation questions . of to picture clear exactly a what of it requires stakeholders on this, and some drafting skills. as All assessed, this can be achieved through a few specific be difficult, to most activities the the probably is step focusing last The y OECD/DAC by should be explored in an evaluation. There are for example the evaluation criteria developed that questions suggesting in helpful be can techniques and approaches used widely Some information insomeway. is there for where look As maximum always, the questions? evaluation define to way best the is What should really beplaced aspossibleintheprocess. asearly humanitarian contexta where time and resources in are usually to lacking, every opportunity focus the but evaluation tools, collection data or indicators e.g. to related closely are questions later stage, usually somewhere in the a methodology. at placed This is is questions not evaluation entirelyspecific of wrong preparation either,the however, guidelines as evaluation evaluation most In address. should it questions what specifically more and evaluation, an of focusstage the planning – preliminary the in – determining of importance critical the stressed have We logic project a of analysis framework logical the of Elements Link. impact. humanitarian complex evaluating for criteria seven to emergencies: relevance/appropriateness, connectedness, coherence, increased coverage, efficiency, effectiveness, and was list The sustainability. and impact Five criteria were defined at first for evaluating development programmes: relevance, efficiency,effectiveness, a rcmedd y G uoei i is vlain guidelines evaluation its in EuropeAid DG by recommended as 15, Link Start • Once • could be of greatest use to them. (“What them. to use greatest of be could when?”). be redundant; inothercases, aconsensus may have to bereached. can concerns Some etc. partners, other donors, potential or donors place, taking is project the where community the of members and leaders staff, policy or HQ wants 20 , at various levels – as well as for acting upon this upon acting for as well as – levels various at learning, for opportunity by you wih il e icse i te ehdlgcl eto o ti Guide this of section methodological the in discussed be will which , to identifying ”). Many people can be concerned, ranging from staff in the field, the in staff from ranging concerned, be can people Many ”). know? know who, ol bs mk ue f vlain nomto (“Who information evaluation of use make best could who ask them what questions do they they want intervention its and 14 Link have, to and know what in priority, information ik 16, Link and 29 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 30 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs Good practice indicates practice Good that evaluation questionsshouldgenerally fulfila few basiccriteria. questions for evaluation: can also be used. The UN Evaluation Group (UNEG) be formulated inseveral ways according to thecontext. Some psychology could sometimes profitably be applied to evaluation questions, which can wasthe (“why late,delivery agencies why was notadapted?”) theproduct etc. about beneficiaries by or late?”), and payment or of capacity signature contract level the was (“why beneficiaries’ donors about agencies by the asked be also could about questions But sustainability. agency the implementing the and about performance, enquire could agency’s donor a example, For bottom-up. or top-down concerns, own their address to questions the of some tobe use and evaluation wish the they in involved if beneficiaries) peers, (donor, stakeholders some by used be to groups, evaluation to approach participatory a In and F.3 (implementing outcome Linkandprocess Linkevaluations). providedbe will illustration,as where feasible, listed forcriteria ofchapters in F.2 variety the criteria discussing be will we when section, methodology the in later mentioned be also will questions evaluation of topic The with riskassessment communities, management, orneutral questionswithdonorsandprofessional peers. beneficiary and staff local with inquiry appreciative example –for circumstances specific in better work to likely are them of some as questions, We recommend that you should consider the three approaches in preparing your evaluation Evaluation • The • Are there better ways ofdoingit?Link18 • Are we doingitwell? • Are we doingtherightthing? • In • Neutrally, • Evaluation questionsmustavoid creating andfrustrations. falseexpectations • In • follow mechanicallythewholelistofOECD/DAC criteria. to need no case any in is focus.There maintain and manageable remain to again good practice recommends to limit the number of questions to a maximum of ten, with the most useful information. In you more provide can elaborate, that two) or ‘standard’one just possible, evaluation(if questions processes, key of number small very a on evaluation the focus should you manageable, and simple things keep the appreciative approach. Link20 inquiry generally leadto vague answers. will questions vague since well, them Articulate terms. simple in know to wants and analysis could gowrong?”). approach Link21. (“What by DGEuropeAid Link19,to optimise theirobjectivity. a a positive number more possibly potentially questions fashion, of evaluation with focusing negative must the , and examples of field-tested evaluation questions evaluation field-tested of examples and Link, optional questions reflect on (“deficit-based”) “What , questions could be subdivided into small into subdivided be could questions Link, precisely assistance should did Link 17 has for its part identified three key we be what of do way kept judgment right?”-type the through to BACK TO TABLE OF CONTENTS initiator a minimum. references the questions of risk the Preferably, as assessment evaluation suggested through to BACK TO TABLE OF CONTENTS F. Tools (looking "outside") (looking Focus onoutcome Tool box A-for orissue: eachsector choice): (single ormultiple sectors Main 1, 2orallthree canbeselected): criteria CriteriaGeneric (notto beappliedmechanically: using results) by andfor target NGOs: (for planning andmanaging evaluations, Capacity building/learningenvironment t

&UDy
t

)FBMUI t

'PPE t

4VHHFTUJPOT
BOE
DBWFBUT t

-JOLBHFT
UP
NPOJUPSJOH XJUI
XFC
MJOLT
tool(s), andindicator(s) benchmark(s) t

3FDPNNFOEFE
FYJTUJOHöFMEUFTUFE t

&YBNQMF
PG
FWBMVBUJPO
RVFTUJPO T 7BMVFT t
3FMFWBODF t
t

&òFDUJWFOFTT Standards, types, planning, methodology, management, reporting, t

0UIFS
DPOEVDJWF
UZQFT
PG
TVQQPSU
y t

1FFST t

.FOUPSJOH t

$PBDIJOH t

5SBJOJOH
SFGSFTIFS
DPVSTFTy t

,FZ
JOUFSOBM
GVODUJPOT
JO
QMBDF (Accountability purposes, and/orlearning from levels…) operational to policy - How are we goingto usetheresults? - Why dowe want to doanevaluation? dissemination anduseofresults multiple choice): issues (singleor Cross-cutting Considering bene ts expected De ning evaluationDe ning questions t

&UDy
t

1BSUJDJQBUJPO t

$IJMESFO or combination ofboth Support by donors: Support /(0T
UP
NFBTVSF
JNQBDU
FUD
evaluations by small/mediumsized post evaluations, to balance single t

(MPCBM
KPJOU
UIFNBUJD
QFFST
NFUB FY t

%JTTFNJOBUJPO
PG
SFTVMUT
TUSBUFHJD
VTF t

)FMQ
JO
öOEJOH
FYQFSUT t

$PPSEJOBUJPO
XJUI
EPOPST
$T t

2"2$
HPPE
FOPVHI
PWFSBMM
RVBMJUZ t

'VOEJOH
BT
SFMFWBOU Generic CriterionGeneric Tool box B:
&óDJFODZ
2VBMJUZ
PG
QSPDFTT t

4VHHFTUJPOT
BOE
DBWFBUT t
-JOLBHFT
UP
BVEJU t
-JOLBHFT
UP
NPOJUPSJOH t
JOEJDBUPS T
XJUI
XFC
MJOLT

tool(s), and benchmark(s) FYJTUJOHöFMEUFTUFE
3FDPNNFOEFE t
question(s) t

&YBNQMF
PG
FWBMVBUJPO (looking "inside") (looking Focus onprocess Table 4 31 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 32 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs F.1. Setting thestandardsSetting case You may already have integrated inyour procedures someoralloftheseissues, inwhich hs ln - hpe wl scesvl rve te aiu sae o preparation, of stages various the issue ofthe core the review include also will successively It evaluation. humanitarian a will of up follow and chapter management - long - This last butnotleast-accountability. Additional sources canbefound inthebibliography. human ofthischapter include: sections management, for standards UNEG manual, ECHO’sevaluation DG own order): alphabetical (by as such These guidance notes are based on the most relevant parts of some widely used guidelines, niety o vlain n t hmntra atvte, n ter ubr s regularly is number increasing. their and activities, humanitarian to and evaluation to indirectly or directly related are which though, exist do standards of sets other Many UN. the by monitoring common We value isassigned according to acontext. A this Guide. Links evaluation comprehensive very the guidelines ofDGEuropeAid. to development), on being focus main (their aid Regular references have also been made, wherever relevant to NGOs and to humanitarian NGOs, have suchasCARE, alsobeenusedwiththeirapproval . major some by prepared guidelines internal guidelines.The evaluation and monitoring quality Planning andmanagingevaluations standard will the a • someadvice regarding follow up, dissemination anduse. • guidance for themanagement and ofanevaluation, includingreporting; • suggestionsto terms drafting ofreference (TOR); • assistance • ofevaluations, atypology withour recommendations; • will that resources see step-by-step and indicators; evaluation; ones, be You discussion “methodology” is an below and an provided UNHCR evaluating ranging will optimal evaluation by to that find agencies, preparing guidance “Step on from throughout standards in value, agency itself, atopic that demandssomeexplanations. this by key the activities”, key step chapter notes an a code “model” standards should evaluation principles have guide the below of short the been conduct chapter,

be which to can aiming URD managing and methodology, for references prepared simply donors, represents for and values, Group’s for. aid in evaluations”, be It workers specifically to performance the is applicable used Quality a including the a selection BACK TO TABLE OF CONTENTS reference bibliography level as to a Compass, the for checklist. UNICEF in of in to among the management to performance evaluating, the humanitarian which use annexed “Managing field, and The of the criteria a and main most fixed WFP e.g. or to of - BACK TO TABLE OF CONTENTS In Compliance also are listed inthebibliography.which protocols and covenants laws, related various the as well as (IHL), Law Humanitarian International the declaration, Rights Human the i.e. standards, and codes international legal documents that are themselves the bases of the various humanitarian reproduced below. are evaluation, humanitarian every integratedby and understood clearly be to need which assistance.disaster in standards,workThese should agencies humanitarian way the govern The code has set, for the first time but on a basis only,voluntary universal basic standards to Relief”). Disaster in NGOs Crescentand Movement Red and Cross Red International common a on agree to the by led NGOs, of networks largest and oldest world’s the of six already, know Asyouof all certainly prompted1994 in genocide disasterthe Rwanda the of valuesHumanitarian standards,Quality for test phaseapplication. Link24 have already discussed we Assistance”which Development of Evaluation for “Principles basic their to addition In referred -where relevant -intheguidance notes. Other UNEG standards concern more specifically the management of evaluations and will be (contribution building). to knowledge 2.1 to 2.4), 10 (transparency and consultation, 11 (ethics – see also standards 2.5 to 2.8) or 13 standards also –see (competencies 9 (quality), 8 (independence), 6 (impartiality), 5 norms following: the particular in NGOs, by performed activities evaluation to relevant quite be Among the norms norms the Among normative andothermandates. of the evaluation units in all UN system organizations spanning development, humanitarian, established has UNEG 17 The Link themselves. A key reference in this field is the is internal field this in reference own key A themselves. their maintain or upgrade evaluators of management the as to well as evaluations,for rules managementorganisations, standards such consider should Agencies General evaluation standards probably have inanupdated to version, beinserted induetime. are however still very much open questions at the time of drafting of this Guide. They will across the UN system, thereby contributing to UN reform and credibility. standards quality professional common establishing in advance major a represents which addition to with the standards, above, n standards and 22 Link Link - Link 56, OECD/DAC has also published Evaluationsome short is ul ae s h Cd o cnut o the for conduct of “Code the is name full (its Conduct of Code it and is essential corresponding Norms and Standards for Standards and Norms that ulse b UE, oe a also may some UNEG, by published 23 Link humanitarian United Nations Evaluation Group (UNEG). Group Evaluation Nations United certification agencies and Evaluation in the UN SystemUN the in Evaluation accreditation movement, Cross Red be The aware UNEG consists of of agencies Link 25 Link the key 33 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 34 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs The Management inthree tiers. action humanitarian Link27 for principles the classified further has below) see International, Partnership Accountability providing also corresponding definitions. be guided by the key principles of principles key the by guided be should action humanitarian that stating for known widely agencies. more implementing usually is all GHD However, to interest of also course of is which action, humanitarian of definitions and objectives reaffirmed has GHD financing, actions, of accountability of and management terms in donors for practices good promoting Humanitarian Beyond ECHO. DG (“Good including Donorship Humanitarian of donors,many by 2003 practice in Stockholm in Donorship”endorsed havebeen GHD) or principles, Good and Principles The Response, firstpublishedin2000. Link 29Agencies committedandto the to theCharter Standards andtheMinimum inDisaster byperspective Charter theSPHEREHumanitarian The Cross Red Code hasbeencomplemented ofConduct inamore operationally oriented Performances inthefield agency.a humanitarian of organisationinternal the of quality the assess to aiming evaluation,process a of case the in relevant quite be can indicators, by complemented has Aid In People which These principles, place’). right the at time right the in people right (‘the staff adequate deploying of human resources by aid agencies. They recognise the crucial importance for any achievement personnel’ The Code's ten principlecommitments: People 7. 6. 5. 4. 3. 2. 1. 10. 9. 8. We shallattempt to builddisaster response onlocalcapacities. We culture shallrespect andcustom. We shallendeavour asinstruments ofgovernment notto act foreign policy. politicalorreligious standpoint. aparticular Aid willnotbeusedto further basis ofneedalone. the on calculated are priorities Aid kind. any of distinction adverse without and recipients the of nationality or creed race, the of regardless given is Aid The imperative Humanitarian comes first. relief aid. Ways shall be found to involve programme beneficiaries in the management of n u ifrain pbiiy n avriig ciiis w sal recognise shall asdignified humanbeings,disaster victims nothopelessobjects. we activities, advertising and publicity information, our In whom we accept resources. We hold ourselves accountable to both those we seek to assist and those from basic needs. Relief aid must strive to reduce future vulnerabilities to disaster as well as meeting Link 28 promotes seven key principles that aim at improving the management of In Aid’s ‘Code of Good humanity, impartiality, neutrality impartiality, humanity, Practice Link 26 It should be noted that HAP-I (Humanitarian in the Management BACK TO TABLE OF CONTENTS and and Support independence, Table 5 of Aid BACK TO TABLE OF CONTENTS 21 high polarisationandpoliticisation. issue in a context of protracted and violent humanitarian crises, of breakdown of institutions, complex extremely an be to prove may beneficiaries However,final towards accountability in thebibliography. Link course manual initiativeson evaluation Link 39 Other of can action. humanitarian be found its in accountability on notes guidance useful produced also has ALNAP 32. link principles whatsummarised should be the approachin particular to accountability through seven key the “Standards in Humanitarian Accountability and Quality management” Link 31. HAP-I has 2007 January in published has which 30, HAP-ILink is field this in organisation leading The shouldbefullyaccountable, action humanitarian butrather how to doit. best it can in addressing their needs. The main question is therefore increasingly not whether the doing for works, is which in communities the and beneficiaries its to accountable customers. be to than rather needs equally agency an that agree donors publicly now actors humanitarian key all Nevertheless, of satisfaction the by ensured generally is operations the of dignity and regarding practice in accountability. privateUnlike humanitarian of business, viability financial humanity the overlook may or needed, beneficiaries. most Despite the various humanitarian codes and standards, is much remains to be done that help of kind provide the to fail may late, arrive indeed may Agencies action. humanitarian of regulation and accountability of lack the about themselves, actors the by often made, been have criticisms of aid funds in Iraq, misuse of some of the international aid for the Tsunami etc) regular victims embezzlement (possible - questions open least at or - scandals recent of number a on Based responsibilities to thosewhoentrusted thoseresponsibilities” use is the following: “the obligation to present an account of and answer for the execution of to what this really means. Probably the most common definition of accountability in general The concept of accountability is talked about frequently, but often with limited consideration Accountability will leadto numerous references intheimplementation chapters. Link Standards are alsoquite helpfulinsuggestingevaluation questionsandindicators, which food, shelter, healthcare) andinanexpanding issues. range of cross-cutting The SPHERE to people’s (water, activities ofhumanitarian needinallmainsectors sanitation, nutrition, particular,principles. In Standards theMinimum aimto quantify requirements withregard and conflict, to promote offundamental orarmed humanitarian calamity theobservance StandardsMinimum are aimingto achieve definedlevelsfor peopleaffected ofservice by Accountability Journal6,3:52-67. Gray, Jenkins, and A. W.I. (1993). PublicSector”.“CodesNew Accountabilitythe of in Considering • Even maximumaccountability cannot guarantee quality! • of most aid actors, the opinion of the final beneficiaries regarding their level of level withtheaiddeliveredsatisfaction boundto isoften benegative. their regarding beneficiaries final the of opinion the actors, aid most of the extent of the needs and the limited 21 . resources Accounting,and Auditing (time, money) 35 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 36 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 22 Deciding about the type ofevaluation aboutthetype Deciding a further classification. further a to leads which programme, or project humanitarian concerned the of aspect specific a on focused be also can Evaluations agencies). peer between learning and resources sharing at emphasised), independent (when accountability is emphasised) or not), is there (when can lead to a typology of e.g. standard approach (when there is enough time), or term, programme a beforeof design starting), and feasibility validate ante(tothe have been contracted outside. From a timeframe perspective, evaluation can be launched called internal be also can Evaluations types. formative and summative or outcome, and process between distinction key the discussed already have Weneeds. learning main the to taken approach the and perspective the to according ways, many in classified be can Evaluations aayt o ognstoa ad prtoa changes” operational and organisational for catalyst of operational responses to emergencies and ensuring that findings were used as immediate wasdesignedat toaiming be byIt UNHCR. “rapidly evaluating effectivenessthe impact and and Kosovo) in 1998-99 (in DANIDA by developed first was approach cycle.RTE The project humanitarian rapid usually the in late too come often evaluations ex-post or standard that effectonkey an have to intervention decisions and to assist in ofahumanitarian improving the existing operation. Experience demonstrates indeed the course in early sufficiently Basically, RTE is a fast very and timely means of providing evaluative feedback. is It launched providing evaluationReal-time (RTE) guidance for improving acurrent operation will follow chapter. themethodologythat willbeoutlinedinthenext might at least start by considering one of the following two approaches.they that Standard suggest trepidation,evaluations some with therefore, would Weevaluation. with or experience capacity their improve to wish that agencies to helpful most being at aims Guide This and thelistisnotlimitative. RTE”, ajoint UNICEF/OCHApaper, November 2006. UNHCR, Jamal and Crisp, 2002. See also “Agency experience with RTE – Towards an approach for inter-agency Some • And, donotforget: • The • or (near or after completion). Special interest in the process of an evaluation an of process the in interest Special completion). after or (near ex-post depending onthelocalculture. thiscase, In beprepared for discussions! contradictory to some core principles -or procurement procedures!- of aid agencies, the beneficiaries to doadecent, fairjob”, details. withoutgoinginto unnecessary accountabilitystop? NGO’sOne e.g.is position that trustedbe should byNGO “the demonstrate respect for beneficiaries is through accountable relations. Where does , when carried out by the agency’s own staff, or ownagency’s the by out carried when self, stated Be verycarefulaboutpromisesthatcannotbekept! NGOs priorities we saeodr ae novd n/r erig is learning and/or involved are stakeholders (when participative argue Link 33 Link that Such classifications can easily become become overlapping, though, easily can classifications Such by it beneficiaries is far from and clear their 22 RE a sne en ple b a by applied been since has RTE . that preferred the , by evaluators who evaluators by external, BACK TO TABLE OF CONTENTS best joint evaluations (aiming approaches way rapid appraisal real-time, to ensure may mid- and ex- be BACK TO TABLE OF CONTENTS 25 24 23 ouet h cnet n cntans ht r fcd hs prah a as so how to show also easier can approach is This faced. it are example, that constraints For and context challenges. the these document address working all help can donors another and/or agencies one different alongside of number a involving evaluation joint A the bestofefforts. even sabotage can control your beyond Events simultaneously. place taking interventions –or difficult be also can intervention, sometimes impossible. own Typically in a humanitarian emergency, your there are multiple actors and of result the is outcome observed any potential of Identifying used. array be can An designs research complicated. sophisticated very get to potential the have evaluations Outcome Joint evaluation at outcomes withotheragencies, looking being addressed). of process the in is people, starving to food getting as such emergency, the in challenges are perceiving the intervention, and if there is any evidence, even if preliminary, that the key whatever consider also beneficiaries how (e.g. effectiveness about can stage early an at available RTE be could information although – issues process on looking’ ‘inside be would approach this of focus primary The memory. institutional with assisting organisation, the perhaps and identify effectiveness, lessons that increase could to be now applied in right other done situations. be The can learning what can rest on within focusing useful, very be to This method can be applied very quickly, and with limited resources limited with and quickly, very applied be can method This (onthereturn flight, ifpossible!). afterwards prepared quickly report brief very and avery been prepared by USAID preparedby been example for has limitations and advantages with methods, appraisal rapid of table useful A review ofkeydocuments, keyinformant interviews, andfocus groups). and using a variety of data-gathering methods that can be applied quickly (typically including other sites and/or for the organisation overall. The evaluation team would visit the operation situation.organisationfor potentiallyand largerimplicationsidentifying in aid also can This the with familiar people by approach positive,constructive a taking time same the at while impartiality, and objectivity of standards the by ‘distance’ necessary required the provide differentfroma but site. relevantorganisationon activities the within working can else This good a As constraints. and potentials example, team an mayRTE include someone from the HQ of the organisation and someone rules, agency’s the with familiar already are who As the technique would apply to an NGO, it would typically involve a small team of evaluators and Assessment (REAM) Methods Evaluation Rapid of variety a evolvedinto has organisations,and other and NGOs of variety assessed activity. Some recent RTEs were launched within large multi-agency UN programmes such as in as the Eastern Africa,such withmixed results. on programmes UN impact multi-agency actual large within an launched have were to RTEs recent time Some appropriate activity. most assessed the at place taking at aims merely that process evaluation mid-term standard-looking almost an into transformed been has cases some in concept UNHCR It should be noted that RTE methodology is currently still evolving. The original «quick and dirty» DANIDA and Performance andevaluation Monitoring TIPS n°5,USAID1996 University, ofEvaluation, Journal American June2007. P.G.and McNall M. appraisal, and Assessment,Evaluation, Rapid Foster-Fishman,of State Methods Michigan 24 A debriefing is provided to field staff departure, before staff field provided to is debriefing A 34 Link 23 . – the extent to which to extent the – attribution 25 . It has the potential the has It . 37 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 38 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 26 eore. o tee n ohr esn, cos uh s .. OECD/DAC e.g. as such actors reasons, other and necessary these the For obtain resources. to easier it make also may and – responsibilities management and contexts. As it is more complex different in in lessons nature, these of taking use a for and joint learning approachfor opportunity can greatest the help provide spread also the workload can show how various actors have contributed to the observed outcomes. This approach can that just Agency A and not Agency B can claim all the credit for what took place. In contrast, it outcomes can arise from the actions of different agencies, without trying to create the fiction Link 232. NGOs Dutch some by out carried evaluations joint concerning reports the in found be can illustration An conflicts. potential avoid to members, its of aspects organisational looking inside- process, investigating - from refrain or - consider carefully also should evaluations Joint time. of investment initial an as well as attitude, right the and compromise requires priorities and approach on agreement common But a to coming itself. and relationships evaluationdeveloping the beyond well to go benefits can many which be others, can with collaboratively There circumstances. working all in suitable or perfect, is approach No prior some by bound be preferably such andmutualconfidence,relations ofpartnership inthe orclusters.MoUs, of form consortia, should if and (even freely donors), so by do promoted to are need evaluations evaluations joint undertake to agree who Agencies andguidance notes.checklists CARE or 39' Link Guidance for ManagingJoint Evaluations, OECD/DAC, 2006. Mar have been promoting joint evaluation practices and publishing and practices evaluation joint promoting been have 36 Link BACK TO TABLE OF CONTENTS 26

ALNAP 35, Link BACK TO TABLE OF CONTENTS Developing amethodology 27 i. saying, address the various questions of the evaluation. Thus a methodological plan is more than n h eauto cmuiy n ltrtr - ees o the to refers - literature and community evaluation the in A evaluation or other purposes, is really a that More single method, orwithatoolbox. which ajudgment ordecisionmay bebased” that theyhadreached thetop oftheHimalaya. Criteria are justcriteria, i.e. “standards on the wide dissemination of e.g. the OECD/DAC criteria may have led some actors to believe Similarly, an evaluation methodology cannot be reduced to using a set of criteria, even if will address theoverall evaluation questions. how they are to be used in phasing and timing, and how the use of each particular method Before of the function Link17. of thefunction aspects various the to 3.10 and 2.1 1.7, 1.6, dedicatedstandards1.5, also has UNEG 38. Link Compass Quality its of evaluation” part “projectthe in tasks related of checklist complete responsibilities and tasks corresponding of table bibliography the through accessed be can EuropeAid The evaluation function is described in details in various guidelines, for example those of DG below), shouldbetheguardian. ofwhichtheevaluation function (see standards UNEG from inspired possibly guidelines, or rules internal adequate adopt to management should not to be allowed to edit/control evaluations. There is therefore a need senior that added be assured”.should is It reporting transparent and unbiased that so and located be to independently from the other management has functions so that it is free from undue function influence evaluation “the 6.1), (norm UNEG to to operation According the evaluated. ‘distance’ from be necessary the and skills ethics, require that tasks key of number a fulfil must - agencies medium-sized or small in e.g. position full-time a not is it if the of aspect crucial a is agency and of theeffectiveness efficiency process. The evaluation manager - or evaluation function an within commissioner) (or manager evaluation The The role oftheevaluation manager(orevaluation function) what To summarise, evaluation managers (or commissioners) are normally responsible for identifying common be measured againststandards, andcomplemented by indicators. (being right “observedownvalues” their than in rather themselves“optimalstandards should values”). They objective”,an necessarily to not relation and in project’sperformance a assess that to order in only “elements to refer operators are Criteria definition. restrictive more slightly a suggests approach Compass Quality recent its in (URD) Développement’ Réhabilitation Urgence ‘Groupethe that note to interesting be may It includes specifically specifically, entering for Links 40, example, mistake the into should specific an 41, but also by most UN agencies and bilateral donors guidelines which that the that evaluation subject, be we methods “we’ll at at evaluated, must do we methodology be that a would survey careful collection of methods, approaches or procedures. will and resources what with time, in point what like . DFID has for example prepared a synoptic a prepared example for has DFID Link. be and to (Webster) to used avoid - , and the URD Group has drawn a drawn has Group URD the and 37, Link emphasise also as most in some is combination, 27 to . commonly equate that focus or design evaluation a “methodology”, groups”, a methodology and used in but and what indicating accepted ways be with plan it over for to a 39 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 40 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs ii. appropriate variations doexist, according to ofasmany establishedpractices agencies. other many though: suggestion, a merely is below shown sections of order sequential The items.or requiresthat followingsections generally the practice TORinclude good Standard in ajoint evaluation). agency lead (or agency commissioning the of behalf designatedon evaluationmanager scope by contributions for opportunities with jointly, made be should TOR of development The record ofagreement asto what willbedone. formal a provide they him/her; of expected is what of idea clear a have to evaluator the the evaluation timetable. They allow the commissioning agency to express its needs, and and considered be to issues the evaluator’smission, the of boundaries the out set TOR Preparing the TOR (Terms ofReference) be usedornot. evaluatorworkcan how and well, evaluationconditionsthe tothe whichlikely and underis an which under conditions the intoplaceresponsiblefor putting is manager evaluation the evaluator, and that competencies of managers are thus at least as Atimportant. a minimum, the as evaluation of use meaningful certainly and failure, or success quality, the with do to much as least at has manager evaluation the that argue well could one above, the Given of and plan, work and methodology detailed the of approval for the for responsible staff), internal and/or consultant[s] (external are managers Evaluation strategy. are and frequently to be addressed, for responsible main questions the at least of specification including what period of time. They are responsible for hy ae lo ed epniiiy o esrn the ensuring for responsibility lead also have They the The • A • The • The • The • of various commissioning agency. actors etc). This section may also stress issues related to mandate, values, etc of the evaluation”?). should accurately reflect the main the reflect accurately should evaluation). the by not or covered are components specific some if precise to relevant if and evaluator to specificperform: tasks (briefing, inception period, desk review, travels, projects.been usedinprior alreadyhave they if purposes comparativefore.g. stage, this at recommended be evaluation process). Corresponding indicators to be followed by the evaluation can the to associated be may who recipients or donors stakeholders, other of perhaps the brief evaluation, of the evaluation (brief description of the of description (brief evaluation the of scope approaches and purpose specific stakeholders eeomn o te ai eauto dsg ad vrl methodological overall and design evaluation basic the of development description of the general the of background description o e nwrd y h eauto. These evaluation. the by answered be to questions evaluation evaluation or f h eauto (wy o e at o o an do to want we do (“why evaluation the of objectives involved ta te omsinn wud xet the expect would commissioning the that “methodology” questions, in the development of the terms of reference (below), concerns project Well-defined work , for process , the of management overall eeac ad s o te evaluation . the of use and relevance (with plan), eeto o te vlain team evaluation the of selection of the commissioning agency (and agency commissioning the of of the crisis (origin, region,(origin, crisis the contextof TOR are therefore crucial. respect under all BACK TO TABLE OF CONTENTS to be evaluated, be to activities interim the to the supervision and objectives final . reports of and the BACK TO TABLE OF CONTENTS 28 detailed inthefollowing chapters ofthisGuide. h cnet f oe f h aoemnind O scin (aa olcin tools, collection (data sections TOR above-mentioned the evaluation of some of content The TOR shouldtherefore alsoinclude, from theoutset, corresponding sections. often left to the ultimate phase of the evaluation process, where they tend to be overlooked. the in considered insufficiently often areresults TOR.of use too and are They participation adequate of aspects key the that indicates practice evaluation of analysis the addition, In to create a “generic TOR” for Assessment inEmergencies itsRapid example of good practice - for internal purposes. It should also be noted that UNICEF intends copied have we Guide, this through 3.11. and 3.7, to 3.1 standards UNEG Link 17 For of the general recommendationsreference only of the those who would e.g. need to access with a practical example accordance in generally is which format, preferred own its already has - ECHO DG including - donor and agency major Every purposes. indicative for even list, a suggest to pressed hard be would Multiple examples of good practices in preparing TOR can be found, so many in that fact we Emergency Field Emergency Handbook–Aguidefor UNICEFstaff, UNICEF, July2005. A • A • An • The required format reporting . • The • A • discussed withtheevaluator, ifhe/sheisdeemedto have adequate skills. tobe procedures, alreadyleft well-established and suggestedonly maybe they or has agency commissioning the if required be may approaches Such used. be to possibly field), (HQ, visit to places debriefing), could beassociated withtheevaluation. Professionalwork. shouldalsobestressed ethicsandcode ofconduct here. orderto in etc), issues achieve a balanced team,with the evaluationin adequacy objectives and scope of cross-cutting some of knowledge specific experience, theinception oftheevaluation.be madeduring period may suggestions alternative and briefing, the during evaluator the with discussed tasks, the locations to be visited, and the reporting deadline. The work plan is often methodology. The work plan usually includes an indicative timetable for the various although it is important that evaluators arealthough itisimportant alsoinformed. agency,commissioning the byimplemented essentially be relevant.to is plan The methodology. about feedback include should and participation, evaluation results. It also requires of prior coordination with evaluation questions leveland the to adapted be itself,beneficiaries,such as are involved theevaluation in process. should plan The corresponding of evaluation results, both internally and externally if externally and internally both results, evaluation of plan use/dissemination , when stakeholders other than the commissioning agency agency commissioning the than other stakeholders when plan, participation team, (usually require work indicative) f the of profile plan, recommended participation, overall an example of TOR that CARE considers as an as considers CARE that TORof example an 42 Link figure to suggest the maximum costs that costs maximum the suggest to figure budget bcgon, sectoral/geographical (background, evaluator(s) work reporting , which is also an integral part of the of part integral an also is which plan, data collection and analysis tools analysis and collection data and follow 28 . up) will be further 41 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 42 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 29 iii. Qualitative methods to try to and data, validate findings by triangulation ofsources. qualitative and quantitative both collect to methods, these of all) (or listed below should probably be preferred. Where possible, an evaluation should use several methods the approach, RTE an relevant, where follow, to suggested have we As allowing. opinions. Answers to evaluation questions must be based on substantiated evidence, not subjective andanalysisChoosing data tools collection (ormethods) have a TOR ofmore thanoneortwo pages. evaluation,e.g. coursethe operation,toduring an out of carried mayit necessary be not and purpose expectations main of the evaluation, the should be developed for minimum all evaluations. Fora a very simple at out setting TOR, of form Some the itself. of evaluation scope the with commensurate be should TOR a of detail of level the particular, In requirements. hard-and-fast as viewed be not however,should guidance above The tutrd prahs sres n mpig ae lo en used been also have mapping and surveys approaches, structured leaders, community local e.g.alwaysnot does or RTE fit assessment rapid approaches.participation with well More rarely, with trust of relations adequate up build to prerequisite a Guide this of section separate a in participation discuss will (we beneficiaries with approaches Participatory below. listed are methods these of Some responsive’ purpose. and their ‘flexible to adapted tools, rapid and simple using usually are methods assessment rapid other and - contexts emergency in NGOs recommendweto which form simple original, their in least at - evaluations time Real Link 193hasprepared alistofquestionsLink194. is tool setting.a world whether To‘gooddecide you enough’,help the Guide’Enough ‘Good that it can take human behaviour out of context in a way that removes the event from its real more subjective, is indeed erroneous. The greatest weakness of the quantitative approach is ones qualitative and alreadyobjective more are methods quantitative that perception the 43, Link as other, each methods appraisal complementrapid on table to USAID the used in demonstrated be should methods both feasible, Where and how many, andqualitative methodshelpto answer questionshow suchas andwhy. who , as such questions answer to help methods Quantitative answer. to seek they questions of types the and approach the in lies categories two difference the between There are two broad categories of data collection methods: Data collection recommendations. and conclusions corresponding draw and patterns, establish to them analyse manner, Cfr footnote 24. • should be avoided. Effective interviewers ask questions with interest, respect and respect interest, with questions ask interviewers Effective avoided. be should be cross-checked – triangulated – with other sources. Leading or biased statements Individual interviews . Their focus should be on obtaining factual information that can Accordingly, evaluations ) must always be favoured. However, since time is often often is time However,favoured.since be always must Link) must be able to collect quantitative and . According to WFP to According 34. Link BACK TO TABLE OF CONTENTS findings 29 tm ad resources and time , in qualitative. The an how much, how organised BACK TO TABLE OF CONTENTS techniques Interviews eety a iia cmr. artv hsoy r trtlig ehius rw out draw techniques and lesscensored andare by notdictated familiarevaluative jargon andprocesses. storytelling or history Narrative information that might be camera. difficult to obtain by other means. Responses areoften fresher digital a – recently • evaluation process. or community teams involved to test out findings to date and update them on the group meetings can also be organised to call together a group of people from the personalities. strong by dominated be to meetings group for risk the be also may There women). (especially groups public in represented be not theremay issues that be are in groupsdiscussed not maypeople some or because because gaps however,informationinterviews,leave group may on Over-reliance issues. individual than rather (below) community of sense be a may getting for which valuable reached be can consensus cases, some In discussion. course group the a in of arise can that information or viewpoints other considering perspectives when initial their modify can frequently people and – context social a in considered responses for providegroups Focusrespondents. of number a among local people or officials) can be useful for on population, the spotaffected cross-checking of information the of members (with groups focus / groups in Interviews processes by agencies. selection beneficiary and targeting the with problems reveal may canalso it as fruitful available be assistance the from benefit not did who those of out seeking Ensuring contexts. emergency confidentiality of in interviews with some individuals may be documentation necessary. The deliberate and keeping record baseline poor of problems the of because assistance development of valuations with case the normally is sourcethan important morerepresentmay a separateInterviews interviews. their in team the of members all by used be should protocols standard of development the interviews, out carrying are people of number a If sensitivity. with such beneficiaries as the ‘Most can Significant often be Change’ made

more (MSC) effective technique by o - more - or 44, Link using Workshops storytelling storytelling and 43 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 44 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs iv. 30 process. Some considerations could have an influence on the timetable, in particular the particular in timetable, the on influence an have could considerations Some process. change and decisions management the of effect actual havean recommendationsto allow to period appropriate most take the at least at must or cycle, project evaluations the in possible that as early as place implies approach assessment rapid or RTE the example, For of use effective results by theconcerned stakeholders. the authorise to evaluation, of aspect crucial a is reporting of Timeliness Defining plan and timetable work found in DG EuropeAid’sguidelines DG in found in used be to mostly - ‘standard’ has evaluation processes forformostly can and development– be example purposes tools analysis and collection data on information Additional would involve a set of rather complex tools which would require trained evaluators.trainedrequire would which tools complex rather of set involvethis woulda although qualitative, or quantitative be also can analysis data that noted be should It Quantitative methods eeomn o a itreto ad t promne Tpcly te nltcl process analytical the Typically, performance. involves its three steps, whichare to befollowed by theevaluator(s): and the about intervention evidence an credible into of data the development transform to is analysis data of purpose The Data analysis own evaluation guidelineslink49. Ministry ofForeignMinistry Danida (2006) Affairs,Denmark, ‘Evaluation Guidelines’. • • • interpreting thedata, e.g. assessingthefindingsagainstcriteria’. • thedata, describing e.g. and generating findingsoffact; • organising thedata for analysis, i.e. data preparation; • population. homogeneous a relatively require they because up set to difficult more also but be implemented to properly. theory sampling of knowledge some also needs It areas. accessible easily usually those with the most resources or power are the ones who settle in the most as results biased to lead often will it that is sampling of type this with danger The etc. settlement, refugee the of centre administrative the near or roadsides, along settled have who those example,access,forto easiest are who respondents those sampling on relies It situations. emergency in method sampling used frequently Conveniencemost haphazardthe accidentalcalled is or (also sampling) sampling behaviours oropinions. characteristics, community on data quantitative some generate can community) specific a on focused groupinterviews basically are (which interviews Community the population. the population. on having is assistance their that effect the and population conditionsaffected the of the observe to evaluators allows interviews, to addition in site), (on observation Direct Mini-surveys are slightly less susceptible to sampling biases Link 45, Link and 46 in Danida’sin 47 BACK TO TABLE OF CONTENTS 30 or in Prolog’sin or 48 Link Link 50 Link BACK TO TABLE OF CONTENTS v. Link 42. This tool doesnotneedto besophisticated, butaccurate. A timetable with key deadlines is therefore an essential - and generally the final - part of the TOR due to e.g. migration patterns. interview to beneficiaries of range of availability or on; clothes many with months, winter in children weighing diseases; periodical account certain and malnutrition into e.g.cyclic as such factors take not would that scheduling by affected be also may findings of Quality local (scheduled)politicalevents. donors’or relevant time,cycles, funding harvest season, rainy the cycles: work and seasons Actual choicesActual shouldbemadeaccording contexts. to actual choosing evaluators. when These qualities are and neither necessary competencies nor sufficient, following though, the but merely considering indicative. recommends practice Good own agency. Link51 your of needs the to adapted be might which grid, selection of model a designed also has example of required qualifications can be found in thetypical TOR ofCARE An them. of track keep evaluators, good know you If promptitude. to suitable with conclusions this able all do be draw should he/she And and reports. good patterns write to detect mind synthetic findings, a and methodically, collect right the to to mind questions analytical right the an ask persons, to mind inquisitive an possess to needs evaluator’ assistance of be perhaps may that organisations of list a provide will – support and capacity with dealing - Guide this of chapter final the and Let us be straightforward again, such things do happen. You will find some guidance below, over-optimistic. the absence of any possible on cross-checking prior achievements, has proven to be slightly the in involved closely too exist do been –bias operations concerned had they because rejected be to have may evaluators yearthe of for period maybusiest evaluators;coincidethe with exceptionallysome suitable of number the drastically potentially suitable profiles; restrict despite a thorough planning the time chosen for sometimes the evaluation may requirements TOR example, For task. in competencies evaluators’ the of easy an always not however, evaluators, is importance skilled evaluation. Tosuccessfulfind a achieving prime the on insisted already have We evaluators Selecting Demonstrable • Inter-personal • -opennessto newideas. Inquiring • Keen observation skills and a deep sense of curiosity are alsoneeded. andadeepsenseof curiosity skills Keen observation organisation. social groups, beneficiary participating among discussion foster to andtact. level ofdiplomacy demandsanextraordinary factions of armed members or military the -increasingly- and population, affected the partner staff, agency implementing authorities, local as groups diverse such with interact to and interviewing analytical ; or the CV that you have finally retained, in retained, finally have you that CV the or 249; Link skills, skills - the particularly willingness . But remember that the ‘perfect ‘perfect the that remember But Link. in complex and ability emergencies, to Link 42. The NGO listen, the the ability need 45 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 46 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 31 vi. Evaluators can be external or internal; when selecting them, it is worth remembering that:Evaluators them,itisworth orinternal; whenselecting canbeexternal eomnain sol b tml, rcia, o h pit ad hud hr feasible where should and point, the to practical, timely, be should recommendations The recommendations. and lessons conclusions, corresponding to leading be themselves should which findings, evidence-based of patterns contain also and should comprehensive It also up-to-the-point. but read, to people for enough executive short all-important be the must which with summary, start should It agency. the outside others for intended primarily a report than different way a in presented be could agency an of staff internal for ‘digestible’ taking into consideration its intended primary audience(s). E.g. a report intended least at or comprehensible, and accessible information the makes that way a presented be donor.should a byreport Theor managementagency’s the byrequired structure common The evaluation report should be logically organised, following for example in such a minimum several keyissues. erig n ato.Ti ivral rqie mr ta js a oml report formal a just than more requires invariably This action. and learning ultimatelycommunicationsand is – Letalwaysus reporting of rememberobjective thatthe Reporting which allevaluators shouldabide. ethical various the defined further evaluations conducting in importance have particular of are that considerations Danida or CIDA as such donors Some 17. Link 3.15. to 3.13 and manager), evaluation the with (relations 3.10 ethics), and (competencies 2.8 to 2.1 standards evaluators: to dedicated are standards UNEG of number large fairly A R.T. Torres, H.S.Preskill, andM. E. Piontek. Evaluation Strategies forCommunicating. Sage, 1996. andReporting Credibility • Familiarity • Strong writing, andaccounting numeric skills. • to Ability thinksystematically andrigorously. • Detachment. • Thoroughness. • internal • external • Sensitivity • Familiarity withthepoliticalcontext. • mandate. and workloads; heavy have already may and methods, evaluation in trained properly generally less expensive; may but their be objectivity questioned, they may not be whether or not planned objectives werewhether ornotplannedobjectives achieved. should be able to work from programme documents and interviews and determine be unfamiliarwiththeagency’s culture. may and expensive often are they but experience,broader on based perspectives evaluators evaluators with with to the the the complexity managers understand day-to-day (when who properly and the operations will constraints organisation, implement skilled) of the associated generally may the programme. evaluation appear BACK TO TABLE OF CONTENTS with offer less Ideally, the recommendations. objective, threatening, , and by and 52, Link organisation’s 31 ad raises and , evaluators fresh are

BACK TO TABLE OF CONTENTS with suitableexperience to provide anadhocassessment. the peers external utilises and/or internal UNHCR of panel responsible,mechanism’, a readingi.e. ‘expertan partner of arbitration implementing the with issue that on dialogue non-conclusive a of case in and reporting), processevaluation(especially the of element an of quality the about staff supervisory donor’s the by doubt of case In UNHCR. by provided be arguably can practice good of example relevant Another judgement. case-by-case a to representa can what appreciationof the ‘goodenough’ left generally is reporting of quality Last but not least, when the report is meant to be delivered to a donor or major stakeholder, parameters to theofagency.final working report the in possible as adapted practically as them make to order in openly, discussed be then would recommendations suggested independence, and objectivity for respect due With findings. their explain would evaluators external where process, evaluation an of end the near workshopsof idea Worldthe in foundparticipatory organise be to VisionInternational perhaps could field this in practice good of example recommendations.An unrealistic with up come may and constraints, and culture organisation, agency’s an with familiar entirely Another potential issue, as already mentioned, is that external evaluators may not always be Link. building andsupport capacity on chapter the in suggested as donors, European among effort “3C”coordination standard or evaluations conducive measures could however be promoted, e.g. by DG ECHO, in the framework joint of the Some solved. either easily be not will make This task. challenging a quite may donors multiple to which reporting format, since ideal long own has agency its or donor designed every nearly that is reporting with problem frequent A bureaucrats thanto practitioners. and academics to more appeal may and information critical mask may that but questions, possible all address may that report technical detailed, a than used, be to likely more and in say may some – almost often all cases – a short, very focused the Very report emphasising implications may be utilisation. more of useful, be scale should the objective with the commensurate all, be Above should evaluation. report a in detail of degree and nature the Again, report. evaluation an of preparation the regarding recommendations sensible 17 link 4.18 to 4.1 and 3.16, n° standards UNEG reference,general a As mustbeableto understand: report be free of information that is not relevant to the overall analysis. The reader of an evaluation concernedissues.placedtofollowthe best wouldon indicatebe who up should Thereport what lessons were distilled. • what recommendations were made;and • what conclusions were drawn; • what evidence was found; • how theevaluation was designed andconducted; • what was evaluated; exactly • oftheevaluation; thepurpose • provide some very very some provide 47 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 48 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs vii. 33 32 for upon laterimplications from arising acting the evaluation. There is increasing evidence often a prerequisite, to development indeed of support and ways,credibility for the best evaluation itself – the and of one is evaluation the planning in involvement and Participation differently inthefuture? done or improved be could what identifying in assist particular in and perceptions these not, or of well some address could feelgoing they do information questions.is What and challenges,perceived feel they what about sessions, of series a or single a in discussion, a in stakeholders Involve approach. participatory a take to is address, should evaluation the One of the best ways of planning an evaluation, and in particular in identifying what questions Taking approach aparticipatory example defined seven practical steps in conducting a participatory evaluation participatory a conducting in steps towards practical seven defined accountability example the in role a play survey. has for USAID satisfaction place a be cases,for there mayonly other In beneficiaries. themselves could evaluations case which in imperative), then is (feedback beneficiaries the by questions evaluation several or one of preparation the with envisaged be could participation full a known- already are principles evaluation–wherefavourable circumstances most the In stages.later in used be will inputs receive rarely they consulted, monitoring and evaluation is and gaining people importance, are increasinglyparticipatory Though evaluation. and monitoring programme in involvement less much however have people initiatives”. Crisis-affected “local “consultation” and to participation” “passive from range probably would programmes of implementation the in participation of typology A stage for credibility oftheevaluation, andfor for ultimate buy-in action. andsupport the set helps importantly,it most But missed. be might otherwise that questions important Involvement of a range of stakeholders can help ensure that the evaluation addresses some countries. developing in practices evaluation introduce gradually to way a as process, the in included be should evaluators local that suggested have actors Some evaluation. the by project when planning an evaluation, to help ensure that their considerations are addressed your of critics potential include to idea good a be also can It considered. not have you that able to pose questions about the appropriateness be or may effectivenessstakeholders of someOther ofregion). youror operations country concerned the in Advisors TechnicalECHO can include community leaders, beneficiaries, other partners, donor representatives is (e.g. DG it rule, general a as But agency. the advisable to involve within key stakeholders from people within and outside key your agency. For of example, this group small a be could this time, little vary.is evaluation? can thereWhere an This planning involvedin be should Who thinkandact. the participants where the act of in part taking planning and the evaluation process leads to changes in how that seven majorNGOs. Link193 Participation is also at the core of the ‘Good Enough Guide’ recently published by a group of Performance andEvaluation Monitoring TIPS n°1, evaluation’,‘Conducting aparticipatory USAID1996. E.g. Patton, op. cit. Note 8. represents one of the main ways in which evaluation is frequently used, frequently is evaluation which in ways main the of one represents use process feedback on the results of the evaluation, and on how their how on and evaluation, the of results the on BACK TO TABLE OF CONTENTS 33

Link 53. Link 32

BACK TO TABLE OF CONTENTS viii. Part 4 “Use andEvaluation ofMonitoring Performance”. Link55 Results, for Evaluation and Monitoring on Handbook its in UNDP them among guidelines, Danida agencies and donors have accordingly established their own procedures, such as for example 3.17 standard UNEG framework, this In an organisation that would beresponsible for: plansare Action condensed within often intothey are listing people orfunctions alsoshort. further consistent,but are be instructions recommendationsor the cases, most In developed,though. to probably need plans action follow-up regarding approaches and Methods found insection VII ofthemodel TOR proposed by Link42. CARE be can example practical A TOR.the of part integral an as beginning, the from established be should plan action the recommended, have we As plan. action appropriate an through This final stage includes the all-important task of effectively using the results of an evaluation, There are however cautionsto theabove. afew important Following up Participation • Time • Different • Participation • In • (i) (iii) (ii) Link 54. A few major agencies have however been developing more comprehensive all possiblestakeholdersinasingleevaluation withoutitgettingoverly complex. evidence most credible to them. It may not be possible to address the questions of organisation ofcommunities, culture, localpolitics, level ofeducation etc). practice, as the to increasing theinvolvement ofcrisis-affected peopleinhumanitarian response. in role active an safety their and affecting decisions welfare. perception in fundamental is shift This take to able and own, their of ideas and capacities with actors social dynamic as them recognising ‘victims’,from or ‘recipients’ to ‘beneficiaries, may into turn problems for theimplementing teams. participating stakeholders that could not be fulfilled. Frustration can appear, which partial results. partial or biased either are and agendas-, political own their -and stakeholders dominant some of opinions the reflecting in result often may rapid which techniques, use appraisal they or cycles, may funding donors’ required most time than important the more case largely which be in cycle, project whole the in their participation and communities beneficiary of empowerment at aiming are approaches oioig hi apiain n proi bss n rprig ak to back reporting and internal structure). basis periodic a management on (through a number of possible functions, according to the agency’s application their monitoring themoutat theconcerned programmecarrying level, and accepting recommendations at theoverall management level, some is definitely stakeholders cases, of also modus operandi may largely depend upon local factors (degree of final evaluations means an beneficiaries may issue shifting have could in could be used as a general reference. Most general a as used be could 17 Link different beneficiaries’ may our contribute be perception questions, difficult

to participation. create to and of implement people may expectations find Either affected in different the participative evaluation from by types crises local of 49 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 50 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs Criteria andindicators i In the new set of criteria, relevance has been complemented by ‘appropriateness’,complemented been three has and relevance criteria, of set new the In Criteria” OECD-DAC the Using Action Humanitarian “Evaluating entitled guide a complex properly more largely been evaluate disseminated has among agencies, essentially thanks to the to efforts of criteria ALNAP which has published evaluation ‘humanitarian’ objective seven of the set with ensuing The 1999 emergencies. in and done – was evaluation aid of This set adapted humanitarian criteria. an required which operate, to often under so had constraints – evaluators humanitarian the that acknowledged aid, rapidly humanitarian DAC of expansion the evaluate to pre-dated designed been essentially mostly had criteria these that era an in developed Initially It The judgement two OECD/DAC (for are sustainability), shown below. ALNAP to according definitions their and criteria These 56 . Link Assistance” Development of Evaluation for the “Principles concerning 1991 in published work their of are the five main criteria which have been defined by OECD/DAC in 1998 criteria evaluation of set - used widely most the still probably and - known widely most The The mostcommonly usedevaluation criteria as criteria one-word pre-set listed below Link221. the than rather questions, evaluation to precisely more elaborate niao (r aibe i a eeet ht n vlao cn s t maue criterion. a measure to use can evaluator an that An element questions. an an evaluation is of suggesting variable) aspect in (or indicator useful every quite systematically be categorising therefore can at they aiming operation; used, being are criteria of on the should which usual final essential • • • • • looks at the wider effects of the project – social, economic, technical, and technical, economic, – social, environmental project the of effects wider the at looks Impact s ocre wt assig hte te rjc i i ln wt local with (aswellneeds andpriorities asdonorpolicy). line in is project the whether assessing with concerned is Relevance (is)theprobability ofcontinuedSustainability long-term benefits. negative, and positive unintended, macro andmicro (sector) (household). and intended be can Impacts institutions. the criterion ofeffectivenessthe criterion istimeliness. or whether this purpose,can be expected to its happen on the achieves basis of the activity outputs. Implicit within an which to extent the measures Effectiveness an output, to seewhetherthemostefficient approach hasbeenused. achieving to approaches alternative comparing requires generally This inputs. of Efficiency measures the outputs – qualitative and quantitative – achieved as a result section be definitions a definition criteria’, judgement noted elements of this or that suggested for short chapter or o idvdas gne- n aegop, omnte and communities age-groups, and gender- individuals, on – a of the criterion decision statements the EuropeAid will by evaluation the provide as can proposed that Quality be evaluation based”) can guidance work. be Compass. by defined We and, guidelines the notes programmes. OECD/ programmes. development have Webster in For on about a already BACK TO TABLE OF CONTENTS an footnote, evaluation recommend ad-hoc dictionary criteria mentioned Link 13, as a result basis the and purposes, (“a rather the and 1 Link to indicators, standard respond Link 57. Link use one more sets of of ‘ BACK TO TABLE OF CONTENTS corresponding processes”“key (no lessthan45). orsub-criteria with equipped themselves are which twelve, to criteria (quality) of number the increasing Group URD the by promoted Compass consideredas by OECD/DAC. arestill differentpursuing Others thought. of lines Quality The of largernumber much a parallel in perhaps and ‘sub-criteria’,situations,various toadapted ‘generic’ criteria as convenient broad, of entry points number into limited more evaluation, a as include suggested may for future. They example near by the IFRC, in defined be to likely are criteria of sets New approaches. their reviewing in engaged actively thereforeare DAC, A number of major actors in the field of humanitarian evaluation, including ALNAP and OECD/ evaluation process. the simplifying to helpful hardly is hand, other the on criteria, Multiplying criteria. distinct the military and making humanitarian principles clear to everybody - would arguably demand disturbing regularity. Some of these challenges - for example streamlining the relations with aid is very much an evolving field of action, in which new challenges tend to be pop up with humanitarian know, to placed well quite be may Guide this of readers the as Furthermore, ‘coordination’”. with confused often is It operationalise… to criteria DAC the of difficult most the proved has that “coherence example for acknowledges readily guide ALNAP’s The use. field easy for than studies academic for suitable more rather appear also may criteria these of Some evaluation don’t specific feel that you needto address allpossiblequestionsinany singleevaluation exercise. more And time. current the articulate at activities your to help relevance particular of to be would that questions and suggest to them Use mechanistically. than rather intelligently – criteria potential of sets other potentially and – these use Thus, “is mechanistic, often andexcludes more creative processes”. every case, this is generally what happens! ALNAP recognises that the use of the DAC criteria in used mechanically NOTbe should criteria of lists whole their that ALNAP OECD/DACand both of insistence the Despite criteria. these using while applied be should caution Some and enlarged theideaofsustainability. new criteria have been added as indicated below, including ‘connectedness’ which has replaced • • • • te ed o ec mjr ouain rus aig life-threatening facing groups suffering wherever theyare. population major reach to need the Coverage: that allpoliciestakeinto account andhuman-rightsconsiderations. humanitarian as well as humanitarian policies, to ensure that there is consistency and, in particular, Coherencetoneed security,the assess : policies developmental, trademilitary and ownership, accountability andcost-effectiveness accordingly. Appropriateness is the tailoring of humanitarian activities to local needs, increasing aue r crid u i a otx ta tks ogrtr ad interconnected and longer-term takes that problems into account. context a in out carried are nature Connectedness refers to the need to ensure that activities of a short-term emergency

a aotd n xadn approach, expanding an adopted has 38 Link 51 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 52 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs ii 34 trbtbe n mr. hs a fr xml be ilsrtd s olw b WP n their in WFP by follows as illustrated andEvaluationMonitoring GuidelinesLink58. been example for has This more. any be not attributable may and activities, humanitarian of end the after well only, term long very the on becomemeasurablemayalso etc.indirect, or It level)microlevel), and/or (householddirect (sector macro negative, or positive unintended, or intended be can Impact interventions. humanitarian emergency of timeframe short rather usually the in misleading be however could impact for Looking agencies. for motivation and purposefulness of potential higher A generic alternative for effectiveness could be’, ‘impact which would arguably a carry much defined hadbeenclearly attheoutset.objectives the that provided - below 6 table in definitions see - operation humanitarian a of outcome the evaluating for appropriate quite also is it but purpose”), its achieves activity an which to extent the “measures (it technical bit a appear may Effectiveness for actors. results humanitarian of importance particular the outline to evaluations, outcome covering criteria generic other the before position, first the in considered be probably should Effectiveness Effectiveness for criteria Generic outcome evaluations cross-references to e.g. Compass Quality criteria for compatibility purposes. eid o tasto ae ao calne fr n gie icuig hs n. da of Ideas one. this including ‘generic’ guide, any for challenges major are transition of Periods The proposed criteria generic might perhapsbemore In parallel, we will also discuss below about some possible alternative generic terms that not adequately theneedto reflect better enforce humanitarianprinciples. one exception for be will There relevance. efficiency, effectiveness, criteria: OECD/DAC existing of names betweendivision main outcomethe processand evaluations. Theyretainmostly will the objectives or standards in humanitarian programmes. The generic criteria will also respect This approach authorises in the two following chapters the use of tables with further further with guidance tables of use the chapters following two the in authorises approach This operations, according to of thespecificfocus ofevaluation chosenbyhumanitarian agencies. of number elements limitative,or components non multiple larger,the to a adapted better into sub-criteria, subdivided be will criteria generic four The form propose will Wepragmatism. cautious by guided be will approach Our compromise. thus us Let you alsowant to bereasonably systematic? And with due caution, while remainingand not simple-not-simplistic reinventing-the-wheel? G CO ugse ta te mgt e ald HC” a arnm o “uaiain i Ciei for Criteria Aid “Humanitarian for acronym an “HACE”, called be Evaluation”. might they that suggested ECHO DG is will it

and even a they notes, few ‘sub’ take? possible - four

hyperlinks criteria ‘values’, aswe think In - the ‘generic’ to ‘motivating’ are avoid meantime, appealing, to

examples completely criteria (or how that theterms ofappropriateness orcoherence do ‘parlant’ 34 but crepnig o h bods ctgre of categories broadest the to corresponding , to of are any tentatively evaluation they inFrench) for agencies. kind going of reflect one-size-fits-all questions to materialise, BACK TO TABLE OF CONTENTS these and trends approach indicators, and in if our yes, Guide when what and BACK TO TABLE OF CONTENTS the demanding needs of conducting humanitarian operations. humanitarian the demandingneedsofconducting fulfilling to adequacy its and effectiveness) to closely deliver it relates to(which pertaining expected as capacity outputs its of components, agency, an of organisation internal number the of quality large the to fairly a ultimately encompasses efficiency generic The it altogether might againbetantamount to ‘reinventing thewheel’. not does input” replacing that used widely so already quality.However, is per good efficiency entail necessarily “output of measurement the Agreed, process”. of “quality brackets between added usually havewesense. meaning, wider expanded Toa this highlight better this Guide suggests to use it also as the sole generic criterion for process evaluations, albeit in Despite the somewhat restrictive definition of efficiency as one of the standard ‘line’ indicators, Efficiency for criterion Generic process evaluations and minds” operations, accountability towards allstakeholders, etc). hearts with “winning space humanitarian encroaching military (the challenges new of face humanitarian community the as a whole by and by criteria the external stakeholders, of has been regularly category increasing in the this - for respect and - to support of importance The all. by understandable easily be should that of “value”,term designation a generic the under gathered been have codes or standards laws, principles, humanitarian various The Values a settle given situation. finally to hope a have to approach strategic comprehensive a up setting of understood need widely the from emanate would that purpose pro-active of sense stronger a An alternative generic term for these issues could arguably be ‘strategy’ itself, as it may entail exit strategies, coverage andcoordination, etc. and entry adequate implications, their and contexts wider or local relevant priorities, and needs identify to effort every comprised be should relevance of criteria generic the Under Relevance Results Inputs Activities Outputs Outcome Impact Logframe an operation. The financial, human and material resources required to implement are inputs which through performed mobilised to produce specificoutputs. work or taken Actions etcThe whichresult products, from services anoperation. outputs. The medium-term orintermediate results ofanoperation's unintended results produced by anoperation, orindirectly. eitherdirectly or intended negative, and positive The Results chain long-term Table 6 53 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 54 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs iii. 36 35 example respond to thefollowing criteria. should they be that ‘SMART’,should meaning indicators that suggest will guidelines Many n niao i sml a aibe lmn ta cn e sd o esr a rtro. A criterion. a measure to used be number can that element variable a simply is indicator An Indicators sub-groups). It is important to use: isimportant It sub-groups). other and age, gender,(e.g. factors of number any by disaggregated be may indicator An are sometimesreferred to as “performance indicators” (seedefinitionsinglossary). indicators such, As framework. logical its in articulated as results desired the achieving and planned as implemented being is operation an whether to as management the informing of role critical the play They change. assess to order in time over compared are Indicators e.g. befound intheEuropeAid guidelinesLink220. A few basicrulesshouldalsoberespected whenconsidering indicators. qatttv o qaiaie atr r aibe ht rvds sml ad eibe en t measure to means reliable and simple a provides that variable achievement orto with an operation. thechanges reflect connected or factor qualitative or quantitative A an implementation process, hasprogressed how faranactivity orto what hasbeenmet. extentanobjective tomeasurecriterion. used a is indicatorvariable) An (or Thismeasurement in givesvariation of indication an • • In • indicators • • Do • indicators that notselect are Do unnecessarily complex. • too many notselect Do indicators. • Do • of measure theindicators andresults. waysto practical arethere coverssystem and the what agreeon parties all that so Measurable: The monitoring system and its indicators are unambiguously specified and clearly by result desired relatingdirectly to achievinganobjective, andonlythat objective. the of essence the captures system The Specific: realistically indeed are there isadangerofsettingyourselfachievable. upfor Otherwise, failure. these that expectation good a and – these establishing In that respect,seealsoourcaveatsIn regarding SMART indicators below. («outcome»). operation the of performance the on information sufficient provide measurement ofthereplies to theevaluation questions; quantitative or measurement and, assuch,shouldbeneitherpositive nornegative. recognised field actors. humanitarian by developed practices good existing Such from drawn be etc). therefore should indicators region, situation, sector, (specific concerned actors humanitarian by by slightly particular, the not not focus state Quality more that the too do qualitative indicators, whenever possible, for better substantiation/ Compass have elaborate target not much identify already achievement on 35 or by WFP or physical definitions quantitative been progress n hv be wdl accepted widely been have and field-tested in 36 can . A typology of evaluation indicators can indicators evaluation of typology A . the targets indicator be («output») found, unless itself: such BACK TO TABLE OF CONTENTS indicators, you the as have those indicator as a proposed firm they is basis simply may not for for a BACK TO TABLE OF CONTENTS do so. ered while using SMART indicators, and that these should be used only when it makes sense to However, good practice suggests also that a number of potential constraints should be consid- SMART • In • • • • could be less rather thanmore attention to results. outcomes andactual inputs than outcomes. An unintended consequence of insisting upon this approach be the majority in emergency humanitarian contexts. humanitarian inemergency be themajority measure; and programmes with duly established and usable track records may not to able are scientists social what just do to them require to be would measurable indicated program. or project the by impacted stakeholder group be to particular the identification of clear with period, set a for frequency desired at manner cost-effective a in tracked Time-bound, of expectations stakeholders. the reflect that and manner, practical are a in that achieved be performance to of likely levels establishes system The Realistic: and Relevant intervention. the to linked be can issue developmental targeted the in changes that requires Attribution realistic. are result(s) the whether and intervention the of result a as anticipated are changes what identifies systemAttributable : The and Achievable practice, indicators ) requiring programmes to only attempt what is specific and specific is what attempt only to programmes requiring 257) Link SMART Timely, may objectives Trackable, become are and limitative. much : The system allows progress to be to progress allows system TargetedThe : easier In to particular, apply for (as measuring Michael outputs Patton and has 55 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 56 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs F.2. Implementing anoutcome evaluation (looking "outside") (looking Focus onoutcome Tool box A-for orissue: eachsector choice): (single ormultiple sectors Main 1, 2orallthree canbeselected): criteria CriteriaGeneric (notto beappliedmechanically: using results) by andfor target NGOs: (for planning andmanaging evaluations, Capacity building/learningenvironment t

&UDy
t

)FBMUI t

'PPE t

4VHHFTUJPOT
BOE
DBWFBUT t

-JOLBHFT
UP
NPOJUPSJOH XJUI
XFC
MJOLT
tool(s), andindicator(s) benchmark(s) t

3FDPNNFOEFE
FYJTUJOHöFMEUFTUFE t

&YBNQMF
PG
FWBMVBUJPO
RVFTUJPO T 7BMVFT t
3FMFWBODF t
t

&òFDUJWFOFTT Standards, types, planning, methodology, management, reporting, t

0UIFS
DPOEVDJWF
UZQFT
PG
TVQQPSU
y t

1FFST t

.FOUPSJOH t

$PBDIJOH t

5SBJOJOH
SFGSFTIFS
DPVSTFTy t

,FZ
JOUFSOBM
GVODUJPOT
JO
QMBDF (Accountability purposes, and/orlearning from levels…) operational to policy - How are we goingto usetheresults? - Why dowe want to doanevaluation? dissemination anduseofresults multiple choice): issues (singleor Cross-cutting Considering bene ts expected De ning evaluationDe ning questions t

&UDy
t

1BSUJDJQBUJPO t

$IJMESFO or combination ofboth Support by donors: Support /(0T
UP
NFBTVSF
JNQBDU
FUD
evaluations by small/mediumsized post evaluations, to balance single t

(MPCBM
KPJOU
UIFNBUJD
QFFST
NFUB FY t

%JTTFNJOBUJPO
PG
SFTVMUT
TUSBUFHJD
VTF t

)FMQ
JO
öOEJOH
FYQFSUT t

$PPSEJOBUJPO
XJUI
EPOPST
$T t

2"2$
HPPE
FOPVHI
PWFSBMM
RVBMJUZ t

'VOEJOH
BT
SFMFWBOU Generic CriterionGeneric Tool box B: BACK TO TABLE OF CONTENTS
&óDJFODZ
2VBMJUZ
PG
QSPDFTT t

4VHHFTUJPOT
BOE
DBWFBUT t
-JOLBHFT
UP
BVEJU t
-JOLBHFT
UP
NPOJUPSJOH t
JOEJDBUPS T
XJUI
XFC
MJOLT

tool(s), and benchmark(s) FYJTUJOHöFMEUFTUFE
3FDPNNFOEFE t
question(s) t

&YBNQMF
PG
FWBMVBUJPO (looking "inside") (looking Focus onprocess Table 7 BACK TO TABLE OF CONTENTS i. Effectiveness 38 37 Examples ofgeneral evaluation questions them allwithduecare. ‘judgement rtra o te cn e ae f hr saeet ta crepn t te ou o an of focus the to correspond evaluation that statements short of made be can they or criteria, OECD/DAC to correspond may they case which in scope, in general be can Sub-criteria could possiblyfocus onwhichanagency itsevaluation.corresponding sub-criteria of list - limitative non - a with table a provide will Guide this criterion, generic each For for than e.g. impact indicators impact e.g. than with Each sub-criterion will be complemented in the tables below, where relevant and feasible, Link 221. of links indicators. In some cases, no corresponding question or indicator could be found, and no as those recommended by the SPHERE standards - are performance indicators performance are - standards SPHERE the by recommended those as such - indicators existing particular,many In purposes. own your to adapted be to need generally would they but use, immediate for copied be probably could indicators and as providedare sanitation”). Effectiveness Effectiveness and theirwelfare systems, institutions orprocesses ofconcern (ibidem). Set at the objective level, impact indicators are the signs of behavioural change in, and impact on, beneficiaries indicators shouldbedisaggregated by sex andage, ifappropriate (UNHCRPractical Guide, March 2001). outputs that can be quantified and/or qualified and readilyplanned of revisitedachievement the towards within performance of measures are a foreseenindicators these level, timeframe. output the at Performance Set Sub-criteria evaluation can comments therefore question criteria’ purposes. In , to better illustrate what could be done. Some questions Some done. be could whatillustrate better to only, examples and the be

------"Were therightthingsdone?" in approach links proposed. Are the services/aid used? Are theservices/aid good? Is quality Are theyaccessible? Are available? theenvisioned services/aid reachingIs theintervention itstarget population? toward itsstated objectives? orprogrammeIs theproject moving satisfactorily non-achievement ofobjectives? What were thereasons for achievement or Were outasoriginally planned? carried theactivities been achieved? To what extent have theagreed (sector) objectives later a specific We to case, 38 . They are therefore often more useful for monitoring than monitoring for useful more often therefore are They . already trust Evaluation question(example) recommended Even sector the your criteria so, existing or please understanding type would and bear by of activity field-tested the be in mind approximately EuropeAid and (e.g. that field evaluation “Level questions evaluation experience /quality/ comparable › to becontinued onpage56› questions and Link 213 Source ALNAP guidelines indicators to adequacy 37 Table 8 handle rather to and the 57 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 58 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs › continued from page55› (connectedness, Sustainability Sustainability Sub-criteria effectiveness Effectiveness Change Impact Impact Impact LRRD) Cost------"Is theprogramme adifference?" making long term, for goodorill? What difference and make, didourinterventions short (see tablewithexamples ofgeneral indicators below) - - - - evaluated in a different environment? What are thecosts ofreplicating being the subject What are the cost implications of scaling up or down? results?expected What isthecheapest ormostefficient way togetthe benchmarks? costsHow compare doactual to othersimilar you thinkwas themostsignificant changeofall?’ ‘From amongallthesesignificant changes, what do of change]?’ domain as themostsignificant changein[particular backover‘Looking thelastmonth, what doyou think impact? Did ourpresence have any unforeseen harmful onthemselves?effects oftheintervention affected perceiveby theintervention to bethe What andotherstakeholders dobeneficiaries the conflict? Did we contribute of to theprotection ofthevictims of thetarget population beattributed to theproject? Can acontribution status to changesinthe(sector) achieved? To whichextenthave theoverall been objectives costs? (environment) What andlong-term environmental are the short- (economic) activities? be covered; what istheeffectonothereconomic How willrecurrent costs andfuture expenditures (social/institutional) does itlinkto localcapacities, power structures? Will theprogramme beculturally accepted, andhow future, at theendofprogramme? (overall) How willthepositive effectsbemaintained inthe effective? What canbedoneto more maketheintervention standards? How well dotheachieved results compare to quality response to changesintheenvironment? Were appropriate andtimelyadaptations madein Evaluation question(example) BACK TO TABLE OF CONTENTS › to becontinued onpage57› Link 213 Link 213 Link 192 Standard Source ALNAP ALNAP UNEG, MSC 3.8 BACK TO TABLE OF CONTENTS ii. Examples ofgeneral indicators › continued from page56› (connectedness, Achievement of Sustainability Sustainability Sub-criteria Sub-criteria effectiveness Effectiveness objectives Impact LRRD) Cost------Link 59 dynamicsand Degree ofchange insocio-economic • - - stakeholders. according to theperceptions of different key Timely goodsandservices, arrival ofsupport, grouping andethnicity.socio-economic Uptake anduseofresources, disaggregated by sex, per sector. Attainment ofSPHEREStandards (SPHERE2000), and males). (e.g. planners, implementers, andaffected females setting andimplementation by different parties inplanning,Degree objective ofparticipation including strength oflivelihood andgenderanalysis. out, ofneedsassessment Adequacy carried pollution, deforestation. ofchangeintheenvironmentExtent , e.g. water participants host government, includinglevels of ofsatisfaction incourses from the and numberofparticipants oftraining out coursesnumber andtype carried at various levels isbuiltby e.g. theintervention, The degree ofgovernment to whichthecapacity and finances andtheirmanagement including choice ofpartners Adequate, institutional review ofpartners hasceased support agency after Resources received by theaffected population to development agencies allocation ofresponsibility anddetailsonhandover Existence ofasoundexit strategy withtimelines, relations androles , gender structure Changes insocio-economic or ethnicrelations? grouping,place interms ofsocio-economic gender to orachangeinthestatus quo? What changes took social structure. leadto Didtheintervention areturn How ofthe docosts affectthesustainability results? economic benefitsoutweigh the costs? beingevaluateddoing?Do Is thesubject worth Evaluation question(example) Indicator (example) › to becontinued onpage58› Standard Source Source ALNAP ALNAP ALNAP UNEG, WFP 3.8 Table 9 59 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 60 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs iii. gender, standards children will be the main issue for UNICEF, and a cross-cutting issue for DG ECHO. SPHERE many,among example an As agencies. of perspectives and mandatescore accordingto SPHERE standards ‘Main (thislistisnotlimitative). sub-criteria Examples ofspecificevaluation questionsandindicators, linked to corresponding › continued from page57› context; DG ECHO’smandate DG The by DGECHO. have We environment. the extended the and list of cross-cutting issues HIV/AIDS; in accordance with a equality; recent review commissioned gender resilience; and strategies concerned genericcriterion. Sub-criteria last Impact sectors’ protection, part human acknowledge of

have the resources; Link 29. - - - - - table HIV/AIDS been degradation of, theenvironment? lead tointervention significant improvements in,or ofenvironmentalExtent change.Didthe ethnicity? grouping,this differ by socio-economic sex, ageor have improved asaresult Does oftheintervention? theaffectedDoes population feel that theirlives Views oftheaffected andperspectives population. women’s? so, whoselivelihoods were e.g. supported, men’s or to and effectiveoflivelihoods?If recovery support population provide. Didtheintervention resources to livelihoodsDegree oftheaffected ofsupport rights? affected population, andanadvancement ofhuman space,in humanitarian greater protection for the rights leadto. Didtheintervention anincrease ofpromotionExtent ofprotectionandhuman policy? asexpressed initsgenderequality of theagency goals thegenderequality support intervention ofpromotionExtent . Didthe ofgenderequality Link 5. Link 7 includes defined Numbers cross-cutting This protection; and list the as the and Indicator (example) corresponds such environment. Quality scope issues: participation in the of Compass children, cross-cutting table also ALNAP to of below, criteria older the primary proposes key issues BACK TO TABLE OF CONTENTS as people, which sectors they stakeholders; can 8 correspond are such vary disabled covered mentioned significantly issues: Source ALNAP people, coping by to local the the in BACK TO TABLE OF CONTENTS Level /quality/ adequacy of adequacy Sub-criteria: and hygiene generation, livelihoods, cash-based sanitation, activities, activities, Food aid, vouchers nutrition security, income Water, Food - Link 190 mortality. and nutrition dedicated to website is Indicators' The 'Smart Comments Main sectors Main (distribution) evaluation questions (rations) Link 61 Link 60 Links to rehabilitation) Link 184(market Link 183(qty, amount) Link 177(income gen.) Link 101(income gen.) Link 100(production) Link 178(training) Link 99(general food sec.) Link 113(distribution) Link 112(supplychain) Link 111(food handling) Link 110(food quality) appropriateness) Link 109(food Link 108(ration planning) Link 157(education) Link 107(micronutrient) feeding programs) Link 159(therapeutic feeding programs) Link 158(supplementary Link 106(severe maln.) malnutrition) Link 105(moderate at-risk)Link 104(nutrition forLink 103(nutrition all) environ-mental health) Link 175(other Link 98(drainage) Link 97(waste disposal) Link 96(chemicals) Link 95(malaria) Link 93(design oftoilets) Link 92(access to toilets) household level) Link 174(sanitation, Link 91(water facilities) Link 90(water quality) Link 89(access to water) Link 176(water) Link 173(hygiene promot) Link 88(hygiene promot.) Links to indicators › to becontinued onpage60› Table 10 61 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 62 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs › continued from page59› Level /quality/ adequacy of adequacy Sub-criteria: generation, livelihoods, cash-based settlement items (NFI) Non-food activities, activities, vouchers security, services income Shelter, Health Food - Link 190 mortality. and nutrition dedicated to website is Indicators' The 'Smart Comments evaluation questions Links to Link 189(vouchers) micro-credit) Link 185(micro-finance, Link 180(cashfor work) Link 179(cashdistrib.) Link 148(eco. recovery) Link 147(veterinary) Link 146(seeds, inputs) Link 145(pests) Link 144(livestock) Link 143(fisheries) access)Link 102(market mitigation, environment) Link 171(hazard environment) onthe Link 119(impact Link 118(construction) Link 117(design) Link 116(covered space) planning) Link 115(physical shelter Link 172(transitional) Link 170(emergency) management) Link 169(campdesign and settlement planning) Link 114(strategic Link 130(HIS) Link 155(pharmaceutical) Link 129(clinicalservices) Link 156(communities) Link 128(PHC) Link 127(coordination) Link 126(localsystems) Link 149(general) Link 125(healthpriorities) Link 124(NFI–tools) Link 123(NFI–stoves) Link 122(NFI–utensils) Link 121(NFI–hygiene) Link 120(NFI–clothing) BACK TO TABLE OF CONTENTS Links to indicators › to becontinued onpage61› BACK TO TABLE OF CONTENTS › continued from page60› Level /quality/ (IDPs, refugees, adequacy of adequacy Sub-criteria: reparedness, Civil-military Civil-military returnees) Displaced DIPECHO interface Disabled Children Disaster services people Health Link 198 UN-CMCoordSee - - - also: See - Link 211 International Handicap See Link 236 DIPECHO See Link212 risks' 'Livingwith See Link 223 Link 222 UNICEF SC-UK Link235 Link 200 for education INEE standards Link 190 mortality. and nutrition dedicated to website is Indicators' The 'Smart Comments Cross-cutting issues Cross-cutting Link 197(UNHCR) Link 196(SC-UK) evaluation questions Links to Link 150(childhealth) Link 141(chronic diseases) Link 140(psychiatry) Link 139(mental/social) Link 151(maternal health) health, GBV) Link 138(reproductive Link 137(injuries) Link 154(HIV/AIDS) Link 136(HIV/AIDS) Link 135(outbr. response) Link 134(outbreak prep.) Link 133(casemgt) Link 132(measles) Link 153(education) Link 131(prevention) Link 152(EPI) Link 197(UNHCR) Link 187(returnees) Link 182(IDPs) mitigation) Link 171(shelter hazard mitigation, management) Link 168(preparedness, Link 167(technological) Link 166(geological) Link 165(hydro-meteo) (friendly spaces)Link 160(friendly Link 150(childhealth) Links to indicators › to becontinued onpage62› 63 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 64 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs › continued from page61› Level /quality/ connectedness) (sustainability, adequacy of adequacy Sub-criteria: Elderly (older) Elderly Environment Protection of Participation beneficiaries (equality) HIV/AIDS Gender Gender people LRRD (USAID), humanitarian aid.humanitarian hazards for environmental studies regarding (IIED) onrecent (IRD) , e.g.See Link 241 DG ECHO UNICEF Link239 Link 238, UNAIDS See - - - - - Link 237 Helpage Link 191 Democracy") and- Development- ("Relief-to- also calledRDD sometimes LRRD is Link 224 UNICEF See Link 226 ALNAP See study Link227 ALNAP See Link 193 for participation Enough Guide' the'Good See Comments Link 233 Link 195 Link 234

Link 62 Link 63 Link 66 Link 65 Link 64 evaluation questions Links to (other environmental health) Link 175 on theenvironment) ofshelter Link 119(impact Link 154 Link 136 Link 181(genderrelations) Link 163(GBV) Link 151(maternal health) health, GBV) Link 138(reproductive Link 148(eco. recovery) spaces, youth centres) Link 160(child-friendly protect.) Link 94(sanitary mainstreaming) Link 186(protection Link 80 BACK TO TABLE OF CONTENTS Links to indicators › to becontinued onpage63› BACK TO TABLE OF CONTENTS › continued from page62› Level /quality/ communication adequacy of adequacy Sub-criteria: Protection of Visibility and beneficiaries below), which can be compared to sub-criteria intheabovebelow), whichcanbecompared table. to sub-criteria The Compass also The Compass provides acompatible approach through column criteria' 'quality (left presents anumberof'keyprocesses' (rightcolumn), corresponding to eachquality Quality criterion C: criterion Quality The project Quality criterion B: criterion Quality The project removes orreduces of therisk Security achieves itsobjectives negative impact criterion. criterion. These keyprocesses indicators. could beusedasquality Link 199 communication' 'A for partnership guidelines DGECHO's See Link 240 guidelines DGECHO's See - - Link 224 UNICEF See Link 226 ALNAP See Comments CompassQuality Link38 C.4. C.3. C.2. C.1. B.3. B.2. B.1. Link 67 Link 66 evaluation questions reduced. isanticipated,security andremoved or The ofnegative onpeople's risk impacts removed orreduced. and politicalcontext isanticipated, and The ofnegative onthesocial risk impacts anticipated, removed orreduced. economy is (from) aiddependency The ofnegative onthelocal risk impact or reduced. environment isanticipated, andremoved The ofnegative onthe risk impacts achieving objectives. The measures project itsprogress towards account. Constraints are analysed andtakeninto Several operational strategies are explored. Links to trafficking) Link 188(slavery, Link 164(psychosocial) centres) Link 163(GBV, women Link 162(familytracing) Link 161(staff) Links to indicators › to becontinued onpage64› 65 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 66 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs i. Relevance Examples ofgeneral evaluation questions › continued from page63› Appropriateness Issue orsub- Coordination aims for beyond positive impacts (coherence) Relevance / Relevance Quality criterion D: criterion Quality The project Coverage criteria Strategy implementation ------"Are we addressing the real needs?" to spendonwhat andwhere? weDo make clear decisions about how much money (the otherstakeholders)? weDo understandthestrengths andstrategies of environment? Are we aware ofwhat ishappeningintheexternal clear? Is ourpurpose agendas? toproportionate theirneedanddevoid ofpolitical population groups withassistance andprotection How well provide didtheintervention major basic goals? How well work towards didtheactors thesame intended effects? plausiblylinkedtoAre the activities theproject (assumptions, risks)? external How didthesituation causingthe problem evolve Was modelsound? theintervention What was thenature andthescope oftheproblem? implementation? been considered planningand intheproject Have issuesof ‘temoignage’ (witnessing)/advocacy itselfappropriate? in linewiththem?Is(our)policy is theproject/program design andimplementation Which (ofour)policiesapplyandto whichextent Evaluation question(example) D.5. D.4. D.3. D.2. D.1. economic andtechnical sustainability. Where appropriate, aimsfor theproject and/or prevention strategies are planned. Where appropriate, disaster preparedness and plannedinadvance. The is thought post-project period about cope withcrises. The strengthens to project people'scapacity The isidentified. purpose project BACK TO TABLE OF CONTENTS › to becontinued onpage65› Link 213 Link 192 Source ALNAP ALNAP ALNAP Table 11 BACK TO TABLE OF CONTENTS ii. Examples ofgeneral indicators › continued from page64› Appropriateness Issue orsub- Issue orsub- Coordination (coherence) Relevance / Relevance criteria criteria ------joint press statements). including level ofmeetings, joint needsassessment, Steps takento improve coherence (e.g. meetings, with othergroups oncooperation/collaborationExistence ofapolicy Views ofstakeholders oncoordination ofcoordinationMonitoring inthefield. structures Degree ofinformation sharing. Existence ofpoliciesfor coordination. channelled. Number ofsources through whichfundsare coordination activities. Percentage fundsspent ofintervention on overincluding authority fundingdecisions. Level provided ofauthority to coordinating body, for staff. accountability ofcoordination inrespect structure ofrolesClarity andresponsibilities and Level oftraining to staff on coordination functions. Level ofstaff released for coordination functions. livelihood strategies.support islikelyto intervention including how external differentiated needsoftheaffected population, fashion,the identifies,clearly inaparticipatory assessment would beconsidered adequate ifit Adequate out. needsassessmentA carried what isacceptable inagiven situation. that theyprovide specific, quantitative indicators of Standards are akeymeasure ofappropriateness in that isequitable, appropriate andaccessible. The state that resources shouldbeprovided isaway Attainment ofSphere Standards. The Standards Is the intervention inlinewithnationalIs theintervention policies? changes intheirneeds(proximity)? to understandtheirsituation aswell aspossible Are we closeenoughto thetarget population to meetthemosturgentprioritised needsfirst? needs? Are choices ourintervention appropriately correspond objectives project Do withidentified population and/oraccording to national policies? perception (expressed needs / demand) of the target appropriateIs theintervention according to the Evaluation question(example) Indicator (example) Link 213 Source Source ALNAP ALNAP Table 12 67 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 68 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs iii. sub-criteria (thislistisnotlimitative). sub-criteria Examples ofspecificevaluation questionsandindicators, linked to corresponding local absorption access, visibility, baseline studies Risk assessment Risk situation, needs other agencies, Security levels,Security (background / Level/quality/ programmatic Entry strategy Entry Coordination, and statistics, (local, EU, US, Specific crisis Specific crisis adequacy of adequacy Coverage by civil-military civil-military Sub-criteria assessment, assessment politics and regional…) impact) of impact) Timeliness priorities, Relevant relations capacity capacity linkages policies EU 3Cs Link 244 ReliefwebSee Link 10 3CswebSee site Link 244 ReliefwebSee Link 243 EURELEX See Link 231 Link 230, DGECHO See Link 229 DGECHO See Comments Link 214 Link 210(Venro) Link 209(Venro) design) -program (WFP Link 73 –targeting) (WFP Link 72 assessment) -needs (WFP Link 71 strategy) -overall (WFP Link 70 (emergency) Link 256 Link 214 Link 208(Venro) Link 76 Link 68 Link 214 Link 77 evaluation questions Links to Link 125(healthpriorities) Link 83(targeting) Link 82(response) (initial assessment) Link 81 (health Link 127 BACK TO TABLE OF CONTENTS Links to indicators coordination) › to becontinued onpage67› Table 13 BACK TO TABLE OF CONTENTS › continued from page66› communication, El Nino, climate children issues) connectedness against natural and mitigation (background / Level/quality/ already learnt Preparedness Local cultural using lessons responds to ademonstrated need. globalisation adequacy of adequacy Exit strategy, perspectives Sub-criteria assessment lessons and acceptance

below), which can be compared to sub-criteria intheabovebelow), whichcanbecompared table. to sub-criteria The Compass also levels (e.g. on gender, impact) of impact) The Compass provides acompatible approach through column criteria' 'quality (left Advocacy, economic presents anumberof'keyprocesses' (rightcolumn), corresponding to eachquality measures changes.. Quality criterion A: criterion Quality The project rules and disasters, Learning media LRRD, WTO, criterion. criterion. These keyprocesses indicators. could beusedasquality Link 228 handbook' 'Exit NRC See See WTO Link 244 ReliefwebSee Comments Link245 CompassQuality Link38 Link 214 A.4. A.3. A.2. A.1. Link 214 evaluation questions identified basic needscanbejustified. The decisionnotto address allofthe needs. The responds project to defined clearly and takeninto account. The origins of people'sneedsare analysed People's needsare identified andmonitored. Links to Links to indicators › to becontinued onpage68› 69 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 70 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs i. Values Examples ofgeneral evaluation questions › continued from page67› (appropriateness) integrated initsinstitutionalcontext Accountability Standards and humanitarian humanitarian Adherence to Issue orsub- Quality criterion H: criterion Quality The is project Quality criterion G: criterion Quality The is project principles / principles Principles criteria quality space in anoptimalmanner. flexible. ------beneficiaries arebeneficiaries saying? Are we listening to from andlearning what were used? Are onhow clearly andhow we well reporting funds succeeding andfailing? Are /donors beneficiaries aware ofwhere we are Can we show that we are livingupto ourvalues? Are we statingwhat clearly we planto do? beenaddressed?conduct Have relevant international standards andcode of technical standards? programme quality, including settingappropriate What systems were putinplace for assuring rights into consideration? How well andhuman didtheytakehumanitarian How consistent were thepoliciesadhered to? population? What istherelationship /proximity to thelocal movement ofourstaff? What are to free therestrictions andsecure intheconflict? other actors Are there any experiences of manipulation by local/ implementation bemonitored independently? Can thesupplyofresources andtheprogram outappropriately?carried Was anindependent needsassessment possibleand the context oftheoperation?' 'Can operational independence bemaintained in Evaluation question(example) H.3. H.2. H.1. G.2. G.1. actors areactors explored andutilised. to cooperateOpportunities withother other interventions. Effective coordination with linkstheproject areActors identified. andtheiractivities changes (reaction). The isadapted project inrelation to context monitored (anticipation). Context changesare anticipated and BACK TO TABLE OF CONTENTS Link 213 Link 192 Source ALNAP WFP Table 14 BACK TO TABLE OF CONTENTS iii. sub-criteria (thislistisnotlimitative). sub-criteria Examples ofspecificevaluation questionsandindicators, linked to corresponding ii. (appropriateness) Accountability principles Humanitarian LawHumanitarian (IHL) Issue orsub- and standards (HAP-I ). DG ECHO'sRegulation Examples ofgeneral indicators Donorship principles Donorship International Human RC CodeRC ofConduct Good Humanitarian Humanitarian Good Principles criteria Level/quality of application of International International Refugee LawRefugee Sub-criteria Rights LawRights - - - - - humanitarian law.humanitarian Correspondence between policiesandinternational and maintained. Cross/NGOas outlinedintheRed Code ofConduct, andproportionality,Degree to whichimpartiality inEHA. stakeholderparticipation facilitating primary onmeansfor themethodsection See intervention. stakeholders, ontheappropriateness ofthe primary Stakeholder views andopinions, includingthoseof or humanrightspolicies. documents, e.g. genderequality, basichumanneeds statements ingovernment anddonorplanning ofpromotion Extent ofgoalscontained inpolicy an evaluation tool innovative example ofhow theCode canbeusedas Code hasbeenfollowed. DEC(2002)provides an to Cross/RedExtent whichtheRed Crescent Link to bibliography - Link 32 evaluation issues. could beconsidered asoutside-looking and6(addressing complaints)(monitoring andreporting) particular,In 5 theoutcome 4(participation), ofprinciples Link to bibliography Link 242 alsoDGECHO’sSee leafleton values andprinciples Link 5 Link to bibliography Link to bibliography Link to bibliography See the 'Good EnoughGuide'for the'Good See accountability Link193 Indicator (example) Comments › to becontinued onpage70› Source ALNAP Table 15 Table 16 71 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 72 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs › continued from page69› on the Rights ofChildren,on theRights Declarations (Convention evaluating agency’s own moral duties (Zakat?) or moral duties(Zakat?) own mandate (relevant Do-No-Harm principles Do-No-Harm values, where relevant (SPHERE, INEE, CIMIC, below), which can be compared to sub-criteria intheabovebelow), whichcanbecompared table. to sub-criteria The Compass also Various religious and The Compass provides acompatible approach through column criteria' 'quality (left presents anumberof'keyprocesses' (rightcolumn), corresponding to eachquality Evaluating agency’s is consistent with the agency's is consistent withtheagency's Quality criterion E: criterion Quality The project Quality criterion F: criterion Quality The project rules to beapplied "No paternalism"? Level/quality of Standards, work Other relevantOther procedures…) application of respects thepopulation.respects mandate andprinciples. Sub-criteria provisions) etc) criterion. criterion. These keyprocesses indicators. could beusedasquality For faith-basedevaluating agencies Link 246 Link to bibliography See UNICEF'sPATHSee Link225 CompassQuality Link38 F.4. F.3. F.2. F.1. E.4. The beingmanipulated oftheproject risk is E.3. makesitspositiononthecrisis The agency E.2. Political andlegalissuesrelating to thecrisis E.1. mandate andprinciples The agency's false expectations. reduce offrustration therisk andofraising measures areNecessary takento remove or population. social andreligious ofthe characteristics takesintothe project account thecultural, involved intheproject. The population isinformed, consulted and show for respect thepopulation. behaviour theyshouldadopt inorder to Teams are aware oftheappropriate identified and, ifpossible, reduced. clear. are analysed. externally. communicated, bothinternally and are definedandare clearly actively Comments BACK TO TABLE OF CONTENTS BACK TO TABLE OF CONTENTS And… The • It • nevnin te "UNICEF the intervention, management. of also “inside-looking”coversissues booklet Thequestions. of remember?” types do,” to with to “what and complete “what M&E), (including staff field for situation copy in your pocket!). This is a step-by-step guide to every aspects of an emergency a useful checklist ofissuesto beremembereda usefulchecklist throughout phases. allproject is Quality always Compass good to consult, Companion Emergency even Book (also available in pocket-size printed form) if it does Field not (ep or w paper own your (keep Handbook” concern directly your sector of 73 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 74 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs F.3. Implementing aprocess evaluation (looking "outside") (looking Focus onoutcome Tool box A-for orissue: eachsector choice): (single ormultiple sectors Main 1, 2orallthree canbeselected): criteria CriteriaGeneric (notto beappliedmechanically: using results) by andfor target NGOs: (for planning andmanaging evaluations, Capacity building/learningenvironment t

&UDy
t

)FBMUI t

'PPE t

4VHHFTUJPOT
BOE
DBWFBUT t

-JOLBHFT
UP
NPOJUPSJOH XJUI
XFC
MJOLT
tool(s), andindicator(s) benchmark(s) t

3FDPNNFOEFE
FYJTUJOHöFMEUFTUFE t

&YBNQMF
PG
FWBMVBUJPO
RVFTUJPO T 7BMVFT t
3FMFWBODF t
t

&òFDUJWFOFTT Standards, types, planning, methodology, management, reporting, t

0UIFS
DPOEVDJWF
UZQFT
PG
TVQQPSU
y t

1FFST t

.FOUPSJOH t

$PBDIJOH t

5SBJOJOH
SFGSFTIFS
DPVSTFTy t

,FZ
JOUFSOBM
GVODUJPOT
JO
QMBDF (Accountability purposes, and/orlearning from levels…) operational to policy - How are we goingto usetheresults? - Why dowe want to doanevaluation? dissemination anduseofresults multiple choice): issues (singleor Cross-cutting Considering bene ts expected De ning evaluationDe ning questions t

&UDy
t

1BSUJDJQBUJPO t

$IJMESFO or combination ofboth Support by donors: Support /(0T
UP
NFBTVSF
JNQBDU
FUD
evaluations by small/mediumsized post evaluations, to balance single t

(MPCBM
KPJOU
UIFNBUJD
QFFST
NFUB FY t

%JTTFNJOBUJPO
PG
SFTVMUT
TUSBUFHJD
VTF t

)FMQ
JO
öOEJOH
FYQFSUT t

$PPSEJOBUJPO
XJUI
EPOPST
$T t

2"2$
HPPE
FOPVHI
PWFSBMM
RVBMJUZ t

'VOEJOH
BT
SFMFWBOU Generic CriterionGeneric Tool box B: BACK TO TABLE OF CONTENTS
&óDJFODZ
2VBMJUZ
PG
QSPDFTT t

4VHHFTUJPOT
BOE
DBWFBUT t
-JOLBHFT
UP
BVEJU t
-JOLBHFT
UP
NPOJUPSJOH t
JOEJDBUPS T
XJUI
XFC
MJOLT

tool(s), and benchmark(s) FYJTUJOHöFMEUFTUFE
3FDPNNFOEFE t
question(s) t

&YBNQMF
PG
FWBMVBUJPO (looking "inside") (looking Focus onprocess Table 17 BACK TO TABLE OF CONTENTS 39 ii. i. Efficiency

Examples ofgeneral indicators See Examples ofgeneral evaluation questions evaluation’). Cost-efficiency Cost-efficiency Issue orsub- Issue orsub- The simple indicator of "unit cost per beneficiary" Thewas not simpleindicatorto be of"unitcostfound amongthe perbeneficiary" examples collected.

Efficiency relevant criteria criteria Quality

guidance 39 ------"Were thingsdoneinthebestway possible?" ------Could we have donemore withthesamebudget? at lower costs? andresultsCould have theactivities beenachieved Were achieved objectives at theleastcost? outputs? What relative are thecosts of inputs(peractivity) to reality? ourpeoplehave toDo makeourplansa theskills improvement? Are we settingandmeetingtargets for process agencies –benchmarks)? How dowe measure upto (othercomparable for ourselves? Are standards we we meetingtheperformance set Are we consistent? How well are we delivering ourservices? goods, transport, etc)?goods, transport, How (e.g. didthelogistics of function timelydelivery How were humanresources managed? to theirmaximumpotential? Were inputsandresources usedappropriately and most efficient way compared toalternative ways? Was theprogramme implemented orproject inthe costs Administration costs asapercentage ofintervention Staff costs, broken down by localand expatriate staff oftransportation type Transportation costs broken down byand sector Costs ofinputslocallyandinternationally Total cost broken oftheintervention down by sector

notes

in Evaluation question Indicator (example)

previous

chapter

(‘implementing

an Link 192 Link 213 Link 213 ALNAP Source Source ALNAP ALNAP

outcome Table 18 Table 19 75 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 76 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs iii. sub-criteria (thislistisnotlimitative). sub-criteria Examples ofspecificevaluation questionsandindicators, linked to corresponding implementation type ofactivity,type or per sector or or persector organisation in General factors General factors Level /quality/ of efficiency in of efficiency and practices adequacy of adequacy general and/ Sub-criteria: Staff policies as relevant Sector and Sector Managing strategies resources regional strategy Internal Internal Human people Link 254 volunteers on humanitarian the'EVHAC'See review Principle 3 PeopleSee Aid – In Principle 2 PeopleSee Aid – In Principle 1 PeopleSee Aid - In - - - - i.a.followingSee tools: Link 28 PeopleSee Aid In a reference Link215 as EU publicservices, Framework (CAF) for Common Assessment methods usedby self-assessment techniques and Total quality tools Link218 USAID recommended by SIDA Link217 ofNGOs,weaknesses of strengths and for assessment OCTAGON, atool Link 216 Networklearning exercise by through aSWOT NGO self-assessment Comments Link28 Link 28 Link 28 Link 78 survey) inquiry, satisfaction Link 204 Link 205 Link 74 Link 69 Links to evaluation questions (appreciative BACK TO TABLE OF CONTENTS Link 203 Link 87 Links to indicators › to becontinued onpage75› Table 20 BACK TO TABLE OF CONTENTS › continued from page74› communication, cost-efficiency, Level /quality/ ability to learn, ability competencies, Accountability activity, where and/or type of and/or type participation / participation principles and principles Health, safety development and selection in protracted effectiveness adequacy of adequacy Sub-criteria: Consultation relevant and Recruitment and security training and feasible, e.g. Accounting procedures Monitoring Monitoring (per sector (per sector Evaluation openness) standards Learning, advocacy advocacy (internal/ external) systems Budget, system (HAP-I) crises) (staff cost- and Link 30 Link 247 guidanceSee notes chapter Link to methodology Principle 7 PeopleSee Aid – In Principle 6 PeopleSee Aid – In Principle 5 PeopleSee Aid – In Principle 4 PeopleSee Aid – In review Link4 DG ECHO'smonitoring To to beconnected "4Es" Link219 management auditand definitionof See Comments Link 28 Link 28 Link 28 Link 28 Link 248 Link 79 Link 206 Link 142 Link 75 Link 207 Links to evaluation questions (WFP) (UNEG) Link 85 Link 86 Link 84 Links to indicators › to becontinued onpage76› 77 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 78 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs › continued from page75› Quality criterion L: criterion Quality uses The agency has theappropriate management below), which can be compared to sub-criteria intheabovebelow), whichcanbecompared table. to sub-criteria The Compass also has the necessary resourceshas thenecessary and The Compass provides acompatible approach through column criteria' 'quality (left makes optimaluseofresources. lessons drawn from experience. presents anumberof'keyprocesses' (rightcolumn), corresponding to eachquality Quality criterion K: K: criterion Quality The agency Quality criterion J: criterion Quality The agency Quality criterion I: criterion Quality The agency criterion. criterion. These keyprocesses indicators. could beusedasquality expertise. capacity. CompassQuality Link38 I.4. I.3. I.2. I.1. J.6. J.5. J.4. J.3. J.2. J.1. K.3. K.2. K.1. L.3. L.2. L.1. resources. The iscompatible project withavailable to eachphase. An appropriate amount oftimeisallocated expertise. haveproject thenecessary Staff andotherpeopleinvolved inthe estimated correctly. andavailable resourcesNecessary are taken into account andmonitored. The faced risks by your team are identified, monitored. are identified, takeninto account and The equipment affectingproject risks management iseffective andtransparent. Administrative, financialandlogistics processing information are appropriate. The methodsusedfor and collecting toproject runsmoothly. teamGood management enablesthe responsibilities are defined. clearly linesanddecision-making Reporting rationally. Available resources are mobilisedandused Project coverage isoptimal. The chosen strategy ensures optimal impact. experience. useslessons learntfrom The agency lessonsfrom learns experience.The agency over thecourse oftheproject. recordsThe agency relevant information BACK TO TABLE OF CONTENTS BACK TO TABLE OF CONTENTS G. Capacity andsupport (looking "outside") (looking Focus onoutcome Tool box A-for orissue: eachsector choice): (single ormultiple sectors Main 1, 2orallthree canbeselected): criteria CriteriaGeneric (notto beappliedmechanically: using results) by andfor target NGOs: (for planningandmanaging evaluations, Capacity building/learningenvironment t

&UDy
t

)FBMUI t

'PPE t

4VHHFTUJPOT
BOE
DBWFBUT t

-JOLBHFT
UP
NPOJUPSJOH XJUI
XFC
MJOLT
tool(s), andindicator(s) benchmark(s) t

3FDPNNFOEFE
FYJTUJOHöFMEUFTUFE t

&YBNQMF
PG
FWBMVBUJPO
RVFTUJPO T 7BMVFT t
3FMFWBODF t
t

&òFDUJWFOFTT Standards, types, planning, methodology, management, reporting, t

0UIFS
DPOEVDJWF
UZQFT
PG
TVQQPSU
y t

1FFST t

.FOUPSJOH t

$PBDIJOH t

5SBJOJOH
SFGSFTIFS
DPVSTFTy t

,FZ
JOUFSOBM
GVODUJPOT
JO
QMBDF (Accountability purposes, and/orlearning from levels…) operational to policy 'Are therebetterways ofdoingit?' And "Whatcanwedo aboutit?" - How are we going to use the results? - Why dowe want to doanevaluation? dissemination anduseofresults multiple choice): issues (singleor Cross-cutting Considering bene ts expected De ning evaluationDe ning questions t

&UDy
t

1BSUJDJQBUJPO t

$IJMESFO or combination ofboth Support by donors: Support /(0T
UP
NFBTVSF
JNQBDU
FUD
evaluations by small/mediumsized post evaluations, to balance single t

(MPCBM
KPJOU
UIFNBUJD
QFFST
NFUB FY t

%JTTFNJOBUJPO
PG
SFTVMUT
TUSBUFHJD
VTF t

)FMQ
JO
öOEJOH
FYQFSUT t

$PPSEJOBUJPO
XJUI
EPOPST
$T t

2"2$
HPPE
FOPVHI
PWFSBMM
RVBMJUZ t

'VOEJOH
BT
SFMFWBOU Generic CriterionGeneric Tool box B: (UNICEF)
&óDJFODZ
2VBMJUZ
PG
QSPDFTT t

4VHHFTUJPOT
BOE
DBWFBUT t
-JOLBHFT
UP
BVEJU t
-JOLBHFT
UP
NPOJUPSJOH t
JOEJDBUPS T
XJUI
XFC
MJOLT

tool(s), and benchmark(s) FYJTUJOHöFMEUFTUFE
3FDPNNFOEFE t
question(s) t

&YBNQMF
PG
FWBMVBUJPO (looking "inside") (looking Focus onprocess Table 21 79 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 80 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs G.1. provide appropriate guidance andsupport. within and agencies expertise themselves, and a range of tasks recommended for donors to This environment could be divided in two main components: internal management support of the findings and implications from arising of evaluation the spectrum activities use and dissemination appropriate identification, maximum forproviding in and tools, and aid interested agencies in being able to most effectively apply the recommended approaches Thus the need was identified for a Capacity Ideally, this person should be at the management level, to provide some ‘clout’ to encourage withotheragencies involvedcontacts inevaluation, etc. events, of and conferences evaluation development training, in the participation through on e.g. skills, priority evaluation place to appropriate be would it not, If evaluation. with full time, or just of part other functions. Preferably this person should have some experience point for evaluation at the HQ level. Depending upon the size of the focal a identify organisation,to is practice, good and standards both step,consistentwith this important One could be of evaluation capacity. can be taken will vary of course, given the size and nature of the agency, and its current level that steps What beneficiaries. assisting in difference a making indeed are efforts evaluation, their that for capacity own their enhance to recognising that this is appropriatean essential tool for improvement, as and also to be able steps to demonstrate take should NGOs plan, the course ofthepreparation ofthisGuide. during consultations numerous the in emphasised was Thisout. carried been have that fromevaluationslearnt lessons of use limitedcould, forthe and otherwise it as useful as commonreasonsmost forevaluation failurethe of the of tobe one as been has capacity effectiveness to create a culture of evaluation with more understanding of its potential to improve the sector. While Guides such as this one can help, we are under no illusions that it is sufficient Internal capacity Internal in 2007 one of their projects fundedbyin 2007oneoftheirprojects DGECHO.) management is a weak point". (D.B., external contractedexpert by an NGO to evaluate including processes, formative access to ability skills, language foreign no or limited is have often staff, who that national on rely staff, who However,NGOs issue. smaller an such international not of pool larger to access have who NGOs large For key. is capacity staff think points…I at weak and limited onhuman/other somewhat is capacity NGOs' ICRC, scarce are and field or UN the with the compared example,For mission. the prepare adequately to resources in capacity limited have NGOs "Often implement, building challengeisto"The linkwhat you learnwithwhat you do" of humanitarian and – strengthening use evaluation conducive environment providing support and assistance to interventions. the – represents understanding Indeed, a key challenge and the lack skills BACK TO TABLE OF CONTENTS of about within relevant (SPHERE) how the humanitarian management to effectively Link 255. BACK TO TABLE OF CONTENTS very specialised. very These are actors notmentioned inthisGuide. be can others techniques), or skills evaluation particular regarding (e.g. or nature in general organisations “commercial” of numbers increasing are providingatinitiativesseminars, aim generallyforand fee.thattraining a are these Manyof there that observe also We evaluation and monitoring Link251. activities i.a. covers which punto.sud by ‘Helpdesk’ a of up setting the Aid’sPeopleCompass or In Quality towards efforts human also resources but management, e.g. improving to of contribution number their significant practices field and management, networking, a support to to years stakeholders, recent and in actors grants humanitarian provided has ECHO DG addition, In based discussiongroups Link. Link 38 – can also be found in the bibliography ALNAP by organised those e.g. various as –such evaluation to the dedicated schemes in mentioned been have tools chapters of and this Guide, in particular in the initiatives sub-criteria tables. Links with some organisations,regular training of number A toolsSelf-development - Who can help? senior management. to presented are evaluations to response in steps action that managementrequire others knowledge and function, a have organisations some example, For done. is this which in out carry to processare forlearnings translated thatthe ensuring into action. Therewaysvarious be can point little is there Guide, way.some in used this not is it if evaluationof formhaverequiresorganisation toThis some the throughout emphasised have we as Also, that could alsobeofassistance. below i.a. listed as organisations, specialised third-parties external, and tools development of development the self- various also activities. are evaluation There specific with in as well capacity,as evaluation below) (see help could Donors etc. guidance updates, exercises, practical peers, with mentoring,cross-fertilisationregular or coaching with needed-, if term long the -on conducive environment”,and comprehensiveeffort a “learning the support to a first step in a process; it is a (learning) activity, not an outcome or a result. There is a need of training a organising of solution course is hardly sufficient. As the usual LFA perspective clearly emphasises, trainingthe in itself is only that stressed be must It ways. of number a in The development and support of evaluation capacity within the organisation can be assisted in themethodologychapter Link. are described evaluation function or evaluation manager the of Thespecific responsibilities prove may that shouldbekeptinplace forfunction at least36months, ifpossible. turnover rapid from evaluation the to assigned protected person the that recommend practices be good Some detrimental. should position the again, Ideally the intheagency’sposition shouldbeacknowledged internal procedures. of independence The organisation. the within evaluation of use facilitate to also and operations, field with maintained is distance necessary that ensure to practice, evaluation . The list includes for example URD’sexample for includes 250. list Link The Link, together with a list of relevant internet- or URD or 39 Link 81 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 82 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 40 G.2. oiy n pormai apoce, niiul tde ae ee ulig blocks building mere are studies individual approaches, programmatic and policy various appropriatenessof the identifying to comes it when recognisedthat widely is it But useful guidance to the concernedagency about extremely how it may be able provideto improve its own operations.can discussed, has Guide this as evaluations, agency Individual which approaches are mosteffective ornotin given circumstances, andthe reasons for this. major donors to focusupon on incumbent the big picture.is it This includes guidance, identifying overalloverall lessons providing learnt about to addition In on. take to donors for role evaluation important an leaves still this interventions, own their of evaluation increasing in engage progressively will agencies aid humanitarian individual that hoped be to is it While will need to undertake thefollowing ofevaluativewill needto types undertake activities. contractors) or agencies support to functions certain out contracting and/or appropriate, as accountabilityand planning needs, experts assistancethe (with donors major external of own their for as Consequently, interventions. well as of types and different sources, setting, from needed is information evaluative from generalised be can that learnt lessons identify ‘Big picture’ Support for evaluation activity anduseby for donors evaluation activity Support Transaction Publishing. This isalsoathemeinALNAP’s ReviewofHumanitarianActions. Link253 E.g. see Ray C. Rist and Nicoletta Stame, eds. (2006). eds. Stame, Nicoletta and Rist C. Ray see E.g. • Undertaking • past actions. past actions. Thus implicationsfrom arising syntheses should: from learned been has what upon acting and learning insufficiently for criticised been recently has sector humanitarian the that given especially use, to lead will Therepoint,little is however, informationthis unless activities such out carrying in improvements. for recommendations making and themselves evaluations the of quality the assessing of means a as reviewers, peer using potentially meta- evaluation, occasional the out carry to appropriate be also may It learnt. lessons practices and good ways, alternative at looks results?”, which the achieving of ways Synthetic data. question: UNEG “Are3rd better the there to responding fragmented in donors aid can reviews of consolidation include can activities of in category undertaken, This recommendations. been and practices, good have learnt, lessons that key capture evaluations to order of syntheses annual out carry To (GHD) Donorship by Humanitarian usingtheGood platform. LRRD) with development donors. Joint evaluations on values could e.g. be launched at (e.g.assessments looking impact donors, humanitarian other e.g.with together some In themselves, site.evaluations joint some undertake to evaluations.wish may joint They single a to limited not undertake to agencies support toencourageto and wish situations,might donors responses varying across look can that evaluation of forms other and evaluations, meta evaluations, cluster evaluations, focusing andcoordination role 2) 1) rarely are adequate technical for reports thispurpose. and audience, intended the reach to media appropriate of use make must dissemination that (ALNAP,recognising networks but global – etc) UN/IASC be used externally, by disseminating them to their and partners through key be usedinternally incontributing to definingdonors’ annualstrategies; (independently, or jointly From Studies to Streams: Managing Evaluative Systems. Evaluative Managing Streams: to Studies From with others) global BACK TO TABLE OF CONTENTS evaluations, thematic 40 To . BACK TO TABLE OF CONTENTS 41 that major donors provide leadership and support ofvariousforms.that majordonorsprovide leadershipandsupport in a number of situations. Thus in order to lead to a broad culture of evaluation, it is important However, we recognise that given and resource capacity limitations, this own.will be challenging their on evaluations undertake can NGOs how about ideas provided has Guide This assuranceQuality /control, to agenciesfor tangible andintangible evaluation support questions (Why anevaluation?basic How to usetheresults?) presentedquestions (Why proposal. withtheproject key the on based approved be should plans evaluation Instead, later). until apparent arenot addressed through evaluationbe often could that questions operationand the about a tospecify TORfortentative evaluation, the except generalformvery and (e.g.a in questions able be to intervention humanitarian the for application of time the prematureat usually is it recognisedthat be should it evaluation, of funding the to approach Taking results-oriented a outcomes.as such questions important more at look to processand of considerationbeyond as a (small) share of the total intervention budget, it will be difficult or impossible forfunds that NGOshave tobeen get devoted to evaluation. Further, if the budget for evaluation is defined only unskilled having evaluators and of results of risk questionable validity a and little be use, thus can representing a there waste evaluation,of those for budget small overly an With budget. meaningful evaluation, rather than based exclusively as a percentage of the overall programme be made available by donors, based upon a realistic budget to address the requirements for a should funding Adequate required. be also will support funding appropriate Tangibleand quality of levelappropriate an reach to NGOs some for time take will it that and time first the perfect rarely is it though even try, who those reward and As part of efforts in promoting a results oriented culture of evaluation, donors should reinforce negative experiences, and how they can use this information to improve future activities. activities. future improve to information this use can they how and experiences, negative from even learned, been has what identifying agencies upon contingent is above the sure, worse. be or To – activities evaluation future undertake to disincentive a as act can findings evaluation negative as perceived be can what for agencies punishing Further, innovation. clear that “failures are accepted”, recognising that punishing failure is equivalent to punishing and thefear ofexposing failure improvements.” have theway to beenblocking further report also indicated that “an overly critical approach can kill [an outcome approach]”and that “risk avoidance Advice from Governments around the World. Available at: 2006 Perrin. (Burt Bank. World the by Outcomes”sponsored to This for example is a major theme in a recent report arising from a Roundtable discussion: “Moving from Outputs Training • To • o FA PM n LA ht a ivle aig pstv apoc t hl in help to approach positive a taking developing consensus. involve can that LFA and PCM FPA, for techniques Inquiry Appreciative as such approaches development organisational funding onevaluations ofmutualinterest). joint (e.g. together work to able be may donors various which in ways identify to positively, more and conflict, even or duplication unnecessary avoid to order in forchannel streamliningrequirements3Cs the regarding e.g. TORs reporting), and bilateralparticular European donors (that might provide to opportunities consider coordinate of donors’ and

own standardise staff (HQ, evaluation field) www.worldbank.org/oed/outcomesroundtable regarding requirements Moving from Outputs to Outcomes: Practical Outcomes: to Outputs from Moving 41 . A corollary to this is making it making is this to corollary A . evaluation, with as other well donors, as related .) This in 83 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 84 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs - could propose suchas: anumberofrelevant services A dedicated website on the CAP model - although of course with a somewhat different focus teams whohavecountry to prepare CAPs at fieldlevel Link252. Section that provides central support – in the form of workshops and training- to the various Appeal) (Consolidated CAP UN-OCHA the in found be could practice good of example An quality assuranceor specificquality control /quality issues, suchasfor example: An independentorganisation support may be able to assist in matters to pertaining general etc.mentoring andcoaching support, NGOs; within functions evaluation appointed newly to process evaluation the of at step any guidance specific as well providing as to: questions limited answering encouragement; be and advice not general but include, can This assistance. advisory relevant provide can that organisation(s) independent several or one contract or support to donors major for appropriate be would it Consequently, source. neutral a from guidance and support require will evaluation in capacity or experience limited with NGOs Many interactive • assisting • a • organising • links • maintaining • ensuring • assuring • providing • executive ontheweb summaries site; for buildingandtraining purposes; capacity discussed. of any ofthenamesondatabase; endorsement no of clear,frameworkis there it the course,that of making 13’), ‘Lot and/or links to some Commission-managed database of possible evaluators (e.g. in of goodpractice, etc; and update relevant sections of this Guide, suggest the inclusion of new tools, examples independence, incaseofe.g. negative feedback aboutanagency. evaluation, ortheevaluators; raised eitherby theagency agencies might findeasier to implement internal process reviews); and some and results/outcomes, about learning in interested more be might donors expert reading mechanismssuchastheUNHCR’sexpert Linkincaseofcontention; be achieved through defining minimum standards, by coaching, and by convening database to existing that in that arbitration relations of a ensuring a the specialised discussion proper participation databases “good with balance transparency in enough” concerned coaches, case or forum sites of between of

quality beneficiaries of where issues facilitators, vacancy agencies and of outcome TOR accountability issues concerning announcement who and mentors, is of carried reporting would and common BACK TO TABLE OF CONTENTS the process supporting by be out independence is ready (e.g. e.g. maintained. with interest evaluations Reliefweb, posting to individualised organisations the contribute could required reports/ This of RedR), (some can the be to BACK TO TABLE OF CONTENTS ANNEXES 85 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs

BACK TO TABLE OF CONTENTS -B- -A- ANNEX A-GLOSSARY directly orindirectly,directly from thedevelopment (OECD/DAC) intervention. : Beneficiaries the individuals, groups, or organizations, whether targeted or not, that benefit, be assessed. (OECD/DAC) can achievements or performance which against standard or point reference : Benchmark against whichchangecanbeassessedorcomparisons made(WFP). Baseline study: the analysis and description of the situation prior to the start of an operation, to setoutinadvance. andcriteria norms (ALNAP) proceduresconformand activities extent what to and whether of determination the Audit: Power of (The Appreciative inquiry)…” appreciation. Positive questionsleadto positive change” and affirmation in grounded inquiry, of process relational a is best, its at change, and organising human that suggests best…(i) their at function they Appreciative is inquiry the study and exploration of what gives life to human systems when development to prior intervention adecisionoffunding. (OECD/DAC) Appraisal: an overall assessment of the relevance, feasibility and potential sustainability of a involves account of, taking andaccounting (HAP-I) to disaster survivors Accountability is the means by which power is used responsibly. A • The • IIA Inc,1975) terms’,n°19, Auditing Internal Report CurrentResearch Selected of ‘AnEvaluation J.E.: (Smith, solutions. recommends and problems identifies report audit resultant objectives…The organisational the to related as decisions management the and the of policies operational the some are evidence of sources primary are evidence.The of sources data Financial development. employee and objectives, social performance of the management function, achievements of programme the the purposes, in improvements increasing through objectives and organisational other profitability the of attainment organisational improving of purposes the for (sic) of of the all activities levels byof management the internalperformed auditor Auditing,(Internal A.D. Chambers, G. G.M.Selim, Vinten, Pitman Publishing 1987) : doingright(beingsociallyresponsibleEquity inasocialauditperspective) Economy: doingthingscheap : doingthingstherightway Efficiency Effectiveness: doingtherightthings Management Audit is a future-oriented, independent, and systematic evaluation “Four E”s inaudit: Humanitarian accountability 87 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 88 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs -E- -C- of theenergy isconverted into Koestler). hot air (Arthur per cent cent. per ninety-five five remaining around The efficiency an has symposium good compared to the energy put in. A good machine can have of per ninety an cent.efficiency A a machine of out get term you The energy of amount ‘efficiency’ the physicist the to means But also… see whetherthemostefficient approach hasbeenused. aresultof as achieved inputs. This − generallyrequires alternativecomparing tooutput, approaches an toachieving and quantitative − qualitative the outputs measures Efficiency : Ameasure ofhowEfficiency economical inputsare converted to outputs. effectiveness. (OECD/DAC) of criteria the under considered be should it importance, its given that suggests Guidance effectiveness is effectiveness of criterion the within Implicit outputs. orwhether the of basis the on happen to its purpose, expected be can this achieves an activity towhich extent the measures Effectiveness be achieved, into account taking theirrelative importance. Effectiveness: operation’sThetothe extent which were objectives achieved, to expected or need anddevoid politicalagendas)(OECD/DAC). ofextraneous their to proportionate protection and assistance with them (providing are they wherever suffering life-threatening facing groups population major reach to need the is Coverage impactin greater had a have would purpose.’achieving theproject that (OECD/DAC) produced been have could outputs different whether to outputs, into converted were inputs how beyond Cost-effectiveness‘… looks take thelonger-term into account (OECD/DAC). need still They sustainable. assessing, however, be in regard to whether, to in responding to acute designedand immediate needs, they not are projects, development to contrast in interventions, humanitarian many … withdrawn. been has to funding likely donor is after impact continue an or activity an whether measuring with concerned is Sustainability are carried out in a context that takes longer term and interconnected problems into account. Connectedness refers to the need to ensure that activities of a short-term emergency nature fromand instability those, from whicharise natural disasters. (ALNAP) conflict from arising needs humanitarian differentiate to used generally is term needs.The authorities, the abuse of human rights and possibly armed host conflict,of that legitimacy createsdisputed humanitarian the structures, state of breakdown the involves which origins, economic and political social, complex, with situation a emergencies : political Complex take into account andhumanrightsconsiderations humanitarian (OECD/DAC). as humanitarian policies, to ensure that there is consistency and, in particular, that all policies well as policies military and security,developmental,trade assess to need the Coherenceis . Although timeliness. is not a formal criterion, the OECD/DAC the criterion, formal a not is coordination BACK TO GLOSSARY INDEX BACK TO GLOSSARY INDEX sustainability (WFP). sustainability and determine impact effectiveness, to efficiency, as is well as aim objectives, of Thefulfilment and results. relevance the and implementation design, its completed policy, or or programme on-going an of assessment objective operation, and systematic the is Evaluation the or something of value ofthatproduct process’ (Scriven1991) or worth, merit, the determining of process ‘The Evaluation: • • • • • • • • • • • • • management practices, and the linkages between these.management practices, (OECD/DAC) and thelinkages organisations, i.e. their instruments,policy mechanisms,their delivery service their primary implementing the the of dynamics internal the of evaluation An Process(evaluation) by, or with, out stakeholders, beneficiaries. usuallytheproject (OECD/DAC) carried Evaluation (evaluation) Participatory of implementation ofanoperation (WFP). period the of middle the towards performed evaluation an Evaluation : Mid-term (OECD/DAC) evaluators. the of performance the assess and/or quality an its of judge evaluation to the evaluation denote to used be also can It evaluations. of series a from aggregatefindings to designed evaluationsfor used evaluationis : Meta term The same area (adapted from OECD/DAC) the in organisationsworking more or two by jointly (evaluation)Conducted Joint or partner, donor, implementing organization. (OECD/DAC) the of management the to reporting individuals and/or a unit by conducted intervention development a of Evaluation evaluation: Internal the of implementation and design development (OECD/DAC) intervention. the for responsible those of control the of free persons and entities by out carried evaluation An evaluation: Independent theimplementation during conducted ofanoperation.often most performance, improve to intended evaluation The evaluation: Formative organizations. (OECD/DAC) implementing and donor the outside individuals and/or entities by conducted intervention development a of evaluation: evaluation External The (WFP). and impacts, and to draw conclusions that results mayof sustainability inform the other assess comparable to failure,operations or success of factors the understand to is intention The completion. after long or after directly undertaken be may It completed. been has it after operation an of evaluation the evaluation: Ex-post development (OECD/DAC) intervention. a of implementation before performed is that evaluation an evaluation : Ex-ante enad/or project programs. (OECD/DAC) activities, related of set a of evaluation an evaluation: Cluster enhance accountability’ (ALNAP) and practice and improvepolicy to lessons draw to intended action humanitarian of examination impartial and systematic ‘A: Action Humanitarian of Evaluation 89 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 90 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs -F- -I- -H- − onindividuals, communities, gender- andage-groups, andinstitutions. (OECD/DAC) looks Impact at the wider effects of the project − social, economic, technical, environmental and natural hazards. (ALNAP) an on emergencies political complex undertaken from resulting needs human actions to response in basis advocacy impartial and protection Assistance, : Action Humanitarian assistance andprotection. (ALNAP) Humanitarian system : the group of organisations involved in the provision of humanitarian : A finding uses evidence from one or more evaluations to allow for a factual statement. (OECD/DAC) factual a for allow to evaluations more or one from evidence uses finding A Finding: means to measure achievement orto the changes reflect to an operation.connected reliable and simple a provides that variable or factor qualitative or quantitative Indicator: Impact • longterm, etc… orindirect,happenonthevery canalsobedirect Impact • Impacts • • • • • • • micro (household). and sectors. (OECD/DAC) regions, countries, across cuts that priority development specific a address which Thematic evaluation: Evaluation of a selection of development interventions, all of value oftheoperation. the about information provide to intended is evaluation were summative resultsA produced. planned the which operation to extent an the determine of to thereof) end phase a the (or at conducted evaluation An evaluation: Summative ofanoperation.with thedesign anddelivery entrustedare who those by managed and planned evaluation an : Self-evaluation to review facilitator. lessonslearned, usinganexternal perhaps (OECD/DAC) Self (evaluation) Short one or two-day exercise managed from within the operation into anongoingoperation to evaluate events astheyunfold. (OECD/DAC) consultant or member staff a of deployment the Involves (evaluation) time Real specific global, regional, country, development orsector objectives. (OECD/DAC) P behavioural change in, and impact on, beneficiaries and their welfare systems, welfare institutions orprocesses ofconcern (UNHCRPractical Guide, March 2001). their and beneficiaries of on, signs impact the and are in, change indicators behavioural impact level, objective the at Set indicator. Impact 8, annexes, – IV Part ECHO. aid. humanitarian Glossary) of Evaluation the for (Manual and (consequences).longer-term impact (results), impact initial (achievements), impact operational between drawn be should distinction A unexpected. or expected and negative, or positive be can Eauto o a e o itretos mrhle t attain to marshalled interventions, of set a of Evaluation evaluation : rogram is can a general be intended term or to unintended, describe the positive effects and of an negative, BACK TO GLOSSARY INDEX intervention. macro (sector) The impact and BACK TO GLOSSARY INDEX -Q- -P- -O- -M- beneficiary. Quality can be measured in terms of efficiency, effectiveness, outcomeand effectiveness, efficiency, (HAP-I). of impact terms in be measured can Quality beneficiary. intended the or consumer the of needs implied or stated the satisfying of purpose the for fit it makes that service or product a of characteristics featuresand of totality the is Quality with stated goalsorplans. (OECD/DAC) operates according to specific criteria/standards/ guidelines or achieves results in accordancepartner development a or intervention development a which to degree PerformanceThe : development a from the achievement ofoutcomes. (OECD/DAC) result which services and intervention; may also include changes goods resulting from the intervention which are relevant to capital products, The : Output outputs. (OECD/DAC) intervention’s an of effects medium-term and short-term achieved or likely The Outcome: to evaluation. complimentary is Monitoring management. project for staff project by out carried normally is It progress. -Monitoring describes the continuous collection and analysis of information to assess project extent ofprogress andachievement ofresults intheuseofallocated fundsandaid. (WFP) indicators to inform management and the main stakeholders of an ongoing operation of the specified on data of systematic the uses thatcollection continuinga function is Monitoring annexes, 8,Glossary) – IV Part ECHO. needs aid. humanitarian of humanitarian Evaluation the for its (Manual operation). or meet plan (global population target a help to taken measures Intervention: (OECD/DAC) : Inputs The financial, human, and material resources used for the development intervention. organisation’sprocesses.managementA quality the of implementation and integration effective the ensure to needed documents other and manual, quality objectives, quality policy, quality documented a comprises It customers. its of expectations the in meeting an organisation of efficiency and effectiveness the improve A n etn te seta nes ad epcig h dgiy o dsse srios (HAP-I) survivors disaster of dignity, the respecting and needs, essential the meeting in is a designated set of processes that enable continual improvement in an agency’s performance is a set of coordinated processes undertaken to continually to undertaken processes coordinated of set a is system management quality • Guide, March 2001). indicators should be disaggregated by sex and age, Performance if appropriate timeframe. (UNHCR Practical foreseen a within revisited readily and qualified and/or quantified be can that outputs planned of achievement towardsthe performance of measures are indicators these level, output the at Set indicator. Performance humanitarian quality management systemmanagement quality humanitarian 91 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 92 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs -R- -T- -S- priorities priorities (as well as donor policy). and needs local with line in is project the whether assessing with concerned is Relevance Relevance /Appropriateness (OECD/DAC) needs,needs, organisational country priorities, andpartners’ anddonors’ policies. Relevance: The extent to which the objectives of an operation are consistent with beneficiaries’ The “Three Cs” intheEU’s Maastricht Treaty mandate”. (OECD/DAC) “evaluationand work” of “scopeare meaning same the with used sometimes other expressions Two requirements. reporting and allocated, time and resources to the are conducted, analyses or be methods assessed be the to is evaluation, performance which the against standard of the used, be scope to and purpose the presenting document Written anevaluation.conducting for parameters and requirements the define reference of terms the Reference: of Terms interest inthedevelopment oritsevaluation. intervention (OECD/DAC) indirect or direct a have who individuals or groups organisations, Agencies, Stakeholders: (OECD/DAC) risks. identified for impacts expected and consequences; probabilities the of quantification undesirable of process the such regarding information provide to process systematic a interventions; development by posed environment the or property, an intervention’s of health, life, human to achievement the successful consequences negative and unwantedaffect potential the of examination to detailed A objectives. are likely which or affecting Risk hoc ad basis (WFP). an on or periodically operation, an of performance the of assessment an Review: to localneeds, increasing ownership, accountability, andcost-effectiveness accordingly. : An analysis or an assessment of factors (called assumptions in the logframe) the in assumptions (called factors of assessment an or analysis An analysis: • • • a fr s osbe plce ad gnis should coherence. agencies and in resulting ideally policies and coordination, of part possible, as efforts, other’s each as complement far as : policy of Complementarity work amongthemselves, withoutconflict. coordinatetheir Commission and Coordinationthe states Treaty: EU thatrequires be coherent withthoseinotherpolicies. also should Treatythe of objectives development the them; to contribute should ideally but objectives, development reaching of possibilities the on infringe not Policycoherence covering differenta policy a : area (i.e. should non-development) Appropriateness is the tailoring of humanitarian activities BACK TO GLOSSARY INDEX BACK TO TABLE OF CONTENTS GENERAL REFERENCES GENERAL ANNEX B-BIBLIOGRAPHY - ACCOUNTABILITY andPARTICIPATION issuesbelow) (seealsocross-cutting ------ALNAP - Caroline &Burall, Simon–One World Trust 2003,London PowerGlobal Accountability Report: withoutAccountability? Kovach; Hetty, Neligan, http://www.undp.org/eo/documents/who.htm Who Evaluation UNDP1997, Are -AParticipatory Handbook, theQuestion-makers? 2006,Care Corps,Oct Intl, Mercy CRS,IRC, Oxfam GB, Save theChildren US, WVI ”Good Enough” Measurement andAccountability Guidefor Impact inEmergencies, draft http://www.oneworldtrust.org/?display=ngoinitiatives One World Trust –NGOAccountability Initiatives index.htm StandardsBOND Quality andAccountability http://bond.org.uk/futures/standards/ http://www.alnap.org/publications/eha_dac/pdfs/eha_2006.pdf Evaluating Using Action Humanitarian theOECD-DAC Criteria, ALNAP2006 Buchanan-Smith, J. Telford An (EHA), introduction to evaluationALNAP&Channelresearch, action ofhumanitarian M. http://www.alnap.org/pdfs/joint_evaluation_meeting_report.pdf June 2006 onJoint Evaluations,dtable Meeting Roun Sponsored by ALNAP, UKDEC,ECBand TEC, ODI http://www.alnap.org/publications/RHA2005/RHA05KMS_English.pdf Evaluation Action: ofHumanitarian ALNAP Review utilisation, December 2006 http://www.alnap.org/publications/eha/index.htm edited by Adrian Wood, publishedby Zed Press, andJohnBorton, Apthorpe Raymond 2001 fromEvaluating Reflections Action: InternationalHumanitarian Practitioners and Evaluation, 1996-http://www.usaid.gov/pubs/usaid_eval/pdf_docs/pnabs539.pdf Evaluation, aParticipatory Conducting TIPS, USAIDCentre for development Information see http://www.peopleinaid.org/pool/files/code/code-en.pdf ofaidpersonnel, haspublished its management andsupport “Code Practice”, ofGood People Aid (www.peopleinaid.or In http://www.oneworldtrust.org/?display=publications 06 04 Dec 2006 GlobalAccountability Repor http://www.hapinternational.org/pdf_word/m466-HAP%202007%20Standard%20lite.pdf HAP 2007Standard Accountability inHumanitarian Management, January andQuality ‘07, HAP-I Accountability (Humanitarian www.hapinternational.or Partnership-International), http://www.odi.org.uk/ALNAP/resources/Downloads/CourseManual.doc , 2007 t Monica Blagescu and Robert Lloyd,, byOne BlagescuandRobert Monica World Trust g), anorganisation promoting inthe goodpractice g

93 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 94 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs - AUDITING ------APPRECIATIVE INQUIRY - - MONITORING DG ECHOandEC - - Int Draft GuidelinesforDevelopmentDraft ofIndicators, theDefinition EC 2002 February Use ofIndicatorsinLocal HealthEvaluation andPlanning, DrCapdegelle, March 2002 Catalogue ofPerformance DGECHO, Indicators (draft), March 2004 http://ec.europa.eu/comm/europeaid/evaluation/intro_pages/methods.htm DG EuropeAid (AIDCO) evaluation guidelines… A ECHO Strategy NeedsAssessment 2005:Global humanitarian (GNA) andForgotten Crisis Commission –AGuide, DGECHO2002(printed version only) FundedEvaluating action Humanitarian ofthe by AidEuropean Office theHumanitarian Catsambass. Publicat Sage Reframing Evaluation Through Appreciative Inquiry. HalliePreskill and Tessie Tzavaras Second InternationalConference 2004 Sep (report), onAppreciative Inquiry The Power –D. ofAppreciative Inquiry Whitney, A. Trosten-Bloom, BKPublishers 2003 LakeshoreB.S. Communications, Kaplin, 2002 ofPositiveEncyclopedia –Vol. Questions 1,D. Whitney, D. Cooperrider, A. Triosten-Bloom, 2001 The First InternationalConference Sep-Oct (report), onAppreciative Inquiry IISD 1999 Locating theEnergy for to Change:AnAppreciative Introduction inquiry, Elliott, Charles The Thin ofAppreciative Book Inquiry, SueAnnis Hammond, TB Publishing, 1998 http://www.odi.org.uk/ALNAP/resources/modules/m3_handouts.doc 3:ManagingandFacilitatingModule Evaluations July2003 Action, ofHumanitarian http://www.odi.org.uk/ALNAP/resources/modules/m2_handouts.doc 2:EvaluationModule – Action ofHumanitarian The Evaluator’s role, July2003 http://www.odi.org.uk/ALNAP/resources/modules/m1_handouts.doc to Evaluation 1:Introduction Module July2003 Action, ofHumanitarian ALNAP Training for Modules Evaluation Action ofHumanitarian end of2007) for Aid,A Methodology ofHumanitarian the Monitoring DGECHO(to becompleted by the Boer,the EC,M.ZabelandS.De EuropeAid, July2005(draft) AssistanceHandbook for ofExternal financed Monitoring Monitors,by Results-oriented Field (ECHO 6),May 1998 Audit –Procedures Manual for theAudit assessment andEvaluation ofPartners, DGECHO ssessment (FCA); Methodologicalnotes. Links229,230,231 ernal Auditing,ernal 2ndedition,Chambers-Selim-Vinten, Pitman 1987 Audit Manual, Audit Questionnaire andcomments, DGECHO ions, 2006. BACK TO TABLE OF CONTENTS BACK TO TABLE OF CONTENTS - OECD/DAC

- - SPHERE ODI/HPN www.odihpn.or OCHA (seealsobelow)

------”Pri Networ Itemid,Array/gid,144/lang,English/ http://www.sphereproject.org/component/option,com_docman/task,doc_details/ Fostering Aid Agencies Q&AamongHumanitarian through aprocess ofAccreditation Itemid,203/lang,English/ http://www.sphereproject.org/component/option,com_docman/task,cat_view/gid,17/ Standards andMinimum inDisaster Response, 2004(Handbook) Charter Humanitarian Section OCHA CAP http://ochaonline.un.org/esu Hofmann aid:areview 17 ofhumanitarian ofcurrent theimpact practice,Measuring HPGReport March 2007 http://ochaonline.un.org/humanitarianappeal/webpage.asp?Nav=_training_en&Site=_training&Lang=en http://www.oecd.org/dataoecd/34/21/37854171.pdf DAC Evaluation Standards Quality (for test phaseapplication),2006 Dec http://puck.sourceoecd.org/vl=642079/cl=34/nw=1/rpsv/~6669/v2001n26/s1/p1l Accountability (Complete Edition) Evaluation andAid Effectiveness No5- Evaluation Feedback for Effective and Learning http://puck.sourceoecd.org/vl=642079/cl=34/nw=1/rpsv/~6669/v2001n25/s1/p1l Development inEnvironment Lessons Learned (Complete Edition), 2001 Evaluation Institutional Capacity for andAid EffectivenessDonor Support No3- http://puck.sourceoecd.org/vl=642079/cl=34/nw=1/rpsv/~6669/v2001n23/s1/p1l 1999 (Complete Edition), 2001 Evaluation andAid Effectiveness No2- Evaluating ProgrammesCountry Vienna Workshop, http://puck.sourceoecd.org/vl=642079/cl=34/nw=1/rpsv/~6669/v2001n22/s1/p1l in Complex Emergencies (Complete Edition), 2001 Evaluation andAid Effectiveness No. 1-Guidance for Evaluating Humanitarian Assistance http://puck.sourceoecd.org/vl=642079/cl=34/nw=1/rpsv/~4292/v6n2/s1/p1 Guidance for ManagingJoint Evaluations, 2006 http://puck.sourceoecd.org/vl=642079/cl=34/nw=1/rpsv/~4292/v6n1/s3/p95 Aid inDACHumanitarian Peer Reviews: ASynthesis ofFindings andExperiences 2004-05 ofKey Glossary Terms inEvaluation basedManagement, OECD2002 andResults oftheDACReview Principles for Evaluation ofDevelopment Assistance, OECD/DAC, 1998 Development Assistance”, OECD/DAC 1991 nciples for Evaluation ofDevelopment Assistance” and “Criteria for Evaluating k Paper by n°58concerning Austen accountability action, inhumanitarian Davis, , C., Roberts, L.,Shoham,J., andHarvey,, C.,Roberts, P., 2004 g 95 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 96 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs OTHERS (miscellaneous) ------UNEG UNHCR UNICEF WFP - 3-4 1998,Reliefweb Coordination-Humanitarian Lessons Seminar, ofaReview Learned, Report Apr Stockholm, Watson for Institute International studies, Brown University, USA,1998 inthe Caucasus: Action Humanitarian aGuidefor Practitioners, Greg Hansen, Thomas J. andAnnexes, Capacity Institutional Measuring “”, 2000 Guidelines for IndicatorandData Quality, “”, 1998 The ofEvaluation Role inUSAID, “ “, 1997 Evaluation, 1996-http://www.usaid.gov/pubs/usaid_eval/pdf_docs/pnabs539.pdf AppraisalUsing Methods, Rapid TIPS, USAIDCentre for development Information and http://do http://www.wfp.org/operations/evaluation/guidelines.asp?section=5&sub_section=8 andevaluationMonitoring guidelines, 2004 http://www.wfp.org/aboutwfp/partners/documents/NGO_handbook_complete.pdf How to work with WFP –Ahandbookfor NGOs, 2005(Chap Dec 8–M&E) indicators for UNHCRstaff Project guideontheuseofobjectives, planninginUNHCR–Apractical outputsand UNHCR Standards andindicators, February 2006 http://www.unhcr.org/research/RESEARCH/3d99a0f74.pdf UNHCR’s Evaluation Policy, Sep. 2002(basedonOECD/DAC criteria) http://www.unhcr.org/research/3b850c744.html (see EPAU web site, Evaluation andPolicy Analysis Unit) de 2005auN Évaluation en Temps Alimentaire àlaCrise del’UNICEF delaRéponse Réel etNutritionnelle PATH (aPrincipled Approach To inCDversion, Link225 action), Humanitarian Assessment July2005–withCDLink222 andMonitoring), FieldEmergency Chap3.1on Handbook–Aguidefor UNICEFstaff(seein particular RTE”,Agency UNICEF/OCHA,Nov 2006 Experience withReal-TimeAgency Evaluation (RTE), “Towards anApproach for Inter- March 2001. http:// Standards for Evaluation intheUNSystem, 2005 April http://www.uneval.org/docs/ACFFC9F.pdf forNorms Evaluation intheUNSystem, 2005 April www.uneval.org/docs/ACFFCA1.pdf cuments.wfp.org/stellent/groups/public/documents/ko/mekb_module_7.pdf iger. UNICEFBureau d’Évaluation. Décembre 2006. and implementing partners, Division of Operational support, DivisionofOperational andimplementingsupport, partners, BACK TO TABLE OF CONTENTS BACK TO TABLE OF CONTENTS HUMANTARIAN VAL ------www.compasqualite.or Compas, Quality GroupeDynamic URD(Urgence RéhabilitationDéveloppement), 2006 http://eipa.euro-ip.net/CAF/Brochure/CAF2002_Eng.pdf Self-evaluation CAF 2004 edition Standards andMinimum inDisaster Response, theSPHEREProject, Charter Humanitarian http://www.ifrc.org/publicat/conduct/code.asp NGOs inDisaster Programmes Response (theCode IFRC-ICRC.. ofConduct), Principles for ofConduct Cross theInternationalRed andred Crescent Movement and http://www.reliefweb.int/ghd/meetings.html#Stockholm (GHD),Principles Donorship ofHumanitarian practice 17June2003 andGood Stockholm, 2007. Utilization-Focused Evaluation: The Century Text. Patton. Quinn Michael Publications, Sage consultancy, staff register) http://www.mango.org.uk/ provides helpto strengthen financialmanagement ofNGOs(training, networking, Accounting (Management Mango for NonGovernmental Organisations): theorganisation confused with«SMART indicators»!) www.smartinidicators.or Standardised andAssessment Monitoring and ofRelief Transition («SMART», notto be Marc-Antoine Pérouse deMontclos, IRDéditions, novembre 2006Link234 leur perception parlesbénéficiaires:l’aide lesenjeuxde internationale auBurundi” par ”Autre Part”, desciences Revue socialesauSud, - l’impartialité“De deshumanitaires etde Strategyframework, Unit, UKPrime Minister’s Cabinet Office, 2003 inqualitativeQuality evaluation: aframework for assessingresearch evidence –AQuality Punto.sud www.puntosud.org/helpdes helpdesk, k NGO Self-Assessment through aSWOT Exercise, www.networklearning.or Centre for Research ontheEpidemiologyofDisasters: www.cred.be 2006www.smartindicators.or1, April (StandardizedMethodology andassessment Monitoring ofRelief Transitions), version Status, Mortality,Nutritional Measuring andFood Situations: inCrisis SMART security –AGlobalreviewLiving withRisk ofdisaster initiatives, reduction ISDR2002 International Legal Standards • - http://www.icrc.org/ihl.nsf/CONVPRES?OpenView 46898c125641e004aa3c5 http://www.icrc.org/ihl.nsf/7c4d08d9b287a42141256739003e636b/6756482d861 Laware collectively asInternationalHumanitarian (IHL). known these 1977: of Protocols Additional two their and War)of Time in Persons Civilian The Geneva Conventions of 1949 (especially the IV Convention on the Protection of UES ANDPRINCIPLE g andwww.urd.or g g g g 97 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 98 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs CROSS - - - IASC/OCHA - ALNAP PROTECTION International Law HumanRights • mlmnig h Claoaie epne o iutos f nenl Displacement, Sep. 2004 Internal of Teams,IASC Country and Situations Coordinators Resident and/or Humanitarian UN to for Guidance Response Collaborative the Implementing http://www.icva.ch/cgi-bin/browse.pl?doc=doc00000717 and Practice Gathered from theField. Inter-Agency Standing Committee, Geneva. Growing the Sheltering Tree: Protecting Rights Through Civilians, Humanitarian Action, of Programmes Protection the OCHA 2004 Pertaining Issues of Consideration the for Aide-Mémoire http://www.alnap.org/pubs/pdfs/protectionbooklet3.pdf Protection, anALNAPguidefor agencies, humanitarian ODIAugust 2005 International Law Refugee • TTING ISSUES UTTING -C ------http://www.unhchr.ch/html/menu3/b/a_cescr.htm International Covenant onEconomic, andCultural Social Rights, 1966. http://www.unhchr.ch/html/menu3/b/a_ccpr.htm International Covenant onCivil andPolitical Rights, 1966. http://www.unhchr.ch/udhr/lang/eng.htm Universal Declaration ofHumanRights, 1948. http://www.unhchr.ch/html/menu3/b/p_genoci.htm Convention onthePrevention andPunishment ofGenocide, oftheCrime 1948. http://www.unhchr.ch/html/menu2/7/b/principles.htm Guiding Principles Displacement, onInternal 1998. =3b66c2aa10 http://www.unhcr.ch/cgi-bin/texis/vtx/protect/opendoc.pdf?tbl=PROTECTION&id of Refugees, 1967. Convention on the Status of Refugees, 1951, and the Protocol Relating to the Status http://www.unhchr.ch/html/menu2/6/crc/treaties/crc.htm Convention oftheChild, ontheRights 1989. http://www.unhchr.ch/html/menu3/b/h_cat39.htm or Punishment, 1984. TreatmentDegrading and Inhuman Cruel, Other and TortureAgainst Convention http://www.unhchr.ch/html/menu3/b/e1cedaw.htm Convention ontheElimination ofAll Forms ofDiscriminationAgainst Women, 1979. BACK TO TABLE OF CONTENTS BACK TO TABLE OF CONTENTS - - - sources onlyistherefore listed below. only- is simply too large to be included in this section. A selection short of the most essential The number of references on the theme of gender the –even most relevant and recent ones GENDER EQUALITY - - - OTHERS - - UNICEF - - - - - UNHCR - - - ICRC - uoei, e gne dcmns nldn Tokt at Toolkit europeaid/projects/gender/index_en.htm including documents gender key Europeaid, acdi-cida.gc.ca/INET/IMAGES.NSF/vLUImages/Africa/$file/Guide-Gender.pdf 2003. Issues, the to Guide A Assistance: Humanitarian and Equality Gender CIDA, reports/re33c.pdf BRIDGE, Gender, Emergencies and Humanitarian Assistance, 1995. http://www.idpproject.org/ GlobalIDPProject: Council Refugee (withUNHCRsupport), Norwegian Protection into Practice, 2005(for Oxfam internal UK, useonly) ‘Improved Protection ofChildren from Violence, Abuse, Exploitation andDiscrimination’. “Core Commitments for Children inEmergencies”, 2004 Persons: Guidelinesfor Prevention andResponse, (UNHCR,May 2003). andGender-based Sexual Violence Displaced andInternally againstRefugees, Returnees United Nations Training Monitoring, onHumanRights (UNHCR,2001. Manual Staff, UNHCR for ofInternationalProtection July2002. Department Checklist Progress: Measuring and Strategies Protection ‘Designing Protecting AField Refugees: Guidefor NGOs. 1999 UNHCR.Geneva, Agenda for Protection. 2002. UNHCR.Geneva, Women Facing War, 2001. ICRC, Geneva, ICRC, Geneva, Protocols, 1983. Additional their and Conventions Geneva the of Rules Basic Geneva, ICRC. Standards. 2001. Professional for Search A War: in Protection Strengthening http://www.reliefweb.int/ocha_ol/civilians OCHA, Protection ofCivilians Conflict. inArmed “Support to Displaced Internally persons–Learning from“Support Evaluations” –SIDA 2005. http://europa.eu.int/comm/ www.bridge.ids.ac.uk/ www. 99 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 100 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs - - CHILDREN isavailableThe IASCtoolkit at www.reliefweb.int/library/GHARkit/ Full available text at www.un.org/womenwatch/daw/beijing/platform/armed.htm Full CEDAW available text at www.un.org/womenwatch/daw/cedaw/frame.htm ------texis/vtx/protect/opendoc.pdf?tbl=PROTECTION&id=3f696bcc4 Displaced People. Guidelines for Prevention and Response, 2003. Internally and Returnees Refugees, against Violence Gender-Based and Sexual UNHCR, f bcpr/ref/gender_manual_uk.pd UNDP, Gender Approaches in Conflict and Post-Conflict Situations, 2002. Web/Eng/siteeng0.nsf/htmlall/p0840/$File/ICRC_002_0840.PDF!Open 2004. Conflict, of Armed Women Needs by Affected Addressingthe ICRC, (on ReliefWeb). assessment, and forintegratinganalysis gender checklist and guidelines Summary IASC, t GHARki IASC, Gender and Humanitarian Assistance Resource Kit, 2001. Programmes: Guidelines. ftp://ftp.fao.org/docrep/fao/007/AD904E/AD904E00.pdf FAO/WFP, Socio-Economic and Gender Analysis (SEAGA) for Emergency and Rehabilitation 2003. http://www.fao.org/sd/seaga/downloads/En/passporten.pdf Programmes, Emergency in Perspective Gender a FAO/WFP,Mainstreaming to Passport http://www.europa.eu.int/comm/external_relations/human_rights/child/caafguidelines.pdf EU GuidelinesonChildren Conflict, andArmed 2003. Children and Young People inDanishDevelopment 2005,Danida. Cooperation. January “Invisible children? – Towards Integration ofChildren’s States’ inEUandMember Rights Save theChildren. Field resource manualfor emergencies, 2003(orlater). Save theChildren. Forgotten 2005. conflict, inarmed Girls casualtiesofwar: by ECHO). (supported children associated withfighting forces.McConnan Isobel Sarah and Uppard, 2001 Save theChildren. Children notsoldiers:Guidelinesfor withchildsoldiersand working monitoring guidelines,Save Globalimpact 2004. theChildren UK. Education inemergencies: Save paper, theChildren policy 2001. 2003. conflict, PracticeODI/ Humanitarian Network. The role ofeducation inprotecting children in INEE(Inter-Agencyfor network Reconstruction, Education inEmergencies), 2004. StandardsMinimum for Education inEmergencies, Chronic andEarly Crises Eliminating theworst forms ofchildlabour. ILO-IPEC, 2002. 2002. on theElimination ofChildLabour(IPEC) childcounts“Every –newglobalestimates onchildlabour”, ILO, InternationalProgramme SCUK-UNICEF-UNHCR-WVI, 2004. onUnaccompaniedInter-Agency Guidingprinciples andSeparated Children, ICRC-IRC- BACK TO TABLE OF CONTENTS www.reliefweb.int/library/ www.unhcr.ch/cgi-bin/ www.undp.org. www.icrc.org/ BACK TO TABLE OF CONTENTS ------OLDER PEOPLE - - HAI. Published by Earthscan Publications,HAI. Published by Earthscan www.earthscan.co.uk The ageing and development report: Poverty, independence and the world’s older people. International HIV/AIDSAlliance, 2003. and HAI children. vulnerable and orphans of carers as people Older families: Forgotten Addressing Older People’s Practice inAfrica: Good Rights Guidelines. HAI. paper.HIV/AIDS andageing:Abriefing May HAI. 2003. researchParticipatory witholderpeople:Asourcebook. HAI.March 2002. State oftheworld’s olderpeople2002.HAI. of olderpeopleinAfrica. 2004. HAI.April vulnerability for factors risk and status Nutritional the FindingsResearchon of Summary February 2005. HAI. report. Summary Darfur: West – people older of assessment nutrition and Health 2001. HAI, Africa. in situations emergency in people older of needs nutritional the Addressing Emergencies andAging: Apositionpaper. HAI.March 2001. Ageing IssuesinAfrica: ASummary. HAI.Undated (likely2000-2001). HAI documents are available at: www.helpage.or other and aboveUNHCR. The and ECHO DG from support 2004. With (HAI) International practice,forHelpAgebest Guidelines Crises: PeopleHumanitarian Older and Disasters in 2004 edition. SPHERE Standard (defined pp. 10-11 and incorporated for all sectors throughout the text. Kenya StrategyMerlin. Country 2003-2006.Updated December 2004. UNICEF. Childre 2004. UNICEF InnocentiResearch Centre. cities:Aframework Buildingchild friendly for action, UNICEF InnocentiResearch Centre. cities, Childfriendly 2004. oftheChild,the Rights 2004. UNICEF InnocentiResearch Centre. Study ontheimplementation oftheConvention on UNICEF. Core commitments for children inemergencies, June2004. UNHCR Education Field Guidelines, 2003. UNHCR GuidelinesonCare &Protection Children, ofRefugee 1994. UN Convention 1989. oftheChild(CRC), ontheRights andminimumstandards indisaster response, charter SPHERE Humanitarian 2004edition. withachildren’s assistance conflicts Humanitarian inarmed rights perspective. Sida,1999. Policies”,Development andCo-operation Save theChildren, 2002. n affected by armed conflict: Unicef actions,n affected Unicef 2002. by armed conflict: g , 1999. 101 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 102 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs ------DISABLED PEOPLE - - - - H gieie. iatr, iaiiy n rhbltto. 2005. rehabilitation. f violence_injury_prevention/other_injury/disaster_disability2.pd and disability Disasters, guidelines. WHO WHO. InternationalClassification of andHealth(ICF). Functioning, Disability especially Rule21)http://www.un.org/esa/socdev/enable/dissre00.htm UN Standard Rules on the Equalisation of forOpportunities Persons with Disabilities. (see all sectors for throughout 2004edition. thetext). intostandards and andincorporated development 10-11 pp. (defined for Standards SPHERE manual training A rights: organisations.humanitarian withSueEnfield. Alison Harris Oxfam GB, 2003. human and equality, Disability, www.itdg.or– Resource pack. Sri Lankan Guidelines on Accessibility – Guidelines for planning in the re-building process solutions. http://www.accessforall.lk/disability_and_emergency_response_tsunami.pd f Barbara Oosters, IDDC, CBM. Overview of problems Asia. faced South-East by in Tsunamidisabled the people by and caused inclusivedisaster the at lens disability a with Looking www.Iddc.org.uk/dis_dev/key_issues/conflict.shtml Oosters, IDDC,March, 2005,andvariousdocuments: disabled including people and for the issue of Guidelines disability in response Interventions. to disasters and Effective humanitarian crises. Barbara for Building Capacity IDDC. http://europa.eu.int/comm/employment_social/events/2004/eddp/index_en.html Disabilities 2004. with People of Day European the for conference Commission European the of Report and Delegations Union EuropeanServices. Commission, DE124,July2004. European for Development and Disability on Note Guidance (COM/2003/650), 2003. Plan Action European A disabilities: with people for opportunities equal establishing on EC the from Communication EU. the in Disabilities with People for Opportunities Equal doc/english/global/david/dwe002/dwe00253.htm#part2chap51 resources: (DFA)http://www.un.org/esa/socdev/enable/designm/AD3-01.ht m All for Design WHO (Ageing andLife Course Unit).HealthandAgeing. 2001. UNHCR Policy onOlderRefugees. Undated. Available at theUNHCRwebsite. UN’s “Principles for OlderPeople” n°46/91).1991. (Resolution of discussion for 8 Issue A. situations.)emergency http://www.un.org/ageing/coverage/action.pd II. f (see 2003. Nations, United 2002. Ageing, on Assembly World Second ageing. on action of plan international Madrid and declaration Political g www.design-for-all.or g; BACK TO TABLE OF CONTENTS www.design.ncsu.edu/cud http://www.dinf.ne.jp/ www.who.int/entity/ BACK TO TABLE OF CONTENTS ------(effectsonenvironmentENVIRONMENT only) projects by humanitarian Some otherresourcesSome onaccessibility: co-operation: development of field the in working networks and organisations umbrella disability Key UNHCR “Environmental Guidelines” of 1996. SPHERE standards, 2004. http://www.europa.eu.int/scadplus/leg/en/lvb/l28119.htm (to access bothdocuments). movements n°1946/2003of15July2003ontrans-boundary ofGMOs Regulation (EC) ProtocolCartagena onBiosafety, 2000. dimension indeveloping countries (RegulationECN°2493/2000of07/11/2000). policy cooperation Community sustainable into Integrating development principle; Precautionary ENV): (DG Environment on Policy EU AIDCO GuidelinesonEnvironment and Technological Means. Pathways • ESCAP, • ITDG-South • UN: • Global • Alliance InternationalDisability (IDA). www.internationaldisabilityalliance.or • EDF. www.edf-feph.or • IDDC.www.iddc.org.uk • Handicap • • • World andDevelopment BankDisability • DFID. • http://www.un.org/esa/socdev/enable/designm/ (By C (By Venter, etal).http://www.globalride-sf.org/images/DFID.pdf http://www.iadb.org/sds/SOC/publication/gen_2547_3206_e.htm DevelopmentAmerican Bank). l 216618~theSitePK:282699,00.htm XTDISABILITY/0,,contentMDK:20192454~menuPK:417739~pagePK:148956~piPK: http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALPROTECTION/E http://www.worldbank.org/disabilit World Bank’s site). Disability Governments, donors, UN agencies, NGOs, DPOs, etc. (see link to its website via the planning post-disaster. www.itdg.or z1500901.htm http://www.unescap.org/esid/psis/disability/decade/publications/z15009gl/ non-handicapping environments. emergency aid.emergency www.hi.org.uk Accessibility Enhanced United Partnership to International. Asia Accessibility: Nations for Accessibility and for the g Rural Regional Disability Disabled: Major Disability Development for . Commission TA and A People . y ,and g Design . provider and Development Team website with the Policy Manual for Itgain f h environmental the of Integration ; worldwide Disabilities Physical Asia Institute for (GPDD), and a Barrier Environment the Living in (2004): Pacific with development Free in members Guidelines guidelines Environment. Urban (The g Areas Inter- from and for on 103 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 104 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs ------CIVIL-MILITARY INTERFACE, ACCESS - - - Conflict in Conflict Afghanistan IMTF (November 2001). Area UNAMA Currentthe during Operations for Humanitarian UN for Aircraft Military of Use on Guidance Guidelines – Afghanistan in Coordinators andotherUNPersonnel (2002). Forces Military with Relationships Civilian and Military and Personnel UN Representatives oftheOccupying Power (8May 2003). inIraq between Interaction for Guidance General March OCHA, 2003 http://ochaonline.un.org/DocView.asp?DocID=426 guidelines”), (“MCDA Emergencies Complex in Activities Humanitarian Nations United Support ToAssets Defence Civil and Military of Use The On Guidelines Binding Guidelines, IASCSeptember2001. Non- and Paper Discussion Convoys- Humanitarian for Escorts Armed or Military of Use “Oslo Guidelines”, IASC1994. Civiland Defence Assets (MCDA) Usethe on Guidelines Military of the Disaster– in Relief in military the and aid agencies far: too bridge response,”humanitarian “A 2002). Jeffreys, Practice Humanitarian paper37(January Network Anna with Barry Jane (GDI),Institute Bonn. State Actor and Non-Governmental Approaches (2004): Claudia Hofmann, Militaires [GRIP], sécurité la humanitaires et (Bruxelles : Éditions Grip, 2002). paix la sur d’informationHumanitaires et recherche de Groupe Binding Guidelines, IASCSeptember 2001. Non- and Paper Discussion Convoys- Humanitarian for Escorts Armed or Military of Use http://coe-dmha.org/Media/Guidance/3MCDAGuidelines.pdf 2003. March IASC Emergencies,ochaonline.un.org/webpage.asp?MenuID=5111&Page=77 4 Complex in Activities United Humanitarian support to Nations (MCDA) Assets Defence Civil and Military of Use the on Guidelines between ECHOandtheCivil ProtectionMoU mechanism(EUMIC/DGENV). Protection mechanism. Civil Community the of establishment the on 2001) (October 2001/792 Decision Council Crisis and http://europa.eu.int/comm/external_relations/cfsp/doc/cm03.pd f Prevention Conflict EC management”, crisis Unit,Management 2003. April EU for instrument “Civilian andtheEnvironment”: www.wfp.org/eb/docs/1998/wfp000272~1.pd“WFP f UNHCR Operations“Refugee andEnvironmental Management”, 2001. UNHCR “Environmental Guidelines–Forestry Situations”, inRefugee 1998. Civil-Military Relationship inComplex Emergencies –anIASCreferenceCivil-Military paper, June2004. Civil-Military Relationship inComplex Emergencies –anIASCreferenceCivil-Military paper, June2004. : À chacun son rôle. Cohérence et incohérences des opérations militaro- opérations des incohérences et Cohérence rôle. son chacun À : Engaging Non-State Armed Groups in Humanitarian Action. Action. Humanitarian in Groups Armed Non-State Engaging , Background Paper, German Development BACK TO TABLE OF CONTENTS http:// BACK TO TABLE OF CONTENTS ------PARTICIPATION, PARTICIPATORY EVALUATIONS - - - - ACCES - - - of Women, and Children, Men 1992. UNHCR, A Framework for People-Oriented Planning in Refugee Situations taking account UNHCR, Situation Analysis: a Tool forAssessment withRefugees, Participatory 2005. 2002. [URL] Practice, Good of Toolkit A – Children with Consult to YouWant So Children, the Save Accountability? Humanitarian Why Contextual andOperational Factors, 2001.www.hapinternational.or Partnership), Accountability (Humanitarian HAP images/pdfs/Participation.pdf 2002. sourcebook, A People: Older with Research Participatory HAI, www.helpage.org/images/pdfs/briefing%20papers/ParticipationPaper.PDF HAI (HelpAge International),Participation andAgeing, 2000. generation andremunerative operations”, schemesinhumanitarian 19/12/2002. cost-recovery, of income on inclusion Staff for the “Procedures Note for ECHO,Guidance ECHO, to Practice AnGood Introduction Aid, inHumanitarian 1998. ALNAP,R.T.Ntata, Pierson 1999 Sudan, South in famine 1998 the to response the during agencies DEC of experience the of review a Operations: Relief Affected in Population the by Participation f www.globalstudyparticipation.org/phase2/english/piloting/handbook.pd for Practitioners, 2003. ALNAP, Participation by Crisis-Affected Populations A Handbook in Humanitarian Action. The World Bank: – CICR OCHA IRIN(Integrated Regional Informationhttp://www.irinnews.org Network) / du confidentialité “La 1997/2 Droit – du etfichespratiques”.Introduction violation DOCT/58 de and cas Humanitaire” en CICR International du “L’action DOCT/15 documents doctrine ICRC EIDHR: http://www.europa.eu.int/comm/europeaid/projects/eidhr/index_en.htm http://www.alnap.org/pubs/pdfs/alnap_protection_guide.pdf advocacy.and checklistHonhumanitarian « “Options for Aid inConflict: Lessons from Field Experience”, Dec.B. Anderson, 2000. Mary Publishers,Rienner February 1999,and Lynne Boulder: Anderson. B. Mary War” Peace-Or Support Can Aid How Operation Harm: No “Do Peacekeeping of Department UN (September 2002). by Policy Coordination Civil-Military Oxfam, What and Men Women Want: APractical Guideto andParticipation, Gender 2004. Protection – an ALNAP guide for humanitarian agencies humanitarian for guide ALNAP an – Protection The World BankParticipation Sourcebook », ALNAP 2005. See page 87 page See 2005. ALNAP », g www.helpage.org/ 105 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 106 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs The • International Fund for Agricultural Development (IFAD) The • International LabourOrganization (ILO) JIU • Unit(JIU) Joint Inspection The • Food andAgriculture Organization (FAO) EVALUATION withinUNAgencies - http://europa.eu.int/comm/europeaid/visibility/pdf/europeaid_guidelines_en.pdf - - andCOMMUNICATIONVISIBILITY - - - - - UNAIDS • Joint UNProgramme onHIV/AIDS(UNAIDS) Also available inFrench and Spanish. summaries and reports can be accessed through their Evaluation evaluation. programme and project for guidance methodological and tools IFAD’s evaluation approach. a includes also site The function. search detailed a including reports evaluation of database a System, also includestheLABORDOCdatabase only). UNICEF’sintranet the on or version printed in (availableBook”, Brand2003 UNICEF “The “EU Visibility Guidelinesfor Actions”, external EuropeAid, November 2002. to relating activities the communication on and aid”,humanitarian partners DGECHO2006. information Commission’s visibility, for of Guidelines implementation - communication for partnership “A and Evaluation, 1996http://www.usaid.gov/pubs/usaid_eval/pdf_docs/pnabs539.pdf Information development forCentre USAID Evaluation, TIPS, Participatory a Conducting http://www.undp.org/eo/documents/who.htm Who Evaluation UNDP1997, Are -AParticipatory Handbook, theQuestion-makers? Planning at People-Oriented Work: Using POPto Improve UNHCR WFP, Approaches, Participatory 2000.www.wfp.org/eb/docs/2000/wfp003920~1.pdf Programming - A Practical Planning Tool for Refugee Workers, B. Anderson, UNHCR, 1994 Mary the progress ontheUNGASSDeclaration ofcommitment onHIV/AIDS. Thelevels. country Also available inFrench andSpanis reports reports as well as reports on implementation of conduct JIU evaluations, recommendations inspections can and be investigations found system-wide. on the The JIU’s mandate, annual

is webpage provides full reports of reports full provides webpage Evaluation

A Eauto Servic Evaluation FAO the vlain Lann ad Impac and Learning Evaluation,

only harmonizes

independent section that includes technical guidance on guidance technical includes that section y Methodolog and Process site provides information on monitoring on information provides site evaluation and monitoring monitoring

external est includes website e

oversight and epg icue the includes webpage t . evaluation

body oiis n Procedures and Policies

of thematic evaluations thematic

the approaches Documents s and database.Report

United BACK TO TABLE OF CONTENTS

Nations at Evaluation Knowledge Knowledge Evaluation global,

system . The ILO website ILO The . a olcin of collection a , regional

mandated site and

to . BACK TO TABLE OF CONTENTS The • United Nations Development Industrial Organization Evaluation (UNIDO) Services The • United Nations Population Plan(UNFPA) This • World HealthOrganization (WHO) The • United Nations Environment Programme (UNEP) • United Nations Educational, andCultural Scientific Organization (UNESCO) The • United Nations Development Programme (UNDP) Evaluation • Office forthe Coordination ofHumanitarian Affairs (OCHA) While • United Nations Capital Development Fund (UNCDF) Emergency • United Nations Children’s Fund (UNICEF) The • United Nations HumanitarianandCrisisResponse(UNHCR) Assessment July2005–withCDLink222 andMonitoring), ND dcmns hc cn e ree oln. n diin ti st lists site s classified by report region andtheme. this addition, In online. ordered be can which documents UNIDO and technical documents from 1985to thepresent. journals WHO-produced from articles and onwards 1948 from publications WHO all of provide hands-on informationprovide onbasicM&Econcepts hands-on andapproaches. The Programme Manager’s Monitoring and Evaluation Toolkit evaluations of report UNFPA-supported projects and programmes. This site also includes methodological guidance underEvaluation Tools The evaluations. The Central Evaluation Databas e contains ofevaluation summaries reports. ongoing and planned agency’s the about information provides and 2002, in published undp.org/undpweb/eo/evalnet/docstore3/yellowbook/documents/comp-series1.pdf completes to listoftheir fullevaluationdate. report sit See: UNHCR’sSee: evaluation policy, sep. 2002 s sinceevaluation 1994. report evaluation. organizedthematically. Theprovidessitealso technicalUNEP’sguidance on approach to e provides M&Etools Internal Internal Oversight Servic site website provides access to all full text UNHCR text full all to access provides website Unit Analysis Policy and Evaluation reports nutil eeomn Asrcs aaae (IDA Database Abstracts Development Industrial vlain sit evaluation est icue fl vrin o their of versions full includes website Resources Evaluation and Monitoring UNCDF provides plays Field section does an access Handbook nlds h “udlns o Otoe vlaos (http://stone. Evaluators” Outcome for “Guidelines the includes e important of not , OCHA specific and Inter-Agency, OCHAspecificand evaluations andstudyreports the to have WHOLIS, the WHO library database which indexes full versions sit e website has links to their recent evaluations and a range of – includes a list of list a includes e role a A section guide as a key for specifically management UNICEF . s report evaluation project and annual staff for otis nee asrcs of abstracts indexed contains ) evaluation, and (see , a collection of tools which learning in particular this tool site in Chap OCHA. evaluation includes 3.1 The on . a ) 107 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 108 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs - Swedish International Development (Sida) Agency - Japan International Cooperation (JICA) Agency - languageonly.in German (German) Federal for Economic Cooperation Ministry andDevelopment (BMZ)* - (Finnish) for International Department Development Cooperation (Finnida) - for International Department Development(UK) (DFID) - Danish International Development (DANIDA) Agency - Australian for Agency International Development (AusAID) BILATERALSOME KEY DONORAGENCIES - Canadian International Development (CIDA) Agency reports from thetwo series.reports newsletter,a has also Sida EvaluationNewsletter “Sida in published are issues methodological Evaluatioconceptualand in n with Studies “Sida Sida’s evaluation arereports published in the series “Sida Evaluations l francais_2379/delegation-action-humanitaire-dah_15669/index.htm http://www.diplomatie.gouv.fr/fr/actions-france_830/humanitaire_1039/acteurs- stocks andfunding.emergency other functions, such as coordination with other concerned institutions, management of Humanitaires desinterventions Direction etdel’Evaluation. Evaluation iscombined with Ministère des Affaires Etrangères (France),Délégation àl’Action Humanitaire (DAH),Sous- olwu ad eut. h st provides site The project results. and plans, and future follow-up and topics evaluation its activities, evaluation history project the its of on information provides website g Monitorin Post-Project and Evaluation evaluations.sectors, aswell andprojects asexamples ofcountry The Finnida are available online, dating from 2000. of Abstracts and evaluation. DFID, provides online and knowledge-sharing learning tools on performance assessment order.to or The report The since 1994. onevaluationfors undertaken 2005-2006,andreports um.dk/en/menu/DevelopmentPolicy/Evaluations/Guidelines/), The guides. management results-based and evaluation other Also available inFrenc as h. well as Evaluations The This s dating backto 1996. report Evaluation and g Reportin site includes access to Evaluation of Developmen Evaluation page provides access to evaluation instruments and evaluations of topics, page lists evaluation reports and summaries as full-text full-text as summaries and reports evaluation lists page g Reportin & Evaluation website offers access to its lengthy manual its lengthy to access offers website Evaluation supported by supported programmes sectoral or programmes country of evaluations Performance Assessment Resource Centre (PARCCentreResource Assessment Performance t section lists DANIDA’s and guides evaluation statistics publications and evaluation guidelines ( ”, selected of summaries including BACK TO TABLE OF CONTENTS vlain programme evaluation ”. Studies concerned s report evaluation o t Perform to How , established by established ), http://www. evaluation .”. .

BACK TO TABLE OF CONTENTS - European Evaluation (EES) Society - Australasian Evaluation (AES Society ) - - Canadian Evaluation (CES Society ) American Evaluation Association (AEA - African Evaluation Association (AfrEA Evaluation Associations andInstitutions EVALUATION SOCIETIES - U.S. for Agency International Development (USAID). events, andlinkto professional andotherevaluation networks sites. h st poie ifrain bu te tutr, ebrhp ad ciiis f the of activities European and Evaluation Society. membership, You can structure, also read their the about information provides site The Also available inFrench. ofProgramJournal Evaluation. Their on standards for reference important an as programme evaluation ingeneral. increasingly serve which Education for Evaluation, Standards on Committee Joint the by developed Standards Evaluation including documents, and publications programs. development professional The The certification; the certification; and co-operation international ethics, on papers position include publications Online online newsletter, are available aswell asarchived issues. its Guidelines anditsdatabase ofevaluators withAfrican evaluation expertise. to access include site this of Highlights evaluators. of networks national 17 together brought AfrEA 2002, June of As discipline. development-oriented every from Evaluators African for association umbrella an as 1999 in founded was AfrEA resources/#grants http://www.usaid.gov/our_work/humanitarian_assistance/disaster_assistance/ alsothe See “Guidelines for Unsolicited Proposals andReporting”, 2006. Dec the database. inparticular See “USAID Performance andEvaluation Monitoring Tips”. included their of Also abstracts are evaluations. ongoing and resources tools, provides “evalweb”USAID’s ordered by contacting the webmaster at webmaster the contacting by ordered section includes up-to-date information on upcoming events. upcoming on information up-to-date includes section Events and Meetings eto poie lns o non dge porm and programs degree ongoing to links provides section Evaluation in Training etos nlds the includes sections publications CES Newsletter CES which are available through a searchable a through available are which publications evaluation ; the ; ) ) CES Annual Repor Annual CES los ces o association-related to access allows Publications vlain ora o Australasia of Journal Evaluation E’ Giig rnils o Eautr and Evaluators for Principles Guiding AEA’s [email protected] newsletter ; and abstracts of the of abstracts and t; . Latest issues of issues Latest . , see upcoming evaluation fia Evaluation African E-new n my be may and Canadian s , their , 109 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 110 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs - Evaluation Francophonie - NetworkNigerian ofMonitoring andEvaluation (ReNSE) - International Organization for Cooperation inEvaluation (IOCE) - International Development Evaluation Association (IDEAS) - Spanish Evaluation Association (SEE)*Onlyavailable inSpanish. - Société française del’Evaluation (SFE)*Onlyavailable inFrench. evaluation; for French-speaking intended participants; events to priority a with professionals evaluation to resources; interest of events evaluation or online and organizations networks, evaluation sections: six includes website The Governanc of Poverty/Vulnerabilit themes the on groups working 7 created ReNSE charge. of free English, The The network has a problems. of world practice solution and global and identification more a the theory take Also available inFrench and Spanish. to to contributing profession evaluation developing evaluation to the in of approach assisting capacity and cross-fertilization and world, the leadership around the around building from to fostering organizations dedicated evaluation countries, is national that and world regional the of coalition Loose good governance. IDEASwas initiated by UNDPandthe World and Bank. transparency of aspect essential an as evaluation development transition. advocating for in platform important an countriesas serves and Africa, and South in based is countries secretariat IDEAS The developing in especially knowledge, and sharing approaches, methodological innovative applying networking, building, to capacity committed evaluators and practitioners development of network global a is IDEAS evaluation withinthefrancophone space. http://www.evaluation.francophonie.org/ Francophonie la de intergouvernementale methodology, and terminology (evaluation standards, sponsors advocacy for evaluation, online reports databases, evaluation etc.). Sponsored by and evaluators to targetedare thatlanguages, other Frenchalso in and documents to references links and evaluation indifferent regions ofthe world. public policies. It also contains a site with links to institutions and associations involved in of evaluation on events training and activities of list a includes SEE the of webpage The public national policies of as evaluation a means promotes of 2001 improving in efficiency and founded effectiveness Association Evaluation of public Spanish interventions. The exchange your evaluation experiences andideas. to SFE by hosted groups discussion the Join member. a become can you how out Find aYahoo and moderates ReNSE Coordinator to email the blank a send is just join, Togroup. discussion UNICEF for Officer Evaluation & section provides access to over a 100 evaluation articles in French and French in articles evaluation 100 a over to access provides section Documents e s Opportunité y , Methodolog CV CV databas e of the evaluators of Niger. y , Health/AIDS, in the evaluation profession; and profession; evaluation the in , andIC Formations or training opportunities in programme/policy T and x Réseau . ReNSE alsopublishesaNewslette. ReNSE r niomna/aua Rsucs Managemen Resources Environmental/Natural ti wbie uprs h dvlpet of development the supports website this , with links to professional to links with Organisations [email protected] Robert Robert Ndamobissi, Monitoring Documentation BACK TO TABLE OF CONTENTS . Manifestations or selected selected or Education, l’Agence t , . BACK TO TABLE OF CONTENTS - EvaluationSri Lanka Association (SLEvA - Monitoring andEvaluation to Assess andUse Results(MEASURE) - UNICEF Innocenti Research Centre (IRC - International Evaluation Research Group (INTEVAL) - International Development Research Centre (IDRC INSTITUTIONS RESEARCH - UK Evaluation (UKES) Society - Swiss Evaluation Association (SEVAL) Training Buildingonmonitoring andCapacity evaluation; andmore. and information onnational andinternational evaluation events Among other things, this site includes a link to a draft of the create customized tables from hundred of indicators; MEASURE indicators; of hundred from tables customizedcreate the including information, of and nutrition programs in developing countries. This site provides access program, to a wide range theMEASURE up make health, population, of evaluation and monitoring for information which provide to established projects five of one is Evaluatio n MEASURE the Commonwealth States ofIndependent (CEE/CIS/Baltics). is a public-use database of indicatorssocio-economic for Central and Eastern Europe and women. and children of rights the promoting in involved those to interest as well as links to a number of databases,external on-line resources and organizations of databases, and resources bibliographic internal of number a to access offers Centre The publications inprogress. Analysis s Serie the of part as or books co-edited as published are results Research meetings.through and e-mail bynature cross-national a of INTEVALissues disseminates developmentIDRC-funded research activities. the materials; research of collection IDRC’s on information provides database Librar The level corporate the and areas programming 3 IDRC’s of each to related links The they face. Highlights of this site include problems the in similarities and differences consider to forum a providing communities evaluation different the between and groups various between bridges builds UKES The for evaluators, commissioners ofevaluations, orresearch partners. of database interactiveFrench). The the Their Also includedisinformation onpastconference Lanka. Sri s and the journal journal the SEVAL-Bulletiand n est cnan IR’ eauto rprs eetoi rsucs and resources electronic reports, evaluation IDRC’s contains website Evaluation evaluation standards are available online in German and French. Publications include y provides public access to development information via two databases: the . The site also provides information on the group’s mandate, members, and STATcompiler - Gesetzgebung & Evaluation (in German and German (in Evaluation & Gesetzgebung - LeGes allows you to post your profile or search or profile your post to you allows evaluators ) ) The Evaluator (UKES Newsletter), links to a oln dtbs to alwn ues to users allowing tool database online an , ) . National Evaluation Polic . New Comparative Policy Comparative s; ; and Publications; TransMONEE on IDRIS s journal BIBLIO IDRC IDRC y for , 111 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 112 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs - Resources for inEvaluation Methods Research andSocial - ReliefWeb - PARC –Performance Assessment Resource Centre - Online Evaluation Resource Library - NewsMandE - Education Resources Information Center (ERIC) - EUFORIC -Europe's Forum onInternational Cooperation EVALUATION CLEARINGHOUSES e.g. surveys, focus groups, sampling, andinterviews. and social research. The focus is on how to do evaluation research and the methods used, Compiled by Gene Shackman, Ph.D., this site lists free resources for methods in evaluation funding resources. to as well as databases training humanitarian other to provided are Links country. and date course keyword, name, agency by searchable is Database TrainingReliefWeb The various parc documents to download, including theirnewsletter. offers section publications the and evaluation development international of world the across learning improving and sharing knowledge to devoted is section E-learning The and annotated listings of search engines to find online evaluation guidelines and reports. terminologies, evaluation used commonly of lexicon a service, news a provides site This (on Evaluations Project for Criteria Criteria for andQuality Reports. design, andutility) technical quality Quality the are interest particular Of community. webthe resourcesusing evaluationforumthe a in forand dialogue ongoing stimulating practice evaluation improving how on guidelines areas; several in evaluations project currentfromand instruments past and plans,of reports provides OERL largecollection a open forum for are opportunities. sections M&Evacancies andconsultancy the in Included objectives. development social with projects development to relevant methods M&E in developments on focuses News MandE graphics), (minimal e-mail by accessible be to designed (NGOs) Organisations Non-Governmental for service news A and thePractical Assessment, Research &Evaluation onlinejournal. Library Internet TextFull ERIC/AE the to access include Highlights use. test responsible encourage to resources and assessment educational on information provides site This the of Department Evaluation Operations and Policy ofForeign IOB Ministry Netherlands AffairsDutchRoyal andthe TropicalInstitute (KIT). the Euforic, of venture cooperative a Forum, Evaluation the including forums multi-actor several hosts Euforic and thematic specific activities. and policies cooperation country development international Europe's briefings, on information dossiers, to access online provides site The BACK TO TABLE OF CONTENTS BACK TO TABLE OF CONTENTS To • Empowerment Evaluation To • EVALCHAT Forum • ARLIST-L INTERNET-BASED DISCUSSIONGROUPS To • EVALTALK EvalNet • EVALNET The • ReBraMA (RedeBrasiliense deMonitoramento &Avaliacao) An • GOVTEVAL To • XCeval To [email protected]. to e-mail subscribe thislistserve, • EVALUER The • PREVAL Australia. SUBSCRIBE message: University, Cross the Southern of Dick Bob by with Sponsored . ARLIST [email protected] e-mail subscribe, To list. mailing the American Evaluationthe American Association. EMPOWERMENT [email protected] (username@hostname). Sponsored by Sponsored by the Office of Evaluation and Oversight (OVE) of of the Inter-American (OVE) Oversight and of Evaluation Development Bank(IDB). Office online. is the Registration by Caribbean. the Sponsored and America Latin in projects development of Sponsored by theU.K. Evaluation Society. address]. e-mail [your evalchat Subscribe only read should message The Subject. the in including thesubscriber’s address andname. Also inSpanish. e-mail America North and Europe. Subscribe online or send a message to [email protected] com. Sponsored by The Association ofInternationalandCross-Cultural Evaluators. .Sponsored by Evaluation theAmerican Association. by Evaluation theAmerican Association. Sponsored address]. e-mail [your GOVTEVAL SUBSCRIBE message: the with nasionet.net majordomo@ evaluation. Toprograme-mail subscribe, sector public to related issues in name, andemailaddress. surname Sponsored by UNICEFBrazil. Segone at msegone@unicef.org or Maria do Pilar Joffily at [email protected] with your Marco e-mail listserve, this to subscribe To([email protected]). subscribe, subscribe, unmoderated join subscribe, NETWORK network for the serves the list go e-mail maintains gathers e-mail Theory to send as global http://home.wmis.net/~russon/icce a [email protected] a forum more [email protected] message and electronic a bilingual than Practice for practitioners 600 to: discussion website [email protected] professionals of Action (English group and Research, with from open academics the and or with Latin e-mail to message: Spanish) a anyone multidisciplinary America the XCeval-subscribe@topica. interested With message: involved SUBSCRIBE and and Evalchat-request a the discussion in or SUBSCRIBE Caribbean, evaluation interested electronic EVALTALK list 113 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs 114 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs deseguimiento, Red evaluación ysistematización Latina enAmérica yelCaribe • ReLAC o Coeain n vlain IC) i coeain ih h ME ewrs f Brazil, of networks M&E Central Colombia America, andPeru, ofUNICEFandPREVAL. andwiththesupport the with cooperation in (IOCE), Evaluation in Cooperation for Organization International the by 2003 September in up Set address. email and surname To subscribe, please send an email to [email protected] with your name, BACK TO TABLE OF CONTENTS BACK TO TABLE OF CONTENTS LIST OFKEYCONTRIBUTORS TO THIS GUIDE(by order) alphabetical ANNEX C • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • World vision WFP VOICE VENRO USAID UNICEF UNHCR UNDP SPHERE SC-UK Reliefweb Punto.sud OXFAM GB OECD/DAC OCHA NRC MSF-B Ministère desaffaires étrangères, France Merlin Malteser International IRD IFRC ICRC HAP-I Groupe URD ECDPM DG EuropeAid DG ECHO DFID DANIDA CARE ALNAP ACF 115 Prolog Consult Evaluation of Humanitarian Aid by and for NGOs

LINKS

Links, hyperlinks and documents of reference (Sequentially numbered as in the text) Link 1: BACK http://www.odi.org.uk/ALNAP/ The Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP) was established in 1997, following the multi-agency evaluation of the Rwanda genocide. It is a collective response by the humanitarian sector, dedicated to improving humanitarian performance through increased learning and accountability. ALNAP publishes annual reviews of the effectiveness of humanitarian actions (its most recent Review suggests that there is not enough learning from past experience, resulting in the repetition of many of the same mistakes). Its website includes a repository of various evaluation reports and other resources. The network has also produced corresponding training modules for both evaluators and agencies. Link 2: BACK Why evaluate? Some triggers that may indicate an evaluation is needed appropriate include the following: • Performance monitoring indicates there are unexpected results (positive or negative) that need to be explained. • A key management decision must be made and there is inadequate information. • Annual performance reviews have identified key questions that need to be answered. • Beneficiary or partner feedback suggests that there are implementation problems or unmet needs. • The contribution of (evaluating agency, donor, stakeholder) activities to results is questioned. • Issues of sustainability, cost-effectiveness, or relevance arise. • The validity of results frameworks hypotheses and critical assumptions is questioned. • Recommendations for actions to improve performance are needed. • Extracting lessons is important for the benefit of other operating units or for future programming. Link 3: BACK The following table highlights some of the differences between audit, monitoring, and evaluation.

Audit Monitoring Evaluation • Episodic • Periodic (systematic) • Usually episodic • Checks against • Assumes appropriateness • Can address a wide compliance to pre- of programme, activities, range of potential determined rules and indicators questions about a policy, procedures, with focus • Tracks progress against programme or project on appropriate use of small number of targets • Can identify what has resources or indicators happened as a result • Standardised procedures • Usually quantitative of an intervention and • Primary focus on • Uses data routinely provide guidance for accountability, assurance, gathered or readily future directions and compliance obtainable • Can address “how” & • Always external to the • Cannot indicate causality “why” questions programme • Difficult to use for impact • Can use data from • Method generally limited assessment different sources and to selective review of • Usually internal from a wide variety document¬tation and of quantitative and perhaps a limited number qualitative methods of interviews • Can identify unintended as well as planned impacts and effects • Can involve internal, external, or self evaluation Link 4: BACK The DG ECHO Monitoring Methodology should be available as from May 2008 on the DG ECHO Thematic Evaluations website: http://ec.europa.eu/echo/evaluation/thematic_en.htm Link 8: BACK

EUROPEAN COMMISSION DIRECTORATE-GENERAL FOR HUMANITARIAN AID – ECHO

ECHO 0/1 – Evaluation Sector

ANNEX I

Terms of Reference for: A methodology for the Evaluation of Humanitarian Aid

Contract n°: ECHO/ADM/BUD/2006/0

Name of consultant(s):

Firm:

1. INTRODUCTION 1. Under this ToR ECHO intends to construct a methodology for the evaluation of humanitarian aid. This work is to adapt and extend the methodology for evaluation recently developed by DG AIDCO, for use in evaluating other types of external aid, to become a tool that can be used in the evaluation of humanitarian actions. DG AIDCO's on-line methodology may be accessed at the following address http://ec.europa.eu/comm/europeaid/evaluation/intro_pages/methods.htm 2. DG ECHO wishes to develop this methodology in consultation with its NGO partners. Among the primary objectives DG ECHO wishes to realise are: • an increase in the timeliness of information available to humanitarian aid decision makers, by increasing evaluation capacity in the sector; and • an improvement in the quality of information available to decision makers, by clarifying issues and promoting the use of a standardised methodology. 3. DG ECHO wishes to promote accountability and lessons learning. In order to promote better accountability, the existing criteria used in humanitarian aid have to be examined. At present, the majority of humanitarian aid evaluations are based on OECD DAC criteria, i.e. relevance, efficiency, effectiveness, impact and sustainability. In the case of DG ECHO, certain criteria are also examined in accordance with the Maastricht Treaty, i.e. coherence, coordination and complementarity. 4. However, it is recognised that there are problems with the application of OECD DAC criteria in humanitarian aid, e.g. impact is hard to measure, is sustainability a pertinent criteria in emergency operations, should not neutrality, independence and impartiality be examined? 5. Neither are cross-cutting issues systematically examined e.g. protection, gender, children, elderly and disabled people, environment, consultation of beneficiaries; civil-military interface, access, advocacy and visibility. Nor are entry and exit strategies systematically appraised. 6. The question of how should humanitarian organisations assess themselves must also be considered? Consider for example, http://eipa.euro-ip.net/CAF/Brochure/CAF2002_Eng.pdf What monitoring or feed-back mechanisms should be used? What are their key features? 7. The consultants chosen will examine the criteria used in humanitarian aid, then produce a methodology for use both by donors and implementing organisations based on the DG AIDCO tool. The methodology is to develop questionnaires based on the criteria to be proposed by the consultants. The questionnaires will contain objectives, judgement criteria and examples of

Terms of Reference 1/5 – ECHO/ADM/BUD//2006/

to be continued >>>

NEXT PAGE >>> continued from link 8

indicators for each criterion, at each phase of intervention. The methodology is to consider the application of the criteria over the whole of the project cycle, from needs-assessment through to exiting. Needs based versus rights based approaches must be considred. The methodology must also cover the characteristics and components of monitoring mechanisms. The questionnaires must also cover issues relating to medical, watsan, nutrition and shelter interventions. 8. To do this the consultants must analyse DG ECHO’s mandate, relevant Maastricht requirements, standards such as the UN’s and SPHERE’s, HAP's accountability initiative, the Good Humanitarian Donorship initiative, DG AIDCO's evaluation methodology http://ec.europa.eu/comm/europeaid/evaluation/intro_pages/methods.htm URD's Compass tool http://www.projetqualite.org/compas/outil/ ODI ALNAP's Guide for evaluating humanitarian action http://www.alnap.org/publications/eha_dac/pdfs/eha_2006.pdf and various other tools in the public domain. 9. The consultants are also to support the methodology by documenting and attaching tools already in the public domain.

2. OBJECTIVES OF THE REVIEW AND TASKS TO BE ACCOMPLISHED 10. Under Article 4 of the Humanitarian Regulation DG ECHO may also finance: general studies... including the exchange of technical know-how and experience by European humanitarian organisations and agencies, or between such bodies and those of third countries. 2.1. Objective of the review 11. The overall objective of this review is to strengthen the evaluation capacity of humanitarian organisations by establishing a standard methodology for the evaluation of humanitarian aid. 2.2. Tasks to be accomplished 12. The basis for the consultants’ opinions shall be: ! their own professional qualifications and experience; ! interviews with key DG ECHO officials, both personnel at DG ECHO headquarters and technical assistants (TAs) based in the field; ! interviews with officials in other EC External Services, DG DEV, DG RELEX and DG AIDCO; ! interviews with officials of UN organisations based in Geneva and elsewhere, and EU NGOs; ! workshops in France, the United Kingdom and at one location in the field, possibly Kenya; and ! reviews of relevant methodologies and tools created by Member States or their agencies, the UN, Red Cross bodies, international and regional entities and NGOs. Consultants will carry out a comparative analysis of the conclusions and recommendations drawn in other publications. 13. Drafting of a Methodology for submission to DG ECHO and its partners. The consultants will need to consider the different criteria used at present and consider whether they should be added to or adapted. Based on this they need to set questions as to objectives, judgement criteria and identify benchmarks and indicators. The experts are required to consider benchmarks and indicators already established by the UN, Red Cross and major NGOs and to incorporate these in the methodology. The consultants must also provide a bibliography of supporting documents and websites that are pertinent for DG ECHO and its partners use. The emphasis is on clarity and conciseness in presentation. 14. In the conduct of their work, the consultants may also be able to draw on the facilities of the French Ministry for Foreign Affairs, which is associated to this review. The consultants may also be able to call upon the support of ALNAP, the secretariat for the 'Active Learning Network for Accountability and Performance in Humanitarian Action. www.alnap.org

Terms of Reference 2/5 – ECHO/ADM/BUD//2006/

PREVIOUS PAGE NEXT PAGE >>> continued from link 8

3. WORK PLAN 15. Briefing in Brussels (maximum 2 days including all travel): A briefing at DG ECHO with the responsible staff, during which all the documents available for the mission and necessary clarifications will be provided by the requesting service and other services of the Commission. 16. Attendance at the 2006 DG ECHO Partners' Conference to host a workshop on evaluation methodologies (maximum 2 days including all travel. 17. Missions to France, United Kingdom, Switzerland, (maximum 8 days including all travel): The senior experts shall undertake these visits in order to have contact with relevant UN and Red Cross officials and workshops with major NGOs. 18. Field work (maximum 7 days including all travel): The senior experts shall undertake a field visit to one DG ECHO office in order to organise a workshop at field level with key international partners and NGOs. They will also have contact with certain of DG ECHO’s technical assistants and officials. 19. Drafting of documents required by the review - the consulting companies bidding will have to propose a number of days for each of the two senior experts. The work has to be accomplished in the budget allowed. The other days are set out to assist with planning and per diems. 20. Presentation of the documents required by the review at DG ECHO (maximum of 3 days for each of the two senior experts including all travel): The two senior experts will make a presentation to DG ECHO management and key staff in 'PowerPoint' of the methodology and supporting documents. This presentation may be combined with a presentation at the Partners' Conference in December 2007. 21. Submission of the final version of documents requested: the experts are to submit their work at least two months before the December 2007 Partners' Conference to allow for review and editing and production of final CD/DVD versions. 4. A METHODOLOGY FOR THE EVALUATION OF HUMANITARIAN AID

22. The review will result in the drawing up of a methodology written in a straightforward manner, in English. The consultants will map relevant supporting documentation in a bibliography and include them on the CD/DVD whenever appropriate. 23. The document format appearing below must be adhered to. • Cover page ! Title: “A Methodology for the Evaluation of Humanitarian Aid”; ! Date of the final version ! Name of the consultants ! Cost of the report in euros ! Indication that “the methodology has been financed by and produced at the request of the European Commission." • Table of contents • Methodology • Annexes, including bibliography and supporting documents 24. DG ECHO requires that 2000 CD/DVDs incorporating the methodology and supporting documentation be supplied. The design quality must be of a professional level (inter alia using a desk top publishing tool to incorporate visual images and clickable links in the final pdf version). The recent DG ECHO Watsan Review may be used as an example of what DG ECHO expects to receive, http://ec.europa.eu/echo/evaluation/thematic_en.htm#water The consultants will have to include this cost in the budget to be submitted. 25. Once established, DG ECHO will have the methodology translated into French and Spanish.

Terms of Reference 3/5 – ECHO/ADM/BUD//2006/

to be continued >>>

PREVIOUS PAGE NEXT PAGE >>> continued from link 8

5. REQUIRED SKILLS FOR THE CONSULTANTS 26. DG ECHO envisages that two senior experts shall carry out the work. DG ECHO considers that the consultants proposed should preferably have work experience in implementing humanitarian aid emergency or relief operations at field level, experience of drafting research, policy or strategy papers and/or substantial experience of evaluating humanitarian aid. 27. All experts should be able to draft in English, but knowledge of French by at least one member of the team is essential. 28. DG ECHO will make available office space at its HQ and one field office to facilitate the consultants’ work. Telephone conferencing to other entities and organisations may also be used where necessary.

6. ASSIGNMENT OF TASKS 29. Each team member is jointly responsible for the final accomplishment of the tasks; however, the separate elements of work necessary for the accomplishment of the tasks may be allocated between the consultants. The members of the team must work in close co-ordination. 30. A team leader shall be named who shall have the added responsibility of the overall co- ordination of the tasks to be completed and of the final coherence of the report and other works, both in terms of content and presentation.

7. TIMETABLE 31. The tasks under this evaluation will be undertaken by two senior experts and will be completed between December 1st 2006 and December 31st 2007, i.e. the consultants will use a workshop at the DG ECHO Partners' Conference 2006 to launch their work, and will finish with a presentation at the Partners' Conference in 2007.

32. Please note that, this contract will have to be presented for liquidation by mid-December 2007 at the latest. After the year-end 2007 liquidation will not be possible.

Terms of Reference 4/5 – ECHO/ADM/BUD//2006/

to be continued >>>

PREVIOUS PAGE NEXT PAGE >>> continued from link 8

Terms of Reference – Annex

Guidelines for the consultants

1. Regulatory basis

The Regulatory basis for the evaluation of the aid provided by ECHO is established in Article 18 of Regulation (EC) 1257/96 concerning humanitarian aid, which states "the Commission shall regularly assess humanitarian aid operations financed by the Community in order to establish whether they have achieved their objectives and to produce guidelines for improving the effectiveness of subsequent operations".

2. Terms of Reference. What are they?

The Terms of Reference set out the scope of the evaluator's mission, the issues to be considered and the evaluation timetable. They allow those commissioning the review to express their needs (guidance function) while providing the consultant with a clear idea of what is expected from him/her (control function).

3. Methodology

For the purpose of accomplishing its tasks, the consultants shall use information available at ECHO HQ, from its experts on the spot, from other Commission services, the relevant Commission Delegation, ECHO partners on the spot and, if necessary, at their headquarters, aid beneficiaries, local authorities and international organisations.

4. Scope of the evaluation and topics of study

In addition to the initial information contained in the ToR, the first briefing session in Brussels provides everyone involved in the review (requesting service, ECHO-Evaluation, consultants and other Commission services) with the opportunity to discuss the contents of the ToR and to establish priorities for the review. This meeting should, as well, allow the consultants to clarify any doubts they might have about the scope of their mission. Any important remark or comment on the content of the ToR at this stage will be considered an integral part of these and will be set out by the consultants' team leader in a note that must be submitted to ECHO-Evaluation at the end of the briefing session, and before the team's departure to other locations in Europe and elsewhere.

During the process of the review the consultants must try to follow all the items listed in the Terms of Reference. Their treatment, the relative importance given to them and their coverage in the final reports will depend, however, on the consultants' own opinion as a result of the information found, both during the desk study phase and in the field. Any decision not to cover one or more of the main task assignments described in the ToR will have to be justified in the text of the reports, if inappropriately justified ECHO may choose to not accept the final report.

5. The report

By commissioning an independent review ECHO expects to obtain an objective, critical, readable and transparent analysis of its policy. This analysis should contain policy recommendations on future courses of action. The report should be, above all, a document that can function as a learning tool. Therefore, while writing it, the consultants should always bear in mind why the report is done, for whom, and how the results will be used.

Furthermore, the report is a working tool of value to ECHO only as long as it clearly reflects the consultant's independent view. ECHO's greatest concern is to respect this independence.

Terms of Reference 5/5 – ECHO/ADM/BUD//2006/

PREVIOUS PAGE Link 12: BACK ICRC check list Evaluation Guidelines (internal), April 2002

Proposing and Deciding to evaluate – A check list Decision while at the planning stage - When planning future components. - When new types of activities are launched. - When activities are in a pilot phase. - When activities and objectives have high political and institutional priority. Decision after the planning stage - When an operation is problematic. - When an operation is running successfully (good practices). - To determine reasons behind unforeseen consequences. - To determine the future of an operation. - When new elements emerge in a given situation. - When a similar action is planned elsewhere. - If requested by donors. Decision not to evaluate - When an operation develops as foreseen, and enough is known about its performance and results. - When enough is known about impact, problems, causes and solutions. - When the context is not conducive to the undertaking of such an exercise (security reasons for example). - When it seems that an evaluation will not be used. Link 14: BACK Setting objectives by using the Logical Framework Analysis (LFA) LFA is a standard feature of DG ECHO's "single form" (proposal format) http://ec.europa.eu/echo/pdf_files/fpa2008/fpa_single_form_en.pdf, so let us presume that every implementing partners is fully conversant with the use of this eminently customer- friendly document. For those who might still nonetheless hesitate, it should be noted that WFP has published an interesting approach on how to link practically the LFA to the methodology of evaluation questions ("Link" supprimé), as illustrated by the table below (http://documents.wfp.org/stellent/groups/public/documents/ko/mekb_module_7.pdf)

Logframe Evaluation questions Impact Impact Sustainability - What changes - Are the benefits did the operation likely to be bring about? maintained for an - Were there any extended period unplanned or after assistance unintended ends? changes? Outcome Effectiveness Relevance Outputs - Were the - Were the operation's operation's objectives objectives achieved? consistent with - Did the outputs beneficiaries' lead to the needs and with intended agency's and outcomes? donor's policies? Activities Efficiency Inputs - Were the goods and/or services available on time and in the right quantities and qualities? - Were activities implemented on schedule and within budget? - Were outputs delivered cost-effectively? Link 15: BACK Articulate the theory of change or INTERVENTION LOGIC The theory of change (frequently referred to as “logic model” or “intervention logic”) graphically illustrates the series of assumptions and links identifying the presumed relationships between inputs (e.g. funding, staff, volunteers, tangible and in-kind support from others, etc.), activities and their immediate outputs (i.e. what is done, such as provision of food aid), intermediate outcomes at various levels (e.g. the right individuals get appropriate food, starvation and malnutrition reduced), and the intended impact – self sufficiency and reduction of vulnerability of the target population. The theory of change also should illustrate the relationship between the various interventions of your organisation itself and those of other humanitarian actors, and how these interact with what else is going on that can affect the lives of your target population and communities. Articulating the theory of change of your intervention represents a crucial step. This is frequently viewed in the evaluation community as a prerequisite to most evaluations. (E.g. see the discussion of intervention logic in the DG EuropeAid evaluation guidelines http://ec.europa.eu/europeaid/evaluation/methodology/methods/mth_log_en.htm). It is not always so easy to develop, as it generally requires identifying hidden assumptions and beliefs, and often reconciling differing views of how the overall programme is expected to work by different stakeholders. Sometimes it can be useful to have an external facilitator or evaluator to help with this process. A model of theory of change represents a tool that is used for programme planning (LFA) and also is essential for designing the approach to evaluation and impact assessment. It can help in understanding the “logic” behind the proposed interventions and how these relate to the overall goal (that can indeed include long-term development and sustainability following alleviation of the immediate emergency situation). This will help in programme development and implementation. Sometimes this process can identify logical gaps, e.g. where it is not clear how a given activity would contribute to the overall desired outcome, or where it becomes apparent that other interventions will also be required. If these gaps can be identified and addressed at an early stage, this can contribute to the success of the intervention. But a theory-of-change model can also represent a very useful tool in simplifying your actual approach to evaluation – and in helping make sure that you do not set yourself up for failure. Humanitarian aid actors are invariably optimistic about what they hope to achieve in the face of formidable challenges. A theory-of-change model, but identifying sequential outcomes at various levels, can be a useful tool to help you focus your evaluation on what is realistic to achieve at the current time – while still showing how this is expected to lead to longer-term goals. Link 18: BACK The first question aims at assessing the appropriateness of what is being done, e.g. taking into account the rationale and relevance of the intervention, and how this is viewed by the intended beneficiaries. The second question addresses "the right way" of doing things or more specifically the implementation of the intervention as well as the results or outcomes, i.e. the efficiency of an organisation, and its capacity to reach expected results, i.e. its effectiveness. The third question gets to the raison d’être of evaluation – how to make use of the evaluation findings in some way, that can vary from minor fine-tuning to current operations, identifying more efficient methods of operating, to identifying very different approaches in the current or perhaps future contexts for addressing humanitarian challenges. Link 21: BACK The key factor of risk assessment for humanitarian activities should be reflected in the evaluation questions and in the overall methodology. There is always a risk, at the time of evaluation, that the expected outcomes will not be realised or maintained. This must be considered in its two dimensions: (a) the likelihood that that some changes may occur that are detrimental to the ultimate achievement of the expected outcome and (b) the effect of the expected outcome if some or all of these changes actually materialise. Risks may be internal to the program or arise from external factors at the local, country, or even global levels. The actual effect of these risks on the ultimate outcomes will depend on both the severity and nature of the changes that occur and on the adaptability (or lack thereof) of the design of the program and its activities. Ideally, the potential risks to the expected outcomes should have been identified at the inception of the program and pertinent indicators for their monitoring included in the LFA (DAC). It should be noted that the URD 'Quality Compass' uses "sentinel indicators", which may be seen as indicators pointing at potential risks. Link 27: BACK Classification of humanitarian principles according to HPA-I Primary principles • Humanity: upholding the right of all persons to receive and give assistance. • Impartiality: providing humanitarian assistance in proportion to need and with respect to urgency, without discrimination based upon gender, age, race, impairment, ethnicity and nationality or by political, religious, cultural or organisational affiliation. Secondary principles • Informed Consent: ensuring that the intended beneficiaries, or their representatives, understand and agree with the proposed humanitarian action and its implications. • Duty of care: ensuring that humanitarian assistance meets or exceeds recognised minimum standards pertaining to the wellbeing of the intended beneficiaries. • Witness: reporting on policies or practices that affect the wellbeing of disaster survivors. Tertiary principles • Transparency: ensuring that all relevant information is communicated to intended beneficiaries or their representatives, and other specified parties. • Independence: acting under the authority of the governing body of the agency and in pursuit of the agency’s mandate. • Neutrality: refraining from giving material or political support to parties to an armed conflict. • Complementarity: operating as a responsible member of the humanitarian assistance community. Link 32: BACK The HAP-I Principles of Accountability 1. Commitment to humanitarian standards and rights • Members state their commitment to respect and foster humanitarian standards and the rights of beneficiaries. 2. Setting standards and building capacity • Members set a framework of accountability (standards, policies, training etc) to their stakeholders* • Members set and periodically review their standards and performance indicators, and revise them, if necessary. • Members provide appropriate training in the use and implementation of standards. 3. Communication • Members inform, and consult with, stakeholders, particularly beneficiaries and staff, about the standards adopted, programmes to be undertaken and mechanisms available for addressing concerns. 4. Participation in programmes • Members involve beneficiaries in the planning, implementation, monitoring and evaluation of programmes and report to them on progress, subject only to serious operational constraints. 5. Monitoring and reporting on compliance • Members involve beneficiaries and staff when they monitor and revise standards. • Members regularly monitor and evaluate compliance with standards, using robust processes. • Members report at least annually to stakeholders, including beneficiaries, on compliance with standards. Reporting may take a variety of forms. 6. Addressing complaints • Members enable beneficiaries and staff to report complaints and seek redress safely 7. Implementing Partners • Members are committed to the implementation of these principles if and when working through implementation partners. Link 33: BACK Evaluation Types (Source: DAC (2001) Glossary of Evaluation and Results Based Management Terms DCD/DAC/EV(2001)3 Working Party on Aid Evaluation, Paris: OECD)

Thematic One theme, such as gender or environment, across a number of projects or country programmes, or across the organisation as a whole. Often called cross-cutting evaluations undertaken to develop or refine policy. Sector Aid to a particular sector, such as health or public nutrition. As with thematic evaluations, these cover a number of projects and country programmes. Sector Programme Sector or programme aid to a particular country. Country Programme All types of aid (project and non-project) to one country. Synthesis A synthesis of the findings from a number of evaluations of individual projects or programmes. Impact Focuses on the impact of the aid, rather than on aid delivery. Usually, but not always, carried out some time after project completion. Policy Examines the framework of understanding, beliefs and assumptions that make individual projects possible as well as desirable. Policy evaluations seek out the inherent tensions or contradictions in policy objectives through tools such as discourse analysis. BACK TO PAGE 34 BACK TO PAGE 35 BACK TO PAGE 40

Link 34: COMMON RAPID APPRAISAL METHODS (source: Performance Monitoring and Evaluation TIPS n°5 "Using Rapid Appraisal Methods", USAID, 1996)

METHODS Useful for Advantages Limitations Providing KEY - general, descriptive - provides in-depth, - does not generate INFORMANT data inside information quantitative data INTERVIEWS - understanding - flexibility permits - susceptible to of attitudes and exploring interviewer and behaviours unanticipated topics selection biases - suggestions and - easy to administer Recommendations - relatively inexpensive - information - takes 4-6 weeks to interpret quantitative data FOCUS - customer views on - can be completed - does not provide GROUP services, products, rapidly (5 weeks) quantitative data INTERVIEWS benefits - very economical - discussion may be - information on - group discussion dominated by a few implementation may reduce individuals problems - inhibitions, allowing - susceptible to - suggestions and free exchange of moderator biases recommendations ideas for improving activities COMMUNITY - village/community - permits direct - can be manipulated INTERVIEWS level data interactions by elites or - views on activities between evaluator monopolized by and suggestions for and large numbers individuals improvements of individuals - cultural taboos or - can generate some norms may inhibit quantitative data discussion of certain on community topics characteristics, behaviours, opinions - participants tend to correct each other, providing more accurate information - inexpensive and quick (5-6 weeks) DIRECT - data on physical - phenomenon can - susceptible to OBSERVATION infrastructure, be examined in its observer bias supplies, conditions natural setting - act of observing can - information about - may reveal affect behaviours an agency's delivery conditions or - distortions can occur systems, services problems informants if sites selected are - insights into are unaware of not representative behaviours or events - can be completed in 3-4 weeks to be continued >>>

NEXT PAGE >>> continued from link 34

METHODS Useful for Advantages Limitations Providing MINISURVEYS - quantitative - can generate - findings are less easy data on narrowly quantitative data tp Generalise from focused questions - reduces non- than those from for a relatively random sampling sample surveys homogeneous errors - requires statistical population - requires limited analysis skills - when probability personnel and is - susceptible to sampling is difficult quick (5-6 weeks) sampling biases - data on attitudes, - inappropriate for beliefs, behaviours gathering in- of customers or depth, qualitative partners information

PREVIOUS PAGE Link 36: BACK Source: Guidelines for CARE Country Offices planning After Action Reviews and/or External Evaluations of Humanitarian Actions. Draft ver. March 3, 2006 Checklist for Joint (Multi-Agency) Evaluations 1. Identify participating agencies and which will be “Lead” Agency to coordinate the evaluation. 2. Lead agency should convene an initial meeting of an Evaluation Steering Committee composed of representatives from each participating agency and they should try and agree on: a. Maximum of 3-4 key objectives (i.e. questions that they would like the evaluation team to particularly focus on). See draft TOR as an example; b. Based on these key objectives, a desired skill set for the team members should be agreed upon and the “TOR for Evaluation Team Members” (also attached) revised accordingly; c. A draft itinerary/timeline that should allow sufficient time for orientations/ briefings, field visits (including to hard-to-access areas), follow up interviews with “key informants”, including in-country senior staff, govt. officials, UN agencies, etc., in-country debrief, telephone interviews with HQ-level key informants, drafting of report, circulation for commentary, and finalization of report; d. Based on the draft itinerary, the Steering Committee should agree on a draft budget and cost-sharing arrangements (typically agencies share consultant costs equally and provide funding for their own agency team member; and e. Decide how the evaluation can best be used to not only improve inter-agency coordination, programme quality, but also advance common advocacy strategies. It’s better to discuss this at an early stage since “use of evaluation” should be included in the draft TOR and such discussions are helpful in identifying who the results of the evaluation should be targeted at and strategies for communicating these results. Look for opportunities where lessons learned could be fed into strategic planning activities. 3. Initiate the search for evaluation team members with suitable profiles. This again should be done as early as possible since good candidates can be booked up many weeks (or months) in advance. Apart from adequate preparation for the evaluation, the selection of a good Team Leader who has significant experience of managing teams will be one of the single most important factors in determining the quality of the result. to be continued >>>

NEXT PAGE >>> continued from link 36

Guidance notes for Joint Evaluations 1. Evaluation Steering Committee: - It’s important for the Steering Committee to meet well in advance of the evaluation because of some important preparations that need to be done at an early stage. One potential spin-off is to encourage agencies to work jointly on M&E and the committee’s focus on programme quality and accountability might actually help in not infrequent cases where the emergencies have caused strained inter-agency relationships. 2. Membership of Evaluation Steering Committee – previously, member agencies of the steering committee have also had staff on the evaluation team, but this does not need to be a requirement. Generally agencies that are willing a) to have their programmes subjected to an external review, b) contribute both in staff time and financially, and c) make a commitment to following-up as needed on results of the evaluation can qualify as members of the Steering Committee. Fewer, shorter meetings with senior staff (in Niger, CD or ACD level was seen as the most useful) are seen to be more productive than more frequent meetings at a junior level. 3. Lead Agency – the lead agency does not “do” the evaluation, but rather facilitates the process. The Lead Agency’s main roles are: a. Facilitation (convene and facilitate meetings, taking minutes, coordination of logistic support for evaluation team); b. Administration (contracting consultants and managing cost-sharing arrangements); and c. Information Management (collation of key background reference material, facilitation of communications between participating agencies and evaluation team). 4. Members of Evaluation Team – a good Team Leader with substantial team management experience and good drafting skills will be critical. A national consultant will ensure that the study remains relevant to the local context. The team members, who have been agency staff from outside the country. Apart from broadening the range of technical skills, agency team members should also help to inform the Team Leader about how their respective organizations function. 5. Mix of technical skills – the Team Leader will be an M&E expert, but this doesn’t mean that all the other team members should be M&E specialists. Of course, it’s useful for them to have a basic knowledge of how M&E processes work but, as an example, the technical expertise that was most appreciated by the Team Leader in Niger within his team was someone with a good finance/admin background and another team member who had a background in economics of local markets. It would certainly be a plus if the person has been previously involved in evaluations (perhaps their programmes have been evaluated), otherwise the ideal profile will be someone who’s worked in the field and has good skills in analysis, observation, interviewing skills (especially open-ended techniques), and drafting. Note, however, that as teams get larger they quickly become difficult to manage. Probably aiming for a total team of 5 with the two external consultants (not including drivers or interpreters for speaking to indigenous groups) would be reasonable.

PREVIOUS PAGE Link 37: BACK Evaluation Stages and Responsibilities Source: DFID evaluation guidelines - www.dfid.gov

STAGE TASKS RESPONSIBILITY OUTPUTS Planning, - Drafting, circulation and - Evaluation Manager - Concept paper commissioning approval of concept - Evaluation Manager - TOR note (+ Steering Group) - Tender documents - Selection and briefing of Steering Group - Drafting, circulation and approval of TOR - Consultant selection Inception - Briefing consultants - Evaluation Manager - Inception report - Initial research - Consultants - Drafting Inception - Consultants report - Evaluation Manager - Circulation and approval and Steering Group of Inception Report Preparation and - Project/programme - Consultants research research - Consultants - Interviews (UK and - Consultants, email) Evaluation Manager - Planning country visits and Steering Group and dissemination strategy Country visits - Preparatory visit: - Consultants - Visit reports - Planning activities (implementation) - In-country - Draft and circulate visit - Evaluation Manager workshops report (Quality of process) - Main visit: - Evaluation research - Workshop(s) - Draft and circulate visit report Reporting - Draft report and - Consultants - Draft Report Evaluation summary - Evaluation Manager - Circulate report for and Steering Group - Revised Report comment - Consultants & editor - Edit and revise report - Evaluation Manager - Circulation and and Steering Group - Final Report comment on revised - Evaluation Manager report - Evaluation Manager - Submission to and Consultants Development Committee - Final amendments to report Dissemination - Publication and - Evaluation Manager - Published Report distribution of report - Consultants, - Evaluation Summary and Evaluation Steering Group and - Workshops summary Evaluation Manager - Workshops Link 39': BACK Managing Joint or Multi-agency Evaluations (MAEs) (source: ALNAP): The following is a short note on managing Multi-Agency Evaluations (MAEs)2. It sets out some indicative considerations and is not comprehensive. It is not an evaluation guide, rather a complement to such guides focussing on the specificity of MAEs. The note is based on the assumption that the two overall differences between MAEs and other evaluations boil down to, firstly, the number of actors involved (especially in the management of the evaluation) and secondly, the potential breadth of the activities, content or programme to be evaluated (e.g. a multi-actor programme rather than a single agency programme, project or activity level). The note takes a chronological approach, through the main evaluation management steps or phases (allowing, however, that phases may overlap and run parallel). It is based on the assumption that the evaluation is a classical evaluation exercise, involving a team of consultants (or mixed with agency staff), going to ‘the field’, returning and reporting. If the exercise involves a number of constituent evaluations or studies, each one might be managed in the following manner, within a larger, overall structure and process. Considerations and key questions are set out as bullet-points under each step: • Assessment and scoping of the value of conducting conduct an MAE - What is the added-value of a joint exercise as opposed to individual exercises, especially in function of its ultimate usage/user-focus? If so, what might those uses be, aimed at what target groups/audiences? - Is there sufficient buy-in for an MAE, including understanding of the complexities, costs and benefits of an MAE? - Can the multiple possible stakeholders to be mapped, prioritised and consulted and by whom and by when will they be consulted? • Establishing a multi-agency management structure - Can the main actors be identified and committed to the process, e.g. through the unambiguous provision of time and resources? - Can a lead or host agency be identified? This is to provide a legally established umbrella organisation e.g. that the MAE be under the aegis of one of the constituent organisations, as a host, providing a physical ‘home’; administrative and contracting body (e.g. for the team); accounts and legal status; etc. - Based on the identification of those actors, what is the most effective and efficient management structure? This will probably be multi-layered, including: an overall MAE group, including an appropriate chairperson; a smaller management sub-committee (e.g. 3 – 5 people, ideally including the overall chairperson); a day-to-day manager (who would also sit on the management sub-committee, but without voting rights); an administrative and coordination secretariat (based in the host agency, to support the manager and entire under-taking); and the evaluation team. - An explicit agreement on roles, responsibilities, rights and obligations of all concerned. This includes fundamentally who holds the ‘ownership’ of the process and its products

2 This short note reflects comments made by the author, John Telford, in response to a number of requests and initiatives; a roundtable meeting on the subject of joint evaluations, sponsored by ALNAP, DEC, ECB2 and the TEC in 2006; a review of the Tsunami Evaluation Coalition (TEC) exercise, also held in 2006; and a draft guide being produced by the Emergency Capacity Building (ECB) project. to be continued >>>

NEXT PAGE >>> continued from link 39'

(e.g. the report). Ownership entails a series of aspects, from legal rights and obligations, to decision-making authority, especially in the event of disputes. - Agreement and formalization of procedures for relevant aspects of the process, including establishment of the management structure; dismantling of the structure (including consideration of future ownership of the products of the MAE); a disputes- management mechanism; and financial, material and human resource management (including contracting the evaluation team and other resource provision, especially for day-to-day secretariat functions). • Designing the MAE and TOR - Managing and focusing multiple possible uses, needs and expectations (links to assessment and scoping of needs above). This includes prioritization of the possible target users and audiences for the possible products coming from the evaluation. - Delegating authority to the management sub-committee to make these decisions with the confidence and support of the broader MAE group, subject their review and approval. • Team selection, preparation and planning - Decision by the MAE group and/or sub-committee on the nature and size of the team. - Delegation of selection to the management sub-committee, based on accepted standards of professionalism, independence and transparency. - Close coordination, preparation and planning among the management sub-committee, contracting (host) entity, MAE manager and the evaluation team. In function of the scoping and TOR, this includes the transparent selection of locations to be visited, possible stakeholders to be involved, • Conducting the MAE, including analysis and reporting - Application of methods and availability of time and resources in accordance with the possible breadth of aspects, issues and locations to be covered. - Allowing sufficient time and ‘space’ (e.g. workshops) for the above and for team and stakeholder analysis of what may be very considerable materials and findings emerging (possibly more than in a single-agency undertaking). - Clear agreements on when, how and by whom the draft reports will be reviewed. Of particular importance is explicit agreement on the authority of reviewers, especially if they are from within the overall management group i.e. are certain types of comments to always to be acted upon (such as errors of fact or inadequate verification) while others are of an advisory nature only (e.g. interpretations or analysis). • Dissemination and use - Agreement (well in advance, preferably at the outset) on the number and type of products that will result from the MAE (according to the diverse sets of target groups/ audiences).

to be continued >>>

PREVIOUS PAGE NEXT PAGE >>> continued from link 39'

- A usage, dissemination and communications plan for the outputs of the exercise. This would explain whether the process is centralized (managed by the sub-committee and/or the secretariat, or some such group), de-centralised (among all the agencies involved and possibly others), or a mixture of both. Given the complex range of possible stakeholders and locations involved, planning may need to be delegated to a sub-committee. - Resources available for the implementation of the plan, and for unforeseen costs. - A follow-up plan, including whatever activities are seen to be relevant, as decided by the overall MAE group. This would be in function of the initial scoping, TOR and results of the MAE. It could imply a new structure and process involving the agencies wishing to take the results forward into a new review and action process. - A review of the MAE itself, recording lessons on the exercise. This would probably require a workshop or one-day meeting of all main actors and stakeholders.

PREVIOUS PAGE BACK TO PAGE 39 BACK TO PAGE 43 BACK TO PAGE 47

Link 42: TERMS OF REFERENCE Independent Evaluation of Care’s Response to the Pakistan Earthquake 1. Background The South Asia earthquake of 8 October 2005 resulted in the loss of an estimated 86,000 lives and considerable damage of the built and natural environment in Pakistan. In all, 4 million people were reported as affected in the north-west frontier (NWFP) and Azad Jammu and Kashmir (AJK). More than 100,000 people were injured, and up to 3 million individuals were in immediate need of shelter and other life-sustaining assistance. An estimated 600,000 housing units were either destroyed or severely damaged. In some areas, close to 100% of the housing stock was destroyed. There was significant damage to roads, schools, health clinics and hospitals and other infrastructure. The vast geographic area affected, along with the rugged mountainous topography and inaccessibility of many populated areas, made a humanitarian response particularly difficult.

CARE Pakistan responded to the Earthquake by mounting a two-pronged relief operation. The first was to work through several strategic partners for immediate distribution of relief materials and provide health care for survivors; and the second to mount an emergency relief operation in the Allai Valley of the North West Frontier Province, one of the hardest hit areas of the Earthquake Zone. CARE Pakistan is currently mounting a reconstruction effort that focuses on shelter support to returning families, livelihood assistance to help families get back on their feet; and reconstruction of schools and start of primary education programs.

This evaluation will assess CARE Pakistan’s immediate response to the earthquake as well the period leading up to the reconstruction phase, with a view to drawing lessons for country office and CI emergency preparedness, disaster risk reduction and future emergency response.

2. Purpose and Objectives of the Evaluation The purpose of the evaluation is three-fold: a) Assess the quality of CARE Pakistan’s response to the earthquake in the Northwest Frontier Province including adherence to Sphere Standards during the response and performance relative to CARE International’s Humanitarian Benchmarks and OECD evaluation criteria. b) Develop lessons learned and recommendations that will assist CARE Pakistan to build disaster risk management and emergency preparedness capacities into future programming in order to help communities better cope with risk, and to enable a more timely and appropriate response to disasters and crises in the future. c) Assess the extent to which CARE Pakistan was able to engage appropriately with the Pakistan military during the emergency response. The evaluation will make recommendations on future CARE policy on civil-military relations. to be continued >>>

NEXT PAGE >>> continued from link 42

Evaluation recommendations will be based on accepted Red Cross Code of Conduct, to which CARE International members are signatories, as well as Sphere Minimum Standards, CARE International’s Humanitarian Benchmarks, CARE International’s Evaluation Policy and OECD evaluation criteria. Some specific areas which the evaluation will examine include: • Timeliness and Appropriateness of response – to what extent did the country office have the capacity, systems and procedures, sufficient human resources and appropriate level of preparedness to facilitate a rapid and appropriate response? How did CARE’s capacity (notably CI members, ARMU and CARE Pakistan) to staff-up affect the quality of the response? Was gender taken into consideration adequately in all relevant areas of the response? • Efficiency – What were the outputs (both qualitative and quantitative) in relation to the inputs? Was CARE Pakistan’s response cost effective? • Impact – Review of the impact of CARE Pakistan’s response in terms of preservation of life and reduction of human suffering. Assessment of the extent to which international standards (e.g., international humanitarian and human rights law; the Red Cross/NGO Code of Conduct) and relevant standards (e.g., Sphere, CI Program Standards) were applied and their impact. Assessment of the impact of the response using a Do-No- Harm lens. • Coverage – scale and ability to reach those most in need, given the political, religious, geographic and social context of the emergency, and providing intended beneficiaries with assistance and protection that is proportionate to that need. • Connectedness and Sustainability – links to local capacity, plans and aspirations and the collaboration and co-ordination with intended beneficiaries (including the effectiveness of communication/feedback systems), within CARE and with external partners.

3. Components of the evaluation report a) Introduction. The context of CARE’s intervention in the earthquake the salient characteristics of the response and their implications. Specific issues for CARE in Pakistan, for example security and profile. b) CARE Pakistan’s decision to engage in the earthquake response. Criteria influencing the decision and the implications of the decision. c) Human resources and management systems. The challenge of expansion from a small development–focused base. Mechanisms used in recruiting or transferring staff. Implications for the organization of the nature of the staff in the short, medium and longer terms. Inter-agency competition/sharing of staff. d) Partnerships. The nature and quality of partnerships with implementing agencies, other NGOs , the UN system and government organizations, including the army. The nature of co-ordination and co-operation and actual modes of operation. e) Community capacities and needs. Community responses in different phases, Building, maintaining and strengthening community capacity. Community participation modes. Community structures. The nature of need assessment at different levels and stages. Prioritisation of needs. to be continued >>>

PREVIOUS PAGE NEXT PAGE >>> continued from link 42

f) Gender. Specific vulnerabilities and limitations on women. Gap identification and gap filling. Specific activities for women. Strategic implications of emergency interventions, Implications for and of human resources past present and future. g) Programming and delivery. Process focus (results in annex). Other stakeholder views, including community. Longer term strategic significance of modes for sustainability. Do no harm principle and accountability. Adherence to codes. h) Logistics. Procurement, delivery mechanisms, accommodation and site development. Specific problems of Allai valley and dependence on scarce helicopter travel and with poor road communication. Telecommunication systems. j) CIMIC. History of relationships and specific problems arising for CARE staff and community. Existing co-ordination mechanisms. Possible future relations in emergency and developmental contexts. k) Preparedness and development. Transition to development. Incorporation of preparedness, risk assessment, vulnerability reduction mechanisms and surveillance systems in the planned development context.

Many issues are relevant in different sections. There will be cross reference between these but no undue repetition. The OECD/DAC evaluation criteria will be used as appropriate in the assessments of each section. Where necessary material will be elaborated in annexes. Findings will be used in the preparation of action-focused recommendations.

4. Evaluation Methodology a) The methodology of the evaluation will include a combination of a desk review of relevant country office documentation, field travel, key informant interviews or focus group discussions with CARE staff in Pakistan (both field and HQ), ARMU and CI. The evaluation team will also interview a selection of beneficiaries in communities and key external stakeholders such as Pakistan government representatives, other international NGOs, and UN agencies. b) Confidentiality of information - all documents and data collected from interviews will be treated as confidential and used solely to facilitate analysis. Interviewees will not be quoted in the reports without their permission. c) Communication of Results – an official report of the evaluation will be prepared. However this report will be supplemented by a presentation of preliminary findings for key stakeholders (both internal and external) to both provide immediate feedback to CARE staff and beneficiaries (?) and give the Evaluation Team an opportunity to validate findings. d) Report: a concise report with focused practical recommendations will be prepared emphasizing both feedback to CARE managers and providing replicable lessons to inform CARE’s disaster risk management and emergency response in future. CARE interviewees will be given an opportunity to comment on the draft reports prior to finalization. While the Evaluation Team will retain responsibility for drafting and editing the report, the Country Office will have the option of making a written response, which will be attached as an annex to the final report. Once finalized, the report will be shared within the CARE world. to be continued >>>

PREVIOUS PAGE NEXT PAGE >>> continued from link 42

5. Evaluation Team Composition CARE Pakistan anticipates that the evaluation team will be made up of 3? to 5 persons including a Team Leader; a Human Resources specialist; a national expert (preferably with expertise in gender); an M&E Specialist and a national M&E Officer (CARE Pakistan staff).

The Team Leader Qualifications : Required : • Extensive experience of emergency management and risk management approaches • Monitoring and evaluation of emergencies • Previous Evaluation Team Leader experience • Good knowledge regarding use of Sphere standards, Red Cross Code of Conduct, beneficiary accountability systems, etc. in humanitarian contexts • First-hand knowledge of the South Asia context • Excellent drafting and communication skills Desired: • Prior experience of CARE relief and development operations • Understanding of the Pakistan context • Experience in managing emergency shelter programs • Gender in emergencies experience • Knowledge of Pashtu and/or Urdu language Other Team member combined experience: • Monitoring and evaluation experience • Strong knowledge of Pakistan context (particularly the Northwest Frontier Province) • Gender in emergencies experience • Strong HR management experience (particularly in emergencies) • Strong emergency management experience (previous experience in earthquake response also desirable) • Knowledge of Pashtu and/or Urdu

6. Use of Evaluation Results The Evaluation will make recommendations to various levels within CARE (e.g. the Country Office, the ARMU, and CARE USA) in order to improve the quality of CARE’s preparedness and response to future emergencies. The target audiences of the evaluation will develop a plan of action based on the evaluation report and its findings within one month of distribution of the final report. An appropriate system for monitoring implementation of recommendations will be agreed by CARE Pakistan, CARE USA/ARMU, and CEG, who will each nominate a focal point to monitor implementation of recommendations. to be continued >>>

PREVIOUS PAGE NEXT PAGE >>> continued from link 42

7. Proposed Timeframe: total of 4 1/2 weeks for the Team Leader and 3 weeks for the other Team Member(s). The evaluation schedule will include:

Activity Approximate Dates Person(s) responsible Desk review End May (2-3 days) Team leader & team members Field Visit to CARE Pakistan First 1/2 of June (3 weeks) Full team (including project sites) Interviews with CI members, Mid-June (2 days) Team Leader RMU Follow-up Interviews Mid-end June(2-3 days) Team leader, M&E and HR Experts Circulation of Draft Report End June Team Leader Final Report Mid-July Team Leader w/ CARE staff (after incorporating feedback on draft) Stakeholder review of End July CO, ARMU, CARE USA, CEG recommendations Stakeholder Plans of Action End July Country Office, ARMU, CARE circulated USA, CEG. Monitoring Implementation of ongoing Country Office, ARMU, CARE Recommendations USA, CEG.

PREVIOUS PAGE Link 48: BACK Examples of data collection methods (Source: Ministry of Foreign Affairs, Denmark, Danida (2006) ‘Evaluation Guidelines’) 1. Literature search Economic and efficient way of obtaining information. Difficult to assess validity and reliability on secondary data. 2. Key informant interviews Flexible, in-depth approach. Easy to implement. Risk of biased presentation/ interpretation from informants/ interviewer 3. Direct measurement Registration of quantifiable or classifiable data by means of analytical instrument. Precise, reliable and often requiring few resources. Registers only facts, not explanations. 4. Direct observation Involves inspection, field visits, observation to understand processes, infrastructure/ services and their utilization. Dependent on observer’s understanding and interpretation. 5. Group interviews Low-cost, efficient. Direct contact with those affected. Susceptible to manipulation and less suitable for sensitive issues. 6. Informal survey Involves quantitative surveys of small samples. Reasonable and rapid. Risk of sampling errors/ biases. Less suited for generalization 7. Case studies In-depth review of one or a small number of selected cases. Well-suited for understanding processes and for formulating hypotheses to be tested later. 8. Observation In-depth observations over an extended period of time, participatory or non-participatory. Well-suited for understanding processes but with limited potential for generalization. 9. Formal survey Oral interviews or written questionnaires in a representative sample of respondents. Data collection is demanding but often produces reliable information. 10. Story-telling/ collection Obtaining participants’ and communities’ experiences of change by collating their observations and stories. Link 49: BACK (Source: Prolog Consult) Some other techniques are more adapted to longer-term evaluations, e.g.: Literature review: comprehensive, often academically oriented, review of relevant references that can be found in international scientific literature, policy papers and guidelines. • focus groups: (if not undertaken in a more rudimentary approach in an emergency context) a qualitative research technique that brings staff of homogeneous background or function together to hold a focused discussion on a specific topic of common concern. The group discussion is led by a skilled moderator whose job it is to gain the confidence of the group members and facilitate their discussion so that they will react and interact freely on a series of questions. A focus group is generally comprised of 7-10 people and does not have to be statistically representative. However, it should involve a good cross section of the groups in the survey population, and separate groups should be formed for the different categories of stakeholders. • Brainstorming sessions: working group generally similar to the focus group above, but without a structured questionnaire to lead the discussion. • Expert panels: a group gathering a limited number of highly qualified external experts, to provide advice or comments on specific issues at some key, pre-defined stages of the evaluation. • Case studies: as a complement to the “horizontal” focus groups, case studies can be seen as the “vertical” type of workshops – or rather in-depth exploration – which typically need to be set up with complete organised communities (or working services) and people from all the various grades and functions. In comparison with focus groups, case studies are therefore more difficult to organise, since they generally demand the presence of a more significant number of interviewees; hence they also usually require more working time. • Benchmarking: comparative survey of external groups, bodies or case studies that seem to present similar characteristics. • Formal surveys. Questionnaires are a valuable tool but also represent lots of work. They are however a systematic way to gather an indication on points in the TOR oriented to the level of awareness within the targeted respondents. Moreover, they often collect comparable information for a relatively large number of people in particular target groups. Questionnaires may have close or/and open questions. The later often provide important depth of information. In the case of internal performance evaluations, administrative arrangements (distribution, how, type – web or other, deadlines) have to be considered and the response rate calculated. • Beneficiary/ Ownership2 / Stakeholder analysis: used to develop and understanding of the power relationships, influence, and interests of the various people involved in an activity, and to design development initiatives, signal constraints to participation, and provide feedback to improve services and activities.

2 Ownership analysis is an important point as ownership by developing countries of their own development processes has been identified as a main guiding principle in the EC Development Policy (COM(2000) 212) Link 50: BACK (Source: Prolog Consult)Data analysis can also be quantitative…. • to describe phenomena in a concise format using statistical tabulation; • to test relationship among variables of interest; • to generalise findings to an overall population; or qualitative (data of this type includes detailed descriptions- patterns, themes, tendencies, trends - and interpretations and explanations of these patterns). • Content analysis, (i.a., through the use of a SWOT table (Strengths, Weaknesses, Opportunities and Threats, see e.g. http://www.m-arc.be/documents/SWOTChart.xls) or a Knowledge, Attitude, Practice and Behaviour table (KAPB), appropriate to assess individual capacities, possibly with a 360° approach. • Analysis of case studies. • Inductive or deductive analysis. • Logical and sequential analysis, including through LFA (logical framework analysis), which may be either the original starting point of the project, or an ex-post frame, re- constructed on purpose by the evaluation. • Peer review analysis through a panel of highly specialised experts, who can be completely external, internal, or mixed. Link 51: BACK

Selection criteria (Sub) Source of Langu.: Sectoral Region Int/Ext Current Nation/ Fe/ TL and Know- proven Name applica- Location -English /pro-gram exper. / expert position resident male evalua- ledge of writing Rates tion -French expe- type of tion mgt man-dates skills -other rience crisis exp.

Shortlist

Longlist (possi- bles)

Not recom- mended Link 52: BACK Ethical considerations Ethical guidance adapted from that provided by CIDA (1990), and as partially reproduced by Danida in its ‘Evaluation Guidelines’ (2006):

Cultural Intrusion Local customs regarding dress, personal interaction, religious beliefs and practices should be respected. Anonymity/confidentiality Evaluators must respect people’s right to provide information in confidence, and must ensure that sensitive information cannot be traced to its source. Responsibility for evaluations In some countries, criticism can have serious consequences for a national. Evaluators must take care that those involved as local evaluators either endorse a report, or that their restricted roles are clearly described in the report. Statements should not be made on behalf of the evaluation team if other team members have not had an opportunity to disagree. Right to privacy Evaluators should realise that people can be extremely busy and their participation in evaluations can be burdensome. Therefore, evaluators should provide ample notice and minimise demands on time. Supremacy of fundamental values There is a delicate balance between certain cultural practices and the deprivation of fundamental rights and freedoms. While evaluators are expected to respect other cultures, they must also be aware of the values affecting minorities and particular groups. In such matters the UN Universal Declaration of Human Rights (1948) is the operative guide. Omissions Ethically, evaluators have a responsibility to bring to light issues and findings which may not relate directly to the Terms of Reference. Certain other issues can cause difficulties for the evaluator and should be acknowledged and discussed with the Evaluation manager as necessary. Evaluation of individuals Performance evaluation is not normally a part of evaluations, though reports will touch on issues such as leadership and management competence that border on evaluation of individuals. The evaluator is not expected to evaluate individuals and must balance an evaluation of management functions with this general principle. Link 53: BACK Steps in Conducting a Participatory Evaluation (source: USAID) Step 1: Decide if a participatory evaluation approach is appropriate. Participatory evaluations are especially useful when there are questions about implementation difficulties or program effects on beneficiaries, or when information is wanted on stakeholders’ knowledge of program goals or their views of progress. Traditional evaluation approaches may be more suitable when there is a need for independent outside judgment, when specialized information is needed that only technical experts can provide, when key stakeholders don’t have time to participate, or when such serious lack of agreement exists among stakeholders that a collaborative approach is likely to fail. Step 2: Decide on the degree of participation. What groups will participate and what roles will they play? Participation may be broad, with a wide array of program staff, beneficiaries, partners, and others. It may, alternatively, target one or two of these groups. For example, if the aim is to uncover what hinders program implementation, field staff may need to be involved. If the issue is a program’s effect on local communities, beneficiaries may be the most appropriate participants. If the aim is to know if all stakeholders understand a program’s goals and view progress similarly, broad participation may be best. Roles may range from serving as a resource or informant to participating fully in some or all phases of the evaluation. Step 3: Prepare the evaluation scope of work. Consider the evaluation approach—the basic methods, schedule, logistics, and funding. Special attention should go to defining roles of the outside facilitator and participating stakeholders. As much as possible, decisions such as the evaluation questions to be addressed and the development of data collection instruments and analysis plans should be left to the participatory process rather than be predetermined in the scope of work. Step 4: Conduct the team planning meeting. Typically, the participatory evaluation process begins with a workshop of the facilitator and participants. The purpose is to build consensus on the aim of the evaluation; refine the scope of work and clarify roles and responsibilities of the participants and facilitator; review the schedule, logistical arrangements, and agenda; and train participants in basic data collection and analysis. Assisted by the facilitator, participants identify the evaluation questions they want answered. The approach taken to identify questions may be open ended or may stipulate broad areas of inquiry. Participants then select appropriate methods and develop data-gathering instruments and analysis plans needed to answer the questions. Step 5: Conduct the evaluation. Participatory Evaluations seek to maximize stakeholders’ involvement in conducting the evaluation in order to promote learning. Participants define the questions, consider the data collection skills, methods, and commitment of time and labour required. Participatory evaluations usually use rapid appraisal techniques, which are simpler, quicker, and less costly than conventional sample surveys. They include methods such as those in the box on page 4. Typically, facilitators are skilled in these methods, and they help train and guide other participants in their use. Step 6: Analyze the data and build consensus on results. Once the data are gathered, participatory approaches to analyzing and interpreting them help participants build a common body of knowledge. Once the analysis is complete, facilitators work with participants to reach consensus on findings, conclusions, and recommendations. Facilitators may need to negotiate among stakeholder groups if disagreements emerge. Developing a common understanding of the results, on the basis of empirical evidence, becomes the cornerstone for group commitment to a plan of action. Step 7: Prepare an action plan. Facilitators work with participants to prepare an action plan to improve program performance. The knowledge shared by participants about a program’s strengths and weaknesses is turned into action. Empowered by knowledge, participants become agents of change and apply the lessons they have learned to improve performance. Link 54: BACK (Source : DANIDA) During the final phases of the evaluation, the evaluators are invited to present their key findings. During this presentation, stakeholders are invited to reflect on recommendations. Danida has a form to facilitate the follow-up to evaluations in general and this form is also applied to evaluations of humanitarian programmes. The form has three columns: the recommendation, the action to be undertaken and the status of this action. • When the final draft of the evaluation report is submitted to the evaluation department, this department transfers the recommendations in the relevant column of the format. • It then contacts the departments concerned to formulate the action to be taken for the follow-up of the recommendation. • The form with recommendations and follow-up action is discussed in the Management Meeting (often) chaired by the Minister. During this meeting the recommendations and the follow-up action indicated are discussed, and if needed adjusted or reformulated. • When agreement is reached, the decision is taken and authorised by the chairman to execute the follow-up action as indicated. It thereby becomes an instruction to the concerned departments • Six months or longer after the follow-up action has been decided, the evaluation unit approaches the departments with the format, and requests to indicate the status of the follow-up action. Link 55: BACK http://www.undp.org/eo/documents/HandBook/part_4.pdf APPLING THE RECOMMENDATIONS FROM FEEDBACK Part of the feedback obtained through monitoring and evaluation will be in the form of recommendations for action. Different types of recommendations from monitoring and evaluation processes should be analyzed separately. This is accomplished by answering a number of questions about the type and consequences of recommendations. The questions address both substantive and implementation recommendations. Feasibilities and priorities based on the answers to these questions must then be set for the short, medium and long terms. The following list of questions concerns the type of recommendation (substantive or implementation) and its consequences. Questions about the consequences of substantive recommendations from monitoring and evaluation: • Who or what will be directly or indirectly affected by the recommendation(s) in terms of planned outputs and outcomes? • How do the recommendations differ from previous ones? • What are the key reasons and approaches used to substantiate the recommendation ( s ) ? • How do the recommendations compare with similar outcomes, projects, programmes or activities in other parts of the country or other countries? • How do recommendations contribute to overall intended outputs and outcome(s)? • Is there a “fit” between intended outcome(s) and actual outcome(s)? • How do recommendations link with regional and global programme objectives? • How does the senior CO management intend to respond to, and follow up on implementation of, the recommendations? Questions about the consequences of i m p l e m e n t a t i o n recommendations from monitoring and evaluation: • What will be the positive or negative effects in terms of key target groups or stakeholders? • What can be done to improve the positive effects and compensate for the negative effects? • What actions are required and by whom? • What is the time frame? • Who has the authority to implement the action? • What are the financial implications? • What are the political implications? • What human resources are needed? • Are special approaches, including training, or new types of partnership required? • What monitoring or follow-up is required? to be continued >>>

NEXT PAGE >>> continued from link 55

Analysis of the above questions will help identify a number of concrete actions through which recommendations may be judged, improved and implemented. While some of these actions will be implemented in the short term, others—especially those requiring political decisions, affecting institutional structure(s) or requiring extensive financial resources—will only be implemented in the medium or long term. A management response to the substantive and implementation issues raised is important for all monitoring and evaluation recommendations. For outcome evaluations in particular, the management response should identify what recommendations are accepted or not accepted and why, and how the accepted recommendations will be implemented and monitored. Publication of evaluation results should follow a clear format in order to treat the evidence fairly, to produce compelling analytic conclusions and to rule out ambiguity. Information may be presented through various analytic techniques. The main point, however, is to make information from evaluations and monitoring user-friendly, easily accessible and advantageous to the user. The characteristics of a good knowledge product, including a good publication, are listed in Box 6.

to be continued >>>

PREVIOUS PAGE NEXT PAGE >>> continued from link 55

Keeping these characteristics in mind even before the start of actual analysis or the preparation of a publication will help organize the evidence in an orderly fashion. The assessments from evaluation should be documented and distributed to stakeholders for feedback. This will help identify information needs. A number of suggestions for improving evaluation feedback are listed in Box 7.

The dissemination of evaluation results is as important as their publication. Only an efficient system of dissemination will ensure that the target recipients receive the evaluation feedback that is relevant to their specific needs. Communicating all that has been learned poses a challenge. The underlying issue is how to capture lessons from experience that are transferable; that is, those lessons that have a broader application as compared to those that are relevant only to a single programme or project. This challenge can be addressed through the institutionalization of learning from monitoring and evaluation feedback. Institutionalization of the learning process can be achieved in part by better incorporating learning into existing tools and processes, such as the project and programme document drafting, etc (according to the agency's own internal organisation). to be continued >>>

PREVIOUS PAGE Link 59: BACK Achievement of objectives (source: WFP) • Are the objectives appropriate and realistic? • To what extent have planned outputs been achieved in terms of quantity, quality, equity and timeliness? • For EMOPs and PRROs: have the targeted beneficiaries received the planned rations? Have the right beneficiaries been targeted? Did the beneficiaries use the food aid as intended? • For PRROs and development activities: have the beneficiaries maintained or improved their livelihoods and what assets have been created/maintained, who have they benefited and how? • What were the most successful recovery activities? What were the least successful? How effectively was a transition made from relief to recovery activities? To what degree has food aid contributed to promoting resettlement and food self-sufficiency (where applicable)? • Have outputs reached the poorest? Women? Vulnerable groups? To what extent? What share of women and vulnerable groups have benefited directly from FFW outputs? • Were resources effectively and efficiently used? Do the results justify the costs? • What evidence is there that the operation’s stated purpose(s) and goal were achieved? • To what extent were WFP’s core mandate responsibilities met? • If at mid-term, what additional measures are needed to improve the chances of achieving the purpose? Factors in the effectiveness and efficiency of EMOPs and PRROs Registration/verification • Were data collected for target populations/groups reliable and disaggregated, and how well were the data maintained? Link 60: BACK Appropriateness of rations (source: WFP) • Was the food ration and basket adequate and acceptable (quantitatively and qualitatively) vis-à-vis the livelihood and coping strategies of both men and women? • Were the nutritional objectives realistic, and to what extent were they achieved? • What has been the nutritional effect/outcome of WFP assistance on beneficiaries? • Have there been ration reductions or phasing-out of food assistance; if so, on what basis? • Was the food aid culturally acceptable and appropriate? Link 61: BACK Final distribution (source: WFP) • Was the delivery system efficient and equitable? Was it transparent? Was a complaints procedure in place? How well were failures in the delivery system addressed? • Did women play lead roles on local decision-making committees for food management? If so, to what extent, and has this resulted in a change in acceptance of women in lead decision-making roles? Link 62: BACK Commitments to Women (source: WFP) • To what extent were WFP’s Commitments to Women met in terms of (i) food access by women; (ii) access to power structures and decision-making; and (iii) access to resources, employment, markets and trade? • What changes are required in a future phase to ensure better compatibility with the Commitments? • What gender training has been carried out, for whom, and what difference, if at all, has such training made? • Was adequate effort been made to mainstream as well as advocate gender considerations? Link 63: BACK Outcome/impact and sustainability/connectedness (source: WFP) • What difference did the operation make to the lives of the beneficiaries (nutritional status, livelihood improvements, etc.)? Who benefited exactly? What would have happened if no assistance had been given? • How do beneficiaries and other stakeholders value the operation? • What impact did the operation have on household food security and self-reliance? • EMOPs and PRROs: What was the impact on host populations? • What unexpected outcomes (positive or negative) have there been? • Have environmental concerns been adequately addressed within the operation/ programme? • What intended or unintended impacts have occurred on the environment as a result of the operation? • What impact have campsites had on the environment? What effect have rations and cooking requirements had? • What was done to ameliorate the operation’s environmental impact and restore the natural habitat? • How long are these benefits likely to last? What are the prospects for the sustainability of activities and outputs? What factors are likely to undermine sustainability of benefits? Are the sustainable benefits likely to exceed the costs? • What costs will have to be borne by government, implementing partners, beneficiaries or other stakeholders in order for the benefits to continue? • Has food aid been used as leverage to obtain complementary national and international resources and recognition to improve the condition of women? Link 64: BACK Beneficiary participation (source: WFP) • In what way have the beneficiaries participated in the design, implementation and monitoring of the operation? Were the arrangements adequate? What was the impact of their participation or non-participation? • Was beneficiary involvement empowering men and women? Link 65: BACK APPRECIATIVE INQUIRY (AI) type of questions AI concentrates on "what works", rather than on what is wrong. For participation purposes with final beneficiaries, e.g. in focus groups meetings (source: Locating the Energy for Change, two AI-focused evaluations in Africa): • How did this community (camp) come to be established? Who founded it? • What do you like best about it? When is the community (camp) happiest? What makes people glad to live here? • What is your favourite activity here? • What would visitors to this community (camp) comment upon most favourably? • When you have visited (other/neighbouring) communities/camps, what has impressed you most about them? • Tell me a story about the best things (this NGO) does for (category of beneficiaries) • Have you a story about the best thing that has happened to (a member of the category of beneficiaries)? • What is it about (this NGO) that makes (category of beneficiaries) have confidence in it? • Some days are better than others for you. Tell me what happens on the best days? • When you are sick, tell me about the best things you can do to help yourself (the NGO can do for you) to get better? Link 66: BACK Protection (source: ALNAP): • What modes of protective practice did the organisation adopt at various stages of the humanitarian response? With what effect? • How well did the programming team understand the affected population’s coping mechanisms, its will and capacities for self-protection and organisation? • What methods were used to assess violence against women throughout the program cycle? • How was the agency’s presence targeted to get close to particular groups of vulnerable people at particular high-risk times and in high-risk places? • How much are existing guidelines on protection disseminated within the organisation concerned i.e. UNHCR guidelines to counter sexual violence? • How aware were staff and field partners as to how to refer women and men seeking redress for human rights violations? How able were staff to link people’s experiences of violations to specific legal standards? • What specific steps were taken to ensure and increase personal safety and security of women, girls, boys and men? • What measures were put in place regarding accommodation, transportation and security to enable women workers to do their jobs as safely as possible? • How much did staff include humanitarian values and principles in educational programmes in health promotion, schools and literacy groups? Link 67: BACK Security (source: WFP) • Have adequate and appropriate measures been introduced and adhered to, in order to minimise the risk to WFP staff and implementing partners involved in the implementation of the operation/programme? • Have measures been taken to provide protection to the men, women and children participating in the operation? • Are there significant security challenges to the smooth functioning of the operation/ programme? Was the safety of the beneficiaries a high priority? Link 68: BACK Questions for evaluation of coordination include (source: ALNAP): • Were policies for coordination in place and followed? What did the policies say? • Were there any incentives to coordinate, for example did donors promote UN coordination through funding arrangements? Or was there competition for funds? • Was a lead agency appointed, and what was the result of this? • Which parties were included and in what manner? Why? • Who took the lead and how effective were they? Why? • Were funds channelled in a coordinated fashion, or individually by donors to suit their own strategic aims? • What were the main constraints and supports to coordination? How was good coordination achieved (What is good coordination?), and is it replicable for other situations. • Did coordination lead to improved effectiveness and impact, for example an increase in humanitarian space? How did this come about? Link 69: BACK APPRECIATIVE INQUIRY (AI) type of questions AI concentrates on "what works", rather than on what is wrong For (local) staff, internal /process review (source: Encyclopedia of positive questions) • What has most attracted you to work (in this NGO) in the first place? • Once you were hired, what were the initial impressions that made you proud of (the NGO)? • What is your sense of the root cause of the ongoing success of (the NGO)? • What do you believe are (this NGO's) most significant strategic (compared) advantages? • Describe one of your earliest positive experiences with (the NGO), and how that reinforced your initial sense that this was a great place to work? • Think of a time when (the NGO) was at its absolute best…when (the donors, beneficiaries, stakeholders) easily saw what (the NGO) really is and what it is about. Tell me about it. • What are the things that make (your function, your department) the best that it can possibly be? • Put yourself in (your donors', your beneficiaries') shoes. What do you think they would say if we asked them what makes (this NGO) best-in-class? • Think about another (NGO, agency) that you admire…one that you either know well or have heard of. What, in your experience, makes that organisation best-in-class? What do they do that (this NGO) could learn to do better? • How do you learn best? Tell me about a time/ a work experience when you learned something very challenging. What contributed to your learning? Who else was involved and what did they do? How did you benefit (for your career development)? How did (the NGO) benefit? • Describe a time when you were involved in a participatory decision-making process in which everyone felt respected. How were you included? How did you contribute? How did people know their voices were heard? What made it successful? • When have you been a part of or seen (this NGO) do something positive for the environment? Who was involved? If there were (2? 3?) things that you would like (this NGO) do that would benefit the environment, what would they be? • Describe a time when you felt you exhibited a clear sense of financial responsibility. What was the situation and what did you do? What was the result of your financial responsibility? • What do you most value and appreciate about leadership in (the NGO)? When people are in leadership positions, what (2? 3?) things can they do that will help you be the best you can be? • What most appeals to you about the concept of continuous learning? What makes (this NGO) such a favourable place for learning? How did you and others grow and change, as a result of being in this environment? • Tell me about a time when (this NGO's) organisational structure most supported you in implementing its core principles and values? • Tell me about a time when you received recognition, appreciation, or acknowledgement for your work. Describe the situation. What did you do? How were you recognised? Describe types of recognition that would inspire you to strive for excellence. Why would these types of recognition be meaningful to you? • Tell me about a time, recently, in which you experienced a comfortable sense of work/family balance. What contributed to it? What is it about (the NGO) that best supports you in creating work/family balance? What (2? 3?) things could be done that would help you create a better work/family balance? Link 70: BACK Strategy (source: WFP) • Was the strategy used the most relevant and appropriate to meet the identified needs? • Were gender issues appropriately reflected in the strategy? • Were alternative approaches to achieving objectives considered? • Was the strategy consistent with WFP’s core mandate, with WFP’s policies and, where relevant, with Country Programme and strategic framework objectives? • What was the link between food aid and other assistance provided? • Was the strategy well prepared and did it provide adequate guidance for implementers? For PRROs: Did the strategy accurately consider the opportunities to introduce recovery activities? • Was an exit strategy developed? • Were other agencies included in the analysis of the problem and the drawing-up of the response plans? • Did the preparation process adequately involve beneficiaries (both men and women)? • Were constraints to implementation adequately assessed from the onset? • Has the strategy been periodically reviewed or modified over the life of the operation in order to maintain its relevance to changing circumstances? Link 71: BACK Assessments (source: WFP) • How was the initial situation assessed? How has the beneficiaries’ food insecurity, vulnerability and nutrition status been assessed? Were the roles of men and women adequately analysed? Have regular reassessments been conducted? Have host communities’ needs been assessed? Link 72: BACK Targeting (source: WFP) • How have beneficiary groups/areas been identified? • Were the special needs of certain groups/areas considered? • Have the needs of the host community been addressed? • Were targeting objectives appropriate? • Were arrangements made to review and update the targeting strategy? • Were male and female beneficiaries involved in the negotiation of beneficiary status? • What was the proportion of male and female beneficiaries participating in food for work (FFW)? Link 73: BACK Design (source: WFP) • Was the design of the operation logical and coherent? Did it provide a good “road map” for implementation and a sound basis for review and evaluation? • Were objectives set at the right level (i.e. in compliance with logframe definitions) and were sound means-to-ends linkages made between inputs, activities, outputs, outcome(s) and the impact? • Was the operation design technically feasible? • Were gender concerns appropriately reflected and in line with the Commitments to Women? • Were assumptions reasonable when they were specified, and were contingency plans made for known risks? • Which external factors were not taken into account during design? • Has the operational context of the operation changed since its design, and if so has the operation adapted effectively? Link 74: BACK Factors in the (effectiveness and) efficiency of all operations Implementation (source: WFP) • Were there any significant delays in approval; what were the causes of this? • Were the implementation schedule and management arrangements realistic (this should include financial management and budgeting systems)? • Were plans followed? If not, why not? • Did objectives change during implementation? Why? On the basis of what analysis or what events? • Have there been delays in implementation? How were these dealt with? What was the impact of such delays? • How well were risks and problems managed? Link 75: BACK Monitoring (source: WFP) • Were baseline data collected and were appropriate indicators identified at the outset for measuring progress and results? • Did the monitoring systems work, including the extent to which gender considerations were built into monitoring arrangements? Were gender- disaggregated data collected in accordance with WFP policy? • For PRROs: have criteria been established to signal when to shift activities from relief to recovery, and from recovery to development? If so, are these being applied effectively? Link 76: BACK Coordination, partnerships (source: WFP) • Were objectives and activities compatible with the policies/programmes of the relevant government(s)? How supportive are WFP’s counterparts? • What mechanisms were put in place for coordination with the Government, donors, United Nations agencies, NGOs, etc. and how effective were they? Were sectorial responsibilities successfully divided among partners? • Did implementing partners fulfil their contractual obligations? How was their capacity assessed? What training of partners has taken place? Has there been a trend towards or away from using local implementing partners? What capacity has been left behind? Link 77: BACK Programmatic linkages (source: WFP) • Are the objectives and activities of the operation/programme compatible with and complementary to those of the other operations currently being implemented by the country office? Link 78: BACK Management, human resources and training (source: WFP) • Did the country office, sub-offices and specialised units fulfil their roles effectively and efficiently? • Did staff have the experience and expertise to carry out the activities envisaged? • Are there staff or skill shortages? What has been done about these? • Have correct financial and administrative procedures been followed? • Is there appropriate gender balance in the operation staff (both WFP and implementing partners)? Have staff been trained in gender issues and are they applying this training to their work? Link 79: BACK Budget and resources (source: WFP) • Operation costs: What was the planned vs. actual expenditure for the operation as a whole, and under each main cost heading? • Was the budget appropriate in relation to its objectives/activities, and what factors affected individual budget items, particularly direct support cost (DSC)? • ITSH: How were internal transport, storage and handling financed? Were the calculated rates adequate? What revisions were required? • To what extent have the resourcing requirements for the operation been met and how has the country office managed shortfalls? How predictably and regularly have resources been supplied to the operation/programme? • Monetization: Was it justified? Were the prices fair? • For PRROs: Has transformation to a PRRO resulted in longer-term (more than one year) financial commitments to the operation? • How successfully has the operation/programme resourced its non-food inputs (NFIs) and what, if any, have been the constraints? Link 80 BACK Participation (source: SPHERE) • Women and men of all ages from the disaster-affected and wider local populations, including vulnerable groups, receive information about the assistance programme, and are given the opportunity to comment to the assistance agency during all stages of the project cycle. • Written assistance programme objectives and plans should reflect the needs, concerns and values of disaster-affected people, particularly those belonging to vulnerable groups, and contribute to their protection. • Programming is designed to maximise the use of local skills and capacities. Link 81 BACK Initial assessment (source: SPHERE) • Information is gathered using standardised procedures and made available to allow for transparent decision-making. • The assessment considers all technical sectors (water and sanitation, nutrition, food, shelter, health), and the physical, social, economic, political and security environment. • Through consultation, the assessment takes into account the responses of the local and national authorities and other actors and agencies. • Local capacities and strategies to cope with the disaster, both those of the affected population and the surrounding population, are identified. • Whenever feasible, data are disaggregated by sex and by age. • The assessment is underpinned by the rights of those affected by disasters, as defined by international law. • The assessment takes into account the responsibility of relevant authorities to protect and assist the population on the territory over which they have control, and also takes into account national law, standards and guidelines applicable where the affected population is found, as they conform with international law. • The assessment includes an analysis of the operating environment, including factors affecting the personal safety and security of the affected population and of humanitarian staff. • Estimates of population numbers are cross-checked and validated with as many sources as possible, and the basis for the estimate made known. • Assessment findings are made available to other sectors, national and local authorities and representatives of the affected population. Recommendations are made on the need for external assistance, and on appropriate responses that should be linked with exit or transition strategies. Link 82 BACK Response (source: SPHERE) • Where people’s lives are at risk as a result of disaster, programmes prioritise life- saving needs. • Programmes and projects are designed to support and protect the affected population and to promote their livelihoods, so that they meet or exceed the Sphere Minimum Standards, as illustrated by the key indicators. • There is effective coordination and exchange of information among those affected by or involved in the disaster response. Humanitarian agencies undertake activities on the basis of need, where their expertise and capacity can have the greatest impact within the overall assistance programme. • Organisations, programmes and projects that either cannot address identified needs or are unable to attain the Minimum Standards make any gaps known so that others may assist. • In conflict situations, the assistance programme takes into account the possible impact of the response on the dynamics of the situation. Link 83 BACK Targeting (source: SPHERE) • Targeting criteria must be based on a thorough analysis of vulnerability. • Targeting mechanisms are agreed among the affected population (including representatives of vulnerable groups) and other appropriate actors. Targeting criteria are clearly defined and widely disseminated. • Targeting mechanisms and criteria should not undermine the dignity and security of individuals, or increase their vulnerability to exploitation. • Distribution systems are monitored to ensure that targeting criteria are respected and that timely corrective action is taken when necessary. Link 84 BACK Monitoring (source: SPHERE) • The information collected for monitoring is timely and useful, it is recorded and analysed in an accurate, logical, consistent, regular and transparent manner and it informs the ongoing programme. • Systems are in place to ensure regular collection of information in each of the technical sectors and to identify whether the indicators for each standard are being met. • Women, men and children from all affected groups are regularly consulted and are involved in monitoring activities. • Systems are in place that enable a flow of information between the programme, other sectors, the affected groups of the population, the relevant local authorities, donors and other actors as needed. Link 85 BACK Evaluation (source: SPHERE) • The programme is evaluated with reference to stated objectives and agreed minimum standards to measure its overall appropriateness, efficiency, coverage, coherence and impact on the affected population. • Evaluations take account of the views and opinions of the affected population, as well as the host community if different. • The collection of information for evaluation purposes is independent and impartial. • The results of each evaluation exercise are used to improve future practice. Link 86 BACK Aid worker competencies and responsibilities (source: SPHERE) • Aid workers have relevant technical qualifications and knowledge of local cultures and customs, and/or previous emergency experience. Workers are also familiar with human rights and humanitarian principles. • Staff are knowledgeable about the potential tensions and sources of conflict within the disaster-affected population itself and with host communities. They are aware of the implications of delivering humanitarian assistance, and pay particular attention to vulnerable groups. • Staff are able to recognise abusive, discriminatory or illegal activities, and refrain from such activities. Link 87 BACK Supervision, management and support of personnel (source: SPHERE) • Managers are accountable for their decisions and for ensuring adequate security and compliance with codes/rules of conduct as well as support for their staff. • Technical and managerial staff are provided with the necessary training, resources and logistical support to fulfil their responsibilities. • Staff working on programmes understand the purpose and method of the activities they are asked to carry out, and receive subsequent feedback on their performance. • All staff have written job descriptions, with clear reporting lines, and undergo periodic written performance assessment. • All staff are oriented regarding relevant health and safety issues for the region and environment in which they are to work. • Staff receive appropriate security training. • Capacity-building systems for staff are set up and these are subject to routine monitoring. • The capacity of national and local organisations is built up to promote long-term sustainability. Link 88 BACK Hygiene promotion standard 1: programme design and implementation (source: SPHERE) • Key hygiene risks of public health importance are identified. • Programmes include an effective mechanism for representative and participatory input from all users, including in the initial design of facilities. • All groups within the population have equitable access to the resources or facilities needed to continue or achieve the hygiene practices that are promoted. • Hygiene promotion messages and activities address key behaviours and misconceptions and are targeted for all user groups. • Representatives from these groups participate in planning, training, implementation, monitoring and evaluation. • Users take responsibility for the management and maintenance of facilities as appropriate, and different groups contribute equitably. Link 89 BACK Water supply standard 1: access and water quantity (source: SPHERE) • Average water use for drinking, cooking and personal hygiene in any household is at least 15 litres per person per day. • The maximum distance from any household to the nearest water point is 500 metres. • Queuing time at a water source is no more than 15 minutes. • It takes no more than three minutes to fill a 20-litre container. • Water sources and systems are maintained such that appropriate quantities of water are available consistently or on a regular basis. Link 90 BACK Water supply standard 2: water quality (source: SPHERE) • A sanitary survey indicates a low risk of faecal contamination. • There are no faecal coliforms per 100ml at the point of delivery. • People drink water from a protected or treated source in preference to other readily available water sources. • Excessive queuing times are indicators of insufficient water availability (either due to an inadequate number of water points or inadequate yields of water points). • Water points should be located in areas that are accessible to all regardless of e.g. sex or ethnicity. Some handpumps and water carrying containers may need to be designed or adapted for use by people living with HIV/AIDS, older and disabled people and children. In situations where water is rationed or pumped at given times, this should be planned in consultation with the users. Times should be set which are convenient and safe for women and others who have responsibility for collecting water, and all users should be fully informed of when and where water is available. • Steps are taken to minimise post-delivery contamination. • For piped water supplies, or for all water supplies at times of risk or presence of diarrhoea epidemic, water is treated with a disinfectant so that there is a free chlorine residual at the tap of 0.5mg per litre and turbidity is below 5 NTU. • No negative health effect is detected due to short-term use of water contaminated by chemical (including carry-over of treatment chemicals) or radiological sources, and assessment shows no significant probability of such an effect. Link 91 BACK Water supply standard 3: water use facilities and goods (source: SPHERE) • Each household has at least two clean water collecting containers of 10-20 litres, plus enough clean water storage containers to ensure there is always water in the household. • Water collection and storage containers have narrow necks and/or covers, or other safe means of storage, drawing and handling, and are demonstrably used. • There is at least 250g of soap available for personal hygiene per person per month. • Where communal bathing facilities are necessary, there are sufficient bathing cubicles available, with separate cubicles for males and females, and they are used appropriately and equitably. • Where communal laundry facilities are necessary, there is at least one washing basin per 100 people, and private laundering areas are available for women to wash and dry undergarments and sanitary cloths. • The participation of all vulnerable groups is actively encouraged in the siting and construction of bathing facilities and/or the production and distribution of soap, and/or the use and promotion of suitable alternatives. Link 92 BACK Excreta disposal standard 1: access to, and numbers of, toilets (source: SPHERE) • A maximum of 20 people use each toilet. • Use of toilets is arranged by household(s) and/or segregated by sex. • Separate toilets for women and men are available in public places (markets, distribution centres, health centres, etc.). • Shared or public toilets are cleaned and maintained in such a way that they are used by all intended users. • Toilets are no more than 50 metres from dwellings. • Toilets are used in the most hygienic way and children’s faeces are disposed of immediately and hygienically. Link 93 BACK Excreta disposal standard 2: design, construction and use of toilets (source: SPHERE) • Users (especially women) have been consulted and approve of the siting and design of the toilet. • Toilets are designed, built and located to have the following features: they are designed in such a way that they can be used by all sections of the population, including children, older people, pregnant women and physically and mentally disabled people; they are sited in such a way as to minimise threats to users, especially women and girls, throughout the day and night; they are sufficiently easy to keep clean to invite use and do not present a health hazard; they provide a degree of privacy in line with the norms of the users; they allow for the disposal of women’s sanitary protection, or provide women with the necessary privacy for washing and drying sanitary protection cloths; they minimise fly and mosquito breeding. • All toilets constructed that use water for flushing and/or a hygienic seal have an adequate and regular supply of water. • Pit latrines and soakaways (for most soils) are at least 30 metres from any groundwater source and the bottom of any latrine is at least 1.5 metres above the water table. Drainage or spillage from defecation systems must not run towards any surface water source or shallow groundwater source. • People wash their hands after defecation and before eating and food preparation. • People are provided with tools and materials for constructing, maintaining and cleaning their own toilets if appropriate. Link 94 BACK Vector control standard 1: individual and family protection (source: SPHERE) • All populations at risk from vector-borne disease understand the modes of transmission and possible methods of prevention. • All populations have access to shelters that do not harbour or encourage the growth of vector populations and are protected by appropriate vector control measures. • People avoid exposure to mosquitoes during peak biting times by using all non- harmful means available to them. Special attention is paid to protection of high- risk groups such as pregnant and feeding mothers, babies, infants, older people and the sick. • People with treated mosquito nets use them effectively. • Control of human body lice is carried out where louse-borne typhus or relapsing fever is a threat. • Bedding and clothing are aired and washed regularly. • Food is protected at all times from contamination by vectors such as flies, insects and rodents. Link 95 BACK Vector control standard 2: physical, environmental and chemical protection measures (source: SPHERE) • Displaced populations are settled in locations that minimise their exposure to mosquitoes. • Vector breeding and resting sites are modified where practicable. • Intensive fly control is carried out in high-density settlements when there is a risk or the presence of a diarrhoea epidemic. • The population density of mosquitoes is kept low enough to avoid the risk of excessive transmission levels and infection. • People infected with malaria are diagnosed early and receive treatment. Link 96 BACK Vector control standard 3: chemical control safety (source: SPHERE) • Personnel are protected by the provision of training, protective clothing, use of bathing facilities, supervision and a restriction on the number of hours spent handling chemicals. • The choice, quality, transport and storage of chemicals used for vector control, the application equipment and the disposal of the substances follow international norms, and can be accounted for at all times. • Communities are informed about the potential risks of the substances used in chemical vector control and about the schedule for application. They are protected during and after the application of poisons or pesticides, according to internationally agreed procedures. Link 97 BACK Solid waste management standard 1: solid waste collection and disposal (source: SPHERE) • People from the affected population are involved in the design and implementation of the solid waste programme. • Household waste is put in containers daily for regular collection, burnt or buried in a specified refuse pit. • All households have access to a refuse container and/or are no more than 100 metres from a communal refuse pit. • At least one 100-litre refuse container is available per 10 families, where domestic refuse is not buried on-site. • Refuse is removed from the settlement before it becomes a nuisance or a health risk. • Medical wastes are separated and disposed of separately and there is a correctly designed, constructed and operated pit, or incinerator with a deep ash pit, within the boundaries of each health facility. • There are no contaminated or dangerous medical wastes (needles, glass, dressings, drugs, etc.) at any time in living areas or public spaces. • There are clearly marked and appropriately fenced refuse pits, bins or specified areas at public places, such as markets and slaughtering areas, with a regular collection system in place. • Final disposal of solid waste is carried out in such a place and in such a way as to avoid creating health and environmental problems for the local and affected populations. Link 98 BACK Drainage standard 1: drainage work (source: SPHERE) • Areas around dwellings and water points are kept free of standing wastewater, and stormwater drains are kept clear. • Shelters, paths and water and sanitation facilities are not flooded or eroded by water. • Water point drainage is well planned, built and maintained. This includes drainage from washing and bathing areas as well as water collection points. • Drainage waters do not pollute existing surface or groundwater sources or cause erosion. • Sufficient numbers of appropriate tools are provided for small drainage works and maintenance where necessary. Link 99 BACK Food security standard 1: general food security (source: SPHERE) • Where people’s lives are at risk through lack of food, responses prioritise meeting their immediate food needs. • In all disaster contexts, measures are taken to support, protect and promote food security. This includes preserving productive assets or recovering those lost as the result of disaster. • Responses that protect and support food security are based on sound analysis, in consultation with the disaster-affected community. • Responses take account of people’s coping strategies, their benefits and any associated risks and costs. • Transition and exit strategies are developed for all food security responses to disaster, and are publicised and applied as appropriate. • When a response supports the development of new or alternative livelihood strategies, all groups have access to appropriate support, including necessary knowledge, skills and services. • Food security responses have the least possible degradative effect on the environment. • Numbers of beneficiaries are monitored to determine the level of acceptance and access by different groups in the population and to ensure overall coverage of the affected population without discrimination. • The effects of responses on the local economy, social networks, livelihoods and the environment are monitored, in addition to ongoing monitoring linked to programme objectives. Link 100 BACK Food security standard 2: primary production (source: SPHERE) • Interventions to support primary production are based on a demonstrated understanding of the viability of production systems, including access to and availability of necessary inputs and services. • New technologies are introduced only where their implications for local production systems, cultural practices and environment are understood and accepted by food producers. • Where possible, a range of inputs is provided in order to give producers more flexibility in managing production, processing and distribution and in reducing risks. • Productive plant, animal or fisheries inputs are delivered in time, are locally acceptable and conform to appropriate quality norms. • The introduction of inputs and services does not exacerbate vulnerability or increase risk, e.g. by increasing competition for scarce natural resources or by damaging existing social networks. • Inputs and services are purchased locally whenever possible, unless this would adversely affect local producers, markets or consumers. • Food producers, processors and distributors receiving project inputs make appropriate use of them. • Responses understand the need for complementary inputs and services and provide these where appropriate. Link 101 BACK Food security standard 3: income and employment (source: SPHERE) • Project decisions about timing, work activities, type of remuneration and the technical feasibility of implementation are based on a demonstrated understanding of local human resource capacities, a market and economic analysis, and an analysis of demand and supply for relevant skills and training needs. • Responses providing job or income opportunities are technically feasible and all necessary inputs are available on time. Where possible, responses contribute to the food security of others and preserve or restore the environment. • The level of remuneration is appropriate, and payments for waged labour are prompt, regular and timely. In situations of acute food insecurity, payments may be made in advance • Procedures are in place to provide a safe, secure working environment. • Projects involving large sums of cash include measures to avoid diversion and/or insecurity. • Responses providing labour opportunities protect and support household caring responsibilities, and do not negatively affect the local environment or interfere with regular livelihood activities. • The household management and use of remuneration (cash or food), grants or loans are understood and seen to be contributing towards the food security of all household members. Link 102 BACK Food security standard 4: access to markets (source: SPHERE) • Food security responses are based on a demonstrated understanding of local markets and economic systems, which informs their design and, where necessary, leads to advocacy for system improvement and policy change. • Producers and consumers have economic and physical access to operating markets, which have a regular supply of basic items, including food at affordable prices. • Adverse effects of food security responses, including food purchases and distribution, on local markets and market suppliers are minimised where possible. • There is increased information and local awareness of market prices and availability, of how markets function and the policies that govern this. • Basic food items and other essential commodities are available. • The negative consequences of extreme seasonal or other abnormal price fluctuations are minimised. Link 103 BACK General nutritional support standard 1: all groups (source: SPHERE) • There is access to a range of foods – staple (cereal or tuber), pulses (or animal products) and fat sources – that meet nutritional requirements. • There is access to vitamin A-, C- and iron-rich or fortified foods or appropriate supplements • There is access to iodised salt for the majority (>90%) of households. • There is access to additional sources of niacin (e.g. pulses, nuts, dried fish) if the staple is maize or sorghum. • There is access to additional sources of thiamine (e.g. pulses, nuts, eggs) if the staple is polished rice. • There is access to adequate sources of riboflavin where people are dependent on a very limited diet. • Levels of moderate and severe malnutrition are stable at, or declining to, acceptable levels. • There are no cases of scurvy, pellagra, beri-beri or riboflavin deficiency. • Rates of xerophthalmia and iodine deficiency disorders are not of public health significance. Link 104 BACK General nutritional support standard 2: at-risk groups (source: SPHERE) • Infants under six months are exclusively breastfed or, in exceptional cases, have access to an adequate amount of an appropriate breast milk substitute. • Children aged 6-24 months have access to nutritious, energy-dense complementary foods. • Pregnant and breastfeeding women have access to additional nutrients and support. • Specific attention is paid to the protection, promotion and support of the care and nutrition of adolescent girls. • Appropriate nutritional information, education and training is given to relevant professionals, care givers and organisations on infant and child feeding practices. • Older people’s access to appropriate nutritious foods and nutritional support is protected, promoted and supported. • Families with chronically ill members, including people living with HIV/AIDS, and members with specific disabilities have access to appropriate nutritious food and adequate nutritional support. • Community-based systems are in place to ensure appropriate care of vulnerable individuals. Link 105 BACK Correction of malnutrition standard 1: moderate malnutrition (source: SPHERE) • From the outset, clearly defined and agreed objectives and criteria for set-up and closure of the programme are established. • Coverage is >50% in rural areas, >70% in urban areas and >90% in a camp situation. • More than 90% of the target population is within <1 day’s return walk (including time for treatment) of the distribution centre for dry ration supplementary feeding programmes and no more than 1 hour’s walk for on-site supplementary feeding programmes. • The proportion of exits from targeted supplementary feeding programmes who have died is <3%, recovered is >75% and defaulted is <15%. • Admission of individuals is based on assessment against internationally accepted anthropometric criteria. • Targeted supplementary feeding programmes are linked to any existing health structure and protocols are followed to identify health problems and refer accordingly. • Supplementary feeding is based on the distribution of dry take-home rations unless there is a clear rationale for on-site feeding. • Monitoring systems are in place. Link 106 BACK Correction of malnutrition standard 2: severe malnutrition (source: SPHERE) • From the outset, clearly defined and agreed criteria for set-up and closure of the programme are established. • Coverage is >50% in rural areas, >70% in urban areas and >90% in camp situations. • The proportion of exits from therapeutic care who have died is <10%, recovered is >75% and defaulted is <15%. • Discharge criteria include non-anthropometric indices such as good appetite and the absence of diarrhoea, fever, parasitic infestation and other untreated illness. • Mean weight gain is >8g per kg per person per day. • Nutritional and medical care is provided according to internationally recognised therapeutic care protocols. • As much attention is attached to breastfeeding and psychosocial support, hygiene and community outreach as to clinical care. • There should be a minimum of one feeding assistant for 10 inpatients. • Constraints to caring for malnourished individuals and affected family members should be identified and addressed. Link 107 BACK Correction of malnutrition standard 3: micronutrient malnutrition (source: SPHERE) • All clinical cases of deficiency diseases are treated according to WHO micronutrient supplementation protocols. • Procedures are established to respond efficiently to micronutrient deficiencies to which the population may be at risk. • Health staff are trained in how to identify and treat micronutrient deficiencies to which the population is most at risk. Link 108 BACK Food aid planning standard 1: ration planning (source: SPHERE) _ Rations for general distribution are designed on the basis of the standard initial planning requirements for energy, protein, fat and micronutrients, adjusted as necessary to the local situation (see guidance note 1; see also General nutrition support standards on pages 137-144 and Appendix 7). _ The ration distributed reduces or eliminates the need for disaster affected people to adopt damaging coping strategies. _ When relevant, the economic transfer value of the ration is calculated and is appropriate to the local situation Link 109 BACK Food aid planning standard 2: appropriateness and acceptability (source: SPHERE) • People are consulted during assessment or programme design on the acceptability, familiarity and appropriateness of food items, and results are factored into programme decisions on the choice of commodities. • When an unfamiliar food is distributed, instructions on its preparation in a locally palatable manner, with minimum nutrient loss, are provided to women and other people who prepare food, preferably in the local language. • People’s ability to access cooking fuel and water, and the duration of cooking times and requirements for soaking, are considered when selecting commodities for distribution. • When a whole grain cereal is distributed, recipients either have the means to mill or process it in a traditional home-based manner or have access to adequate milling/ processing facilities reasonably close to their dwellings. • People have access to culturally important items, including condiments. • There is no distribution of free or subsidised milk powder or of liquid milk as a single commodity. Link 110 BACK Food aid planning standard 3: food quality and safety (source: SPHERE) • Food commodities conform to national (recipient country) and other internationally accepted standards. • All imported packaged food has a minimum six-month shelf life on arrival in the country and is distributed before the expiry date or well within the ‘best before’ period. • There are no verifiable complaints about the quality of food distributed. • Food packaging is sturdy, convenient for handling, storage and distribution, and is not a hazard for the environment. • Food packages are labelled in an appropriate language with, for packaged foods, the date of production, the ’best before’ date and details of the nutrient content. • Storage conditions are adequate and appropriate, stores are properly managed and routine checks on food quality are carried out in all locations. Link 111 BACK Food aid management standard 1: food handling (source: SPHERE) • There are no adverse health effects resulting from inappropriate food handling or preparation at any distribution site. • Recipients of food aid are informed about and understand the importance of food hygiene. • There are no complaints concerning difficulties in storing, preparing, cooking or consuming the food distributed. • Every household has access to appropriate cooking utensils, fuel and hygiene materials. • Individuals who cannot prepare food or cannot feed themselves have access to a carer who prepares appropriate food in a timely manner and administers feeding where necessary. • Where food is distributed in cooked form, staff have received training in safe storage, handling of commodities and the preparation of food and understand the potential health hazards caused by improper practices. Link 112 BACK Food aid management standard 2: supply chain management (source: SPHERE) • Food aid resources reach the intended beneficiaries. • An assessment is made of local supply chain management (SCM) capabilities and logistics infrastructure and a co-ordinated, efficient SCM system is established, using local capacity where this is feasible. • The assessment considers the availability of locally sourced food commodities. • The award of contracts for SCM services is transparent, fair and open. • Staff at all levels of the SCM system are adequately trained and observe procedures relating to food quality and safety. • Appropriate inventory accounting, reporting and financial systems are in place to ensure accountability at all levels of the SCM system • Care is taken to minimise losses, including through theft, and all losses are accounted for. • The food pipeline is monitored and maintained in such a way that any interruption to distribution is avoided. • Information on the performance of the supply chain is provided to all stakeholders on a regular basis. Link 113 BACK Food aid management standard 3: distribution (source: SPHERE) • Recipients of food aid are identified and targeted on the basis of need, by means of an assessment carried out through consultation with stakeholders, including community groups • Efficient and equitable distribution methods are designed in consultation with local groups and partner organisations, and involve the various recipient groups. • The point of distribution is as close as possible to recipients’ homes to ensure easy access and safety. • Recipients are informed well in advance of the quality and quantity of the food ration and the distribution plan. • The performance and effectiveness of the food aid programme are properly monitored and evaluated. Link 114 BACK Shelter and settlement standard 1: strategic planning (source: SPHERE) • Affected households return to the site of their original dwellings where possible. • Affected households who cannot return to the site of their original dwellings settle independently within a host community or with host families where possible. • Affected households who cannot return to the site of their original dwellings or who cannot settle independently within a host community or with host families are accommodated in mass shelters or in temporary planned or self-settled camps. • Actual or potential threats to the security of the affected population are assessed, and the dwellings or settlements are located at a safe distance from any such external threats. • Risks from natural hazards including earthquakes, volcanic activity, landslides, flooding or high winds are minimised, and the area is not prone to diseases or significant vector risks. • Locations are free of potentially hazardous equipment or material, and existing hazards such as dangerous structures, debris or unstable ground are identified and made safe, or access is restricted and guarded. • Land and property ownership and/or use rights for buildings or locations are established prior to occupation and permitted use is agreed as necessary. • Water and sanitation services, and social facilities including health care, schools and places of worship, are available or can be satisfactorily provided. • The transportation infrastructure provides access to the settlement for personal movement and the provision of services. • Where possible, households can access land, markets or services for the continuation or development of livelihood support activities. Link 115 BACK Shelter and settlement standard 2: physical planning (source: SPHERE) • Area or cluster planning by family, neighbourhood or village groups as appropriate supports existing social networks, contributes to security and enables self- management by the affected population. • All members of the affected population have safe access to water, sanitary facilities, health care, solid waste disposal, graveyards and social facilities, including schools, places of worship, meeting points and recreational areas. • Temporary planned or self-settled camps are based on a minimum surface area of 45m2 for each person. • The surface topography is used or augmented to facilitate water drainage, and the ground conditions are suitable for excavating toilet pits where this is the primary sanitation system. • There are roads and pathways to provide safe, secure and all-weather access to the individual dwellings and facilities. • Mass shelters have openings to enable required access and emergency evacuation, and these openings are positioned so that access is well supervised and does not pose a security threat to occupants. • Vector risks are minimised. Link 116 BACK Shelter and settlement standard 3: covered living space (source: SPHERE) • The initial covered floor area per person is at least 3.5m2. • The covered area enables safe separation and privacy between the sexes, between different age groups and between separate families within a given household as required. • Essential household activities can be carried out within the shelter. • Key livelihood support activities are accommodated where possible. Link 117 BACK Shelter and settlement standard 4: design (source: SPHERE) • The design of the shelter and the materials used are familiar where possible and culturally and socially acceptable. • The repair of existing damaged shelters or the upgrading of initial shelter solutions constructed by the disaster-affected population is prioritised. • Alternative materials required to provide temporary shelter are durable, practical and acceptable to the affected population. • The type of construction, materials used and the sizing and positioning of openings provides optimal thermal comfort and ventilation. • Access to water supply sources and sanitation facilities, and the appropriate provision of rainwater harvesting, water storage, drainage and solid waste management, complement the construction of shelters. • Vector control measures are incorporated into the design and materials are selected to minimise health hazards. Link 118 BACK Shelter and settlement standard 5: construction (source: SPHERE) • Locally sourced materials and labour are used without adversely affecting the local economy or environment. • Locally derived standards of workmanship and materials are achieved. • Construction and material specifications mitigate against future natural disasters. • The type of construction and materials used enable the maintenance and upgrading of individual household shelters using locally available tools and resources. • The procurement of materials and labour and the supervision of the construction process are transparent, accountable and in accordance with internationally accepted bidding, purchasing and construction administration practices. Link 119 BACK Shelter and settlement standard 6: environmental impact (source: SPHERE) • The temporary or permanent settling of the affected population considers the extent of the natural resources available. • Natural resources are managed to meet the ongoing needs of the displaced and host populations. • The production and supply of construction material and the building process minimises the long-term depletion of natural resources. • Trees and other vegetation are retained where possible to increase water retention, minimise soil erosion and to provide shade. • The locations of mass shelters or temporary planned camps are returned to their original condition, unless agreed otherwise, once they are no longer needed for emergency shelter use. Link 120 BACK Non-food items standard 1: clothing and bedding (source: SPHERE) • Women, girls, men and boys have at least one full set of clothing in the correct size, appropriate to the culture, season and climate. • Infants and children up to two years old also have a blanket of a minimum 100cmx70cm. • People have access to a combination of blankets, bedding or sleeping mats to provide thermal comfort and to enable separate sleeping arrangements as required. • Those individuals most at risk have additional clothing and bedding to meet their needs. • Culturally appropriate burial cloth is available when needed. Link 121 BACK Non-food items standard 2: personal hygiene (source: SPHERE) • Each person has access to 250g of bathing soap per month. • Each person has access to 200g of laundry soap per month. • Women and girls have sanitary materials for menstruation. • Infants and children up to two years old have 12 washable nappies or diapers where these are typically used. • Additional items essential for ensuring personal hygiene, dignity and well-being can be accessed. Link 122 BACK Non-food items standard 3: cooking and eating utensils (source: SPHERE) • Each household has access to a large-sized cooking pot with handle and a pan to act as a lid; a medium-sized cooking pot with handle and lid; a basin for food preparation or serving; a kitchen knife; and two wooden serving spoons. • Each household has access to two 10- to 20-litre water collection vessels with a lid or cap (20-litre jerry can with a screw cap or 10-litre bucket with lid), plus additional water or food storage vessels. • Each person has access to a dished plate, a metal spoon and a mug or drinking vessel. Link 123 BACK Non-food items standard 4: stoves, fuel and lighting (source: SPHERE) • Where food is cooked on an individual household basis, each household has a stove and fuel to meet essential cooking and heating needs. • Environmentally and economically sustainable sources of fuel are identified and prioritised over fuel provided from external sources. • Fuel is obtained in a safe and secure manner, and there are no reports of incidents of harm to people in the routine collection of fuel. • Safe fuel storage space is available. • Each household has access to sustainable means of providing artificial lighting, e.g. lanterns or candles. • Each household has access to matches or a suitable alternative means of igniting fuel or candles, etc. Link 124 BACK Non-food items standard 5: tools and equipment (source: SPHERE) • Where responsible for constructing part or all of their shelters or for carrying out essential maintenance, each household has access to tools and equipment to safely undertake each task. • Training or guidance in the use of the tools and in the shelter construction or maintenance tasks required is provided where necessary. • Materials to reduce the spread of vector-borne disease, such as impregnated mosquito nets, are provided to protect each member of the household (see Vector control standards 1-3). Link 125 BACK Health systems and infrastructure standard 1: prioritising health services (source: SPHERE) • The major causes of mortality and morbidity are identified, documented and monitored. • Priority health services include the most appropriate and effective interventions to reduce excess morbidity and mortality. • All members of the community, including vulnerable groups, have access to priority health interventions. • Local health authorities and community members participate in the design and implementation of priority health interventions. • There is active collaboration with other sectors in the design and implementation of priority health interventions, including water and sanitation, food security, nutrition, shelter and protection. • The crude mortality rate (CMR) is maintained at, or reduced to, less than twice the baseline rate documented for the population prior to the disaster. • The under-5 mortality rate (U5MR) is maintained at, or reduced to, less than twice the baseline rate documented for the population prior to the disaster. Link 126 BACK Health systems and infrastructure standard 2: supporting national and local health systems (source: SPHERE) • Representatives of the Ministry of Health lead the health sector response, whenever possible. • When the Ministry of Health lacks the necessary capacity, an alternate agency with the requisite capacity is identified to take the lead in the health sector. • Local health facilities are supported and strengthened by responding agencies. • Local health workers are supported and integrated into health services, taking account of gender and ethnic balance. • Health services incorporate or adapt the existing national standards and guidelines of the disaster-affected or host country. • No alternate or parallel health facilities and services are established, including foreign field hospitals, unless local capacities are exceeded or the population does not have ready access to existing services. The lead health authority is consulted on this issue. Link 127 BACK Health systems and infrastructure standard 3: coordination (source: SPHERE) • Coordination mechanisms are established at central level (national or regional) and at field level within the health sector, and between health and other sectors. • Specific responsibilities of each health agency are clarified and documented in consultation with the lead health authority to ensure optimal coverage of the population and complementarity of services. • Regular health sector coordination meetings are held for local and external partners at both central and field levels. Link 128 BACK Health systems and infrastructure standard 4: primary health care (source: SPHERE) • All people have access to health information that allows them to protect and promote their own health and well-being. • Health services are provided at the appropriate level of the health system: household/ community, peripheral health facilities, central health facilities, referral hospital. • A standardised referral system is established by the lead health authority and utilised by health agencies. Suitable transportation is organised for patients to reach the referral facility. • Health services and interventions are based on scientifically sound methods and are evidence-based, whenever possible. • Health services and interventions utilise appropriate technology, and are socially and culturally acceptable. Link 129 BACK Health systems and infrastructure standard 5: clinical services (source: SPHERE) • The number, level and location of health facilities are appropriate to meet the needs of the population. • The number, skills and gender/ethnic balance of staff at each health facility are appropriate to meet the needs of the population. • Adequate staffing levels are achieved so that clinicians are not required to consistently consult on more than 50 patients per day. If this threshold is regularly exceeded, additional clinical staff are recruited. • Utilisation rates at health facilities are monitored and corrective measures taken if there is over- or under-utilisation. • Standardised case management protocols are established by the lead health authority, and adhered to by health agencies. • A standardised essential drug list is established by the lead health authority, and adhered to by health agencies. • Clinical staff are trained and supervised in the use of the protocols and the essential drug list • People have access to a consistent supply of essential drugs through a standardised drug management system that follows accepted guidelines. • Drug donations are accepted only if they follow internationally recognised guidelines. Donations that do not follow these guidelines are not used and are disposed of safely. • Bodies of the deceased are disposed of in a manner that is dignified, culturally appropriate and is based on good public health practice. Link 130 BACK Health systems and infrastructure standard 6: health information systems (source: SPHERE) • A standardised health information system (HIS) is implemented by all health agencies to routinely collect relevant data on demographics, mortality, morbidity and health services. • A designated HIS coordinating agency (or agencies) is identified to organise and supervise the system. • Health facilities and agencies submit surveillance data to the designated HIS coordinating agency on a regular basis. The frequency of these reports will vary according to the context, e.g. daily, weekly, monthly. • A regular epidemiological report, including analysis and interpretation of the data, is produced by the HIS coordinating agency and shared with all relevant agencies, decision-makers and the community. The frequency of the report will vary according to the context, e.g. daily, weekly, monthly. • Agencies take adequate precautions for the protection of data to guarantee the rights and safety of individuals and/or populations. • The HIS includes an early warning component to ensure timely detection of and response to infectious disease outbreaks (see Control of communicable diseases standard 5). • Supplementary data from other relevant sources are consistently used to interpret surveillance data and to guide decision-making. Link 131 BACK Control of communicable diseases standard 1: prevention (source: SPHERE) • General prevention measures are developed and implemented in coordination with other relevant sectors. • Community health education messages provide individuals with information on how to prevent common communicable diseases and how to access relevant services (see Health systems and infrastructure standard 4). • Specific prevention measures, such as a mass measles vaccination campaign and Expanded Programme on Immunisation (EPI), are implemented as indicated. Link 132 BACK Control of communicable diseases standard 2: measles prevention (source: SPHERE) • An estimation of measles vaccination coverage of children aged 9 months to 15 years is made at the outset of the emergency response, to determine the prevalence of susceptibility to measles. • If vaccination coverage is estimated to be less than 90%, a mass measles vaccination campaign for children aged 6 months to 15 years (including administration of vitamin A to children aged 6-59 months) is initiated. The vaccination campaign is coordinated with national and local health authorities, including the Expanded Programme on Immunisation. • Upon completion of the campaign: – at least 95% of children aged 6 months to 15 years have received measles vaccination; – at least 95% of children aged 6-59 months have received an appropriate dose of vitamin A. • All infants vaccinated between 6-9 months of age receive another dose of measles vaccine upon reaching 9 months. • Routine ongoing vaccination of 9-month-old children is established to ensure the maintenance of the minimum 95% coverage. This system is linked to the Expanded Programme on Immunisation. • For mobile or displaced populations, an ongoing system is established to ensure that at least 95% of newcomers aged between 6 months and 15 years receive vaccination against measles. Link 133 BACK Control of communicable diseases standard 3: diagnosis and case management (source: SPHERE) • Standardised case management protocols for diagnosis and treatment of the most common infectious diseases are consistently used (see also Health systems and infrastructure standard 5). • Public health education messages encourage people to seek early care for fever, cough, diarrhoea, etc., especially children, pregnant women and older people. • In malaria-endemic regions, a protocol is established to ensure early (<24 hours) diagnosis of fever cases and treatment with highly effective first-line drugs. • Laboratory services are available and utilised when indicated. • A tuberculosis control programme is introduced only after consideration of recognised criteria. Link 134 BACK Control of communicable diseases standard 4: outbreak preparedness (source: SPHERE) • An outbreak investigation and control plan is prepared. • Protocols for the investigation and control of common outbreaks are available and distributed to relevant staff. • Staff receive training in the principles of outbreak investigation and control, including relevant treatment protocols. • Reserve stocks of essential drugs, medical supplies, vaccines and basic protection material are available and can be procured rapidly. • Sources of vaccines for relevant outbreaks (e.g. measles, meningococcal meningitis, yellow fever) are identified for rapid procurement and use. Mechanisms for rapid procurement are established. • Sites for the isolation and treatment of infectious patients are identified in advance, e.g. cholera treatment centres. • A laboratory is identified, whether locally, regionally, nationally or in another country, that can provide confirmation of diagnoses. • Sampling materials and transport media for the infectious agents most likely to cause a sudden outbreak are available on-site, to permit transfer of specimens to an appropriate laboratory. In addition, several rapid tests may be stored on-site. Link 135 BACK Control of communicable diseases standard 5: outbreak detection, investigation and response (source: SPHERE) • The health information system (HIS) includes an early warning component. • Initiation of outbreak investigation occurs within 24 hours of notification. • The outbreak is described according to time, place and person, leading to the identification of high-risk groups. Adequate precautions are taken to protect the safety of both individuals and data. • Appropriate control measures that are specific to the disease and context are implemented as soon as possible. • Case fatality rates are maintained at acceptable levels: – cholera: 1% or lower – Shigella dysentery: 1% or lower – typhoid: 1% or lower – meningococcal meningitis: varies. Link 136 BACK Control of communicable diseases standard 6: HIV/AIDS (source: SPHERE) • People have access to the following essential package of services during the disaster phase: – free male condoms and promotion of proper condom use; – universal precautions to prevent iatrogenic/nosocomial transmission in emergency and health-care settings; – safe blood supply; – relevant information and education so that individuals can take steps to protect themselves against HIV transmission; – syndromic case management of sexually transmitted infections (STIs); – prevention and management of the consequences of sexual violence; – basic health care for people living with HIV/AIDS (PLWH/A). • Plans are initiated to broaden the range of HIV control services in the post- disaster phase. Link 137 BACK Control of non-communicable diseases standard 1: injury (source: SPHERE) • In situations with a large number of injured patients, a standardised system of triage is established to guide health care providers on assessment, prioritisation, basic resuscitation and referral. • Standardised guidelines for the provision of first aid and basic resuscitation are established. • Standardised protocols for the referral of injured patients for advanced care, including surgery, are established. Suitable transportation is organised for patients to reach the referral facility. • Definitive trauma and surgical services are established only by agencies with appropriate expertise and resources. • In situations with a potentially large number of injured patients, contingency plans for the management of multiple casualties are developed for relevant health care facilities. These plans are related to district and regional plans. Link 138 BACK Control of non-communicable diseases standard 2: reproductive health (source: SPHERE) • An organisation(s) and individual(s) are identified to facilitate the coordination and implementation of the MISP in consultation with the lead health authority. • Steps are taken by health agencies to prevent and manage the consequences of gender-based violence (GBV), in coordination with other relevant sectors, especially protection and community services. • The number of cases of sexual and other forms of GBV reported to health services, protection and security officers is monitored and reported to a designated lead GBV agency (or agencies). Rules of confidentiality are applied to data collection and review. • The minimum package of services to prevent the transmission of HIV/AIDS is implemented (see Control of communicable diseases standard 6). • Adequate numbers of clean delivery kits, based on the estimated number of births in a given time period, are available and distributed to visibly pregnant women and skilled/traditional birth attendants to promote clean home deliveries. • Adequate numbers of midwife delivery kits (UNICEF or equivalent) are distributed to health facilities to ensure clean and safe deliveries. • A standardised referral system is established and promoted within the community, incorporating midwives and skilled/traditional birth attendants, to manage obstetric emergencies. Suitable transportation is organised for patients to reach the referral facility. • Plans are initiated to implement a comprehensive range of reproductive health services integrated into primary health care as soon as possible. Link 139 BACK Control of non-communicable diseases standard 3: mental and social aspects of health (source: SPHERE) • People have access to an ongoing, reliable flow of credible information on the disaster and associated relief efforts. • Normal cultural and religious events are maintained or re-established (including grieving rituals conducted by relevant spiritual and religious practitioners). People are able to conduct funeral ceremonies. • As soon as resources permit, children and adolescents have access to formal or informal schooling and to normal recreational activities. • Adults and adolescents are able to participate in concrete, purposeful, common interest activities, such as emergency relief activities. • Isolated persons, such as separated or orphaned children, child combatants, widows and widowers, older people or others without their families, have access to activities that facilitate their inclusion in social networks. • When necessary, a tracing service is established to reunite people and families. • Where people are displaced, shelter is organised with the aim of keeping family members and communities together. • The community is consulted regarding decisions on where to locate religious places, schools, water points and sanitation facilities. The design of settlements for displaced people includes recreational and cultural space (see Shelter and settlement standards 1-2). Link 140 BACK Key psychological and psychiatric intervention indicators (source: SPHERE) (to be read in conjunction with the guidance notes) • Individuals experiencing acute mental distress after exposure to traumatic stressors have access to psychological first aid at health service facilities and in the community. • Care for urgent psychiatric complaints is available through the primary health care system. Essential psychiatric medications, consistent with the essential drug list, are available at primary care facilities. • Individuals with pre-existing psychiatric disorders continue to receive relevant treatment, and harmful, sudden discontinuation of medications is avoided. Basic needs of patients in custodial psychiatric hospitals are addressed. • If the disaster becomes protracted, plans are initiated to provide a more comprehensive range of community-based psychological interventions for the post-disaster phase. Link 141 BACK Control of non-communicable diseases standard 4: chronic diseases (source: SPHERE) • A specific agency (or agencies) is designated to coordinate programmes for individuals with chronic diseases for which an acute cessation of therapy is likely to result in death. • Individuals with such chronic diseases are actively identified and registered. • Medications for the routine, ongoing management of chronic diseases are available through the primary health care system, provided that these medications are specified on the essential drug list. Link 142 BACK Reporting (source: WFP) • What are the type and frequency of reporting for the operation, including periodic participatory appraisals? Were reports submitted on time? Were they complete? What problems were experienced? Is the information analysed and used to make decisions regarding the management of the operation? Link 143 BACK Fisheries (source: USAID/OFDA, Program guidelines 2006) Capture Fisheries: - Number of people provided fishing or processing equipment - Number of people trained - Number of kilograms of fish captured/fisher/month - Average number of kilograms of fish consumed/week/fisher household - Average monthly income from captured fish/fisher or household Aquaculture: - Number of fish farmers provided equipment - Number of fish farmers trained - Number of kilograms of fish harvested/fish farmer or household/six-month period - Average number of kilograms of fish consumed/week/fish farmer household - Average income (in USD) from fish sales/fish farmer/week Link 144 BACK Livestock (source: USAID/OFDA, Program guidelines 2006) - Number and percentage of targeted animals vaccinated - Number of CAHWs trained - Number of beneficiaries provided equipment Link 145 BACK Pests and Pesticides (source: USAID/OFDA, Program guidelines 2006) - Number of hectares treated against target pests - Number of types of crops/pasture saved/protected against - Liters/kilograms of obsolete and unusable pesticides removed and/or disposed safely - Number of empty pesticide containers collected and disposed safely - Number of farmers/pastoralists trained in ETOP operations Link 146 BACK Seed Systems and Agricultural Inputs (source: USAID/OFDA, Program guidelines 2006) - Actual number of hectares (ha) planted with distributed seeds - Number and percent of seed recipient farmers with sufficient seeds to plant fields for next agricultural seasons - Number of months of food self-sufficiency due to distributed seed production for beneficiary families - Number of gardens planted with distributed seed - Number of seedlings provided Link 147 BACK Veterinary Medicines and Vaccines (source: USAID/OFDA, Program guidelines 2006) - Number of veterinary diseases/conditions identified - Prevalence of disease in area of intervention, represented as both the percentage of total herd as well as the total number of affected animals. - Number of interventions, treatments or vaccinations administered. - Increase in number and percentage of animals vaccinated, by type, from baseline of pre-program numbers vaccinated Link 148 BACK Economic Recovery (LRRD) (source: USAID/OFDA, Program guidelines 2006) Individual Level - USD amount increase in purchasing power per beneficiary. This may consist of the total voucher amount per beneficiary, cash distribution amount, or cash-for-work payout. Note: For Cash-for-work payout, the “average pay/beneficiary over the life of the project” is preferable to “pay/beneficiary/day.” Market Level - Number of markets restored - Kilometers of roads rehabilitated - Number of fairs - Total USD grant amount of fairs Local Economy Level - Total value in USD of all assistance provided directly to beneficiaries. This may consist of the sum of all cash-for-work salaries, cash distribution, vouchers, and value of locally-procured commodities. This indicator is designed to estimate the amount of money that is infused into the local economy through beneficiary-oriented activities. The indicator should not include staff salaries, operation expenses (office rent, vehicle rental, etc.), office supplies, etc. Link 149 BACK Health (General Indicators) (source: USAID/OFDA, Program guidelines 2006) - Number of USAID/OFDA-supported health facilities providing services to manage the most common diseases causing morbidity and mortality in the target population, including maternal and neonatal health - Number of health providers and/or community members trained in areas such as the prevention and management of the most common diseases causing morbidity and mortality, maternal and neonatal health, EPI, HIV/AIDS prevention, and health education - Utilization rates of USAID/OFDA supported health facilities - Reduction in CMR or CDR with a target of less than 2x baseline or ≤1/10,000/day if baseline unknown) - Reduction in U5MR or 0-5DR maintained or reduced (target less than 2x baseline or U5MR ≤2/10,000/day if baseline unknown.) Link 150 BACK Child Health (source: USAID/OFDA, Program guidelines 2006) Malaria (in malaria endemic countries) • Number and percent of total population, pregnant women, and < 5 effectively treated for malaria with appropriate anti-malarial -Percent of coverage with ITNs of target population (total, <5 and pregnant women) -Percent utilization of ITNs in target population (total, <5 and pregnant women) -Incidence rate of malaria (total, <5 and pregnant women) -Proportional morbidity from malaria (total, <5 and pregnant women) -Proportional mortality from malaria (total, <5 and pregnant women) Acute Respiratory Tract Infections (ARI) -Number and percent of population < 5 effectively treated for ARI with appropriate antibiotic -Proportional morbidity of children < 5 years of age from ARI -Proportional mortality of children < 5 years of age from ARI Diarrhea -Number and percent of population < 5 effectively treated for diarrhea with ORT and zinc (antibiotic for dysentery) at the health facility and/or community level -Percentage of children exclusively breastfed for at least 6 months or continued feeding -Incidence rate of diarrhea -Proportional morbidity of children < 5 years of age from diarrhea -Proportional mortality of children < 5 years of age from diarrhea Link 151 BACK Maternal Health (source: USAID/OFDA, Program guidelines 2006) • Number of providers trained in management of clean deliveries and detection of danger signs. (Indicate the type of provider, such as midwife or TBA.) • Percentage of pregnant women who have attended at least 2 ANC visits • Percentage of pregnant women receiving iron and folic acid at the ANC • Percentage of pregnant women who received a clean delivery kit • Percentage of women vaccinated with tetanus toxoid (TT2) • Percentage of pregnant women in their 2nd and 3rd trimester attending an ANC who receive intermittent presumptive therapy for malaria -Percentage of pregnant women utilizing ITNs • Percentage of pregnant women who deliver assisted by a trained provider (indicate type of provider such as midwife, TBA) • Referral system for obstetrical emergencies in place • Percentage of women who attend at least one postpartum visit with trained provider (including provision of Vitamin A where appropriate) • Maternal mortality ratio (MMR) -Neonatal mortality rate Link 152 BACK Expanded Program of Immunization (EPI) (source: USAID/OFDA, Program guidelines 2006) • Immunization coverage <1 year age (DPT3, Polio3). • Measles immunization coverage, ages 6 month to 15 years old. • Vitamin A coverage, 6 to 59 months. Link 153 BACK Health Education (source: USAID/OFDA, Program guidelines 2006) • Number of CHW trained in the prevention and community based management of diarrhea, ARI, malaria, measles, and maternal and infant health • Number of health education and promotion sessions (or number of population targeted) provided to communities • Percentage of target population properly washing hands at appropriate times (see hygiene section of WASH for more details) • Percentage of target population with knowledge of and practicing two methods to prevent diarrhea • Percentage of population properly managing diarrhea at home • Percentage of women exclusively breastfeeding for six months • Percentage of target population able to identify the transmission and prevention of malaria • Percentage of utilization rate of ITNs in total population, children under 5, and pregnant women • Percentage of target population with knowledge of when to seek care for children with respiratory difficulty, fever and diarrhea (dehydration), complications of pregnancy Link 154 BACK HIV/AIDS source: USAID/OFDA, Program guidelines 2006) • Number of providers (indicate type of provider, such as CHW, midwives, or nurses) trained in the prevention of HIV/AIDS and STIs • Number of health education sessions (or number of population targeted) provided to communities • Percentage of target population with the knowledge of two types of transmission and prevention of HIV/AIDS • Percentage of health facilities were standard precautions are routinely practiced • Percentage of referral centers with safe blood supply (based on laboratory validation) Link 155 BACK Essential Medicines / Pharmaceuticals (source: USAID/OFDA, Program guidelines 2006) • Number of patients treated for a particular targeted disease or condition. • Number of facilities where prescription records kept for medicines dispensed • Number of healthcare providers involved in project, including Physicians, Pharmacists, Nurses, Dentists, and Midwives. Link 156 BACK Community Therapeutic Care (source: USAID/OFDA, Program guidelines 2006) • General -GAM and SAM rates decreased to pre-crisis level. (Provide current crisis level and pre-crisis data.) • CTC program as a whole: - Coverage rate: in rural areas >70%; in urban areas >70%; in IDP camps > 90% • Number of beneficiaries treated in the community-based therapeutic care program, disaggregated by age, under- fives, and adults and disaggregated by patients treated in the stabilization centers and the outpatient therapeutic programs Particular to Outpatient Therapeutic Programs (OTP) -Default rate: <15% • Death rate: <10% -Average length of stay in OTP: <60 days -Weight gain: >4g/kg/ day Particular to Stabilization Centers (SC) -Average length of stay in SC 4-7 days • Referrals to hospital are <10% of exits Link 157 BACK Nutrition Education (source: USAID/OFDA, Program guidelines 2006) • Number of beneficiaries receiving nutrition education. -Percentage change in practice pertaining to nutrition education topics. -Number of health care providers trained in the treatment of moderate and severe acute malnutrition Link 158 BACK Supplementary Feeding Programs (source: USAID/OFDA, Program guidelines 2006) • MAM rates decreased to pre-crisis level. (Provide current crisis level and pre-crisis data, focusing on moderate acute malnutrition rates.) • Stand Alone SFP: Coverage rate: in rural areas >50%; in urban areas >70%; in IDP camps >90% (Sphere Guidelines) • If the SFP is part of a CTC program then the coverage rate should be as follows: in rural areas >70%; in urban areas >70%; in IDP camps >90% -Default rate: <15% • Cure rate: >75% -Death rate: <3% -Number of beneficiaries treated in the SFP (disaggregated by under-fives and adults) Link 159 BACK Therapeutic Feeding Programs (source: USAID/OFDA, Program guidelines 2006) • SAM rates decreased to pre-crisis level (provide current crisis level and pre-crisis data) • Number of beneficiaries treated in the TFP (disaggregated by under- fives and adults) • Coverage rate: in rural areas >50%; in urban areas >70%; in IDP camps > 90% • Default rate: <15% -Death rate: <10% -Cure rate: >75% Link 160 BACK Child-Friendly Spaces and/or Youth Centers (source: USAID/OFDA, Program guidelines 2006) • Number of spaces created. • Percent of targeted children using the spaces/centers. -Number of youth trained in a vocation. Link 161 BACK Coordination, Personnel, and/or Advocacy (source: USAID/OFDA, Program guidelines 2006) • Number of protection officers provided. • Numbers of policies or procedures modified as a result of this program to include protection language. Link 162 BACK Family Reunification and/or Child Tracing (source: USAID/OFDA, Program guidelines 2006) • Number of children united with their families. • Number of systems established to coordinate reunification and tracing. Link 163 BACK Gender-Based Violence and/or Women’s Centers (source: USAID/OFDA, Program guidelines 2006) • Number and type of GBV services made available to targeted population. • Number of complex GBV cases successfully referred to specialists. • Number of women trained/sensitized in GBV issues • Number of men trained/sensitized in GBV issues. Link 164 BACK Psychosocial Services (source: USAID/OFDA, Program guidelines 2006) • Percentage of targeted participants returning to productive family and community roles and responsibilities. • Numbers of individuals experiencing acute mental distress provided psychological first aid at health service facilities or in the community. • Number of individuals with psychiatric disorders referred to specialists and equipped facilities. Link 165 BACK Hydrometeorological Disasters (source: USAID/OFDA, Program guidelines 2006) • Percent improvement in the lead time for potential hydrometeorological hazards. • Number and percentage of participants retaining information gained during training. • Number of policies/procedures improved as a result of the program. • Number of people trained in hydrometeorological -related activities. • Number of people who will benefit from proposed hydrometeorological activities. Link 166 BACK Geological Disasters (source: USAID/OFDA, Program guidelines 2006) • Number of people who will benefit from proposed geologic-disaster related activities. • Percent improvement in warning time for potential geologic events. • Number of policies or procedures modified as a result of the program to increase the preparedness for geologic events. • Number of people trained to reduce the impact of geologic events. Link 167 BACK Technological Disasters (source: USAID/OFDA, Program guidelines 2006) • Number of people who will benefit from proposed activities. • Number of people who have retained knowledge obtained during training three months after the training. • Number of policies/procedures that have been modified to reduce risks to technological disasters. • Number of people trained on responding or prevention of technological disasters. Link 168 BACK Disaster Preparedness, Mitigation, and Management (source: USAID/OFDA, Program guidelines 2006) • Percent improvement in time in responding to disaster. • Percent reduction in disaster response time. • Number of people retaining knowledge gained three months after training. • Change in the policies, legislation, regulations or behavior of communities, local and regional entities on disaster risk reduction and management. • Number of national hazard risk reduction plans or policies developed. • Number of people trained in disaster risk reduction and management. • Number of people who will benefit from reduced risk. Link 169 BACK Camp Design and Management (source: USAID/OFDA, Program guidelines 2006) • Number of households receiving shelter according to Sphere Guidelines. • Percentage of total affected population receiving shelter assistance. • Amount and percentage of approved project budget spent in affected local economy Link 170 BACK Emergency Shelter (source: USAID/OFDA, Program guidelines 2006) • Number of households receiving shelter according to Sphere Guidelines • Percentage of total affected population receiving shelter assistance • Amount and percentage of approved project budget spent in the affected economy Link 171 BACK Shelter Hazard Mitigation and Environmental Management (source: USAID/OFDA, Program guidelines 2006) • Number of shelters incorporating hazard mitigation measures • Number of settlements adopting hazard mitigation measures • Number of people receiving training • Number of project trainees retaining knowledge received for identified activities • Number of non-beneficiaries who learned and applied mitigation measures Link 172 BACK Transitional Shelter and Settlements (source: USAID/OFDA, Program guidelines 2006) • Number of households receiving shelter according to Sphere Guidelines • Percentage share of total affected population receiving shelter assistance • Amount and percentage of approved project budget spent in the affected economy Link 173 BACK Hygiene Promotion (source: USAID/OFDA, Program guidelines 2006) • Number of water and sanitation committees revitalized, trained and/or established • Average cleanliness of water points three months after their completion (reported as percent of water points) • Average increase in good hand washing practices (reported as percentage of people) • Average increase in correct water usage practices (reported as percentage of people) Link 174 BACK Sanitation (Household Level) (source: USAID/OFDA, Program guidelines 2006) • Number of household latrines constructed • Number of beneficiaries benefiting from household latrines • Number of communal/institutional latrines constructed • Number of beneficiaries benefiting from communal/institutional latrines • Number of household hand washing facilities introduced • Number of communal hand washing facilities introduced • Number of household waste management pits dug Link 175 BACK Other Environmental Health (source: USAID/OFDA, Program guidelines 2006) • Number of community waste management facilities constructed • Average cleanliness of community waste management facilities 3 months after their completion (reported as the percent of facilities) • Number of animal pens constructed -Number of debris cleanups undertaken Link 176 BACK Water (source: USAID/OFDA, Program guidelines 2006) • Number of protected water points established • Geo-coordinates of every protected water point established • Number of beneficiaries receiving water from protected water points • Number of liters available per person per day before the intervention • Number of liters available per person per day after the intervention • Number of minutes a family takes to collect water each day before the intervention • Number of minutes a family takes to collect water each day after the intervention • Percent of water points with 0 fecal coliforms per 100 ml • Percent of household water supplies with 0 fecal coliforms per 100 ml Link 177 BACK Artisanal Production (source: USAID/OFDA, Program guidelines 2006) • Number and percentage of participants with increased production • Number and percentage of participants with increased sales • Average increase in income of participants in USD over the life of the program Link 178 BACK Capacity Building / Training (source: USAID/OFDA, Program guidelines 2006) • Number of people trained • Number of people hired post-training • Percentage of beneficiaries using their skills training within their livelihood activity Link 179 BACK Cash Distribution (source: USAID/OFDA, Program guidelines 2006) • Amount of money distributed per beneficiary. Link 180 BACK Cash for Work (CFW) (source: USAID/OFDA, Program guidelines 2006) • Number of people employed in the CFW. • Percentage of beneficiaries reporting increased access to food and non-food items as a result of the program. Link 181 BACK Gender Relations (source: USAID/OFDA, Program guidelines 2006) • Percentage of the men and women in the targeted population or who report improvements in gender relations as a result of the activities Link 182 BACK Internally Displaced Populations (IDPs) (source: USAID/OFDA, Program guidelines 2006) • Total number of IDPs assisted • Number of male IDPs assisted • Number of female IDPs assisted. Link 183 BACK Livelihoods / Income Generation (source: USAID/OFDA, Program guidelines 2006) • Number of people assisted with livelihoods interventions • Average amount of income in USD generated by participant/month Link 184 BACK Market Rehabilitation (source: USAID/OFDA, Program guidelines 2006) • Number of markets rehabilitated/created • Number of cooperatives created • Number of participants trained in marketing skills • Percentage of targeted beneficiaries with increased engagement in economic activities • Number/percentage of targeted small businesses with increased revenues relative to baseline Link 185 BACK Micro-Finance / Micro-Credit (source: USAID/OFDA, Program guidelines 2006) • Number of individuals receiving micro-credit assistance • Number and percentage of micro-credit recipients who rehabilitate or establish new businesses • Average amount (in USD) of increase in income of participants over the life of the program (based on pre-program baseline) Link 186 BACK Protection Mainstreaming (source: USAID/OFDA, Program guidelines 2006) • Coordination / Information Management Sector: - Percent of disseminated documents that include protection information - Percent of disseminated documents that safeguard sensitive information. • Economy and Market Systems Sector: - Percent of income-generating activities that reduce risks for targeted population • Agriculture and Food Security Sector: - Percent of vulnerable populations receiving full designated rations - Percentage of women escorted to agricultural fields in insecure areas • Health and Nutrition Sectors: - Number and types of gender-based violence (GBV) services provided - Percentage of targeted staff trained in protection sensitivity - Number of systems of privacy protocols established and adhered to • Shelter and Settlements Sector: - Number of settlement solutions provided that reduce the need for large camps - Percent of camps sited away from conflict and borders • WASH Sector: - Percentage of latrines built with separate facilities for men and women - Percentage of latrine facilities providing escort services for women and girls where necessary Link 187 BACK Returnees (source: USAID/OFDA, Program guidelines 2006) • Number of returnees assisted • Number of returnees settling permanently in home village or new area Link 188 BACK Slavery / Trafficking (source: USAID/OFDA, Program guidelines 2006) • Number of people trained in slavery and trafficking sensitivity • Number of individuals removed from a slavery/trafficked status Link 189 BACK Vouchers (source: USAID/OFDA, Program guidelines 2006) • Number of people receiving vouchers • Number of merchants participating in the voucher activity • Percentage of vouchers redeemed • Total monetary value of vouchers redeemed • Percentage of types of goods procured Link 192: BACK Organisational Performance, a discussion paper, S. Lawry White, 1998. Link 193: BACK the Good Enough Guide – Impact measurement and Accountability in Emergencies, the Emergency Capacity Building project (CARE; CRS; IRC; Mercy Corps; Oxfam GB; Save the Children; WVI), 2007 http://publications.oxfam.org.uk/oxfam/display.asp?K=_2006111410173391&sf_01=CTITLE &st_01=the+good+enough+guide&sort=SORT_DATE%2FD&x=16&y=12&m=1&dc=1 Link 194: BACK Questions to help test whether a tool is 'good enough' (source: The Good Enough Guide, link 193) • Can we use this tool without endangering field staff and the people affected by the emergency? • Does it meet essential requirements in this context at this time? • Is it realistic? • Do we have the resources – time, staff, volunteers and money – to use it? • Is it useful for those applying it? • Is it as simple as necessary? • Have we referred to widely accepted humanitarian values, standards and guidelines? • Will it be 'good enough' tomorrow? When will we review our use of this tool? Link 195: BACK USAID/OFDA has embarked on a programme in collaboration with CARE International and the Benfield Hazard Research Centre in UK to develop an appropriate analytical framework to identify significant environmental issues in emergencies. Member states (specifically Norway) also contribute to this programme. The programme has developed a simple methodology, known as Rapid Environmental Assessment (REA) which is now in its second phase and has been field tested and improved in a number of situations. Training courses in the use of REA are run regularly in different parts of the world and several of DG ECHO’s staff working on DIPECHO projects have attended these courses (see for more details http://www.benfieldhrc.org/disaster_studies/rea/virtual_conf/index.htm). Link 196: BACK A number of examples of potential evaluation questions regarding children can betaken from the Save the Children UK 'Global Impact Monitoring Guidelines', published in 2004. Questions are divided into 5 main dimensions of change: changes in the lives of children and young people; changes in policies and practices affecting children's and young people's rights; changes in children's and young people's participation and active citizenship; changes in equity and non-discrimination of children and young people, and changes in civil societies' and communities' capacity to support children rights. No web link could be found. Link 197 BACK 'Practical Guide to the Systematic Use of Standards and Indicators in UNHCR Operations', UNHCR, 2006 http://www.unhcr.org/statistics/STATISTICS/40eaa9804.pdf, pages 63-69. Link 198 BACK The objective of UN-CMCoord IMPACT ("Integrated Missions Practical Approach to Coordination Tools") is to raise awareness on civil-military coordination guiding documents among military and civilians personnel working in multidimensional peace missions or deploying to countries hosting peace operations. http://ocha.unog.ch/uncmcoord/ Link 199 BACK DG ECHO's booklet 'A partnership for communication' no web link Link 200 BACK INEE (Inter-Agency Network for Education in Emergencies) Minimum Standards for Education in Emergencies, Chronic crisis and Early Reconstruction, in particular the M&E page http://www.ineesite.org/page.asp?pid=1066 Link 202 BACK

When evaluation undertaken Some key characteristics Evaluation during the Evaluation at the end of a programme/operation programme/operation Alternative names Midterm, formative Final, ex-post, summative evaluation evaluation Main purpose and use of To identify ways in which To identify what can the evaluation the current operation can be be learned from the improved overall operation to inform planning and implementation of future programmes and operations Main user of the Managers and staff of Depends – can be local evaluation the current operation managers if a follow-up (and perhaps others for operation is planned, agency reassurance) HQ staff and managers/staff at other sites undertaking or contemplating similar interventions Methodological approach Quick, simple and easy to Use of more robust methods apply, e.g. RTE possible Link 203 BACK Source: VENRO (Verband Entwicklungspolitik Deutscher Nichtregierungsorganisationen e.v.) working paper no. 14 " Humanitarian aid put to the test - Principles, criteria and indicators to ensure and monitor quality in humanitarian aid", August 2005

4.2 Quality criteria: Briefing and preparatory measures for staff, staff care Indicators: 1. Guidelines in place on the selection, briefing and preparing and care of the staff assigned and the local staff 2. Training of staff regarding Code of Conduct, minimum standards (Sphere Project), International Humanitarian Law and its application as well as security training for assigned specialists 3. Are the staff informed about measures to prevent sexual abuse? (Also see UNHCR guideline on “Sexual and gender based violence against refugees, returnees and internally displaced persons”, May 2003 and “Prävention von sexuellem Missbrauch von Abhängigen in der Projektarbeit”, DWHH July 2004.) 4. Descriptions of workplaces for assigned and local staff are available 5. Sufficient health and accident insurance for assigned and local staff tailored to the respective situation 6. Orientation on guidelines and standards on staff care such as the People in Aid Code of Good Practice, cf. www.peopleinaid.org and VENRO’s “Minimum Standards for Staff Security in Humanitarian Aid” and “Criteria for Staff Planning in Humanitarian Aid”, cf. http://www.venro.org. Link 204 BACK Source: VENRO (Verband Entwicklungspolitik Deutscher Nichtregierungsorganisationen e.v.) working paper no. 14 "Humanitarian aid put to the test - Principles, criteria and indicators to ensure and monitor quality in humanitarian aid", August 2005 Key questions: • Are the organisation’s strategy, content and regional priorities documented e.g. in Internet presentations, activity reports and profiles? • Are all the organisation’s levels aware of the strategy and sectoral and regional orientation? • Does knowledge of regional dangers and risk factors exist, and is it considered in the concept of the measure with a view to reducing the target group’s vulnerability? • Are there concepts to integrate preventive components vis-à-vis existing local and regional risk factors into project development and implementation? Link 205 BACK Source: VENRO (Verband Entwicklungspolitik Deutscher Nichtregierungsorganisationen e.v.) working paper no. 14 " Humanitarian aid put to the test - Principles, criteria and indicators to ensure and monitor quality in humanitarian aid", August 2005 • Are guidelines on staff care in place? • Do workplace descriptions exist for the various positions? • How is the security and the safeguarding of staff seen to? • Do recognised standards form the basis of staff care? • Are the conditions of assignment considered in social security arrangements (insurances, etc.)? Link 206 BACK Source: VENRO (Verband Entwicklungspolitik Deutscher Nichtregierungsorganisationen e.v.) working paper no. 14 "Humanitarian aid put to the test - Principles, criteria and indicators to ensure and monitor quality in humanitarian aid", August 2005 • In selecting staff, is intercultural competence and experience with emergency and development aid considered in addition to subject qualifications? • What is the ratio of local to assigned staff like in filling posts? • Does the profile of the staff assigned correspond to the defined requirements? • Are there regular (several times a year) upgrading and further education programmes? • Are local partners and their staff considered when applications are reviewed? Link 207 BACK Source: VENRO (Verband Entwicklungspolitik Deutscher Nichtregierungsorganisationen e.v.) working paper no. 14 "Humanitarian aid put to the test - Principles, criteria and indicators to ensure and monitor quality in humanitarian aid", August 2005 • Is there a consistent accounting system? • Are calls for tenders made according to the requirements of the financiers when material is procured, and is it documented? • Is the handling of financial affairs subject to regular external checking? • Is the financial situation made public? Link 208 BACK Source: VENRO (Verband Entwicklungspolitik Deutscher Nichtregierungsorganisationen e.v.) working paper no. 14 "Humanitarian aid put to the test - Principles, criteria and indicators to ensure and monitor quality in humanitarian aid", August 2005 • Are there local co-ordinating structures? • Does the organisation participate in co-ordinating meetings and structures? • Is the organisation actively involved in co-ordination? • Which tasks has the organisation performed to promote co-ordination? • How has the organisation co-ordinated its activities with other organisations and local authorities? • Has the organisation been recognised as a constructive and reliable partner by the organisations responsible for co-ordination (“co-ordinating organisations”)? • What impact has co-ordination had on the programmes of the organisation in question and those of other organisations? • What were the reasons not to take part in coordinating or to do so only marginally? • Is the organisation integrated into international co-ordinating committees at home or at regional or international level, and does it participate actively in these committees? Link 209 BACK Source: VENRO (Verband Entwicklungspolitik Deutscher Nichtregierungsorganisationen e.v.) working paper no. 14 "Humanitarian aid put to the test - Principles, criteria and indicators to ensure and monitor quality in humanitarian aid", August 2005 • Was there a needs assessment? • Did the needs assessment establish the causes of the present disaster and existing risk factors (hazards and vulnerability)? • Were the goals sufficiently unambiguously formulated (SMART), and were appropriate indicators defined and applied to measure attainment of the goal? • To what degree was the defined goal attained? • What were the essential framework conditions that influenced the attainment of the goal? • Were deviations from the objective explained in a comprehensible manner? • Were the causes established and was a risk assessment made? Were these aspects considered in the project concept, and were preventive measures integrated into the project design? • Has the project contributed to boosting the target group’s self-help potential to face new risks? • Did the measures reach the target population? Who was reached? Who was not reached? Link 210 BACK Source: VENRO (Verband Entwicklungspolitik Deutscher Nichtregierungsorganisationen e.v.) working paper no. 14 "Humanitarian aid put to the test - Principles, criteria and indicators to ensure and monitor quality in humanitarian aid", August 2005 • What would have happened if the project had not been run? • Were the framework conditions (e.g. political, economic, environmental, conflict potential) established and analysed with regard to possible interaction with the aid measures? • Were such assumptions and risks sufficiently represented and considered in planning? • Were possible negative impacts considered in designing the concept? • Was there a monitoring system to ensure that negative impacts could be established and counter-measures could be initiated as early as possible? • Were local risk factors established and considered? • Were there special plans to enhance the target group’s self-help capacity vis-à-vis the risk factors or was the development of dependence accepted? • Have the project results raised or reduced the target group’s vulnerability? • Were measures to reduce the risks integrated into planning? Link 212 BACK 'Living with Risk', A global review of disaster reduction initiatives (preliminary version), ISDR Geneva, July 2002 Link 213 BACK MSF-B, Evaluation Manual (internal document), December 2005 Link 214 BACK Source: MSF-B, Evaluation Manual (internal document), December 2005 Coverage - To which extent does the project activities reach the specific target population? Are there any factors that hinder us in reaching the most needy population? - To which extent do beneficiaries have access to project services? - Is anyone, or any particular group excluded from our services? Connectedness - Is a phasing out strategy designed and achieved? What does it consist of? - Which interconnected and longer-term problems (possible negative effects of the project) can be identified, and how have they been taken into consideration? - Which local capacities and resources were identified? How does the project connect with them? Timeliness - Was the timing of the intervention adequate? - What were reasons for timely or delayed response? Coordination - Have other actors in the area of intervention been identified? - Which contacts / coordination happens with other actors? - Is coordination adequate? - Has independence from other actors / coordinating bodies need maintained? Learning lessons - What issues emerged during the setting up and management of the program, which might be generalised to other situations? - What policy lessons can we learn from the program? Link 215 BACK The Common Assessment Framework – Improving an organisation through self-assessment, October 2002 http://www.eipa.nl/CAF/CAFmenu.htm Link 216 BACK NGO Self-assessment through a SWOT exercise, Networklearning 2006 http://www.networklearning.org/download/SWOT.pdf Link 217 BACK The OCTAGON tool for assessment of strengths and weaknesses of NGOs, SIDA 2002 http://www.sida.se/sida/jsp/sida.jsp?d=118&a=2745&language=en_ US&searchWords=octagon Link 218 BACK Source: USAID TIPS n°15, 2000 "Measuring Institutional Capacity" –annex with tools Link 219 BACK See also glossary. A Management Audit is a future-oriented, independent, and systematic evaluation (sic) of the activities of all levels of management performed by the internal auditor for the purposes of improving organisational profitability and increasing the attainment of the other organisational objectives through improvements in the performance of the management function, achievements of programme purposes, social objectives, and employee development. Financial data are some of the sources of evidence. The primary sources of evidence are the operational policies and the management decisions as related to the organisational objectives…The resultant audit report identifies problems and recommends solutions. (Smith, J.E.: 'An Evaluation of Selected Current Internal Auditing terms', Research Report n°19, IIA Inc, 1975)

The "Four E"s in audit: - Effectiveness: doing the right things - Efficiency: doing things the right way - Economy: doing things cheap - Equity: doing right, i.e. being socially responsible in a social audit perspective. (Internal Auditing, A.D. Chambers, G.M. Selim, G. Vinten, Pitman Publishing 1987)

Remark: the use of the fourth audit 'E' (Equity), should be probably avoided in the larger humanitarian context, as it could be confused with some notions of humanitarian principles (e.g. rights-based approach), whereas its audit sense is much more restrictive. Link 220 BACK Evaluation indicators (source: EuropeAid evaluation Guidelines ) http://ec.europa.eu/europeaid/evaluation/methodology/methods/mth_ind_en.htm - The main evaluation indicators are those related to judgement criteria, that specify the data needed to make a judgement based on those criteria. - An indicator can be constructed specifically for an evaluation (ad hoc indicator) and measured during a survey, for example. It may also be drawn from monitoring databases, a performance assessment framework, or statistical sources. - A qualitative indicator (or descriptor) takes the form of a statement that has to be verified during the data collection (e.g. parents' opinion is that their children have the possibility of attending a primary school class with a qualified and experienced teacher). - A quantitative indicator is based on a counting process (e.g. number of qualified and experienced teachers). The basic indicator directly results from the counting process. It may be used for computing more elaborate indicators (ratios, rates) such as cost per pupil or number of qualified and experienced teachers per 1,000 children of primary-school age. - Indicators may belong to different categories: inputs, outputs, results or impacts. Link 221 BACK Judgement criteria, from evaluation questions to indicators (source: EuropeAid evaluation guidelines) http://ec.europa.eu/europeaid/evaluation/methodology/methods/mth_qci_en.htm Link 222 BACK Emergency Field Handbook – A guide for UNICEF staff, UNICEF July 2005 http://www.unicef.org/publications/files/UNICEF_EFH_2005.pdf Link 223 BACK Core Commitments for Children (CCC) in Emergencies, UNICEF 2005 http://www.unicef.org/publications/files/CCC_EMERG_E_revised7.pdf Link 224 BACK Growing the Sheltering Tree - Protecting Rights Through Humanitarian Action, UNICEF 2002 http://www.unicef.org/publications/index_4397.html Link 225 BACK PATH – a Principled Approach To Humanitarian Action (UNICEF) training course available on CD http://www.unicef.org/path/ Link 226 BACK Protection - An ALNAP Guide for Humanitarian Agencies, by Hugo Slim and Andrew Bonwick, 2005. ALNAP http://www.odi.org.uk/ALNAP/publications/protection/alnap_protection_guide.pdf Link 227 BACK ALNAP Global Study on Consultation and Participation of Disaster-affected Populations, ALNAP, 2003 – 2004 http://www.odi.org.uk/ALNAP/publications/participation_study.htm Link 228 BACK Exit Handbook, Norwegian Refugee council (NRC), May 2007 Link 229 BACK Assessment of humanitarian needs and identification of “forgotten crises”, technical note, DG ECHO August 2006 http://ec.europa.eu/echo/pdf_files/strategic_methodologies/methodology_2007_en.pdf Link 230 BACK Global Needs Assessment (GNA) 2006 - 2007, DG ECHO 2007 http://ec.europa.eu/echo/pdf_files/strategic_methodologies/glob_needs_ass_2007.xls Link 231 BACK Forgotten Crises Assessment (FCA), DG ECHO 2007 http://ec.europa.eu/echo/pdf_files/strategic_methodologies/forgotten_crisis_data_2007.xls Link 232 BACK http://www.partos.nl/index.php?page=5_2_3 (please follow the links in the middle frame of the website to access the joint evaluation reports). Link 233 BACK IIED (International institute for Environment and Development), www.iied.org Link 234 BACK Information on climate, risks, natural resources management in developing countries and humanitarian settings can also be drawn from a number of sites, such as e.g. (in French) the IRD (Institut de Recherche pour le Développement), web site http://www.bondy.ird.fr/recherche/themes.htm Link 238 BACK UNAIDS, catalogue of publications http://www.unaids.org/DocOrder/OrderForm.aspx Link 239 BACK UNICEF, Enhanced Protection for Children Affected by AIDS http://www.unicef.org/publications/files/Enhanced_Protection_for_Children_Affected_by_AIDS.pdf Link 247 BACK • Cost effectiveness analysis entails comparing costs across different strategies for achieving a given outcome, with a view to determining the lowest cost approach. For example, cost- effectiveness analysis might explore three different approaches to getting girls working in the informal sector back into school. As compared to cost-efficiency analysis, it is wider in scope, looking beyond outputs to outcomes. (M&E Training Resources, UNICEF, 2004) • Cost effectiveness looks beyond how inputs were converted into outputs, to whether different outputs could have been produced that would have had a greater impact in achieving the project purpose. (ALNAP Quality Pro-forma). • Cost analysis in evaluation builds on financial information, but may also involve calculating “economic costs” such as human resources, labour-in-kind, opportunity costs, etc. The scope of cost analysis, i.e. whether cost comparison is made concerning impacts, outcomes or outputs, will depend on the purpose of the evaluation and the evaluation questions posed. Cost analysis must be explicit in terms of the different perspectives from which costs are analysed (donors, a single organization, primary stakeholders) and the limitations – the complexity of the subject (multiple programme objectives, partners, financial systems), the availability of data and the time and resources invested. • Cost analysis is not always feasible. Where no cost analysis is included in an evaluation, some rationale for this exclusion should be included in the objectives or methodology section. It is expected that evaluators point out areas of obvious inefficient use of resources. (UNEG, standard 3.8) Link 248 BACK Source: UNEG, Standard 3.8: an evaluation should assess cost effectiveness, to the extent feasible. Using a range of cost analysis approaches, from the elaborate cost-effectiveness and cost- benefit analysis, to cost-efficiency analysis, to a quick cost comparison, an evaluation should, to the extent possible, pursue the following broad questions: • How do actual costs compare to other similar benchmarks? • What is the cheapest or most efficient way to get the expected results? • What are the cost implications of scaling up or down? • What are the costs of replicating the subject being evaluated in a different environment? • Is the subject being evaluated worth doing? Do economic benefits outweigh the costs? • How do costs affect the sustainability of the results? Link 249 BACK Some forms of bias: Spatial Issues of comfort and ease determine the assessment site Project The assessor is drawn toward sites where contacts and information is readily available and may have been assessed before by many others Person Key informants tend to be those who are in a high position and have the ability to communicate Season Assessments are conducted during periods of pleasant weather, or areas cut off by bad weather go unassessed, thus many typical problems go unnoticed Diplomatic Selectivity in projects shown to the assessor for diplomatic reasons Professional Assessors are too specialised and miss linkages between processes (preceding biases, Chambers, 1983) Battle Assessors go only to areas of cease-fire and relative safety. (Barakat and Ellis, 1996) Political Informants present information that is skewed toward their political agenda; assessors look for information that fits their political agenda. Cultural Incorrect assumptions are based on one’s own cultural norms; Assessors do not understand the cultural practices of the affected populations. Class/ethnic Needs and resources of different groups are not included in the assessment Interviewer or Investigator Tendency to concentrate on information that confirms preconceived notions and hypotheses, causing one to seek consistency too early and overlook evidence inconsistent with earlier findings; Partiality to the opinions of elite key informants. Key informant Biases of key informants carried into assessment results Gender Assessors only speak to men, or male interviewers survey women, or vice versa. Mandate or speciality Agencies assess areas of their competency without an inter- disciplinary or inter-agency approach. Time of day or schedule bias The assessment is conducted at a time of day when certain segments of the population may be over- or under- represented. Sampling Respondents are not representative of the population.

(Source: Monitoring and Evaluation Guidelines. A training module prepared for the University of Wisconsin-Madison Disaster Management Center by InterWorks.) Link 252 BACK http://ochaonline.un.org/humanitarianappeal/webpage.asp?Nav=_training_en&Site=_training&Lang=en CAP workshops are usually facilitated by two people from different humanitarian organisations, based either in the field or at headquarters. They are seen as a key step in an on-going process of discussion around humanitarian effectiveness, with the aim of increasing the accountability of the humanitarian system. The objectives of a workshop are to debate openly and critically about humanitarian response and mechanisms; analyse contexts and build scenarios; clarify roles and responsibilities, etc. Schedules of workshops, guidelines and toolkit, a roster of skilled facilitators and reports are available on OCHA's website. Annual training of trainers sessions and a training workshop for OCHA's own staff are also organised. Link 255 BACK The importance of such a need has recently been identified by the OECD/DAC, e.g. in the following analysis. "Current efforts to devise more strategic approaches to aid allocations are subjecting DAC evaluation departments to demands to assess the performance of external organizations, such as multilateral organizations and NGOs. ... While some evaluation departments are devising ways to rely on multilateral existing systems to capitalize on them and extract strategic lessons, in other cases these demands are pushing evaluation towards an audit-like function. The same considerations apply to the need for assessing NGOs' programmes and activities. Several donors are concentrating on requiring NGOs and implementing agencies to step up their systems, standards and practices in order to conduct more rigorous self-evaluations. Once again, the challenges here lie in supporting the development of NGOs' evaluation capacities to help them demonstrate the impact and sustainability of their programmes autonomously." (emphasis added) (Source: Liverani and Lundgren - Evaluation systems in development aid agencies: An analysis of DAC peer reviews 1996-2004. Evaluation, vol. e, no. 2 April 2007) Link 256 BACK Assessment questions regarding emergencies: • How urgently is help needed? (is it a life-saving emergency or not?) • Who needs help? • Which groups are affected? • Which groups are worst off? • What is their situation now? • What resources do they have? • What resources do they need? (Source: Assessment and Analysis V110, J. Cosgrave et al, Interworks Europe, Oct. 2002) Link 257 BACK This finding must be attributed to Michael Patton, one of the best known writers about evaluation, who in particular has strongly advocated a utilisation-focused approach to evaluation, such as this Guide also recommends. His writing, by far, the most entertaining of evaluation theorists. If there is one book on evaluation to read, his Utilization-Focused Evaluation (1997) is the one to get.