Training On M&E For Peacebuilding

Evaluation and Impact Assessment of Peacebuilding Projects Advanced course by Emery Brusset & Tony Vaux September 2006 Summary of the course Rachel Goldwyn - CIUK

INTRODUCTION 1

METHODOLOGIES PRESENTED IN THE COURSE – 2 DFID’S STRATEGIC CONFLICT ANALYSIS FRAMEWORK 2 MACRO IMPACT ANALYSIS 3 CONFLICT SENSITIVE APPROACHES 4 NARRATIVE DISAGGREGATION 5 REFLECTING ON PEACE PRACTICE 6 QUANTITATIVE SURVEY APPROACH 7 THE UK GOVERNMENT’S PUBLIC SERVICE AGREEMENT APPROACH 7 THE CASCADE OF OBJECTIVES 7 OPPORTUNITY MAPPING 9 HERWEG STEINER METHOD 12 PRACTICALITIES OF EVALUATING PEACEBUILDING PROGRAMMES 14

Introduction This document summarises the Channel research advanced course in Evaluation and Impact Assessment of Peacebuilding projects, run in September 2006. It explains the alternative methodologies presented, and captures some of the discussions. It builds on the 2005 paper ‘INCORE training summary’ – which was a preliminary course.

The objective of the course was to help practitioners answer the following questions:  What difference is an intervention making in peace/conflict situation?  How do we know?  How can it be argued (justified)  How can it be verified? METHODOLOGIES PRESENTED IN THE COURSE – an overview of the options for evaluating peacebuilding programming

DFID’S STRATEGIC CONFLICT ANALYSIS FRAMEWORK Tony has often used DFID’s strategic conflict analysis (SCA) framework for evaluating strategies of DFID country assistance programmes. He has augmented the standard SCA approach into the following stages specifically for evaluation purposes: 1. Conflict analysis: analysis of causes (using the grid framework of security, political, economic & social factors, divided according to local, regional and national), analysis of actors (in particular identifying those driven by greed and those motivated by grievance) and from these generate principles for conflict sensitive action – i.e. principles that should inform project design generated by the conflict context. They are the reformulation of the key issues in the conflict in the form of a hypothesis. Example: the conflict analysis identifies that women are marginalized & tend to support the Maoists as the Maoists promote women’s empowerment, thus a conflict sensitive principle derived from this would be ‘all development projects should support the empowerment of women’. Or if the SCA identifies that the political elite excluding others is a major conflict cause, and that aid tends to strengthen the elite, a conflict sensitive principle would be along the lines of how project interventions could help support the growth of representation. These conflict sensitive principles can then be used for evaluating specific interventions.

2. Mapping of current responses: using the same grid framework as above, identify all the different project components to be able to compare the causes and the responses. Alternatively, if reviewing a portfolio of interventions, map out all the different projects/programmes on the grid (not just the intervening agency but also all other interventions). This helps us see the project in relation to all other interventions.

3. Ways forward: using the same grid, identify the areas of potential activity to help understand how well aligned the intervention is to the conflict needs. This could identify an overall strategic direction, or to make specific recommendations for a project/programme.

Discussion/questions: 1. The conflict sensitive principles involve implicit theories of change, but do not state them, nor evaluate them. Such principles ought to be examined for appropriateness and effectiveness. 2. Is there sufficient attention paid to actors? 3. To evaluate an individual project, only stage one seems to be relevant. The 2 nd stage – mapping of current responses – will identify the areas of intervention, but will not give any further insights into the appropriateness or effectiveness of the specific intervention(s). This requires an analysis of all the peacebuilding needs of that situation.

Nepal case study – application of the SCA to a portfolio review Evaluation and impact assessment of peacebuilding projects – Advanced course summary of the course Page 2 Using the SCA in Nepal, Tony reviewed DFID’s portfolio of interventions. The SCA analysis was undertaken, and from this a set of ‘conflict sensitive’ principles were developed (principles of good practice in a conflict situation). These principles were then bought into the strategic planning process, and were the yardstick for the portfolio review, evaluation of specific projects and overall impact assessment (of all the different projects & strategies in relation to conflict).

MACRO IMPACT ANALYSIS CDA, in conjunction with CARE Kosovo, Search for Common Ground and KU Leuven University have recently undertaken a macro impact analysis of peacebuilding impact in Kosovo. The purpose of the study is to understand whether and how peacebuilding programming in Kosovo contributes to communities’ non-participation in violence, and through this, to improve the impact of peacebuilding programming in Kosovo. The innovative research methodology designed specifically for this study (see box) could be applied to other conflict settings, and study findings offer important insights into peacebuilding programming that are relevant across the globe.

The study posed two questions: What have been the factors that have enabled communities to resist or not to participate in violence? To what extent & how has peacebuilding contributed to these factors?

The specific intention was not to evaluate the outcomes/outputs of individual projects/programmes and sum these together, but to instead to shift to the strategic (impact) level to ascertain if programming across the entire array of peacebuilding actors are actually contributing to the achievement of peace. Thus the study could not be an evaluation of specific peacebuilding programmes, but was instead an in-depth analysis of what enables people to resist violence, and whether peacebuilding as a whole supported the factors that enabled people to resist violence. The riots of March 2004 were the entry point, giving the study a focal point to explore where violence did or did not occur & why.

The study involved 3 phases: 1. Mapping of violence through existing data sets to quantify and categorise physical and psychological violence, & initial consultations to develop initial hypotheses for research & identify case study sites. 2. 7 in-depth narrative case studies that encompassed those with/without extensive peacebuilding activities, those with/without high levels of inter-ethnic violence, and those with/without violence in March 2004. The case studies involved interviews with 20-40 people by two local researchers. 3. Analysis of the cases, including discussing emergent findings in several consultations, to identify common themes, issues & questions

Discussion/questions: 1. How much overlap is there between this approach and the strategic conflict assessment approach above? Could the Kosovo approach be used for a portfolio review by a donor?

Evaluation and impact assessment of peacebuilding projects – Advanced course summary of the course Page 3 2. Evaluation timeframe: Over what period is it fair to judge impact? Was 2004 a fair evaluation point for pre-2004 programming? Did it represent a trend or a spike? How far can conclusions re a single event be transferable/generalized (beyond Kosovo, as well as within Kosovo?) 3. Recommendations on methodology: could broad population assessments also augmented the deep case studies? Could the information have been disaggregated by gender? Can the different data sets actually be compared?

CONFLICT SENSITIVE APPROACHES Several conflict sensitive approaches, designed for the development/humanitarian sector, can provide insights for the evaluation for peacebuilding projects. Emery presented 2 different methodologies – PCIA and Do No Harm – which he described as ‘plausible narrative’ analysis processes – meaning that it gives evidence in a text form (this was explained as qualitative research, but does not lend itself easily to argument, defence and consensus).

The PCIA methodology uses 4 areas to consider the wider peacebuilding/conflict generating impacts of development projects:

 Did the project produce substantial or politically significant changes in access to individual or collective material and non-material resources?  Did the project create, exacerbate, or mitigate socio-economic tensions?  Did the project produce substantial changes in the material basis of economic sustenance or food security?  Did the project produce challenges to or changes in content of or control over existing political, economic &/or social systems?

Discussion/questions: 1. These questions appear more useful for evaluating the conflict sensitivity of development projects than for evaluating specifically peacebuilding projects. 2. A concern with narrative approaches is that they hinge on a subjective agreement on what is the conflict, and create complexities for analysis by generating different analysis from different perspectives. Evaluations are contentious. The analysis of the problem is often used to undermine the findings. So this analysis must be done in a value-free manner. The use of issues which have been agreed as key by participants, real or potential is a way out of ‘plausible’ stories about the conflict which seem biased and oppressive to others. 3. Conflict analysis seeks to simplify & rationalize reality, although the most convincing reality is usually from those who can present best and whose explanations links most closely with our experiences. 4. The micro-macro link remains unclear with this approach.

Do No Harm can offer insights to peacebuilding evaluation through the observable aspects of inter-group relations in terms of dividers and connectors. For example if inequality in housing has divided groups, changes in access can be assessed. Thus the dividers and connectors

Evaluation and impact assessment of peacebuilding projects – Advanced course summary of the course Page 4 framework can provide observable indicators. Further, it can be observed to what degree people are using areas that connect people, such as market places.

Discussion/questions: 1. How do we know if the dividers/connectors identified and observed are the key ones?

NARRATIVE DISAGGREGATION An overarching framework for evaluating conflict resolution work was developed by Church and Shouldice which is structured around 3 thematic areas. The framework is to guide the preparation before an evaluation, rather than to provide a blueprint for all evaluations:

Of particular interest is the ‘impact’ criteria – which evaluates how the intervention has contributed to peace at the macro level. Church and Shouldice introduce several concepts to help evaluators (and project designers) model and assess this link.

Tiers of influence – this describes the target of the intervention, for instance the family unit or social network (micro level), the community or sub national region (meso-level) or society at large, regional group of countries (macro level).

Focus of change – an intervention is usually targeted at any one target group described above. The change we are trying to achieve through the intervention is the focus of change.

Transfer or ripple effect - While programming tends to target one group we tend to assume that it will have some kind of ripple effect to other groups. Impact is expected to decrease as we move further away from the target group, but causal chains can show the process of transfer between groups.

Evaluation and impact assessment of peacebuilding projects – Advanced course summary of the course Page 5 Discussion/questions: 1 The 3 tier evaluation method, when compared to the OECD DAC criteria, risk increasing confusion, as they use related concepts but in a different way. The five OECD DAC criteria represent an emerging consensus of language, which is rare and should be preserved.

REFLECTING ON PEACE PRACTICE The RPP work stressed the importance of assessing effectiveness at both the programme level (i.e. is the intervention achieving its intended goal through examining project design, implementation, and immediate results) and also at the macro level – also called ‘peace writ large’ (i.e. does meeting the specific programme goals contribute to the construction of peace at the broader levels of society). While no one agency should be held accountable for creating ‘peace writ large’, all ought to understand whether interventions are making a contribution to ‘peace writ large’.

General criteria for evaluating the effectiveness of peacebuilding programming were developed through the RPP project:

1. The effort caused the participants & communities to develop their own initiatives for peace. 2. The effort resulted in the creation or reform of political institutions to handle grievances that fuel the conflict. 3. The effort prompts people increasingly to resist violence and provocations to violence. 4. The effort results in an increase in peoples’ security.

The criteria are additive – thus the more criteria an intervention fulfils the more effective it is. Moreover these are context-linked, i.e. a change is more successful if:  It occurs sooner rather than later;  If it is sustained over time;  If it is proportional to & on the same scale as the conflict.

It is noted that the criteria developed by RPP are loosely linked to the OECD DAC criteria, but come from a different angle as with the Church model above.

Some questions on narrative approaches: Having a conflict analysis as the basis of an evaluation is a key means to address the question of relevance of the intervention. However experience from the field is that if a narrative conflict analysis is part of the evaluation then the client tends to attack that analysis, and in so doing undermines the evaluation findings.

However narratives are crucial means to draw perspectives from different actors in or affected by conflict. The complex pictures they provide are a real reflection of the complexity that is conflict. Managing complexity is thus a problem of coping with narrative approaches.

Evaluation and impact assessment of peacebuilding projects – Advanced course summary of the course Page 6 QUANTITATIVE SURVEY APPROACH In the Macedonia OTI initiative evaluation, while intended outcomes to increase trust between ethnic communities could be measured quantitatively (in terms of observable changes in attitudes, behaviours and knowledge of the participants), it was not possible to establish whether this was the result of the project.

The evaluators argued that an intermediary objective was missing between the specific objective (project purpose) and the general objective (overall conflict resolution). This later led to the development of the cascade of objectives method.

The evaluators found there to be four levels of objectives in the OTI project:

1 stability in the country 2 support peace agreement 3 diffuse tension 4 create community development projects

A quantitative survey method was used to examine the relationship between levels 3 and 4, while narrative methods were used for higher levels.

Discussion/questions: 1. Tracking the perverse effects of projects, or the actions of spoilers still requires qualitative evidence, for instance the use of surveys.

THE UK GOVERNMENT’S PUBLIC SERVICE AGREEMENT APPROACH These use quantitative targets – both general and country specific, drawn from the DFID country strategies. During the Conflict Prevention Pools evaluation 3 main problems were found with using such measurements: 1. Data problems – there are no specific national data sets, and data sources are problematic. It is difficult to link the data sets to war stress, and further difficult to show a clear relationship between UK outputs/results with outcomes/impacts. 2. a qualitative measure – relating to the UK’s ability to mobilise an effective international response – is very difficult to evaluate. 3. The PSA covers 3 different departments (MOD, DFID and FCO) all of which have different approaches to conflict prevention. The contributions are so varied that they become non-comparable.

THE CASCADE OF OBJECTIVES This seeks to identify the links between the project’s outputs and contribution to peace. The key element is to clarify the intermediary objectives, and then to work through the different tiers of objectives, assess whether they have been successful, and whether these have contributed to the achievement of the next tier up objective(s).

Evaluation and impact assessment of peacebuilding projects – Advanced course summary of the course Page 7 It is drawn as a flow chart, with multiple objectives drawn at each tier, and arrows drawn on to link between tiers. Drawing the arrows can prove difficult, particularly as lower tier objectives can contribute to multiple higher tier objectives. Each objective is then re-phrased as a question, for which 3-4 evaluation questions are developed, and recommendations are developed on the basis of the findings for each of the objectives. This methodology differs from the conventional log frame as there are 4 tiers in a log frame, while there can be over 4 in the cascade, and multiple causality.

In evaluating projects from Search for Common Ground & USAID, the evaluators sought to reconstruct the implicit objectives of the programmes. They used surveys and qualitative semi- structured interviews to draw out stakeholder opinions about impacts and causation. 4 evaluative questions were used relating to the different intermediary impacts:

1. Strength How influential is the programme in the society, compared with other actors? Which trends or drivers of peace and conflict does it influence?

2. To catch people’s imagination (short term impact) Has there been a propagation of new models of social interaction which are attractive and reproduced and emulated further within society? What are these models, how attractive are they? Were they proposed at the right time, or allowed the organisation to buy time while contradictory messages of incitation to violence and tension were being spread?

3. To create new methods of interaction (medium term impact) Have the activities allowed groups which did not have contact talk to one another, either through political representatives, community to politicians, or between communities? Have these contacts been of a new nature? Were they noticed as important by a significant number of people?

4. Capacities (long term impact) What institutions have been created, for example new radio stations? What personnel have been trained and what skills have been improved which will allow the society to respond to a new upsurge in violence? What sustainability have these new capacities got?

In the Macedonia project, in which children’s television programming sought to promote inter- group relations, the evaluation included consideration of the following factors:  Discussions in the households following the programmes;  The degree to which adults were also exposed to the messages (indicated by specialised advertising targeting adults which was only broadcast with this programme);  Whether the children were the most important target at the time;  What follow up occurred when the broadcasts ended;  Comparison to an area where there was no broadcast as a control group.

The cascade of objectives may also help us look at situations where things went wrong, and try to unpack what led to what.

Evaluation and impact assessment of peacebuilding projects – Advanced course summary of the course Page 8 The European Commission External Relations evaluation unit has developed a version of the cascade of objectives approach, but based on the principle that evaluator shouldn’t reconstruct or invent objectives, and instead should stick entirely to the written evidence of the project. Thus the evaluation process is to examine all the documents relating to project objectives, programme objectives and intervention objectives. This can go up to high level documentation, such as the objectives of the EC itself. The key bonus of such an approach is that the client cannot later challenge the findings, as the evaluation is based entirely on their documentation. However it is a very rigid blueprint approach.

Discussion/questions: 1. The cascade does not create any space for capturing unexpected outcomes, nor for project learning, thus those who correct mid-project would likely be penalised unless they go through a process of getting the project documents revised and accepted as such. Others contested this saying as an evaluation method, this process reconstructs the reality developed in the course of implementation. Therefore, documents do not have to be accepted by a donor, rather they are valid as long as they are considered legitimate and important by the implementing agency concerned. 2. Drawing the arrows between tiers of objectives appears a subjective process. While the objectives themselves are evaluated, the arrows are not. However it seems that this would be crucial to evaluate our theory of change, and to help indicate some level of attribution. 3. The attribution problem also remains – it is still difficult to gauge the sphere of influence of different actors, thus it was impossible to attribute effectiveness of the projects where there were multiple other factors at play, although for discrete projects where the objectives are very clear this can be a useful methodology.

OPPORTUNITY MAPPING This methodology, developed from the MOD Strategic Assessment Method, deliberately shifts away from analysing root causes, on the basis that trying to get a definitive explanation of causes is difficult/impossible among conflict parties, and does not expose the evaluator to criticism by the client along the lines of an inadequate/incorrect analysis. The method analyses the events & trends in a situation, and aims to bring together divergent views in doing so.

Opportunity mapping allows you to consider sectors of activity and relate them to an analysis of conflict. It uses an entirely different approach from objectives based evaluation. Objectives based evaluations consider the current state, the ideal state, and give a technique for how to get from here to there. Opportunity mapping instead focuses on the current environment & what leads to change. There is no ideal state and no end state, instead the drivers of conflict are identified along with where spheres of influence exist. The premise is that the conflict should evolve towards its own conclusion, rather than have agencies define specific end states, with objectives that are small compared the actual conflict. It can be used in project design, monitoring, and evaluation.

The methodology uses two sets of maps, and various quantitative calculations. My belief is that using numbers does not enhance the process (and may in fact detract from it) however the

Evaluation and impact assessment of peacebuilding projects – Advanced course summary of the course Page 9 process itself could be undertaken with qualitative statements to replace the numerical calculations. The mapping is then supplemented by outcome definition and linkage through field interviews. It can be used to consider a single intervention, or a range of interventions by an agency.

Step 1 – mapping trends and events Chronological events & trends are mapped on a large space – from left to right. Events (single finite events) are shown in squares, trends (continuous processes) in circles. Where linkage can be established between any of these – i.e. that one event/trend has increased or decreased the probability of another event/trend occurring – a link is drawn. A + or – on the link indicates whether it is increasing or decreasing the probability. There is no need to show different levels (local, national, international etc)

ANALYSIS Count up the number of arrows leading to/from events & trends – doesn’t matter if the line is increasing/decreasing possibility. Use these values to rank all trends and events – capturing the top 5 as a list. These are nodes, and indicate the priority events/trends in this context.

Returning to the map, consider which events & trends were amenable to change – i.e. that we could expect the agency/intervention being evaluated to have been able to influence. Circle these events/trends on the map. Note – this represents a hypothesis about theories of change (how we think influence could be bought to bear on these events/trends).

Next step back, look for the nodes, write them down, and check how a project, a series of projects, have impacted these issues. This is tested by asking about targeting, extent, and duration the main areas of activity, either of projects, programmes or a whole strategy according to the following criteria:

 How well targeted were the activities to the issues?  What was the extent (or scope) of the effects of the activities on the issues?  What was the duration of the effects of the activities on the issues?

If you are going the numerical route then multiply together the scores for each component to find the significance rating (a score out of 64) If you are not going to use the numbers precisely, you can still use the numbers in general terms, for example when defining the nodes. The grading of performance as per the three bullet points can be done narratively (as long as you use verifiable indicators).

The results that emerge should evaluate both the qualitative effect of the programme (for example deeper relations established), and the quantitative effect (for example level of employment of demobilised groups).

VARIATION – MAPPING OF THE INTERVENTION A similar map of events/trends could also be drawn up to map out the intervention being evaluated. This could help show the links between the intervention and the conflict more systematically.

PROCESS POINTS Evaluation and impact assessment of peacebuilding projects – Advanced course summary of the course Page 10 People tend to be more open to this workshop if it is described as something that will improve the intervention. You need a large wall to do the mapping on. It is best done by two people – one to write (don’t let people come up and write up things themselves!) and the other to facilitate, specifically to help people understand what is an event or trend rather than prompting. It is best if you can get the different conflict parties (or their proxies) to the same workshop, but people do need to be comfortable with each other. This should be eased by the process not seeking to get the root causes nor into value judgements – the process tends to lead people into discussions about precise details or probability links, thus is presenting their world view rather than an objective truth. If there are competing views about whether a probability line exists it can simply be added. Separate workshops to repeat the process could be conducted where parties cannot be bought together. The participants can stay for the analysis phase but it isn’t necessary to involve them.

Step 2 – stakeholder mapping The mapping is to identify both priority stakeholders and amenability to influence, and specifically to find the intersection between the two, thus focussing us on who we could/should have had an impact on. Actor mapping is useful in defining what change occurred in the positions of key people, and then the questions is: how much was this change attributable to an agency/strategy/project? The key element that is added with the mapping methods is not just to be stuck with the change and the outputs separated from each, but to link them. This linking has to be done through careful analysis based on indicators. So the mapping is only half the process

The map is drawn on a quadrant chart – with the horizontal axis showing people’s position for or against a project or an issue, and the vertical axis showing their ability to influence the project/issue. Those at the lowest point cannot influence the project at all. [if you go the numerical route mark the lowest point as 0, the intersection of the axes at 2, and the top of the vertical axis 4.]

Participants identify all the people who can influence the project/issue, and place them on the chart. Then consider who is influencing who, both formally and informally, and show these as links. Arrows can be used to show direction of influence if you choose.

ANALYSIS Count up the arrows to/from any stakeholder and rank the top 5 in terms of the number of arrows they have, and note these 5 as a list. Two way influence could be considered as two separate arrows. For each of the ranked stakeholders note the level of influence they have according to the vertical axis. [if you go the numerical route add their number of arrows to the level of influence]

Consider what influence the agency under evaluation has on these actors. This is a subjective judgement. [If you go the numerical route score a figure out of 3, where 1 is no influence. If you multiply this influence rating with the summed figure above, this gives the significance rating of that actor]. If you are not going the numerical route then analyse how much people moved visually, and describe it in a paragraph.

For each of the ranked stakeholders note in a new column the programming that is targeting them, and repeat the analysis according to the criteria under step 1.

Evaluation and impact assessment of peacebuilding projects – Advanced course summary of the course Page 11 PROCESS POINTS The debate can become heated – this is a less transparent process and should only be undertaken with client staff.

Discussion/questions: 1. The use of numbers gives this methodology a more ‘rigorous’ appearance. However I am unconvinced by the numbers – significance can be considered using narrative terms, and I feel these would be more appropriate. 2. Identifying the sphere of influence on the events/trends map involves considerable assumptions about theories of change, as there are implicit assumptions about what could be done to affect those events/trends. However these theories of change aren’t articulated, nor are they assessed. We also make considerable assumptions about the potential reach of the agency – how do we know that these events/trends were something the agency could have influenced? 3. Would it be useful to separate the stakeholders into those with hierarchical power and those with ideological power? Can different sorts of stakeholders be compared (for instance in the stakeholder mapping can you compare an individual with a party?) 4. Does an analysis that focuses our attention on track 1 & 2 (as this mapping inevitable does) lead us to a situation where we are actually strengthening of the powerful by focussing all our energies on them? Does it also bias us towards better evaluations for track 1 & 2 interventions, and against track 3 interventions? 5. Will using a trends & events mapping actually result in agreement among conflict parties? It would be a very interesting experiment to try! Would there be other possible spin offs resulting from this? 6. As yet, the methodology doesn’t capture what was done inadvertently. Undertaking a mapping of the intervention (as described under ‘variation’ above) might help us identify the inadvertent negative impacts of a project. It would be interesting to experiment with this to see if the more nuanced means of interactions (such as implicit ethical messages etc) would also emerge through such a process.

HERWEG STEINER METHOD This methodology was not designed specifically for conflict situations, however the ‘participatory systems analysis’ component is useful to understand which root causes are more important in a conflict, as they influence other core problems more than they are influenced by them. It uses a non-linear approach to cause and effect, and is a quantitative methodology. The explanation below relates exclusively to this section of their impact monitoring and assessment methodology.

All different root causes of conflict are listed in a matrix – with each root cause listed both horizontally and vertically. The matrix is filled out to show the influence of one root cause on another. The influence of each root cause on the other root causes is given a value, with 2 as a strong influence, 1 a moderate influence, o.5 a weak influence, and 0.1 a very weak influence. These ratings are entered along the horizontal line for each root cause.

Evaluation and impact assessment of peacebuilding projects – Advanced course summary of the course Page 12 No. 1 2 3 4 5 6 7 8 9

Core Problems Active Degree of Sum Interrelation

1 2 3 4 5 6 7 8 9 Passive Sum Activity Ratio

By adding all the ratings along a horizontal line we get the ‘active sum’ – the influence that specific root cause has on other root causes. By adding together all the ratings on a vertical line we get the ‘passive sum’ – the influence all other root causes have on the root cause in question. By multiplying together the active sum and the passive sum we get the ‘degree of inter-relation’ – i.e. how strongly or weakly the root cause is networking within the conflict system. A high degree of interrelation implies that there are many ways to influence this root cause. Dividing the active sum of a root cause by its passive sum gives its ‘activity ratio’ – which indicates whether a root cause plays an active role (>1) or passive role (<1) in the conflict. Passive elements are poor starting points to change the conflict.

Using the activity ratio and the degree of interrelation, a chart can be plotted to demonstrate the functions of the different root causes within the conflict:

 A symptom is an element that is greatly influenced by other elements but may not have much power to change the system itself. Symptoms can be useful indicators of context changes, but development activities in this sector may only amount to a "treatment of the symptom, not the cause".  A buffer is characterised by low importance in the context. It is rather unremarkable because it neither influences other elements much nor is it influenced much by others. Development activities in this sector are expected to have little impact on the context.  A critical element is an accelerator or catalyst in the system. It changes many things quickly, but may also create many unexpected and undesired side effects. Development activities in this sector can be highly uncertain, and impacts may be unpredictable. Therefore, critical elements have to be treated very carefully. It is particularly important to formulate impact hypotheses for this sector.  A motor or lever is an active element with predictable impacts. This is the most important sector for intervention, as it can be influenced and has multiple effects.

Evaluation and impact assessment of peacebuilding projects – Advanced course summary of the course Page 13 • Elements in the two sectors on the left (symptom & buffer) are rather passive, i.e. they are influenced by other elements more than they influence others. • Elements in the two sectors on the right (critical element & motor) are rather active, i.e. they influence other elements more than they are influenced. • Elements in the two lower sectors (buffer & motor) are rather weakly interrelated. • Elements in the two upper sectors (symptom & critical element) are rather highly interrelated.

The same analysis can be applied to stakeholders using the stakeholder map of position & influence (see opportunity mapping above)

PRACTICALITIES OF EVALUATING PEACEBUILDING PROGRAMMES The following captures discussions on practicalities of evaluating peacebuilding programmes. These practicalities are relevant to whichever method is chosen above.

Agreeing on terms To be able to evaluate a peacebuilding project there needs to be agreement on the following terms: Conflict Peace – often the definition of peace is unacceptable to many conflict parties – thus you cant even share the log frame with them. Peacebuilding – not only in relation to what kinds of activities constitute peacebuilding, but also to shift the focus from outputs to process. All three terms will likely be very context specific.

Evaluation and impact assessment of peacebuilding projects – Advanced course summary of the course Page 14 There also needs to be a clear understanding within the implementing agency on the relationship between humanitarian, development & peacebuilding objectives, otherwise the evaluator will have to try to construct this.

Attribution The problem of attribution – cause and effect – differs significantly from the debate over qualitative and quantitative approaches, and should not be confused. Using numbers does not lead you to a better understanding of attribution. Numbers are there to facilitate comparison and analysis only. They are not more reliable than qualitative data.

Conflict blind impacts Qualitative methods should be used to track the perverse effects of projects and the actions of spoilers. This can be done through surveys for instance. Where this is not included in the T of R for an evaluation the evaluator ought to ensure it gets included.

Consultation – good practice The use of local consultants is essential, as they know the situation. When talking to primary stakeholders don’t assume anything about them. Disaggregate them as much as possible and cross check responses. People will be interested to know which side you are on.

Focus group discussions are only useful to get an idea of the interactions between people rather than getting any specific information. One-to-one meetings are best, and can help clarify which questions ought to be probed in any subsequent joint meetings. The presence of powerful actors in focus groups will simply close everyone else down.

Control groups How can we know the counterfactual? Is it possible to have control groups? If so, is it moral to have control groups? How do we cope with situations where external events overwhelm the intervention, and the conflict actually worsens?

Criteria The OECD DAC criteria are used extensively by donors but much less so by implementing agencies. They are currently being revised by the OECD DAC, from which a framework or guidelines or standards on how to evaluate conflict and peace work may emerge. They expect to be doing field tests of a framework 2007-8. Thus far work has focussed on security sector reform, with guidelines being developed in a project lifecycle approach. They will next focus on dialogue activities.

Most evaluations tend to consider the output and outcome, but not the impact. Impact assessments tend to not consider attribution. The DAC criteria can be used at each of these different levels, and could be supplemented with theories of change to make the links between the different levels.

Currently the OECD DAC criteria could capture key issues in peacebuilding evaluation, for instance: Appropriateness criteria – would consider whether the analysis was/is right. Would analysis of whether the theory of change is appropriate fit within the appropriateness criteria? Relevance/connectedness – would consider the micro macro linkage Evaluation and impact assessment of peacebuilding projects – Advanced course summary of the course Page 15 Coherence – would consider the link with military or other interventions. However this means that the interests of security and political interest get mixed in with peacebuilding.

Do we need to develop specific guidance to help direct evaluators to this specific points within the criteria?

Indicators Emery’s position is that you can have indicators of peacebuilding, so long as you define peace. The indicators should be verifiable, but they don’t need to be measurable.

Activities – can develop indicators for this Specific objective – harder to develop indicators General objective – much harder to develop indicators

For example: If the objective was to support a peace process then the indicator wouldn’t be to do with the number of meetings held, but about how well the parties were prepared for the meeting.

Indicators will not show attribution – they can only show that there has been change, not to be able to track back causality. Examining the theory of change could be better at doing this.

Several different approaches were discussed as possible indicators:  Results based management. This has been discussed in the humanitarian sector, however it presents a moral hazard that could undermine the humanitarian imperative, and thus is being resisted.  Chaos theory (not discussed in detail)  Violence as an indicator (as in the Kosovo research). Would this depend on the objective – are you trying to reduce violence? Does violence occur in proportion to the level of peace in a society? Could we get around this by considering Galtung’s 3 levels of violence, and link all different peacebuilding interventions to work on cultural, structural or direct violence?  Freedom from fear. Although might people be less fearful because they are heavily armed?  A human security index that could be the basis of evaluation. UNDP’s central evaluation unit are working on this.

Several challenges were noted:  Indictors developed from the objectives assume that that the objective was based on an analysis of the situation. This isn’t always the case.  Collecting information for indicators of peacebuilding can take a lot more time that conventional indicators.

Evaluation and impact assessment of peacebuilding projects – Advanced course summary of the course Page 16