Evaluation of the CAFOD
2008-2011
Partnership Programme Arrangement
November 2010
FINAL REPORT
Trish Silkin
with Georges Tshionza Mata
This is an independent evaluation report of CAFOD’s 2008-2011 Partnership Programme Arrangement which has been prepared for CAFOD and for the UK Department for International Development. The views expressed herein are those of the author and do not necessarily represent the views of CAFOD or of DFID.
.
Table of contents
Acronyms and terms ...... i
Summary...... ii
Introduction...... 1
Results ...... 3 CAFOD’s approach ...... 3 Impact on people’s lives ...... 3 Building Church capacity to address poverty and injustice ...... 3 Building interfaith alliances to increase reach and effectiveness ...... 5 Supporting partners to improve responses to HIV...... 6 Impact on policy ...... 7 Impact of the PPA on gender and social exclusion ...... 9 Impact of the PPA on CAFOD’s organisational development ...... 10
Value for money ...... 11 Finance management...... 12 Audit ...... 12 Procurement...... 13 Public financial reporting ...... 13 Effectiveness in international programmes ...... 13 External validation of CAFOD standards...... 14
Lesson Learning...... 14
Building support for development...... 16
Issues to be addressed ...... 18
Annex 1: Terms of Reference for 2010 CAFOD PPA Evaluation...... 21
Annex 2: Persons consulted...... 26
Annex 3: Selected Documents Consulted...... 28
Annex 4: Timeline for the Evaluation...... 32
Annex 5: CAFOD PPA Performance Framework...... 33
Annex 6: Results against PPA indicators ...... 41
Annex 7: Case study of CAFOD’s PPA work ...... 45
Annex 8: Engagement Framework...... 51
Acronyms and terms
ACEAC Association des Conférences Episcopales de l’Afrique Centrale (Central Africa Bishops’ Conference)
AIDS Acquired Immunodeficiency Syndrome
CAFOD Catholic Agency for Overseas Development
DEC Disasters Emergency Committee
DFID UK Department for International Development
DRC Democratic Republic of Congo
ECHO European Commission – Humanitarian Aid and Civil Protection
HAP Humanitarian Accountability Partnership
HIV Human Immunodeficiency Virus
IDP Internally Displaced Person
INSET In-Service Training (for teachers)
LRA Lord’s Resistance Army
MDG Millennium Development Goal
NBRIA National Board of Religious Inspectors and Advisers
NGO Non-Governmental Organisation
OECD- Organisation for Economic Cooperation and Development-Development DAC Assistance Committee
PCM Programme Cycle Management
PPA Partnership Programme Arrangement
SPLA Sudan People’s Liberation Army
UN United Nations
CAFOD_PPA evaluation report_101130.docx i
Summary a) This is the report of an external evaluation of the April 2008 – March 2011 Partnership Programme Arrangement (PPA) of the Catholic Agency for Overseas Development (CAFOD), funded by the UK Department for International Development (DFID). CAFOD has received a total of £13.18 million in PPA funding during this funding period. The evaluation’s objective was to assess progress towards outcomes in the PPA Performance Framework. The evaluation involved document review and consultations with key stakeholders within and outside CAFOD. UK-based work was enriched by fieldwork carried out in the Democratic Republic of Congo.
Impact on people’s lives b) CAFOD programmes presented within this PPA are concerned with governance, humanitarian response, HIV and AIDS, and advocacy. Partnership with Church organisations is central to CAFOD’s approach because the Church’s reach and authority makes it a key actor in combating poverty and injustice. CAFOD recognises that Church organisations need to be strengthened to fulfil roles in development, humanitarian response and advocacy. The PPA highlights work in northern Kenya and in Eritrea where the Church is almost the only provider of services. In the first case, CAFOD’s partner supported communities to make local government more effective; benefits included improved health and sanitation services. In the second, CAFOD built its partner’s capacity almost from scratch so that it was able to work to international standards in managing nutrition programmes. c) CAFOD’s approach in the DRC has focused on establishing effective and well- coordinated national and diocesan structures. This is consistent with principles of engagement with fragile states, adopted by the OECD-DAC. The national Justice and Peace Commission has played a key role in rolling out decentralisation, from developing a decentralisation handbook (adopted by the government), to promoting effective governance at local level that has led to improved infrastructure and security, and a greater willingness by citizens to pay taxes1. CAFOD partners have often been the first to respond to humanitarian crises in eastern DRC and the only agencies willing to work in highly insecure areas. They are now recognised by the international community as key humanitarian actors and are providing models of good practice in humanitarian response that are being taken up elsewhere in the DRC. d) CAFOD’s partners are working with more than 100,000 men, women and children living with or affected by HIV. CAFOD promotes holistic responses to HIV and has developed a tool that enables partners to map the comprehensiveness of provision. This helped a Ugandan partner specialising in home-based care to secure the services of a legal NGO to provide rights education and legal representation for the women and children within their programme. A partner in Mexico set a legal precedent by winning a case to reinstate a soldier who had been discharged after testing HIV-positive. As a result of lobbying by another partner, the Mexican Bishops’ Conference published a pastoral letter opposing stigma and calling for a more inclusive response to HIV from the Church. e) CAFOD wishes to increase the scope and effectiveness of its work through forming interfaith alliances. The main activities under this objective have been support for Islamic Relief’s emergency programmes and to African Church leaders to combat religious conflict. Some useful activities have been carried out (support to emergency programmes in Gaza; a reflection on Christian/Muslim relations in Africa) but CAFOD’s strategic intentions for this objective are not well-defined and its impact is difficult to judge. CAFOD is reviewing its approach with a stronger focus on multi-faith initiatives that add value to programmes.
1 This work was carried out within the framework of CAFOD’s Action for Better Governance Programme with support from DFID’s Governance and Transparency Fund.
CAFOD_PPA evaluation report_101130.docx ii
Impact on policy f) CAFOD has developed an Engagement Framework2 to assist in building partners’ capacity and to monitor their progress in advocacy. CAFOD’s partners have had significant success in advocacy. In coalition with around 100 other CSOs, partners in Brazil were able to delay a reform to social security spending that would have adversely affected more than 20 million people and stalled the progress that is being made in reducing inequalities. CAFOD’s partner in Honduras built a case that enabled proceedings to be initiated against an international mining company for causing severe environmental damage, although the case was later dropped after a change in government. In both cases, CAFOD was able to provide technical support that enabled partners to build credible technical arguments to support their campaigning.
Impact on gender and social exclusion g) The evaluation found some examples of work that show positive impact on women and members of socially excluded groups. However, CAFOD lacks data to show whether these are representative of CAFOD’s work or are exceptional cases. This is recognised in CAFOD and is now being addressed.
Impact on organisational development h) PPA funding has helped to speed up or to extend changes that were already under way. These include the development of tools for monitoring progress in HIV and in developing advocacy strategies. The demand to demonstrate results under the PPA has also encouraged CAFOD to strengthen its PCM and country planning systems and its evaluation policy towards a much stronger focus on outcomes.
Value for money i) CAFOD has a financial framework that limits the proportion of resources that can be spent generating income and that stipulates a minimum of 55 percent to be spent on grants to partners and third parties. During 2009-10, CAFOD’s Trustees took steps to reduce overhead costs to protect funding for programmes. Peer reviews indicate that CAFOD is one of the best performers in terms of keeping fund-raising and overhead costs within bounds. CAFOD’s decentralised budgeting, centralised treasury and real time accounting mean that the organisation is able to manage its financial resources efficiently. Small variances in the last two years between budgets and expenditure and between income forecasts and actual income indicate that this is the case. CAFOD’s processes for internal and external audit provide a satisfactory level of control. j) During this PPA period, CAFOD developed and rolled out standard supply chain procedures. CAFOD has ECHO ‘P’ status which indicates that its procedures and control mechanisms are accepted as satisfactory for managing ECHO resources. k) Examples of the effectiveness of CAFOD programmes are provided in the report. The organisational changes that are being introduced to strengthen CAFOD’s focus on outcomes are enabling CAFOD to have a better picture of how effective it is and to demonstrate effectiveness more systematically.
Learning l) In 2008, CAFOD set up a Programme Learning Team tasked with improving programme quality and accountability. Initiatives include a fund for staff and partners to pilot innovations in programme practice. There has been cross-fertilisation in the use of the HIV and advocacy tools, with advocacy tools being used to assist planning in HIV work and HIV tools being
2 The Engagement Framework is also called the Voice and Accountability tool but is referred to as the Engagement Framework throughout this report.
CAFOD_PPA evaluation report_101130.docx iii
used to assess effectiveness and to analyse power in other types of programme. Communities of practice (e.g. for HIV practitioners) have provided support and have raised standards of practice. A recent review of CAFOD’s livelihoods work noted that a lack of relevant data, inconsistency in approaches, and a failure to make links with other sectors had compromised the impact of and learning from this work.
Building support for development m) For the PPA, CAFOD highlights how its work in schools is raising public awareness about global social justice. Specific work highlighted includes organising sixth form conferences, providing in-service training and tracking the attitudes of a sample of primary school children. There were mixed results from the sixth form conferences: schools that were already active were enthused to increase their efforts but follow-up was less in schools where teachers did not provide strong support. CAFOD seeks to influence teachers through in-service training and through the national network of religious inspectors and advisers. CAFOD school resources are included in standard curricula for religious education. Teachers rate CAFOD resources highly in terms of quality. A survey of the use of post-16 resources showed that the majority of schools surveyed had used CAFOD materials and taken action as a result. Preliminary findings from the research being carried out into the attitudes of primary school children show positive results from exposure to CAFOD materials. This is confirmed in Section 48 reports on religious education in Catholic schools3.
Issues to be addressed No issues emerged during the evaluation that need to be addressed within the framework of the current PPA. Issues flagged for the future are for CAFOD to (i) improve the coherence of the performance framework, (ii) be clearer about whether strengthening partners is an end in itself or a means to greater effectiveness, (iii) revisit and clarify the approach to interfaith work, (iv) incorporate sustainable livelihoods more effectively in a future PPA, and (v) be more ambitious in the PPA indicators that it sets for itself. An issue flagged for DFID to address is for DFID to be clearer about its expectations for performance measurement under the PPA, bearing in mind that performance frameworks are not logframes – indicators are examples of work rather than milestones towards objectives – and conclusions on progress cannot be drawn with the same degree of precision as with a logframe.
3 Section 48 reports are based on inspections carried out by the Catholic Education Service of England and Wales.
CAFOD_PPA evaluation report_101130.docx iv
Introduction 1. This is the report of an external evaluation of the April 2008 – March 2010 Partnership Programme Arrangement (PPA) of the Catholic Agency for Overseas Development (CAFOD), funded by the UK Department for International Development (DFID). PPAs were introduced by DFID in 2000 for some of the more trusted and respected UK development NGOs with which DFID shared common goals and objectives. PPAs provide the holders with three-year unrestricted funding that support outcomes mutually agreed between the holder and DFID. During the period 2008-2010 CAFOD has received a total of £13.18 million in PPA funding, which represents around 9 percent of CAFOD’s general income over the period. 2. The evaluation’s Terms of Reference identify its purpose as being to re-confirm CAFOD and DFID’s shared ethos and vision in recognised priority areas, which led to the formulation of the current PPA. The evaluation’s objective is to evaluate how far CAFOD has gone towards achieving the mutually agreed outcomes as stated in the PPA Performance Framework. 3. The Performance Framework sets out the overall purpose of CAFOD’s PPA and the strategic objectives that contribute to the achievement of the PPA purpose. CAFOD’s purpose statement is that “CAFOD will develop its relationships and collaboration with key church partners in order to strengthen their actions targeted at enabling individuals and communities to participate fully in decision making processes (in the south and in the north), to reduce vulnerability to disasters, conflict and the effects of HIV and AIDS, and to develop sustainable livelihoods.” 4. CAFOD’s PPA purpose is to be delivered through the following five strategic objectives: 1. Drawing on its faith identity CAFOD will access, support, mobilise and influence the Catholic Church to be more effective in its efforts to reduce poverty and injustice. 2. CAFOD will develop ecumenical and inter-faith alliances with organisations and networks in order to increase the reach and effectiveness of joint developmental and humanitarian responses. 3. Southern civil society organisations, including agencies of the Catholic Church, are better able to represent constituent communities in advocacy work, and are more effective in engaging with and influencing the policies and practices of government and business that affect them. 4. CAFOD will support Church partners to develop improved responses to HIV that both reduce vulnerability to infection and improve the quality of life for people infected and affected by HIV. 5. CAFOD will increase development awareness in England and Wales working through the Catholic schools’ network with a focus on global social justice and the MDGs.
5. Each strategic objective is supported by between two and five indicators. The unrestricted nature of PPA funding means that PPA holders are not required to track DFID funds through to output and outcome level (in other words, in relation to individual indicators and strategic objectives). In the same way, CAFOD’s PPA Performance Framework structure of purpose, objectives and indicators is not a logframe in the conventional sense and the indicators do not represent milestones for achieving their related strategic objective. Rather, they highlight or showcase examples of the type of work that CAFOD is doing to achieve the strategic objective. In that sense, they cannot simply be aggregated to provide comprehensive evidence of impact at strategic objective level. Instead, the impact achieved under individual indicators can be used to show evidence of patterns or trends in making progress towards strategic objectives.
6. During 2009-2010, DFID supported a consultancy to review PPA performance frameworks in order to make indicators more measurable. This led to the quantification of
CAFOD_PPA evaluation report_101130.docx 1
some of CAFOD’s indicators, although the previous structure of purpose, strategic objectives and indicators remained as before. The revised (March 2010) Performance Framework is at Annex 5. 7. The evaluation was carried out as follows. The evaluation was led by a UK-based evaluator supported by a local consultant in the Democratic Republic of Congo (DRC). The evaluation reviewed progress, first, by assessing whether CAFOD is on track to achieve the targets set out in the indicators in the Performance Framework. The findings from this assessment are provided in a table in Annex 6. The evaluation then considered selected indicators in greater detail, by reviewing programme materials and evaluation reports, and through discussion with CAFOD staff, and with selected partners and external stakeholders. For the UK-based work, these included telephone interviews and email exchanges as well as face-to-face interviews. Some fieldwork was also carried out in UK schools where there has been a relationship with CAFOD. 8. The UK-based work was supplemented and enriched by fieldwork carried out in the DRC in the first half of October 2010. This included discussions with CAFOD staff and partners based in the capital. In addition, the local consultant travelled to the eastern DRC to see examples of CAFOD-supported work carried out by CAFOD’s diocesan Justice and Peace and Caritas partners. Interviews and focus group discussions were held with partner representatives and beneficiaries. The findings from the DRC fieldwork are included as a case study presented in Annex 7. Selected examples from the case study are included in the discussion of results in the main report. (The case study provides information on additional activities funded by CAFOD that are relevant to other indicators in the Performance Framework (such as advocacy on gold-mining) but that have not been showcased in the Performance Framework.)
9. To complement the case study’s focus on an African country, discussion of several of the other indicators in the main report are taken from CAFOD’s programmes in Latin America. These exemplify CAFOD’s long history of working with the Latin American Church and with the strong civil society organisations and networks that are a distinctive feature of the continent. Latin America is a broadly coherent region, socially and culturally, springing from a common colonial experience and elements of shared political history. These examples from the Latin American programme therefore also serve to illustrate the increasingly regional focus of much of CAFOD’s work.
10. The report follows the structure specified in the evaluation’s Terms of Reference. This introduction is followed by a section on results. The assessment of results has been informed by the evaluation criteria of relevance, effectiveness, impact and sustainability. As required, the section is divided into results of the PPA on people’s lives, results in terms of policy influencing, and results on CAFOD’s own development, including on monitoring and evaluation systems. In the discussion of results, impact on people’s lives is treated through discussion of findings on CAFOD’s PPA strategic objectives 1, 2 and 4 and impact on policy through a discussion of strategic objective 3. The section on results is followed by a section on value for money, which is informed by the evaluation criteria of efficiency and effectiveness. The last two sections are concerned with lesson learning and building support for development. The latter is addressed through considering the progress that has been made under CAFOD’s PPA strategic 5.
11. Terms of Reference are attached as Annex 1, People Consulted as Annex 2, Selected Documents Consulted as Annex 3 and the Timeline for the Evaluation as Annex 4.
CAFOD_PPA evaluation report_101130.docx 2
Results
CAFOD’s approach
12. CAFOD is mandated by the Bishops’ Conference of England and Wales as the official overseas development and relief agency of the Catholic Church in England and Wales. Its faith identity is evident throughout its work; for example, out of CAFOD’s approximately 675 partners (of which 155 are based in Europe, the US or Australia), more than 388 are Church-related. The rationale for this lies not just in CAFOD’s own identity but in the inclusive nature of the Church, the local legitimacy that Church organisations enjoy, their networks, which enable them to reach and provide services to people living in the most remote regions, and their long-term presence in and with communities. The Church has particular authority to speak out on issues concerning human dignity, human rights and the responsibilities of states towards their citizens. In development terms, this means that the Church and its institutions are able to offer relevant, effective and durable solutions to the unjust conditions that keep people poor, disempowered and marginalised. CAFOD recognises that there are challenges to working with the Church, including inflexible and bureaucratic hierarchies, weak technical capacities, and in some cases a reluctance to collaborate with secular organisations or government programmes. Its capacity building approach to partnership aims to address these.
13. CAFOD’s approach is reflected in the PPA purpose which emphasises CAFOD’s role in supporting key Church partners to become more effective agents of change in ways that promote better governance, that reduce the vulnerability of individuals and communities to disasters, conflict and the effects of HIV/AIDS, and that develop sustainable livelihoods. The logic of the PPA purpose, objectives and indicators is broadly coherent. However, there is some overlap between the PPA purpose and strategic objective 1 (discussed below) and there is relatively little in the PPA that directly concerns sustainable livelihoods, although at 23 percent this sector represents CAFOD’s second largest share of expenditure (after disaster relief).4
14. CAFOD has made significant progress in delivering the PPA outputs (or indicators). This is shown in a summary of the progress that has been made so far towards each of the PPA indicators, which is provided in Annex 6. This summary of progress shows that, two thirds of the way through this current PPA, CAFOD had already met its quantified targets for some indicators and was well on track to achieving the remainder by March 2011.
Impact on people’s lives
Building Church capacity to address poverty and injustice (strategic objective 1) 15. As noted, partnership with and capacity building for Church organisations is embedded in the PPA purpose. To some extent therefore there is a significant overlap between the PPA purpose and its first strategic objective. This also means that objective 1 has become something of a catchall category that includes programmes concerned with the Church’s involvement in governance, humanitarian response, sustainable livelihoods, disaster risk reduction and campaigning on climate change. The diversity of work showcased under strategic objective 1 creates some ambiguity about whether the objective is chiefly concerned with strengthening Church structures or is concerned with achieving specific development objective(s). This presents problems in drawing conclusions about the impact of the objective as a whole, whatever one can say about impact at the level of individual indicators and programmes.
4 The absence of examples in the PPA of work in sustainable livelihoods may reflect a conclusion of a recent review of CAFOD’s livelihoods work that CAFOD has not applied a consistent approach in this sector.
CAFOD_PPA evaluation report_101130.docx 3
16. The work carried out under objective 1 provides credible evidence to support the argument that the Church is often uniquely placed to take action to combat poverty and injustice. This is demonstrated in the following two examples: • The Church is almost the only service provider in Dukana, a remote area of Marsabit, which is ranked as the poorest rural district in Kenya with approximately 88 percent of the population living below the poverty line. The Church provided support and encouragement to communities in Dukana to demand that a corrupt and ineffective local committee be replaced. The new committee has successfully lobbied the District Commissioner to provide a trained nurse for the local health clinic, has had three new boreholes drilled, and has introduced more transparent and competitive tendering procedures that have reduced the cost of one project from US$40,000 to US$12,0005. • In Eritrea, a long history of government restrictions on the operations of all types of non-government agency left the Eritrean Catholic Secretariat (ErCS) as one of the few agencies able to respond to a nutritional crisis in 2009 when, according to UNICEF, admissions to therapeutic feeding centres were six times higher than in the previous year. The isolation of Eritrea, the result of two decades of government policies, also meant that the ErCS was trying to tackle the crisis without either adequate funding or relevant up-to-date technical expertise. CAFOD funding and training in SPHERE standards enabled the ErCS to achieve an internationally acceptable level of proficiency in running a nutrition programme, and to exceed its target number of beneficiaries by almost 50 percent.
17. CAFOD’s work in the DRC originates in its response to the Rwanda crisis during the 1990s and is supported by work with ACEAC6 (the Central Africa Bishops’ Conference), which is an important regional actor in efforts to reduce conflict in the Great Lakes. CAFOD’s principal partners in the DRC are the Bishops’ Conference, the national Justice and Peace and Caritas offices, and local Justice and Peace and Caritas offices in the dioceses of eastern DRC. CAFOD’s approach in the DRC has focused on establishing effective and well-coordinated national and diocesan structures after the damage caused to the Church, as to other institutions, by years of conflict and neglect. This approach is consistent with principles of engagement with fragile states, adopted by the Organisation for Economic Cooperation and Development-Development Assistance Committee (OECD-DAC), which give first priority to helping to “build legitimate, effective and resilient state institutions”7. Partners give CAFOD much of the credit for the fact that they are now much stronger institutions than before, in particular for the fact that: • the Church takes a lead on issues of conflict, the management of natural resources, and decentralisation. • a functioning national and diocesan Justice and Peace structure has replaced previous fragmented diocesan initiatives • diocesan Caritas offices are increasingly providing models of good practice in humanitarian assistance in areas of eastern DRC where other agencies are unwilling or unable to work. 18. Annex 7 contains a detailed case study of CAFOD’s work in the DRC. Specific results of work by the Congolese Justice and Peace Commission and by Caritas are shown in Box 1 below.
5 This work was carried out within the framework of CAFOD’s Action for Better Governance Programme with support from DFID’s Governance and Transparency Fund as well as CAFOD general funds 6 Association des Conférences Episcopales de l’Afrique Centrale. 7 OECD-DAC. 2007. Principles for Good International Engagement in Fragile States and Situations. http://www.oecd.org/document/46/0,3343,en_2649_33693550_35233262_1_1_1_1,00.html
CAFOD_PPA evaluation report_101130.docx 4
Box 1: Promoting good governance and reducing vulnerability in the DRC
Decentralisation:8 The national Justice and Peace Commission developed a guide to decentralisation for civic education in dioceses that was adopted by the government as the basis for the national programme for educating citizens about the new decentralisation law. Diocesan structures throughout the DRC are working to ensure that decentralisation leads to tangible benefits for citizens. They have set up consultative bodies of citizens with local authorities, which have helped to reduce conflict (through establishing multi-stakeholder security committees; disarming returning militia; engaging in reconciliation) and to improve infrastructure (health centres, electricity, roads). These bodies have played an important role in overcoming distrust among citizens and between citizens and local government. The Justice and Peace Commission in Bukavu (pop. 2,34 million) conducted research which showed that corruption was the main reason that people were withholding tax payments. They ran an awareness campaign involving citizens, local officials and MPs which led to corrupt tax officials being replaced and participatory structures set up to scrutinise how tax revenues are collected and disbursed. This has resulted in a higher level of payments and a significant improvement in the provision of public infrastructure.
Humanitarian response: Between 2008 and 2010 Caritas provided emergency assistance to more than 55,000 households (around 275,000 individuals) in conflict-affected zones in eastern DRC. In 2009, Caritas was the first agency to respond to the displacement caused by LRA and SPLA9 military activity, and to initiate a wider international response to the resulting humanitarian crisis. In order to reach affected populations Caritas staff have to face armed hold-ups and looting and to negotiate with a myriad of armed militia and local authorities. The speed and effectiveness of Caritas’s humanitarian response in eastern DRC, together with its ability to work under conditions that other agencies are unable or unwilling to accept, mean that it has achieved recognition as a key humanitarian actor in the region, invited to participate in inter-agency coordination and planning mechanisms under the UN. In 2009, UN Humanitarian Coordinator Ross Mountain singled out Caritas Dungu for the speed and effectiveness of its work, and for its ability to reach communities which better resourced and equipped INGOs had been unable to assist10. Similarly, Caritas-Goma has also become a model for how other diocesan Caritas’s in the DRC can develop humanitarian assistance programmes.
Building interfaith alliances to increase reach and effectiveness (strategic objective 2) 19. This strategic objective has two components. The first concerns building a strong partnership with Islamic Relief in order to increase the reach and effectiveness of the work of both organisations. Through this partnership, CAFOD has been able to facilitate emergency assistance to populations in Gaza, Pakistan and pastoral regions of Ethiopia where the Church (and CAFOD) has little or no presence. Through this means, for example, CAFOD was able to assist 5,572 pastoral and agro-pastoral households (33,432 individuals) to recover from drought through provision of animal health services. The other component is concerned with supporting Catholic Church leaders in Africa to engage in interfaith initiatives that bring communities together to work on reconciliation and/or development. Work under this component has included providing support to the Archbishop of Jos (Nigeria) in his efforts to confront inter-religious violence, helping to fund and establish an interfaith organisation in Ethiopia, and bringing African Catholic leaders to the UK to reflect on the Church and Christian/Muslim relations in Africa.
20. These have all been practical interventions but it is difficult to assess what their actual impact has been, mainly because it is not clear what strategy linked the individual initiatives and what overall impact was intended. For example, to what extent is the
8 This work was carried out within the framework of CAFOD’s Action for Better Governance Programme with support from DFID’s Governance and Transparency Fund as well as CAFOD general funds 9 Lords Resistance Army (Northern Uganda) and Sudan People’s Liberation Army (South Sudan). 10 Reported in CAFOD 2010 self-assessment.
CAFOD_PPA evaluation report_101130.docx 5
partnership with Islamic Relief about CAFOD being able to demonstrate that it is supporting Muslim communities and to what extent is it concerned with getting aid to people that otherwise would not be reached at all? Given the high profile of some of Islamic Relief’s emergency work (e.g. in Gaza) it is unlikely that the work would have remained unfunded if CAFOD had not provided support. Moreover, while CAFOD’s funding for these emergency programmes is evidence of the strong mutual respect that exists between the two agencies, it is essentially a funding relationship rather than a joint interfaith initiative.
21. The initiatives that CAFOD has taken to support Church leaders in Africa have been useful in themselves and have been welcomed by the Church leaders concerned. The reflection on Christian/Muslim relations was regarded as an important first step towards building stronger relations between the two faith groups. However, with the possible exception of the inter-faith organisation in Ethiopia, they fall some way short of being real interfaith initiatives and there has been little or no follow-up from them. While this lack of progress has largely been due to externalities beyond CAFOD’s control, it also reflects a lack of clarity on CAFOD’s part about what this objective is trying to achieve.
22. An interfaith adviser was recruited by CAFOD to develop policy thinking in this area. During this PPA period, this resulted in a realisation that, while CAFOD’s faith identity meant that it was trusted and well-placed to participate in multi-faith initiatives, interfaith working was not, in most cases a useful end in itself. Rather, it was felt that CAFOD should seek to engage in multi-faith initiatives where this would add value to specific selected programmes of work within CAFOD’s portfolio.
Supporting partners to improve responses to HIV (strategic objective 4) 23. CAFOD’s main quantitative target for the PPA 2008-2010 was for 100,000 women, men and children who are living with or are affected by HIV and AIDS to have improved quality of life as a result of more holistic approaches to care and mitigation being provided by CAFOD’s partners in nine countries. By October 2010, this target had formally been surpassed, with more than 118,000 people being reached through CAFOD-supported programmes. (This indicator refers only to the numbers being reached and not to the quality of services being provided.) Although not all data provided by partners are disaggregated, it appears that more than 60 percent of those benefiting from these programmes are women and children.
24. The components of CAFOD’s approach to improving responses to HIV include promoting holistic care provision, working with Church partners to reduce stigma, improving approaches to prevention, and promoting mainstreaming of HIV into other programmes and sectors. CAFOD is on course to meet all its PPA targets with respect to these components, for example in the number of partners adopting relevant approaches to prevention and the number of programmes being adapted to incorporate responses to HIV. Some examples of the impact of CAFOD’s approach to work in HIV follow.
25. CAFOD has developed a mapping tool that helps partners to assess the degree to which a holistic response is being provided. The mapping tool measures and ranks care and mitigation services across the dimensions of health, psycho-social-spiritual support, human rights and legal support, and livelihood security. The four dimensions are broken down into 30 specific outcomes that map the extent of provision11. Partners who have used the tool have found it helpful in identifying gaps in provision and how to redress them. For example, a partner in Uganda specialising in providing home-based care identified that women and children were experiencing discrimination because they were
11 Examples of outcomes are ‘people with HIV have access to clean water and appropriate sanitation’ and ‘there are active initiatives seeking to change stigmatising legislation’.
CAFOD_PPA evaluation report_101130.docx 6
unaware of their legal rights. They reached an agreement with the Uganda Legal Network to provide legal education on how women could claim rights to family property and how children could seek redress for physical, emotional or sexual abuse. The Uganda Legal Network has also provided pro bono legal representation in individual cases. By using the mapping tool, CAFOD partners in Tanzania identified a problem of school drop-out among children with or affected by HIV. Work with teachers to show them how they could provide better support to these children has meant that during the PPA period no drop- outs have been recorded.
26. To complement the mapping tool, CAFOD is piloting a Quality of Life tool to assess the well-being of people who access care and mitigation services. By asking people to assess their level of well-being across the same four dimensions as the mapping tool, the Quality of Life tool helps to show how services are experienced in practice by clients. The Quality of Life tool uses a simple visualisation method to show well-being across the four dimensions, to track changes across these dimensions and to assess the reasons for change (whether related to service provision or to other factors). The tool has been well- received by clients who have used it, with preliminary results confirming that it is important to clients that responses to HIV and AIDS address these wider aspects of care, as well as health. So far, the tool has been piloted with small groups of self-selected clients in seven African countries and in Cambodia. Challenges remain in finding ways to systematise and scale up the application of the tool so that it is able to provide evidence of the impact of care and mitigation services on a representative scale.
27. In Central America CAFOD’s HIV programme has both national and regional dimensions. The regional dimension is particularly important for Central American partners who can feel isolated in their own countries, given that rates of prevalence are relatively low and stigma and discrimination are pervasive. As a means of encouraging mutual support, CAFOD brings its partners together for an annual meeting to discuss the past year’s experiences, and to see whether approaches that have worked well in one country are applicable in others. Out of this, a Mexican partner benefited from the extensive experience of a Guatemalan partner’s work with indigenous peoples. The latter learned about promoting the legal rights of HIV-positive people from the Mexican partner’s success in winning a case to reinstate a career soldier who had been discharged from the army after testing HIV-positive. This case set a legal precedent in Mexico as a result of which the employment policy of the army was changed. CAFOD enabled its partners to participate in the 17th International AIDS Conference in Mexico in 2008. Caritas Mexico used the opportunity that the conference presented to convince the Mexican Bishops’ Conference to publish a pastoral letter calling for support, inclusion, opposition to stigma, and holistic care and prevention responses to HIV from all Church initiatives.
Impact on policy
28. CAFOD’s policy influencing work has mainly been carried out under its strategic objective 3, which aims to build advocacy capacity in southern partners to influence the policies and practices of government and the private sector. Some policy work has also been carried out under strategic objective 1, concerned with building capacity in Church structures (e.g. the DRC Justice and Peace Commission’s work on gold mining described in Annex 7), and under CAFOD’s work to help Church partners to develop improved responses to HIV (e.g. the legal precedent concerning the armed forces that was set in Mexico). This section of the report only deals with work done under strategic objective 3, which has been organised around CAFOD’s advocacy monitoring tool, the Engagement Framework.
29. The Engagement Framework was developed to assist partners in monitoring the progress that they are making in developing advocacy capacity and strategies. The
CAFOD_PPA evaluation report_101130.docx 7
Framework enables partners to monitor progress along four dimensions: involvement in government processes; involvement in corporate structures; advocacy strategy development; and community/constituency building. Each dimension has five levels; for example, the dimension of involvement in government processes starts from ‘beginning a dialogue with government’ and proceeds to the top level of seeing ‘measurable impact from government reforms that partners have helped to influence’. Each of the dimensions pays particular attention to the potential impacts of reform on women. For example, data gathered as part of an advocacy campaign should be disaggregated by gender and there should be evidence that this has been reflected in gender-sensitive policy reforms. (The Engagement Framework is attached as Annex 8.)
30. CAFOD has achieved its targets for progress made under the Engagement Framework with nine partners in seven countries having progressed one level by March 2010 on at least one of the dimensions of the Framework. Five partners made this progress in their involvement in government processes but CAFOD’s role in influencing the private sector has also been commended. (See Annex 7 for commentary on influencing mining companies in the DRC). CAFOD also aimed to provide four examples of specific changes made by a government or company as a result of actions by citizens mobilised by partners during the PPA period. The examples reported to date have been from Bolivia (successful lobbying of local government to provide animal health services in an area where livelihoods are dependent on livestock), Brazil (delay in a proposed regressive tax reform), Honduras (initiation of legal proceedings against a mining company for environmental damage) and Sierra Leone (lobbying local government to involve young people in construction of a local road). The examples from Brazil and Honduras are described in more detail in Box 2.
Box 2: Protecting social benefits in Brazil; campaigning against environmental damage in Honduras Brazil – Social protection: Church partners led a coalition of more than 100 CSOs which mobilised hundreds of thousands of people to successfully delay a proposed tax reform that would have removed R$ (real) 60 billion (more than £22.5 billion) from the social security budget. This would have adversely affected more than 20 million people or around seven percent of the population. Support from the Bishops Conference was critical in giving the campaign legitimacy. Social security has played an important role in reducing inequality in Brazil (which is still ranked as one of the most unequal countries in the world) and, had the reform gone through, progress in reducing inequalities would have stalled. By definition, many of those who would have suffered were women, children and members of socially disadvantaged groups. The campaign worked with experts to present evidence on the implications of what was a technically complex subject. Technical expertise was also helpful in enabling the different sectoral interests represented in the coalition to reach a consensus around the objectives and strategy for the campaign. As the proposed legislation has been delayed, rather than thrown out, the coalition will need to maintain the momentum of the campaign.
Honduras – Environment: Caritas Tegucigalpa has campaigned on open cast mining since 2000; it has a proven track record in this field in Honduras, has developed a good understanding of the technical issues involved and has strong links with affected communities. With support from experts at Newcastle University, Caritas Tegucigalpa assembled evidence to support claims that operations at a gold mine managed by a subsidiary of the Goldcorp mining company were causing long-term significant environmental damage in the Siria valley and were presenting health risks. Newcastle University’s experts concluded that the company’s plans for mine closure fell far short of international standards. These findings assisted the Honduran Environmental Prosecutor to initiate proceedings against the mining company and against the state body responsible for regulating mining operations. In 2009, political crisis in Honduras brought in a government sympathetic to mining interests and, in late 2010, the Environmental Prosecutor’s case was dismissed. Given the current unfavourable national political context, campaigners in Honduras are now considering whether to launch coordinated action with campaigners in other countries in Central America where Goldcorp also has mining operations. Caritas Tegucigalpa’s work had previously been an important influence on partners in El Salvador in their work on gold
CAFOD_PPA evaluation report_101130.docx 8
prospecting.
31. In both these examples, CAFOD had strong partners with a long track record in campaigning. CAFOD’s contribution focused on assisting them to develop credible technical arguments to back up their lobbying. In Brazil, CAFOD helped to strengthen the coalition by encouraging partners to include organisations that they did not see as natural allies, such as feminist movements and trade unions. CAFOD’s partners in Brazil noted that using the Engagement Framework was not only useful in strategising and in monitoring progress but it also helped to counteract a tendency for different sectoral interests within the coalition to work in separate silos. In the case of Honduras, CAFOD secured the services of the UK experts, whose international reputations meant that their evidence could not easily be ignored. CAFOD also provided international legitimacy to the Caritas campaign by organising visiting delegations of journalists and MPs from the UK; this was a particularly valuable form of support since the campaign was directed against the operations of an international company.
32. These two examples demonstrate the commonly-encountered difficulty of attributing results and of measuring impact in policy work. There are several reasons for this difficulty. First, successful campaigns almost always rely on working in alliance with others, which means that success can rarely, if ever, be attributable to just one party. In both of the examples given in Box 2, CAFOD’s partners worked within coalitions and in the case of Honduras CAFOD was similarly coordinating the support that it was providing with sister agencies in the alliance of European and North American Catholic development agencies. In relation to measuring impact, progress in influencing policy is also strongly affected by externalities and is rarely linear. The coalition in Brazil benefited from operating in a relatively open representative democracy, although the success that the coalition has achieved so far is only provisional. After initial success, the campaign in Honduras suffered a major set-back as a result of the change of government. This set- back meant that partners in Honduras assessed that between 2009 and 2010 they regressed from level 4 to level 1 on the dimension of involvement with government on the Engagement Framework.
Impact of the PPA on gender and social exclusion
33. This evaluation revealed some good qualitative examples from PPA programmes that have had a specific and positive impact on women and on members of socially excluded groups. Examples include measures to address sexual violence in eastern DRC, enabling women IDPs to start small businesses in Colombia, protecting social benefits for the poorest in Brazil, and providing legal services for women and children living with or affected by HIV in Uganda. The evaluation has found however that CAFOD lacks data that are able to show whether these examples are representative of CAFOD’s work as a whole or are exceptional cases. This means that it is not possible to reach meaningful conclusions about the impact of the 2008-2010 PPA on gender and on social exclusion.
34. This finding confirms CAFOD’s conclusions from its own research, which shows that its programme database provides little in the way of quantitative evidence for CAFOD’s impact on socially excluded groups. Information in the database showed that “less than 30% of current live projects explicitly target relevant beneficiary categories (disaster victims, elderly, ethnic groups/tribes, homeless people, internally displaced persons, landless people, people with disabilities, persons living with HIV/AIDS, prisoners & ex- prisoners, refugees, returnees, shanty town dwellers, un(der)employed, war-affected communities and war-victims).”12 Neither the evaluation nor the research findings
12 James Marchant. July 2, 2010. CAFOD programmes – reaching the poorest and most disadvantaged. Internal document.
CAFOD_PPA evaluation report_101130.docx 9
suggest that PPA programmes do not have a positive impact on gender and social exclusion. Rather, both investigations show that CAFOD’s systems are unable to generate evidence to demonstrate whether they do or not.
35. The main exception to this comment on the quality of data is in work on HIV and AIDS where CAFOD has made systematic efforts to collect data that is disaggregated by gender and age. Under its work on HIV mainstreaming, CAFOD has also developed and is using a tool that analyses who has power and who is excluded from power in given situations.
36. CAFOD’s strategic framework 2010-2020 includes ‘change for the poorest and most disadvantaged’ and ‘the needs of women and girls and the equality of women and men’ among its immediate four priorities. In line with this, programmes will be required to set relevant baselines and indicators and systematically to collect disaggregated data. These will be reflected in Country Strategy Papers and Programme Frameworks. Terms of Reference for all final evaluations will include issues of gender equality and where relevant, HIV.
Impact of the PPA on CAFOD’s organisational development
37. The impact of PPA funding on organisational change in CAFOD has mainly been to speed up or to extend changes that were already under way.
38. The tools that CAFOD has developed for monitoring progress in HIV and advocacy programmes are a direct result of the PPA’s demand to demonstrate results. The flexibility of PPA funding has also enabled CAFOD to take forward these new initiatives more rapidly and comprehensively than would otherwise have been the case. For example, PPA funding made it possible for the development phase of the advocacy and HIV tools to include a long process of peer review and piloting with partners. The need to demonstrate results under the PPA has been a significant focus of discussion among the HIV focal points, and this has helped to consolidate this community of practice (described in the section on Learning).
39. In order to map the results from its engagement with the Catholic community in England and Wales, CAFOD’s education team has developed a dashboard which monitors visits to Catholic schools and levels of fundraising activity. The dashboard collates comprehensive information on schools participating in continuing professional development with CAFOD. The effectiveness of their participation together with pupil involvement in community advocacy action is captured and regularly reviewed.
40. The PPA has encouraged CAFOD to strengthen its focus on outcomes and on measurable results. CAFOD’s International Division has three mechanisms for monitoring and evaluating progress at different points in the project cycle. These mechanisms are: • Programme Cycle Management (PCM) System: which operates at the level of projects and programmes and which replaced processes that were mainly about managing grants with processes that are focused on enhancing and documenting effectiveness. • 3-year planning and review: which introduces an outcomes-oriented approach at country and regional levels linked to the global outcomes of CAFOD’s strategic framework. • Evaluation Policy: which defines approaches to and standards for evaluations of international programmes and the contribution that these can make to CAFOD’s organisational learning and the external validation of CAFOD’s work.
CAFOD_PPA evaluation report_101130.docx 10
41. The PCM system was introduced in 2007, following the recommendation of a previous PPA review13. It is designed to support the dialogue with partners on developing realistic projects and programmes, and on tracking outcomes and changes in capacity. The main focus during this PPA period has been on institutionalising the PCM system and improving its quality and use.
42. Three-year country-level planning and the new evaluation policy were introduced in the 2010-2011 financial year. CAFOD has reviewed the first round of three-year plans and has identified the main areas where they need to be strengthened. The first is the need for plans to make a clearer distinction between outputs (specifically, improved partner capacity) and outcomes (improved services). This finding mirrors the finding from this evaluation about the ambiguity in strategic objective 1 about whether it is concerned chiefly with strengthening Church structures of with achieving specific development outcomes. Second, there is the need to better integrate planning and budgeting – in several plans there is a disparity between the outcomes identified in the three-year plan and actual programme activity as judged by the number of grants made and the levels of expenditure. Finally, country plans need to incorporate the monitoring tools that CAFOD has developed more explicitly into their monitoring proposals, with better tracking of change in beneficiaries lives. CAFOD plans to follow-up from this review by doing further work on matching outcomes in country programmes with CAFOD’s corporate strategic outcomes and on developing core indicators for specific sectors that could provide an aggregate picture of the impact of CAFOD’s work.
Value for money
43. By the end of this funding cycle, CAFOD will have received more than £13 million of public funds through the PPA. The evaluation was required to consider whether CAFOD is able to show value for money from this funding, including specific details about the organisation’s procurement processes and procedures, and measures in place to manage overall fiduciary risk.
44. Value for money assessments consider how funds have been used, employing the criteria of economy (minimising the cost of resources used while having regard to quality); efficiency (the relationship between outputs and the resources used to produce them); and effectiveness (extent to which objectives are achieved)14. Fiduciary risk assessments are concerned with whether there is ‘leakage’ of resources, due either to weak finance management or because of deliberate misuse. Fiduciary risk assessments consider finance management from the perspective of efficiency, control and transparency. Together, value for money and fiduciary risk ask questions about whether funds are being used for the purposes for which they were provided and whether they are achieving their intended objectives at reasonable cost.
45. This section of the report assesses value for money and fiduciary risk by considering CAFOD’s finance management (economy and efficiency); audit and procurement procedures (control; efficiency); financial reporting (transparency); and management of overseas programmes (effectiveness). A caveat needs to be issued in relation to this exercise because: “[m]easuring fiduciary risk is not an exact science. By definition, risk involves uncertainty and hence, ultimately, a portion of subjective judgement.”15 The same holds true for value for money – to take the most obvious example, there is an
13 PARC/IOC. November 2005. Evaluation of the DFID/CAFOD PPA. 14 National Audit Office http://www.nao.org.uk/images/commission_flow_chart_large.jpg 15 Bernasconi J. L and Tediosi F. No date. Current issues and future challenges in health SWAps: Fiduciary Risk Assessment & Financial Management – Issues Note. Swiss Centre for International Health/Swiss Agency for Development and Cooperation
CAFOD_PPA evaluation report_101130.docx 11
unavoidable measure of subjectivity in the judgements that different stakeholders make about the relevance and quality of particular services or development interventions.
Finance management 46. CAFOD prepares budgets within a financial framework that is formally approved by CAFOD’s Trustees. The current financial framework stipulates that not less than 55 percent of total spend must be for grants to partners and other third parties, and that not more than 15p can be spent to raise £1 of income. In order to protect budgets for partner grants, CAFOD’s Trustees undertook a financial review during 2009-2010 that identified savings of £1.8 million that will be made through reducing staffing and activity costs from the next financial year. CAFOD also reviews quarterly to ensure that its planning takes into account possible financial downturns or economic changes that may compromise funding for programmes. 47. CAFOD has traditionally kept the cost of generating funds as a fairly low proportion of spend. In 2007, at 8.5 percent of income, CAFOD was ranked 9th lowest out of 10 international development NGOs in terms of the proportion of income that it spent on generating funds16. This compared with the lowest spend of 8 percent, the highest spend of almost 25 percent and an average spend of just over 20 percent. CAFOD and some of its sister agencies have set up a benchmarking exercise through which they are continuing to monitor this indicator (and also the proportion of income for salaries). The most recent results from this exercise are not yet in the public domain, but CAFOD is reported to have retained its good ranking. 48. CAFOD’s finance management systems encourage efficient use of resources. CAFOD prepares annual budgets and forecasts of general income (i.e. excluding income received for emergencies). Budgeting is decentralised with programme teams and country offices being required to prepare budgets in line with annual operational plans and according to a coding structure that captures expenditure at divisional, departmental, regional and programme team/office level. CAFOD operates a centralised treasury, with most payments to partners and for UK payroll and CAFOD activity being paid from head office. The central accounting system is in real time, allowing for flexibility in the reallocation of funds when necessary and timely accounts reconciliations17. Detailed annual budgeting and real time analysis of budget against expenditure mean that variances tend to be relatively small. In the last two financial years, there was a variance in expenditure compared to budget of 1.6 percent and a variance in actual over projected income of 1 percent. This is one indicator of the efficient management of resources.
Audit 49. CAFOD has internal systems for scrutinising its accounts, including a specialist Finance and Internal Audit Committee as a sub-committee of its Board of Trustees. CAFOD’s Corporate Leadership Team, Finance and Internal Audit Committee and Trustees carry out formal quarterly monitoring and mid-term review.
50. The Trustees have mandated the Finance and Internal Audit Committee to oversee a risk based internal audit function. A formal internal audit programme was introduced in 2006. Two internal audits a year are carried out of overseas offices and the programmes that they manage, which include an assessment of a sample of partners. At a minimum, the audit examines efficiency, risk management and procurement. A recent partner audit considered organisational management, programme management (including beneficiary selection procedures), supply chain and stock management procedures, and finance management, audit and control systems.
16 Charities Aid Foundation. 2007. Charity Trends. 17 CAFOD is exploring options for country offices to move away from monthly returns towards real time accounting.
CAFOD_PPA evaluation report_101130.docx 12
51. A full external audit of CAFOD is carried out annually and the implementation of recommendations is tracked by the Finance and Internal Audit Committee.
Procurement 52. Procurement is concerned with both efficiency (ensuring that goods and services are procured at the least practicable cost) and with transparency (preventing or mitigating corrupt practice). As CAFOD began to open overseas offices, increasing numbers of staff in different parts of the organisation became involved in procuring goods and services, and it became necessary for CAFOD to standardise procedures and practice across the organisation.
53. During 2009-10, CAFOD introduced standard supply chain procedures and began a roll- out of these within international programmes, with a training programme for overseas staff and some partners. The procedures are set out in a comprehensive manual or toolkit (finalised March 2010) containing guidance, templates and regulations. It covers all stages in the supply chain including identifying suppliers, completing a purchase, ensuring delivery (including warehousing, transport and storage), receipt of goods, services and supplies, and making payments. The procedures are to be used for all projects and programmes covered under institutional contracts, including whole or part funding from the Disasters Emergency Committee (DEC); all projects where the total value of approval within the same grant to the same partner exceeds £20,000; and all procurement directly managed by CAFOD staff. The manual defines different financial thresholds requiring different tendering processes.
54. Since 2010, the supply chain procedures have been extended to cover UK CAFOD team and office budgets. This yielded a 22 percent saving on CAFOD’s audit fee.
55. CAFOD has ECHO ‘P’ status which indicates that its own procedures and control mechanisms, including in procurement, are accepted as satisfactory for managing ECHO resources.
Public financial reporting 56. CAFOD provides all financial information required by the Charity Commission and Statement of Recommended Practice 2005 charity accounting regulations in its annual returns and published Trustees’ report and financial statements. In 2006, the Charity Commission cited CAFOD as an example of good practice for the overall quality and timeliness of its annual financial statements. The Charity Commission noted that: “The timely submission of accounts says a great deal about how your charity is run. Demonstrating your efficiency in this way may help you access funding and will help you avoid criticism from those whose views you care about.”18
57. For the next financial year CAFOD is committed to extending expenses reporting, which currently covers Trustees only, to include the Chief Executive, and is also considering extending the current practice of publishing information on grants over £50,000 to publishing information on all grants made.
Effectiveness in international programmes 58. Effectiveness is measured in part by the results of projects and programmes, and examples of results have been provided in this report. The report has also described organisational changes that are being introduced with the aim of strengthening the focus on outcomes at programme, country and corporate levels. These organisational changes
18 CAFOD. 15 May 2006. Charity Commission praises CAFOD for coming top on financial accountability. Press release.
CAFOD_PPA evaluation report_101130.docx 13
have followed moves to decentralise programme management, by opening overseas offices, with the aim of working more closely with partners and helping them to become more effective. A partner representative at regional level in Africa told the evaluation that she considers that decentralisation has significantly improved partners’ capacity to plan, implement, monitor and evaluate their work, partly through the tailored training that CAFOD has provided but largely through CAFOD’s close accompaniment and introduction of new systems and procedures19.
59. CAFOD is taking a further step towards ensuring minimum standards of performance in partners’ programmes through establishing ‘strategic partnerships’ with key partners. Candidates for strategic partnerships are identified from among partners who are funded above a specified level and where there is a minimum two-year track record of working with CAFOD. The components of the strategic partnership include organisational development, skill sharing, joint advocacy initiatives and a dedicated programme of learning and feedback mechanisms to improve performance. CAFOD is currently in the process of defining the minimum standards that will apply to these partnerships, which will be formalised through a mutual Statement of Commitment.
60. Finally, it should be noted that PPA funding (at around 9 percent of CAFOD’s income during this PPA period) is complemented by significantly greater funds from CAFOD’s supporter base (at around 50 percent of CAFOD’s income in 2010) and from institutional donors. These other funding sources also make a contribution to PPA programmes and, taking these into account, the results achieved under CAFOD’s PPA represent good value for money from public funds.
External validation of CAFOD standards 61. CAFOD has official recognition by the following external agencies: • ECHO ‘P’ status (as indicated above). • DEC – compliant with DEC accountability priorities, which include sound financial management at agency and partner levels (green on all indicators) • HAP – compliant with HAP Principles of Accountability and Quality Management in Humanitarian Action • People in Aid – Verified Compliant in the management of human resources (for humanitarian and development agencies).
Lesson Learning
62. As other organisations, CAFOD has faced challenges in ensuring that lessons learned in individual programmes are picked up, distilled and disseminated so that overall programme quality is enhanced. In 2008, CAFOD formalised its learning systems by establishing a Programme Learning team within the International Division with a remit to promote learning as a means of improving quality and accountability within programmes; to promote thematic learning; and to help consolidate and develop programme management systems so that they make a stronger contribution to organisational learning. Much of the period under review has been concerned with setting up new learning systems and developing tools that provide support and incentives to staff to learn from one another’s work. During 2010, the Programme Learning Team also commissioned a thematic review of CAFOD’s work in HIV and AIDS on behalf of the International Programmes Committee of the board.
19 She commented also that she believes that CAFOD’s practice of involving partners in budget preparation has not only developed partners’ skills in finance management but has also made international programme budgets more relevant and accurate.
CAFOD_PPA evaluation report_101130.docx 14
63. In 2009, the Programme Learning Team established a £25,000 fund to enable CAFOD programme staff and partners to carry out small pilot projects with an emphasis on innovation, learning and improving programme practice. In the first year, eight projects were supported including an exchange of learning between partners in Colombia and Peru (described below), use of new advocacy media in Brazil and the DRC, piloting of monitoring and evaluation tools in Southern Africa, learning on sexual violence in the DRC, and young people’s learning on traditions and cultural practices in Cambodia. It is too early to see how far the learning from these initiatives is influencing other international programmes, but priority is being given this year to proposals that replicate or scale up the ideas that were developed in last year’s funding round. Other priorities for this year are for proposals to relate to the international priorities in the new Strategic Framework (poverty and disadvantage; gender; working with the Church; demonstrating effectiveness and impact)20 and/or that involve cross-country programme collaboration.
64. The HIV and advocacy tools developed under the PPA have been described above. There has been considerable cross-fertilisation between sectors in how these tools have been used. For example, partners in Central America have used the Engagement Framework for planning and developing strategies for work on HIV. The HIV Quality of Life tool has been adapted for measuring the effectiveness of Disaster Risk Reduction plans in the Philippines and of capacity-building programmes in Ethiopia. The four Ps analysis (potential, power, protection, and priority stakeholders/groups) that is used in HIV mainstreaming has been used in gender analysis and in other thematic areas. The main lessons learned from using the HIV mainstreaming that are transferable to other contexts are shown in Box 3 below.
Box 3: Lessons learned from using the HIV mainstreaming tool
• HIV mainstreaming must be applied to wider development work if this is to remain effective and do no harm in contexts also affected by HIV and AIDS • Mainstreaming needs to be driven by senior managers and budget-holders • Understanding the need to mainstream is not enough. People need tools to make mainstreaming happen in practice • The tools are effective in both low-prevalence & high-prevalence countries. In low- prevalence countries it is essential to give time to developing people’s understanding of HIV before proceeding with mainstreaming • The tools should also be applied to analyse the management of HIV-focused programmes. • The challenge remains of how these tools can be adapted to work with communities. The resource indicates points to raise, but tools should be supplemented with other participatory processes for communities.
65. The HIV mainstreaming tool was used in a workshop at the end of the first phase of work on small enterprise development with IDPs in Colombia to analyse power relations in the project. As a result of the analysis the second phase of work had a more explicit gender focus and policies were developed concerning the rights of beneficiaries to acquire some of the equipment procured during the project. This was part of a wider lesson learning exercise which identified that, although the project had worked effectively with women and people with disabilities, the project approach had tended to exclude young people. A learning event at the end of 2009, brought the Colombian partners together with partners from Peru who have specific expertise in how to include young people in livelihoods work.
20 The development of the strategic framework was used as an opportunity for developing the skills of programme staff in 3-year planning and for assisting staff to work through the programmatic implications of the four international programme priorities.
CAFOD_PPA evaluation report_101130.docx 15
66. CAFOD has also disseminated these tools more widely. The Quality of Life tool and the Engagement Framework have been shared with sister agencies in the UK AIDS Consortium and in BOND, respectively. The Engagement Framework has been taken up in a modified form by Progressio whose partners have found it a simple and systematic way of measuring the progress that they are making in advocacy initiatives. CAFOD’s approach to mainstreaming HIV was presented at the 17th International AIDS Conference in 2008 and its Quality of Life tool at the 18th International AIDS Conference in 2010.
67. CAFOD has set up communities of practice to provide support to and develop the professional expertise of staff. So far, communities of practice exist for the staff who are working on HIV and AIDS and for a group of Asia Programme Accompaniers. Led by the 2-person HIV support team in London, CAFOD staff and partners working across 8-9 countries meet virtually and face-to-face to share knowledge and experience and raise the level of practice of the team as a whole. The community of practice provides a collegiate environment where problems can be openly discussed and practical solutions suggested and debated. Being members of the HIV community of practice has been particularly helpful to staff and partners working in low prevalence regions like Central America, who benefit from the greater experience of their colleagues working in Africa.
68. As noted, the livelihoods sector is hardly represented in the PPA. With a view to planning for the next PPA, the Programme Learning Team recently commissioned a review of CAFOD’s livelihoods programmes. The review noted that data on livelihoods was used inconsistently or was absent and that CAFOD’s livelihoods framework is not applied consistently across programmes, resulting in considerable variations in practice and a missed opportunity in terms of cross-organisational development and programme learning. The review also noted a tendency for livelihoods work to be developed in isolation from other core sectors (such as economic justice) which further compromises organisational learning and programme development.
69. The main exception to this has been the Disaster Risk Reduction (DRR) programme in Central America, where CAFOD’s partners have worked with communities and local government to adapt farming practices to reduce the likelihood of losses in the event of a disaster and to develop emergency planning should a similar disaster occur. The main driver for DRR in Central America was the environmental damage caused by Hurricane Mitch in 1998, which prompted governments in the region to develop relevant legal and policy frameworks and training materials. The main lesson learned from the CAFOD- funded programme was the need to incorporate DRR into sustainable livelihoods programmes. The results of the programme were produced in the form of a learning document, and CAFOD staff and partners in the Philippines are now using this to identify how these lessons can be taken forward in Asia. In a related initiative, programme staff from Kenya and East Timor were involved in a review of learning on livelihoods in Ethiopia.
Building support for development
70. Under the PPA, CAFOD showcases the support that it is providing to Catholic schools to build development awareness. The specific results indicated for the current PPA are (i) that pupils in 50 sixth forms will take action on global justice21 and (ii) that teachers from at least 280 schools will participate in INSET (in-service training) on issues related to the global dimension of the curriculum. As a third indicator, CAFOD is building a body of qualitative information on children’s attitudes and knowledge by tracking a sample of children in selected primary schools over the period of the PPA.
21 A sample of 50 sixth forms represents around 17 percent of the total of Catholic sixth forms in England and Wales.
CAFOD_PPA evaluation report_101130.docx 16
71. By April 2010, CAFOD had worked with 45 sixth forms, compared with its target of 50 for the whole PPA period. One of the main activities reported under this indicator has been the organisation of a major sixth form conference in 2008 attended by 259 students and 40 teachers/chaplains from 24 schools and a leadership day in 2010, involving more than a hundred students and 16 teachers from 13 schools. The main aim of these events is to enthuse the sixth formers so that when they go back to their schools they share with others what they have learned on the day. Completed evaluation forms from the 2008 conference indicated that participants had been stimulated and inspired by the day’s activities and that around half of them planned to take some immediate action in their schools on their return.
72. A follow-up review by CAFOD indicated a mixed picture in terms of whether participating schools and students had followed through on these intentions. Some students were very active, with sixth formers in one of the most committed schools using ideas and materials from the conference to organise two-day retreats for two groups of younger pupils on the themes of environmental change and of ‘Live simply, sustainably and in solidarity’22. The same school organised several other activities, including transforming the sixth form common room into a soup kitchen for Harvest Fast Day (one of CAFOD’s main annual fund-raising events). In this school, as in others, the pupils have formed an active Justice and Peace (or ‘CAFOD’) group, and successive sixth forms have continued to be involved in similar ways, for example, by organising retreats for local primary schools using CAFOD themes and materials.
73. CAFOD is the almost automatic choice for teachers in Catholic schools who are looking for resources on development issues. However, in schools visited during the evaluation, teachers also rated CAFOD materials highly compared to resources from other agencies in terms of the interest of their content, relevance to different age groups and attractive presentation. Since most materials are available from CAFOD’s web-site, their accessibility and ease of use were additional points in their favour. A small survey of how CAFOD’s units for Post-16 General Religious Education had been used in Catholic schools and colleges found that of the 65 schools surveyed, two thirds had taken action as a result of studying these materials and that about the same number had taken action as a result of using other CAFOD materials. Actions included campaigning, fund raising, peer education and lifestyle changes.
74. CAFOD’s assessment after the 2008 conference was that the main deciding factor in whether sixth formers followed through on their commitment was the level of support provided by the teacher or chaplain who accompanied the students to this event. CAFOD’s INSET days address the need to develop teachers’ interest and knowledge. By 2010, teachers attending INSET days had come from 283 schools, compared with the target of at least 280 for the whole of the PPA period. CAFOD surveys of teachers who have attended INSET indicate that they feel they have a much better understanding of the relevance of the global dimensions for the curriculum.
75. School conference and INSET days take place within a much broader approach to influencing what is taught in schools. CAFOD has good contacts with NBRIA (National Board of Religious Inspectors and Advisers) which acts as a resource for indicating to CAFOD what types of materials would be most useful in schools and as a network through which materials can be publicised and disseminated. In 2008, CAFOD took three Diocesan Religious Education Advisers on an exposure visit to meet partners and see programmes in El Salvador. On their return, they used materials from their visits for delivery in their own INSET days and in talks with more than 90 schools. This is in
22 CAFOD’s Live Simply resources for groups encourage reflection on social justice and poverty.
CAFOD_PPA evaluation report_101130.docx 17
addition to the 283 reached by CAFOD. CAFOD materials have been referenced within standard curricula for religious education at secondary level (such as those from the National Open College Network and from Edexcel).
76. Preliminary findings from the research being carried out into the attitudes of primary school children highlighted that the children concerned have a well-developed sense of fairness, understand that countries as well as individuals need to be treated fairly, and are conscious of the fact that this does not always happen. Related to this children were aware of their relatively privileged position in the world and the social responsibility that goes with privilege. The children had some understanding of poverty as a multi- dimensional concept and they were particularly concerned about climate change. Most of the children had heard of CAFOD and the more familiar that they were with CAFOD, the more that they were able to discuss these topics.
77. Discussions during the evaluation with pupils in schools where there is a relationship with CAFOD also indicated a relatively good understanding of CAFOD and a concern for global social justice. Pupils at primary level were able to discuss and make distinctions between basic human needs and lesser needs, and between emergencies and development work. They were also able to say something about how and why CAFOD works in partnership with local people. The sixth formers interviewed had quite a sophisticated grasp of a concept such as partnership and of its implications in terms of their own responsibilities.
78. Section 48 reports on religious education in Catholic schools23 bear out these findings. CAFOD was mentioned as having been a source of information and ideas for action in reports on more than a third of schools in 10 dioceses. For example, a report on a school in Clifton diocese noted: “There is a strong emphasis on social and moral responsibility taught using CAFOD resources. This contributes well to the development of pupils’ social conscience.” A report on a school in Lancaster diocese observed that staff were linking fund-raising for CAFOD to issues such as equality and were using these links to influence children’s learning and attitudes.
79. Fund-raising for CAFOD is not the main objective of CAFOD’s educational work, but the numbers of schools that have raised funds for CAFOD and the level of funds that they have contributed can be taken as a rough proxy for the level of development awareness in schools strongly influenced by regular exposure to CAFOD materials. CAFOD records show that for the academic year 2008-2009, 1,214 primary and secondary schools raised £613,851 and in the academic year 2009-2010, 1,425 schools raised £1,133,117.
Issues to be addressed
80. No issues emerged from this evaluation that CAFOD needs to address within the framework of the current PPA. This final section of the report therefore focuses on issues that CAFOD needs to consider mainly for the purposes of designing the next PPA. This section draws on points that have been made throughout the report. In most cases, the issues identified support initiatives that CAFOD is already beginning to take to strengthen its work under the PPA and across the international programme as a whole.
81. Coherence: The performance framework is broadly coherent, with objectives and indicators being related in a generally logical way to the PPA purpose. However, there is some confusion between the overall purpose, which is concerned with strengthening the Church to be an effective agent of change, and strategic objective 1, which is similarly concerned with enabling the Church to be more effective in addressing poverty and
23 Section 48 reports are based on inspections by the Catholic Education Service of England and Wales..
CAFOD_PPA evaluation report_101130.docx 18
injustice. A related aspect, is that strategic objective 1 is a catchall category (including governance, DRR, humanitarian response, and campaigning on climate change) and this presents difficulties for CAFOD in demonstrating the impact that the objective as a whole has achieved. In reality, strengthening Church organisations to become effective change agents is a cross-cutting theme within the PPA, and should be treated as such. The strategic objectives would then be defined in terms of outcomes to be achieved within specific domains or sectors (as with strategic objective 4 concerned with HIV and AIDS).
82. Partnership: Related to the previous point, CAFOD needs to further clarify and specify its understanding of partnership. CAFOD’s new 10-year strategic framework reconfirms partnership as a central feature of CAFOD’s approach, and CAFOD’s claims for the efficiency, impact and durability of its programmes are in large measure based on this approach. It is all the more important therefore for CAFOD to make a clear distinction between strengthening partner capacity as a means to an end (an output) and strengthening partner capacity in order to achieve positive change in people’s lives (an outcome).
83. Interfaith. CAFOD needs to clarify the salience of interfaith work within its organisational values and the implications of this for programming. At present, the strategic objective appears to be something of an ’orphan’ with no clear ownership in any part of CAFOD’s structure and with no clear sense of what type of change interfaith work is expected to produce. CAFOD has begun to review this objective, focusing more on embedding multi- faith approaches within programmes. The evaluator learned that during negotiations with DFID for the current PPA, DFID expressed a strong interest in having a strong faith focus within CAFOD’s PPA. It may therefore be relevant for CAFOD to ask DFID to clarify its interest in this as it may help CAFOD to clarify its own understanding and to see whether this has implications for programming within the PPA or would be better addressed in other ways outside the PPA.
84. Sustainable livelihoods: This sector is highlighted in the PPA purpose and having livelihoods that provide an adequate level of economic security over the long-term is clearly fundamental to people’s well-being. This sector needs to feature much more strongly in the next PPA. As a longer-term project, the Programme Learning Team needs to develop strategies that will enable CAFOD to reflect on and to begin to address systematically the recommendations of its recent review of livelihoods work (concerning the importance of data collection, the need to adopt consistent approaches, and the need for stronger linkages between work in sustainable livelihoods and work in other sectors).
85. Targets: CAFOD has made good progress in achieving the quantitative targets specified in PPA indicators. By October 2010, CAFOD had met most of these targets, had surpassed some, and was well on the way to meeting the remainder. Considered more critically, this may be an indication that CAFOD was over-cautious in developing the 2008-2011 Performance Framework. Based on the experience of this PPA, CAFOD can afford to be more ambitious in the targets that it sets for itself for the next one.
86. Assessing performance in the PPA: CAFOD has made a significant investment in developing organisational systems and tools for measuring whether its programmes are having a beneficial effect on people’s lives. CAFOD’s ability to demonstrate effectiveness has made major advances since the 2005 review of CAFOD’s PPA noted that there was a considerable gap between CAFOD’s rhetoric and its practice. These organisational developments are broadly in line with DFID’s demand during the current PPA for PPA holders to develop more robust monitoring and evaluation systems by more clearly defining baselines, by sharpening indicators and by more firmly rooting accounts of achievement in evidence. However, it would be wrong if robust were simply to be equated with results that are readily capable of measurement and aggregation. As stated
CAFOD_PPA evaluation report_101130.docx 19
at the outset, a PPA performance Framework is not a logframe and conclusions about results at objective and purpose level cannot be drawn with the same degree of precision as when assessing performance in relation to a logframe. It will be important for the next PPA, for DFID to be much clearer about its expectations for performance measurement under the PPA.
CAFOD_PPA evaluation report_101130.docx 20
Annex 1: Terms of Reference for 2010 CAFOD PPA Evaluation
1. Background
Partnership Programme Arrangements (PPAs) were introduced by the UK Department for International Development (DFID) in 2000 as direct replacements for the former Block Grant and Volunteer Grants. These new models of Civil Society Organisation (CSO) support allowed DFID to enter high level strategic partnerships with some of the most trusted and respected CSOs with whom DFID shared common goals and objectives.
PPAs provided the holders with longer term, unrestricted strategic funding in support of mutually agreed outcomes, enabling them to undertake those potentially higher yielding areas of their work which were innovative and higher risk but were, as a result, often under- funded. Annual self-assessments by the CSO are carried out against a set of strategic objectives and indicators, in order to demonstrate impact, accountability and value for money.
The unrestricted nature of the PPA funding meant that, although high level outcomes were agreed between the two parties, the PPA holder was not required to track DFID funds through to output and outcome level.
Over the past couple of years DFID has, following National Audit Office recommendations, undertaken a comprehensive reform of the PPA process. One of the main outcomes of this process has been the introduction of more rigorous performance frameworks for all PPAs.
The Catholic Agency for Overseas Development (CAFOD) is the international agency of the Catholic Church in England and Wales. Its mission as part of the Catholic community, and together with partners and the global Caritas family is to: • work with poor and disadvantaged communities in the global South to overcome poverty and bring about sustainable development and well-being. • protect lives and relieve suffering during emergencies; reduce the risks to vulnerable communities as a result of conflict and natural disasters. • raise awareness and understanding of the causes of poverty and injustice to inspire a commitment to lasting change. • challenge those with power to adopt policies and behaviour that promote social justice and end poverty.
CAFOD first received DFID PPA support in 2001 and our current Programme Partnership Arrangement (PPA) with DFID runs until 31 March 2011.
The agreed purpose of this current PPA is: CAFOD will develop its relationships and collaboration with key Church partners in order to strengthen their actions targeted at enabling individuals and communities to participate more fully in decision making processes (in the south and in the north), to reduce their vulnerability to disasters, conflict and the effects of HIV/AIDS, and to develop sustainable livelihoods.
2. Purpose, Objective and Scope of the Evaluation
CAFOD_PPA evaluation report_101130.docx 21
As part of the PPA funding arrangement with DFID, it was agreed that all PPA holders would, in the last year of funding, arrange for an independent external evaluation of the PPA for the period 2008-11. This evaluation will use the most recent version of the CAFOD PPA performance framework.
The purpose of this evaluation is to re-confirm the common ethos and vision in recognised priority areas which exist between CAFOD and DFID and which led to the formulation of the current PPA.
The objective of the evaluation is to evaluate how far CAFOD has gone towards achieving the mutually agreed outcomes as stated in the PPA performance framework, which are as follows:
Strategic Objective 1 (international): Drawing on its faith identity CAFOD will access, support, mobilise and influence the Catholic Church to be more effective in its efforts to reduce poverty and injustice.
Strategic Objective 2 (international): CAFOD will develop ecumenical and inter-faith alliances with organisations and networks in order to increase the reach and effectiveness of joint developmental and humanitarian responses.
Strategic Objective 3 (Bolivia, Brazil, Cambodia, East Timor, Ethiopia, DR Congo, Honduras, Liberia/Sierra Leone, Mexico, Mozambique): Southern civil society organisations, including agencies of the Catholic Church, are better able to represent constituent communities in advocacy work, and are more effective in engaging with and influencing the policies and practices of government and business that affect them.
Strategic Objective 4 (international): CAFOD will support Church partners to develop improved responses to HIV that both reduce vulnerability to infection and improve the quality of life for people infected and affected by HIV
Strategic Objective 5 (England & Wales): CAFOD will increase development awareness in England and Wales working through the Catholic schools’ network with a focus on global social justice and the Millennium Development Goals.
The evaluation will also derive lessons which will enable CAFOD to inform its future strategies, programmes, approaches and organisation.
The target audience and main users of the findings of the evaluation will be CAFOD and the Civil Society Team at DFID. CAFOD partners and other stakeholders will also benefit from the findings (this will include but is not limited to southern partners of CAFOD, other donors, other parts of DFID, fellow PPA holders).
In terms of scope, the evaluation will focus only on the most recent PPA period ie from 1st April 2008 to the present therefore it should cover developments in CAFOD’s strategies, programming, structure and context over this time frame.
CAFOD_PPA evaluation report_101130.docx 22
The evaluation will consider the PPA in relation to the DAC24 Criteria:
Relevance: The extent to which programmes conform to the needs and priorities of their beneficiaries, and to the policies of partners, donors and other stakeholders.
Effectiveness: The extent to which programmes have achieved their objectives, considering their relative importance. Measuring effectiveness includes taking the perspectives of beneficiaries into account.
Efficiency: Efficiency compares the outputs against the inputs. It illustrates the relation between means and ends. To what extent can the costs of programmes be justified by their results, taking into account alternatives?
Impact: The totality of the effects of programmes, positive and negative, intended and unintended. Impact is wider and distinct from effectiveness where the focus is on intended and positive effects. Impact assessment starts from the perspective of the intended beneficiaries, and moves on to see how others are affected by the intervention.
Sustainability: The extent to which the positive effects of the work of CAFOD and its partners will continue, taking into account the degree of local management, ownership and resource allocation.
3. Methodology
Using the most recent agreed version of the CAFOD PPA performance framework and the agreed PPA baseline (both attached), the successful bidder will:
• Review the annual self-assessment reports which CAFOD has submitted to DFID since 2008 and their appendices (approximately 3 days)
• Review a representative sample of supporting documentation including partner reports, information from the Webpromise database and evaluations relating to the five strategic objectives of the PPA (approximately 5 days)
• Interview key CAFOD staff, partners, DFID staff and external stakeholders to clarify and amplify the research already carried out from reports (approximately 5 days)
• Field visits to validate findings and to gather evidence of change and of CAFOD’s added value (approximately 10 days) − to England/Wales Education partners (strategic objective 5) − to one country programme (to demonstrate as much as possible of strategic objectives 1-4)
• Review documentation of CAFOD’s internal systems, procedures and governance and interview CAFOD’s Finance and Human Resources directors (approximately 2 days)
• Prepare and present draft and final reports (approximately 5 days)
24 Development Assistance Committee of the Organisation for Economic Co-operation and Development
CAFOD_PPA evaluation report_101130.docx 23
4. Outputs
The successful bidder(s) will be expected to produce a report of no more than 20 pages of A4, in Plain English. This report will be written in a format which is easily accessible to all stakeholders.
This report will follow the following format:
1. Summary section (approx 2 pages). This summary will focus on the main findings and will specifically state how far the evaluator(s) feel CAFOD has gone towards achieving the mutually agreed outcomes as stated in the PPA performance framework.
2. An Introductory section (approx 2 pages) which should include the methodology used in undertaking the evaluation.
3. A section on Results (approx 6 pages) – this section should include (i) results, including impact, of the activities of CAFOD and its partners ‘on peoples’ lives’ including any specific impacts on gender relations and social exclusion issues (ii) results, including impact, of CAFOD on relevant policy issues (iii) the impact of the PPA on the overall organisational development of CAFOD. It should also comment on the overall health of CAFOD monitoring and evaluation systems.
4. A section covering Value for Money (approx 3 pages). In this section the evaluator will provide specific evidence to show whether or not CAFOD is able to show value for money for DFID’s PPA inputs. This should include specific details about the organisation’s procurement processes and procedures, and measures in place to manage overall fiduciary risk.
5. A section covering Lesson Learning (approx 3 pages). This section will highlight lessons learned throughout the 3 year PPA period and how/whether these have been taken up across CAFOD.
6. A section covering Building Support for Development (approx 2 pages) showing what progress CAFOD has made against strategic objective 5 of the performance framework and how this has contributed towards (i) building public knowledge and awareness of global poverty; (ii) generating public support and momentum for action to reduce global poverty, (iii) stimulating the public and/or organisations to act to reduce global poverty..
7. A section on Issues to be addressed (approx 2 pages). This section will highlight any specific issues which arose during the evaluation which the evaluator feels need to be addressed by CAFOD or DFID.
In submitting the final report we would ask that you include annexes which contain: • details of the final agreed Terms of Reference • a list of people and organisations interviewed • a list of documentation reviewed • a timeline of the evaluation process
CAFOD_PPA evaluation report_101130.docx 24
5. Timetable
It is envisaged that the evaluation will involve 30 days consultancy including preparation, surveys, interviews, field research, production and presentation of the final report. The 30 days may be shared between more than one consultant with one clearly designated leader. The exact timeframe will be finalised with the consultant(s) when the contract is agreed.
The successful bidder(s) will be expected to produce an initial draft report for discussion with CAFOD by no later than 31 October 2010.
The successful bidder(s) will be expected to produce a final report, agreed with CAFOD, for submission to DFID. CAFOD will submit this finalised evaluation report to DFID, together with a 4 page management response, by no later than 30 November 2010.
6. Skills and Qualifications
The Evaluator(s) will be appointed by negotiated procedure: a minimum of three suppliers will be approached simultaneously in writing and invited to negotiate for the contract. Selection will be carried out by a bid committee consisting of CAFOD’s Programme Learning Manager, Monitoring and Evaluation Adviser and one further member of staff closely involved with the PPA.
Consultant(s) must be impartial and independent in the sense that they are not current employees although they may have previous experience of CAFOD.
The successful bidder(s) will be expected to have:
• A strong understanding and experience of evaluation methodology
• A comprehensive background in working with civil society organisations in the field of international development and advocacy
• Experience and understanding of partnership and faith-based organisations
• Understanding of schools and development education in the UK (desirable)
• Portuguese and/or Spanish (desirable)
CAFOD_PPA evaluation report_101130.docx 25
Annex 2: Persons consulted
Name Designation CAFOD Chris Bain Director James Steel Director of Finance and Services Andy Waites Head of Partner and Programme Support Section Robert Sweatman Interim Head of ATLE Ivan Kent Programme Monitoring and Evaluation Adviser James Marchant Programme Learning Manager Ann Smith HIV Corporate Strategist Harriet Jones HIV Knowledge Management Coordinator Matthew Carter Head of Humanitarian Department Jennifer Hadley Emergency Response Officer ATLE Laura Donkin Emergency Officer, Programme Development and Funding Gwen Barry Head of Public Policy Janet Gunter Senior Advocacy Accompanier Anne Lindsay Public Policy Adviser, Private Sector Lucy Hurn Campaigns Manager, Climate Change Jo Kitterick Head of Education, Communities and Supporters Division Bridget Fenwick Schools Programme Team Leader Ann Wilson Schools Outreach Coordinator (North) Maurice McPartlan Head of International Programmes, Africa Zara Benosa Action for Better Governance Programme Manager Catherine Ogolla Programme Manager, HIV and AIDS, East Africa Désiré Tchato Regional Representative, Great Lakes Joanna Heil Program Development and Funding Officer, Great lakes Nicolas Programme Manager, DRC Ntemuanengu Lambert Biyempo Action for Better Governance Project Officer, DRC Michel Monginda Humanitarian/Early Recovery Coordinator, DRC Clare Dixon Head of Latin America and Caribbean Sarah Smith-Pearse Communications Officer, Latin America and the Caribbean Barbara Davies Programme Officer, Andes Cecilia Iorio Programme Manager, Brazil/Southern Cone Lorna Fielding Programme Officer, Brazil/Southern Cone Montserrat Programme Officer, Central America Fernandez Julia Metcalfe Programme Manager, Cambodia, Philippines, China and Burma
Partner representatives Margaret Mwaniki Regional Representative, Caritas Africa, CAFOD Trustee Jacques Dinan Executive Secretary, Caritas Africa Luciane Udovic Coordinator, Grito des Excluidos, Brazil Padre Bernardo Jubileu Sul, Brazil Lestienne Staff, Rainbow Diocese of Moshi, Kenya Centre Christophe Katembo Liaison Officer, Caritas-Goma, DRC Abbé Jean-Claude Deputy Director, Caritas-Congo, DRC Mbu-Mubel Sr Marie-Bernard Coordinator, Diocesan Justice and Peace Commission, DRC Alima Henri Muhiya Secretary, Episcopal Commission on Natural Resources, DRC
CAFOD_PPA evaluation report_101130.docx 26
Musabate Sylvestre Imbese Secretary, Episcopal Commission for Justice and Peace, DRC Abbé Come Director, Caritas Dungu, DRC Mbolingaba Sr Claire Nkunzi Director, Diocesan Health Office, Dungu, DRC Abbé Oswald Director, Caritas Goma, DRC Félicien Nzitatira Coordinator Justice and Peace Commission, Caritas Goma, DRC Roger Ndagije Coordinator Diocesan Health Office, Caritas Goma, DRC Célestin Twisenge Coordinator, Diocesan Development Office, Caritas Goma, DRC Eddy Yamwenziyo Emergency Coordinator, Caritas Goma,DRC Abbé Romain Ngulu Curate, Rugari Parish, DRC Abbé Salomon Vicar, Rugari Parish, DRC Habimana Parishioners Rugari Parish, DRC Abbé Alfred Ndrabu Diocesan Justice and Peace Commission, Bunia, DRC Buju Jean-Paul Lonema Assistant, Bunia Justice and Peace Commission, DRC Richard Mugisa Animator, Bunia Justice and Peace Commission, DRC François Bura Animator, Bunia Justice and Peace Commission, DRC Dhengo Jimmy Munguriek Secretary. Cadre de Concertation, DRC Sylvie Buma Local Committee for Participatory Governance, DRC Trésor Kafwaki Local Committee for Participatory Governance, DRC Charles Djawiyambe Local Committee for Participatory Governance, DRC Danks Mandama Local Committee for Participatory Governance, DRC Adjundro Jacques Avetse Local Committee for Participatory Governance, DRC Rudjoko Christian Vangazi Local Committee for Participatory Governance, DRC
External stakeholders Jo Bezzano Former CAFOD Interfaith Adviser Affan Cheema Partners and Institutions Manager, Islamic Relief Worldwide Bernard Stuart Assistant Director of Education, Diocese of Salford Peter Ward Education Officer, Westminster Diocese; member of the Executive Board of the National Board of Religious Inspectors and Advisers Rev. David Harrison Chaplain, All Hallows Catholic College, Macclesfield Fiona Turner Year 13 student, All Hallows Catholic College, Macclesfield Ashley Geldard Year 13 student, All Hallows Catholic College, Macclesfield James Cocker Year 13 student, All Hallows Catholic College, Macclesfield Christine Beswick Head Teacher, Our Lady’s Catholic Primary School, Stockport Anna Core Deputy Head, Our Lady’s Catholic Primary School, Stockport Sr. Eleanor RE Coordinator, Our Lady’s Catholic Primary School, Stockport O’Gorman Astrid Schwietering Campaign and Advocacy Officer, CIDSE James Whitehead International Programmes Director, Progressio Guilherme Delgado Technical Consultant, Economic Justice Programme Brazil Alain–Yvon Kabeya Former CAFOD Advocacy Officer, DRC Peter McNichol CONCERN Director, DRC Professor Paul Director, Newcastle Institute for Research on Sustainability, Younger Newcastle University
CAFOD_PPA evaluation report_101130.docx 27
Annex 3: Selected documents
This evaluation has had access to a very large number of monitoring reports and other written materials covering all of CAFOD’s work under the PPA. Only the main documents are listed here.
CAFOD PPA Report 2009
2008-2009 Self-assessment
List of supporting documentation: Strategic objective 1: 1.1a ACEAC Bishops’ statement on governance 1.2a ABColombia policy document : Fit For Purpose 1.2b CIDSE Learning document on Extractives campaign (partner work reported in previous years): http://www.cidse.org/uploadedFiles/Publications/Publication_repository/EPLA%20analysis%2 0final%20ENG(2).pdf?n=72 YouTube link - http://www.youtube.com/watch?v=UsT6ye5-G7o 1.4a CIDSE Policy position: http://www.cidse.org/uploadedFiles/Areas_of_work/Climate_Change/cidse_policy_paper_cli mate_justice_dec08_EN.pdf 1.4b Online international action now live: http://www.cidse.org/Area_of_work/?id=84 1.4c Caritas Internationalis support the campaign: http://www.caritas.org/activities/climate_change/action.html 1.4d CAFOD to launch Climate Justice campaign at 12 events from March 12 across England and Wales: www.cafod.org.uk/climatejustice 1.4e Catholic Bishops sign statements to support campaign: http://www.cafod.org.uk/news/climate-justice-2008-12-08 1.4f Bishops’ statement
Strategic objective 2: 2.2a Steps to conflict transformation in Jos Visits to DFID Abuja to discuss cooperation on health in Northern Nigeria on 22nd Feb 2009 and to discuss the Jos crisis specifically on 23rd Feb 2009.
Strategic objective 3: 3.1a Engagement Framework (Voice and Accountability Monitoring Tool) 3.1b Letter to CAFOD partners explaining the Engagement Framework 3.1c Partner baseline assessment using the Engagement Framework 3.2 a Case study: CIPCA Bolivia , with pictures 3.2 b Case study: Ethiopia 3.2 c Guidance notes for donors on working with NGO networks in Ethiopia 3.2 d Honduras advocacy on gold mining March 2009
CAFOD_PPA evaluation report_101130.docx 28
Strategic objective 4: 4.1a Data for participating programmes disaggregated by gender and age 4.1b Mapping tool template 4.1c Completed mapping tools from Ethiopia, Kenya and Mozambique (3 docs attached) 4.1d Photos from workshops in Mozambique and Kenya (4 photos attached) 4.1e Video clip from Kenya 4.2a English translation of Mexican Bishops’ pastoral letter “Putting Our Faith into Action for A Dignified Life for our Brothers and Sisters Living with HIV” (CEPS...) 4.3a Case Study – Faith based organisations and HIV in Nicaragua 4.4 a CAFOD HIV mainstreaming toolkit http://tinyurl.com/4d3lvg 4.4 b CAFOD poster presented at the International AIDS Conference, México, August 2008 http://www.aids2008.org/Pag/AbstractPlus/AbsDoc_7258_1.pdf 4.4.c Handouts for same event 4.4d Staff report – CAFOD’s HIV mainstreaming work in the Philippines
Strategic objective 5: 5.1a Internal CAFOD report on student activity since the liveit! Conference 5.1b Programme of day of Liveit conference 5.1c Video clip from event on YouTube http://www.youtube.com/watch?v=1pWR3opd7f8
CAFOD PPA Report 2010
2009-2010 Self-assessment
List of supporting documentation:
Part A A Explanation of Church Structures
Strategic objective 1: 1.1 Justice Magazine article – governance in Kenya 1.2a Evaluation of the Colombia IDP programme 1.2b Lessons for a disaster risk reduction regional programme – Luis Felipe Ulloa 1.3a Mid-term review of the Eritrea Nutritional Support programme
Link to the statement Southern partners prepared in Malawi http://www.cidse.org/uploadedFiles/Publications/Publication_repository/Climate%20Justice% 20Statement%20CIDSE_CI%20Southern%20Partners_June%202009.pdf Policy reports Reducing Vulnerability, Enhancing Resilience: The importance of Adaptation Technologies for the post-2012 Climate Agreement - http://www.cidse.org/publications/publications.aspx?id=&eid=1213 Launched in Bonn, June 2009 with press conference and panel discussion – launch invite: http://www.caritas.org/includes/pdf/bonn09paneldiscussion.pdf Link to CIDSE webpage on Climate Change including the bell ringing event http://www.cidse.org/Area_of_work/Climate_change/?id=1472
Strategic objective 2:
CAFOD_PPA evaluation report_101130.docx 29
Presentations from the African perspectives on Christian/Muslim relations conference http://www.cafod.org.uk/about-us/what-we-do/conflict-and-peace/faith-in-africa http://www.cafod.org.uk/news/nigeria-religious-conflict
Strategic objective 3: 3.1a Edited Voice and Accountability tool 3.1b Monitoring results table 3.2 Case studies from Honduras, Brazil, and Sierra Leone
Strategic objective 4: 4.3: Approaches to HIV Prevention: Key findings, from a review of six Church-based programmes 4.4: Application of CAFOD’s mainstreaming tool in Colombia and Kenya
Strategic objective 5: 5.1 Research on CAFOD’s post-16 General RE Resources Management Summary, carried out by David Burton Associations, November 2009
‘Leading for a Change’ conference http://www.youtube.com/watch?v=ok6w0_53_1U and www.cafod.org.uk/secondary/post-16)
5.2: Written report on the INSET with technology teachers
Selected additional reports and evaluations
Brehony, E. 2010. Report on Review of MMM Counselling and Social Services Centre, Addis Ababa, Ethiopia.
CAFOD. CAFOD 2010. Building communities for change.
CAFOD. June 2010. Just One World.
CEREAL. November 2009. Labour Rights in a Time of Crisis: Third Report on working conditions in the Mexican electronics industry.
Charities Aid Foundation. 2007. Charity Trends.
Disasters Emergency Committee. No date. DEC Accountability Framework. Member Agency Assessment 2009/10. CAFOD.
Humanitarian Accountability Partnership. April 2009. Audit Report of HAP 2007 Standard.
Islamic Relief Worldwide. 2009. End of Project report. Emergency Assistance to Displaced Christian Community in Iraq.
Islamic Relief Worldwide. 2009. Final Report. Ader Livestock Recovery Project.
Islamic Relief Worldwide. 2009. Final Report. Emergency Intervention in Gaza Strip.
Islamic Relief Worldwide. 2009. Project Completion Report. Emergency Transitional Shelter Project for Earthquake Affectees. Ziarat (Balochistan-Pakistan).
Islamic Relief Worldwide. 2009. Project Completion Report. Support Community Resilience in Emergency Areas of Mardan (SCREAM)).
CAFOD_PPA evaluation report_101130.docx 30
Islamic Relief Worldwide. 2010. Final Report. Child and Family Psychosocial Counselling Centre.
LAPPA. 2010. LAPPA Regional Meeting Report. Social Policies and Civil Society in Latin America.
Nana, R and Rudolf van den Boogaard. July 2008. Final Report. Evaluation of the DFID funded CAFOD Rapid Reaction Fund. Eastern DRC. CAFOD/MDF-Afrique Centrale.
National Audit Office. http://www.nao.org.uk/images/commission_flow_chart_large.jpg showing Value for Money flow chart
PARC/IOC. November 2005. Evaluation of the DFID/CAFOD PPA.
Ruiz Mendieta, B. March 2010. Systematized Learnings and Experiences of the Disaster Risk Reduction Programme: A Regional Glance 2008-2010. El Salvador, Honduras and Nicaragua.
Shonko, N. et al. September 2010. Eritrea Community Based Nutrition Project. Project Evaluation Report. July 2009-April 2010.
Van der Wijk, J. et al. January 2010. Evaluation of the DEC-funded CAFOD Health and WASH project in the DRC. CAFOD/MDF-Afrique Centrale.
CAFOD_PPA evaluation report_101130.docx 31
Annex 4: Timeline for the Evaluation
Date Activity August 25th 2010 First Steering Committee meeting with UK consultant
September 1st 2010 Formal start date for the evaluation
September 10th 2010 Inception report submitted by UK consultant
September 13th 2010 Steering Committee meeting to review inception report
October 4th 2010 DRC consultant begins fieldwork
October 12th 2010 Steering Committee meeting with UK consultant
October 21st 2010 French version of report from DRC fieldwork submitted by DRC consultant
October 29th 2010 English version of report from DRC fieldwork submitted by DRC consultant
November 1st 2010 Draft evaluation report submitted by UK consultant
November 8th 2010 Presentation and discussion of draft evaluation report with CAFOD staff
November 16th 2010 CAFOD comments on draft report submitted to UK consultant
November 19th 2010 Final evaluation report submitted to CAFOD by UK consultant
CAFOD_PPA evaluation report_101130.docx 32
Annex 5: CAFOD PPA Performance Framework
CAFOD_PPA evaluation report_101130.docx 33
CAFOD PPA Revised Logframe 2008-2011
Niche: CAFOD is the international aid agency of the Catholic Church in England and Wales. It establishes long-term relationships with people, communities, and organisations, and particularly organisations of the Catholic Church, to support and strengthen their work to relieve suffering, bring about sustainable human development and social justice. The Catholic Church is an important actor in many developing countries, having an extensive network that extends into areas where other agencies, including national governments, find it hard to work. Its constituency continues to grow, particularly in sub-Saharan Africa, and for many poor people the Church represents not just a source of spiritual support but also a key provider of basic services, their first point of refuge and support in emergencies and, through its hierarchy, their main voice with governments and policy makers.
CAFOD supports both Church and other partners in the south through a variety of means including direct funding, technical and moral support, facilitating networking and collaboration, and joint advocacy. Learning and evidence gathered from the international programme is used to inform development awareness and advocacy work in England and Wales.
Purpose: CAFOD will develop its relationships and collaboration with key Church partners in order to strengthen their actions targeted at enabling individuals and communities to participate more fully in decision making processes (in the south and in the north), to reduce their vulnerability to disasters, conflict and the effects of HIV/AIDS, and to develop sustainable livelihoods.
CAFOD_PPA evaluation report_101130.docx 34
Strategic Objectives Indicators Means of Verification Assumptions/Risks 1. Drawing on its faith identity 1.1 At least two examples per Case-studies showing change CAFOD can only influence and CAFOD will access, support, year of the Catholic Church against 2009 baseline studies and support; it cannot control the mobilise and influence the Catholic demonstrating increased capacity assessments of Church advocacy actions taken by partners. Church to be more effective in its to influence key governance work in Central, East and efforts to reduce poverty and processes in Africa which result Southern Africa. injustice. in tangible benefits for the poor and disadvantaged
1.2 By 2011 the Catholic Reporting from partners and Church in Latin America will have programme staff captured supported over 330 communities through CAFOD’s PCM system; to increase their resilience to independent evaluations. conflict and natural disasters.
1.2.1 People in 40 communities affected by the armed conflict in Colombia are able to rebuild a sustainable livelihood
1.2.2 In Central America 292 communities at risk of environmental disaster are able to incorporate DRR techniques into their livelihoods work.
1.3 At least two examples per Independent evaluations of Capacity of national agencies year of humanitarian responses emergency response programmes; is variable. being led by local Caritas partners feedback from other agencies
25 CIDSE – Coopération Internationale pour la Développement et la Solidarité – an alliance of 15 Catholic development agencies in Europe and N. America
CAFOD_PPA evaluation report_101130.docx 35
Strategic Objectives Indicators Means of Verification Assumptions/Risks (supported by CAFOD) involved; one case study per year. demonstrating the use of and adherence to SPHERE standards Number of Catholics engaged in 1.4 By 2010, CIDSE25 taking action on the campaign Assumes continued campaigning on climate change led across Europe and North engagement by Catholic by CAFOD, will have mobilised America, statements from agencies in CIDSE/CI over 100,000 individual Catholics Bishop's conferences; monitored campaign and eight national Catholic by CIDSE hierarchies to call on their governments for an equitable post-Kyoto treaty on Climate Change.
2. CAFOD will develop ecumenical 2.1 Memorandum of Case studies and reports MoU between CAFOD and and inter-faith alliances with Understanding with Islamic Relief produced by the two agencies Islamic Relief due for renewal organisations and networks in results in at least three new joint (2008) order to increase the reach and programme initiatives by 2011 effectiveness of joint that enable one or other of the developmental and humanitarian agencies to increase the reach or responses. effectiveness of their work.
2.2 By 2011 at least two Reports from partners and Conflict between communities examples of Catholic Church programme staff as captured mean that there is a lot of leaders in Africa engaging in through CAFOD’s PCM system antipathy towards interfaith interfaith initiatives that bring working in many parts of communities together to work on Africa programmes promoting development and/or reconciliation.
CAFOD_PPA evaluation report_101130.docx 36
Strategic Objectives Indicators Means of Verification Assumptions/Risks 3. Southern civil society 3.1 By 2011 at least eight Engagement Framework26 used The institutional framework of organisations, including agencies of partners from the countries with partners on an annual the state will allow them to the Catholic Church, are better listed, including at least four basis. respond to the voice of able to represent constituent from Latin America will have citizens communities in advocacy work, and moved up at least one level on the are more effective in engaging Engagement Framework (CAFOD’s with and influencing the policies advocacy monitoring tool) and practices of government and demonstrating more effective business that affect them. engagement with policy makers or key corporate actors.
3.2 At least four examples of Partner reports or public Companies responsive to specific changes made by statements of government or local pressure. companies or government (at the companies. appropriate level) in response to citizen voices following partner Stories of Change collected work. through CAFOD’s PCM system, demonstrating the role of CAFOD’s partners in the change.
Countries: Bolivia, Brazil, Mexico Cambodia, East Timor, Ethiopia, , Honduras, Mano River (Sierra Leone/Liberia), DRC, Mozambique
26 The Engagement Framework was submitted to DFID as an appendix to the 2008/9 PPA report
CAFOD_PPA evaluation report_101130.docx 37
Strategic Objectives Indicators Means of Verification Assumptions/Risks 4. CAFOD will support Church 4.1a Holistic care and mitigation 4.1a The range of services CAFOD support is one factor partners to develop improved (C&M) responses are provided provided directly or through that will influence the work of responses to HIV that both reduce directly or through referral by referrals under 4 domains its partners, and the lives of vulnerability to infection and Church partners in 9 countries, (health, psycho-social, their beneficiaries. improve the quality of life for serving 100,000 women, men and legal/human rights and people infected and affected by children affected by HIV economic/livelihoods security) The use of condoms is HIV. identified using CAFOD’s C&M forbidden by the Catholic mapping tool Church and the stance of individual Church leaders can 4.1b Men, women and children 4.1b Use of CAFOD’s Quality of limit, or enhance, the space accessing these services report Life (batteries) monitoring tool for open discussion on the improved quality of life. with a sample of beneficiaries. subject within communities.
4.2 At least four examples (over 4.2 Case studies document the 3 years) of Church partners initiatives undertaken, the role of directly contributing to policy CAFOD’s partners, and evidence changes at local or national levels of change in either stigma to reduce HIV-related reduction or accessibility of stigmatisation and/or enable services. access to HIV-related services in hard-to-reach communities
4.3 Seven Church programmes 4.3 CAFOD’s comprehensive demonstrate more comprehensive prevention monitoring tool approaches (combining initiatives (modelled on NGO Code of Good to reduce risk, decrease Practice) documents evidence of vulnerability and mitigate progress towards this from staff impact)to HIV prevention interviews, programme materials and activities review, and feedback from programme clients.
CAFOD_PPA evaluation report_101130.docx 38
Strategic Objectives Indicators Means of Verification Assumptions/Risks
4.4 At least three examples of 4.4 Use of CAFOD HIV development/ emergency mainstreaming tools and tracking response programmes through Programme Cycle demonstrating evidence of Management system provides adaptation to improve evidence of adjustments made to effectiveness in contexts also programme practices and their affected by HIV. effectiveness.
In 2010/11 CAFOD proposes commissioning an independent evaluation of the impact of its HIV work.
CAFOD_PPA evaluation report_101130.docx 39
Strategic Objectives Indicators Means of Verification Assumptions/Risks 5. CAFOD will increase 5.1 By 2011 sixth form Records of the number of Catholic schools remain development awareness in England students in 50 schools will have students participating receptive to CAFOD and Wales working through the taken action for global justice as Monitoring by diocesan offices participation in their Catholic schools’ network with a a result of CAFOD material and activities. focus on global social justice and volunteer visits the MDGs. 5.2 By 2011 number of schools Reports from school volunteers Other issues, including change attending INSET to support in government policy, do not global social justice in their have greater influence on the school life and curriculum will curriculum increase from 140 to at least 280 in CAFOD’s four focus dioceses.
LAC: By 2011 number of schools Reports from Education Advisors. attending in-service training to support global social justice with a focus on Latin America in their school life and curriculum will reach at least 100 in CAFOD’s four focus dioceses. Research findings on children’s 5.4 Changes in children’s attitudes and understanding of Three years is sufficient time attitudes and knowledge over a global issues commissioned by to record changes in three-year period CAFOD in collaboration with the attitudes. Catholic Education Service (Baseline 2008/9) Continued commitment from CES to collaboration on this work.
CAFOD_PPA evaluation report_101130.docx 40
Annex 6: Results against PPA indicators – November 2010
Strategic objective 1: Drawing on its faith identity CAFOD will access, support, mobilise and influence the Catholic Church to be more effective in its efforts to reduce poverty and injustice.27 Indicator Results 1.1 At least two examples per year of the Law on Decentralization in DRC (see main report Catholic Church demonstrating increased and Annex 7). capacity to influence key governance processes in Africa which result in tangible benefits for the Mobilising citizen action on good governance in poor and disadvantaged Marsabit, Northern Kenya (see main report).
1.2 By 2011 the Catholic Church in Latin America will have supported over 330 communities to increase their resilience to conflict and natural disasters. 1.2.1 People in 40 communities affected by the In last 3 years, 254 displaced and vulnerable armed conflict in Colombia are able to rebuild a families have set up 25 businesses. 20 community sustainable livelihood organisations have been supported and 10 community action plans have been developed. Most businesses are run by women.
Significant progress made in terms of confidence- building at the level of individuals and in working jointly with local and IDP communities, but progress in developing sustainable livelihoods has been slower than anticipated.
1.2.2 In Central America 292 communities at Partners in El Salvador, Honduras and Nicaragua risk of environmental disaster are able to have learned to incorporate techniques into their incorporate Disaster Risk Reduction techniques work on sustainable livelihoods that pre-empt and into their livelihoods work. mitigate the effects of natural disasters. Work is ongoing to elaborate emergency plans and protocols, to strengthen partner capacities to analyze needs, to work to Sphere standards; and to promote coordination and learning between partners and other networks. 1.3 At least two examples per year of Caritas, eastern DRC emergency response (see humanitarian responses being led by local main report and Annex 7). Caritas partners (supported by CAFOD) demonstrating the use of and adherence to Eritrean Catholic Secretariat nutrition programme SPHERE standards (see main report). 1.4 By 2010, CIDSE campaigning on climate For the Copenhagen Summit, 158,000 change led by CAFOD, will have mobilised over campaigners in Europe were mobilised by CIDSE 100,000 individual Catholics and eight national agencies to sign up to the demands of the climate Catholic hierarchies to call on their governments change campaign, of which 55,000 were mobilised for an equitable post-Kyoto treaty on Climate in the UK by CAFOD Change. Individual Bishops in Europe issued a joint letter calling on EU leaders ‘to be morally courageous at this crucial time within the climate change talks in Copenhagen’ but it was not possible to engage Bishops’ Conferences as planned.
Strategic Objective 2: CAFOD will develop ecumenical and inter-faith alliances with organisations and networks in order to increase the reach and effectiveness of joint
27 Indicators for this strategic objective were revised in 2010 and so the base year for examples is 2009-2010. Several comparable examples were provided in the 2008-2009 self-assessment report.
CAFOD_PPA evaluation report_101130.docx 41
developmental and humanitarian responses. Indicator Main results 2.1 Memorandum of Understanding with Support to Islamic Relief projects in reporting Islamic Relief results in at least three new joint period: programme initiatives by 2011 that enable one Ader Livestock Recovery Project (Ethiopia) June or other of the agencies to increase the reach or 2008-January 2009: 5,572 pastoral and agro- effectiveness of their work. pastoral households (33,432 individuals) assisted to recover from drought through provision of animal health services.
Child and Family Psycho-social Counselling Centre in Gaza (OPT) April 2009-May 2010: more than 600 children and 200 families provided with home visits and individual, family and group counselling.
£60,000 provided to support 1,303 internally displaced households (9,121 individuals; 60% female and 40% male) with food and nutrition support in Gaza January-March 2009.
Emergency Transitional Shelter Project for earthquake affected populations in Balochistan (Pakistan) January-March 2009: 70 households (560 individuals; 374 male and 186 female)
2.2 By 2011 at least two examples of Catholic CAFOD organised a conference in the UK for Church leaders in Africa engaging in interfaith African Catholic leaders to discuss The Church and initiatives that bring communities together to Christian/Muslim relations in Africa. This followed work on programmes promoting development the October 2009 Synod for Africa on the theme of and/or reconciliation. “the Church in service to Reconciliation, Justice and Peace”. CAFOD funded a follow-up meeting to the synod in Mozambique in May 2010 where representatives of Caritas Africa and of African Bishops’ Conferences met to discuss how to put the decisions of the synod into practice.
CAFOD provides funding to support the work of the Archbishop of Jos (Nigeria) whose Archdiocese has suffered regular outbreaks of religious violence. The support includes funding for a priest to study for an MA in Peace Studies with the intention that on return he will assist in training priests and religious in inter-religious dialogue.
CAFOD provides funding to the Ethiopia Interfaith Forum for Development Dialogue and Action, where the Bishop of Adigrat has developed strong links with Muslim and Orthodox leaders, including in tackling stigma against people living with HIV and AIDS. Strategic Objective 3: Southern civil society organisations, including agencies of the Catholic Church, are better able to represent constituent communities in advocacy work, and are more effective in engaging with and influencing the policies and practices of government and business that affect them. 3.1 By 2011 at least eight partners from The following partners moved up at least one level the countries listed, including at least four on the Engagement Framework: CIPCA and CIPE from Latin America will have moved up at (Bolivia); Economic Justice Programme (Brazil); least one level on the Engagement Banteay Srei, CCSP, DPA and NGO Forum Framework (CAFOD’s advocacy monitoring (Camobdia); Cadre de Concertation (DRC); Luta tool) demonstrating more effective Hamutuck and MDI (East Timor); Caritas
CAFOD_PPA evaluation report_101130.docx 42
engagement with policy makers or key Tegucigalpa (Honduras); CJPS and Don Bosco corporate actors. (Liberia); CEREAL (Mexico); UNAC (Mozambique); and NMJD (Sierra Leone).
3.2 At least four examples of specific Brazil: delay in tax reform (see main report) changes made by companies or government (at Honduras: initiation of legal proceedings against an the appropriate level) in response to citizen international mining company (see main report). voices following partner work. Sierra Leone: a youth group was fully involved in implementing a road construction project after successfully advocating local authorities. Strategic Objective 4: CAFOD will support Church partners to develop improved responses to HIV that both reduce vulnerability to infection and improve the quality of life for people infected and affected by HIV. 4.1a Holistic care and mitigation (C&M) Support provided to 38 projects with C&M responses are provided directly or through component in 13 countries. Mapping tool referral by Church partners in 9 countries, introduced to partners in 8 countries serving more serving 100,000 women, men and children than 118,000 people. affected by HIV.
4.1b Men, women and children accessing these QoL monitoring tool has been used with self- services report improved quality of life. selected clients in 8 countries. Qualitative reporting.
4.2 At least four examples (over 3 years) of Reinstatement of HIV positive soldier in armed Church partners directly contributing to policy forces as a result of action by Catholic FBO and changes at local or national levels to reduce others in Mexico set a legal precedent. HIV-related stigmatisation and/or enable access to HIV-related services in hard-to-reach Home-based care NGO successfully lobbied communities. Nigerian Federal Medical Centre to provide ART in rural areas. A second unsuccessful HBC NGO obtained ART through a government-approved Catholic hospital.
4.3 Seven Church programmes demonstrate Assessed through administering protocol with 6 more comprehensive approaches (combining partners in 5 countries. Mixed results provide initiatives to reduce risk, decrease vulnerability indications of where further investment by CAFOD and mitigate impact)to HIV prevention required e.g. lobbying bishops for more inclusive approaches.
4.4 At least three examples of development/ Use of CAFOD’s HIV mainstreaming tools show emergency response programmes evidence of adaptation in Caritas Colombia IDP demonstrating evidence of adaptation to programme and in responses to displacement in improve effectiveness in contexts also affected Kenya following disputed presidential elections. by HIV. Strategic Objective 5: CAFOD will increase development awareness in England and Wales working through the Catholic schools’ network with a focus on global social justice and the MDGs. 5.1 By 2011 sixth form students in 50 schools CAFOD worked directly with 28 6th forms in will have taken action for global justice as a reporting year, making a total of 45 schools during result of CAFOD material and volunteer visits current PPA. Research showed that 68% of 65 surveyed schools had taken action as a result of CAFOD resources for post-16 education.
5.2 By 2011 number of schools attending INSET attended by teachers from 164 schools in INSET to support global social justice in their reporting year, making a total of 283 schools during school life and curriculum will increase from 140 current PPA. to at least 280 in CAFOD’s four focus dioceses.
5.3 LAC: By 2011 number of schools attending in-service training to support global social justice with a focus on Latin America in their
CAFOD_PPA evaluation report_101130.docx 43
school life and curriculum will reach at least 100 in CAFOD’s four focus dioceses.
5.4 Changes in children’s attitudes and Preliminary research results indicate that children knowledge over a three-year period have a well-developed sense of fairness as this applies to how countries as well as people should be treated; have some understanding of the global issues and some knowledge of CAFOD. Children exposed to CAFOD materials had better understanding than others.
CAFOD_PPA evaluation report_101130.docx 44
Annex 7: Case study of CAFOD’s PPA work in the Democratic Republic of Congo
The following case study is drawn from fieldwork and a report compiled by Georges Tshionza Mata.
1. CAFOD’s work in the Democratic Republic of Congo (DRC) originates in its response to the Rwanda crisis during the 1990s and is supported by work with ACEAC28 (the Central Africa Bishops’ Conference), which is an important regional actor in efforts to reduce conflict in the Great Lakes. CAFOD’s principal partners in the DRC are the Congo National Bishops’ Conference (CENCO), the national Justice and Peace and Caritas offices, and local Justice and Peace and Caritas offices in the dioceses of eastern DRC. CAFOD’s approach in the DRC has focused on establishing effective and well- coordinated national and diocesan structures after the damage caused to the Church, as to other national institutions, by years of conflict and neglect. This approach is consistent with principles of engagement with fragile states, adopted by the Organisation for Economic Cooperation and Development-Development Assistance Committee (OECD- DAC), which give first priority to helping to “build legitimate, effective and resilient state institutions”29. CAFOD’s support to its Church partners has focused on assisting them to become effective agents for change in the domains of mineral exploitation and its relation to the persistence of conflict in the region, of government decentralisation, and of humanitarian response.
Advocacy on mineral exploitation 2. CAFOD’s partners in the DRC that are active in advocacy on mineral exploitation are the national Justice and Peace Commission (CEJP), the ad hoc Episcopal Commission on Natural Resources (CERN), the diocesan Justice and Peace Committee (CDJP) in Bunia, and the Cadre de Concertation (CdC), an organisation established by the Church for the purpose of monitoring the activities of mining companies and of raising awareness in local communities about the social, economic and environmental effects of mining operations. The programme of work on mineral exploitation was initiated in 2006 through a workshop convened by CAFOD with other agencies, such as Human Rights Watch, to consider the impact that the exploitation of natural resources was having on communities in the eastern DRC.
3. The national Justice and Peace Commission began work on mineral exploitation by carrying out research to establish the link between the exploitation of natural resources and the persistence of conflict in the DRC. Other pieces of work undertaken by the CEJP and the CERN include the production of training materials for pastoral leaders, such as the Vade Mecum (handbook) for Congolese citizens on natural resource management published in 2009, and Terms of Reference for a review of mining contracts, which the government has used in the revision of many of the existing mining contracts. At local level, the Diocese of Bunia, which is situated in the heart of the gold-mining district of Ituri, has become a recognised source of expertise on the impact on local communities of the gold-mining exploration being carried out by the Anglo Gold Ashanti and Ashanti Goldfields Kilo (AGK) company and its subsidiaries.
28 Association des Conférences Episcopales de l’Afrique Centrale. Throughout this case study the French acronyms are used because these are the acronyms by which the relevant bodies are known and recognised in the DRC. 29 OECD-DAC. 2007. Principles for Good International Engagement in Fragile States and Situations. http://www.oecd.org/document/46/0,3343,en_2649_33693550_35233262_1_1_1_1,00.html
CAFOD_PPA evaluation report_101130.docx 45
4. One of the obstacles that agencies encounter in trying to monitor mineral exploitation is a lack of transparency about mining company operations, including about contracts made between the government and private companies and any agreements that have been reached with government concerning the companies’ obligations towards local communities. One of the successes that has been achieved in this regard as the result of lobbying by the CdC, the Bunia CDJP and other agencies is that a Tripartite Forum has been set up to act as a channel of communication between AGK and the people living in its areas of operation. Tension in the area is also reported to have significantly declined as a result of CdC’s intervention to improve AGK performance in some key policy areas, such as the company’s responsibilities towards redundant workers and its attitudes and practices towards those who engage in small-scale mining.
5. The CdC and the Bunia CDJP have also made progress in increasing people’s awareness of the link between the exploitation of the valuable mineral resources in Ituri and the prevalence of conflict in the area. People are slowly coming to understand that divisions between social groups are caused by economic factors and not by ethnic differences, and the level of mistrust between ethnic groups is beginning to decline, one indicator of this being that several cases of marriage between members of different ethnic groups have been reported. Participation in the Tripartite Forum appears to have increased people’s interest in engaging more actively in local government in general, for example in the forums and processes around decentralisation.
6. The CdC is now seen as a credible interlocutor by the mining companies and local and national governments, and as a credible spokesperson on mining issues by international NGOs and other sectors of Congolese civil society. This position has not been achieved without a struggle since the government and mining companies are wary of the involvement of civil society actors in monitoring their activities. At the same time as having to persuade the government and private sector that they can be trusted, the CdC and the diocese have also had to manage local people’s often unrealistic expectations about how much progress can be made in a short period of time vis-à-vis the mining companies.
7. CAFOD’s partners’ ability to represent local interests to the mining company is still somewhat dependent on CAFOD’s support, expressed by members of the Bunia CDJP as: "If CAFOD weren’t on our side, would Anglo Gold Kilo listen to what we have to say or accept our role as watch dogs? " CAFOD’s approach to advocacy with AGK has been commended by external commentators who noted that CAFOD’s report on AGK’s operations in the DRC was “one of the better examples of NGO commentary/pressure on the corporate world, a real improvement, in terms of clarity and quality, over some of the similar documents we’ve seen”30
8. Such visible support from CAFOD – and the international legitimacy that this brings – is extremely important to the work that CAFOD’s partners are doing on mining. One aspect of this is the networking that CAFOD facilitates with other international NGOs, such as Human Rights Watch. CAFOD has also linked its partners in the DRC with Catholic Relief Services and with the US Catholic Bishops Conference to campaign for a new bill that would improve mineral trade transparency, assist the DRC to improve mineral management and export, and impose punitive measures for commercial activities by US companies that support illegal armed groups.
9. CAFOD’s partners are planning to work further on educating and mobilising citizens about the importance of transparency in how revenues from the mining sector are collected and about the need to ensure that revenues are used in the interests of the
30 Measured Pressure. February 15 2010. http://www.co3.coop/blog/measured-pressure
CAFOD_PPA evaluation report_101130.docx 46
public. There is also work to be done to ensure that Anglo Gold Kilo honours its commitments to maintain the investments in health and education that it has made in Ituri, thanks to lobbying by CSOs.
Support to decentralisation 10. In 2007, the National Assembly in the DRC adopted a new law on decentralisation. The CEJP and the diocesan Justice and Peace Committees have been heavily involved in helping to ensure that decentralisation is rolled out locally in such a way as to bring tangible benefits to the population. In 2009, as part of CAFOD’s Action For Better Governance Programme31, the CEJP produced a guide to decentralisation for facilitators, which was used in training for pastoral agents. This guide was later adopted by the Ministry of Local Government as the model for its own national programme of civic education on decentralisation.
11. The main focus of the Church’s work on decentralisation has been at diocesan and parish levels. It has provided support to the Local Committees for Participatory Governance (CLGP), which are newly-established bodies that aim to engage citizens and local authorities in a constructive dialogue. The Diocese of Goma organised 200 CLGPs and the Diocese of Bunia organised 40. The CLGPs provide a platform where citizens are able to express their concerns about conditions in the locality and identify where action needs to be taken, for example to upgrade public infrastructure, to improve security, to increase access to health care, and to address the prevalence of sexual violence. Local authority representatives and citizens then identify what are the immediate priorities for action and they jointly develop strategies to put these priorities into action. In some places, joint committees have also been created to monitor how the decisions that have been taken are implemented.
12. The Justice and Peace Commission in Bukavu (pop. 2.34 million) conducted research which showed that corruption was the main reason that people were withholding tax payments. They ran an awareness campaign involving citizens, local officials and MPs which led to corrupt tax officials being replaced and participatory structures set up to scrutinise how tax revenues are collected and disbursed. This has resulted in a higher level of payments and a significant improvement in the provision of public infrastructure.
13. The CLGPs have contributed to building a stronger sense of community and civic engagement. They have proved to be helpful in breaking down the long-standing mistrust that has existed between the authorities and the public. They have facilitated dialogue between civil society representatives and local political and administrative authorities, and have encouraged a more constructive partnership to develop. They have also provided an effective setting within which disputes between different social groups can be settled, and have had significant success in bringing young people and women into initiatives concerned with peace-building and reconciliation.
Managing humanitarian response 14. CAFOD first provided assistance to Caritas-Goma in 1995 in response to the humanitarian crisis caused by the influx of more than one million Rwandan refugees into North Kivu. At this time, Caritas-Goma had little experience in managing emergency response programmes and, as well as providing immediate material and logistic support, CAFOD’s longer-term focus was on building Caritas-Goma’s technical capacity to enable it to perform to internationally recognised standards, independent of external technical support. Over the past 16 years Caritas-Goma has built itself from an 8-person team – initially ignored as a possible player in the response to the Rwandan genocide – to an
31 The Action For Better Governance Programme is supported by DFID’s Governance and Transparency Fund as well as CAFOD general funds
CAFOD_PPA evaluation report_101130.docx 47
organisation with over 100 staff that is recognised as one of the reference NGOs of Goma, not only by sister agencies in Caritas International but also by UN agencies and donors such as the EU, ADB, DFID, Belgium and Germany. Its detailed local knowledge and ability to access people living in conflict-affected areas are recognised as exceptional.
15. Caritas offices in Bukavu, Bunia and Dungu are similarly coming to be seen as important players in the management of humanitarian response programmes. Both Caritas-Goma and Caritas-Dungu have successfully participated in tenders in competition with international NGOs. Recently, Caritas-Goma won a competitive tender for $3 million to be disbursed within one year and Caritas-Dungu received a grant from USAID via CRS to work on food security.
16. During the PPA period, Caritas-Goma has responded to several major emergencies. In 2008, the agency distributed food rations and non-food items to more than 100,000 IDPs living in camps. In 2009, Caritas-Goma managed a health and water programme that kept 43 health facilities supplied with drugs and clean water. Caritas also rehabilitated and equipped the pharmaceutical laboratory in Goma. As a result of these interventions, there has been a significantly increased take-up of health services in the locality and in the number of women having safe deliveries in health facilities. During 2009, in the wake of atrocities committed by government soldiers and rebel forces, Caritas-Goma was the only agency that distributed food to 7,000 households in Walikale, a remote area with almost impassable roads that other agencies were unwilling to go to.
17. Similarly, Caritas-Dungu was the first agency to provide assistance in Haut Uélé to people who had been displaced by attacks and killings carried out by LRA and SPLA militia32. Initially, CAFOD was the only agency willing to provide funds to enable the relatively untried Caritas-Dungu to respond to this crisis. As a result of these interventions, Caritas-Dungu was then able to alert the international community to the humanitarian crisis that was taking place in Haut Uélé, which prompted a major response by UN agencies and international NGOs. Caritas-Dungu was able to initiate the international response through CAFOD’s membership of the international Caritas network.
18. Given the pervasiveness of sexual violence as a weapon of war in eastern DRC, special attention needs to be paid to the security of women and children displaced by conflict. To reduce the risk of sexual assaults, Caritas-Goma and Caritas-Dungu ensure that in the IDP camps that they manage, sanitation facilities for women, children and girls are segregated from those for men. Where relevant, a similar segregation is applied to the provision of shelter.
19. As a result of its recognised professionalism, Caritas is one of the few Congolese NGOs that participates in the various coordination mechanisms organised under UN auspices. Caritas is a member of the UN Inter-Agency Standing Committee (the main coordination mechanism for humanitarian agencies), attends weekly meetings chaired by OCHA, and participates in the different clusters that group together UN agencies, international NGOs and others around a sector or service provided during a humanitarian emergency. Caritas also participates in the needs assessments for the Rapid Response Mechanism (RRM), which identifies the priorities for humanitarian response. Caritas-Goma and Caritas Dungu are among the few national NGOs that have direct access to OCHA’s multi-donor Pooled Fund, through which these priorities are funded, and Caritas Goma also has access to DGECHO resources. UNICEF have recently notified Caritas-Goma that they no longer require full justification of their accounting for UNICEF programmes. Caritas-Goma
32 LRA : Lord’s Resistance Army (Uganda). SPLA : Sudan People’s Liberation Army (South Sudan).
CAFOD_PPA evaluation report_101130.docx 48
is the only Congolese NGO that is regularly invited to participate in meetings of the elite Heads of Mission Forum.
CAFOD’s approach to partnership and capacity building 20. Partners give CAFOD much of the credit for the fact that they are now much stronger institutions than before, in particular for the fact that: • the Church takes a lead on issues of conflict, the management of natural resources, and decentralisation. • a functioning national and diocesan Justice and Peace structure has replaced previous fragmented diocesan initiatives • diocesan Caritas offices are increasingly providing models of good practice in humanitarian assistance in areas of eastern DRC where other agencies are unwilling or unable to work.
21. Partners appreciate the decentralisation of CAFOD’s programme management, through the opening of a regional office in Kinshasa. This has significantly helped to strengthen the relationship between the parties. CAFOD has a better understanding of the contexts in which partners are working and is able to respond more rapidly and constructively to partners’ requests and proposals. For example, Caritas-Goma noted that CAFOD accepted their arguments that a grant originally provided for a single school in an area of IDP return would be sufficient for several schools if local people were involved in some of the construction and provided some of the materials. In the event, six schools were built with the grant. CAFOD's public endorsement of its partners’ advocacy in international arenas (particularly in relation to conflict, mineral exploitation and gender-based violence) was cited as further evidence of CAFOD’s recognition of its partners’ priorities.
22. A key element in CAFOD’s approach to partnership is the long-term accompaniment that it provides to individual partners, sometimes in the form of a dedicated accompanier and sometimes through close follow-up by programme staff in the CAFOD office. In either case, CAFOD gives priority to investing time (rather than money) in partners as a means of building partners’ management and technical capacity. The results of this are seen in the confidence that other donors develop in partners’ capacities and the success that partners then have in attracting funding from sources other than CAFOD. The examples given above of Caritas-Goma and Caritas-Dungu are cases in point.
23. Capacity building begins with an organisational diagnosis through which the partner’s strengths and weaknesses are analysed and priorities for support and attention are identified. A Memorandum of Understanding is signed between CAFOD and each partner, which sets out the obligations and expectations of each side. CAFOD funding has helped partners’ to acquire the staff and the equipment (including IT equipment) that they need to perform to a professional standard. As a first step, partners learn the procedures and acquire the tools that are applied in running an organisation and in managing projects. Partners have found CAFOD’s monitoring procedures helpful to them both in helping them to see whether progress is being made in projects and also in developing their own monitoring skills. CAFOD monitoring includes regular assessment visits to partners’ operating areas, and participatory mid-term and final evaluations of projects.
24. Financial accountability and control is extremely important, particularly for the Caritas agencies, which are handling high levels of emergency supplies and funding. CAFOD has provided its partners’ with appropriate accounting software and the training that goes with it. CAFOD has also trained partners in the procedures that they need to follow in order to fulfil the requirements of other donors. This has been particularly helpful to the Caritas agencies and, as noted above, Caritas-Goma’s own procedures are now accepted by UNICEF and DGECHO. The introduction of regular internal financial audits
CAFOD_PPA evaluation report_101130.docx 49
has been a significant factor both in increasing partners’ financial skills and in developing the culture of control and accountability.
25. Partners also learn through the regular exchange visits and the networking that CAFOD facilitates. CAFOD has encouraged its Church partners to network at the diocesan level (for example, exchange visits between Goma and Bunia) and with other actors in civil society (such as the CdC). Networking also happens at the national level and regionally through ACEAC, which brings together the Churches in Burundi, Rwanda and the DRC. Regional networking has been important for the three Churches that are all working to engage young people in peace-building efforts and for international advocacy for greater transparency and accountability from international mining companies.
26. Regional partners have gone through a joint training in advocacy, focussed on peace- building, the management of natural resources, and implementation of decentralisation. The initial training was provided at provincial church level and was then cascaded to diocesan and parish levels. The training aimed to help pastoral workers see the links between some economic issues (management of natural resources) and some political processes (decentralisation), and to understand the importance of managing natural resources so that they contribute both to poverty reduction and to good governance at local as well as at national levels. Joint training of pastoral agents from the three countries has helped to build greater trust and confidence between people coming from countries that have been in open conflict with one another and who are among the key opinion leaders in their localities. It has also helped to create a network within the Church working on democracy, peace-building and natural resource management. This network is particularly strong in the DRC.
27. The Congolese Church has long been recognised as one of the most important voices stressing the fundamental importance of good governance in all its dimensions. The support and training that Church organisations have received under CAFOD’s auspices have enabled them to overcome many of the challenges that they faced in ensuring that this advocacy could be translated into government policies and programmes that have a tangible and positive impact in people’s daily lives. Overall, CAFOD’s partners judge that CAFOD’s approach to partnership and to capacity building has helped to make them more professional and credible agents of change. In order for this positive result to be sustained over the long-term and independent of CAFOD support, CAFOD’s partners are looking for further support in developing their skills in monitoring and evaluation; results- based management; and in leadership, advocacy and communication.
CAFOD_PPA evaluation report_101130.docx 50
Annex 8: Engagement Framework
CAFOD_PPA evaluation report_101130.docx 51
Voice and Accountability Tool Engagement Framework
This tool can be used in different places and for different types of advocacy, and helps CAFOD and our partners to think about the wider context of their work as well as their achievements. The levels are cumulative, and require partner organisations to think through the reasons why they might be classed at one level or another by providing anecdotal evidence in each case. The tool does not prescribe which elements to emphasize in any given level. Organisations may find elements of their work in more than one box but are asked to best approximate their level on a yearly basis. Involvement in Advocacy Strategy Community & Involvement in Government Processes Development Constituency Building Corporate Structures
Benefits from Strategic Partnership Active recommended changes and Responsive Engagement 5 • Formal structures for consultation • Regular assessment of the dynamics • Citizens and constituents develop • Companies responding promptly are recognised, their decisions between stakeholders (including the work on their own issues or lead the to cases and proactively seeking have influence and can withstand dynamics between genders) and process of deciding what issues the options and information from political interference changes in context and their impact CSO will address communities or female / male Level • The recommendations of the CSOs on opportunities and risks different groups (such as women workers have not only led to changes in • Continuous strategic thinking or youth) are able to bring in their • Joint decision-making structures legislation but real benefits for men adapting activities to challenges perspectives and women and opportunities with milestones • Ownership of messages is strongly on the ground planned felt by all groups (including those of • Decision makers engage with CSOs • Clear recommendations for action are different genders / ages) in determining the policy agenda communicated to identified targets • Representative structures act in appropriate ways, based on good effectively reporting back and evidence and refined over time bringing issues from all the • Strategic engagement with others perspectives of their members through strong coalitions in (including men and women which the CSO has a leading role, and different ages) in regular involving a range of actors such as consultation academia/ private sector where appropriate
Influence Strategising Empowering Action
• Formal structures for consultation • Strong analysis of political context • Citizens / constituents play • Partial, corrective response
4 are established with an understanding of how an active and informed role in from companies to individual • Evidence that CSO input is different stakeholders interact and assessing issues and identifying instances of problems brought by influencing policy their relative power with a risk strategies to address them which communities/female and male analysis included reflect the needs of different groups workers • Government acts on the demands Level (such as women or youth),guided of CSOs through changes in rhetoric • Strong strategies identified that use • Some consultation mechanisms by the CSO used and changes in policy appropriate methods for different established with participation of targets • Structures are representative of women and men • Recommendations are well all groups (including women and • Companies willing to discuss targeted and based on good men and different age groups) and changes in procedures or polices evidence report back effectively • Leading role setting agendas for, or forming, relevant coalitions / networks Involvement in Advocacy Strategy Community & Involvement in Government Processes Development Constituency Building Corporate Structures
Transparency Analysing Consulting Acknowledgement
3 • Formal structures for consultation • Analysis of political context • Citizens / constituents (including • Companies acknowledge problems are established and stakeholders different genders and ages) for women and men but action • Evidence that CSO gender sensitive • Drafting of influencing strategies, participate in the process to is limited input is influencing policy including identification of targets identify and prioritise issues • Some ad hoc discussions or
Level • Government acts on the demands • Recommendations are based on • Formal gender-sensitive structures engagement with civil society of CSOs through changes in rhetoric solid evidence and research of representation have been / workers / communities used and changes in policy • Pro-active participation within developed or existing structures • Company starts to review policies networks or coalitions engaged which allow for the participation of all
Representation Engaging Awareness Raising Response
• Some research and discussions • CSO raises awareness of the systems • Response in the form of
2 • Existence of ad-hoc fora for consultation with CSOs undertaken on issues (with a which impact on citizens (recognising statements by companies • Consultation processes are gender lens) the different impacts on different but not through action transparent and government • Recommendations developed groups such as women and youth) • Some informal meetings with CSOs and opportunities for change Level provides subsequent feedback • Attendance at meetings and if pushed by external stakeholders • Decision makers are open to sharing with relevant networks • Citizens / constituents come • Companies share information challenge or organisations together in an organised way to on policies and practices explore issues and action or existing groups begin this discussion
Voice Exploring Speaking For Evidence Presented
• The CSOs internally establish policy • Issues identified and influencing • CSO bases its positions on its • CSOs gather evidence of problems 1 positions through discussion and recognised as a way to generate experience of work with citizens or for women and men of different debate change constituents ages and presents to companies • Beginning of conversation/ • Awareness of other organisations • The men and women that the or other responsible bodies
Level dialogue with government and networks working on similar CSO is claiming to represent (their issues constituency) are clearly identified