Speak the Language of Lusophone Africa Advances have been made in monitoring monitoring in made been have Advances Lusophone African countries. African Lusophone (Portuguese-speaking) countries, but the road road the but countries, (Portuguese-speaking) and evaluation activities and systems in in systems and activities evaluation and contributing to the strengthening of monitoring while challenges current of characterization brief a provides article This long. still is ahead and evaluation activities in Lusophone Lusophone in activities evaluation and eVALUation Matters Third Quarter 2019

Elsa de Morais Sarmento; Carla Félix; and Mariana Branco

Key Messages

❚ Lusophone countries are still lagging behind in Monitoring and Evaluation (M&E), as initiatives to establish a shared culture of evaluation practice, in particular in Africa, are still emerging. ❚ In several Lusophone Africa countries, policy decisions are mostly driven by values rather than outcomes, compounded with the lack of an evaluation culture and the set of skills needed to be!er use rigorous evidence. In parallel, methodologies for establishing guidelines for community-based research and evaluation, which should build on the culture of local communities, are a rare find. All of this adds to the lack of evaluation learning materials available in the Portuguese language. ❚ The greatest practical challenge remains the language, given the predominance of foreign evaluators with a minimal mastery of Portuguese and countries´ history and local contexts, not to mention local languages. Furthermore, trained native evaluators do not abound. ❚ There are no ready-made fast-tracking solutions for achieving M&E goals in support of the global development agenda and the Sustainable Development Goals (SDG). Lusophone African countries ought to be aware that it is essential to take the lead in this process and map their own needs, building on their own internal resources and on international initiatives, so as to strengthen their own national evaluation systems and capacities. Sound M&E systems and evaluation capacity development are urgently required in most African countries, and much more in those speaking Portuguese. A participatory diagnosis study of the issues at hand in these countries is required to understand which kind of capacity and resources should be developed, who the champions are and what steps are to be followed. Local stakeholders must play a central role in this exercise.

Evaluation as an essential are home to 267 million people located in component of public policy, four continents with a common language, programs and projects in a shared history, and cultural similarities. Lusophone countries Evaluation is known to be of critical he Community of Portuguese importance for policy- and decision- L a n g u a g e C o u n t r i e s making, underlying country strategies (Comunidade dos Países de and processes for accelerated growth Língua Portuguesa or CPLP) and inclusive sustainable development grew from seven countries worldwide. However, Lusophone countries T(Angola, Brazil, Cabo Verde, Guinea-Bissau, are still lagging behind. Many policy Mozambique, Portugal, and São Tomé decisions are still driven by values rather and Príncipe) to the current nine, a"er than outcomes, compounded not only with the self-determination of Timor-Leste the lack of an evaluation culture and the set in 2002 and the accession of Equatorial of skills needed for using rigorous evidence, Guinea in 2014. Portuguese is the sixth but also with a mismatch between political most natively spoken language in the timetables and the time-frame of evidence world and Portuguese-speaking countries producers. Initiatives to establish a

Speak the Language of Lusophone Africa 49 eVALUation Matters Third Quarter 2019

common culture of evaluation practice have to do with weak technical capacity, a among Portuguese-speaking countries and reactive institutional culture, the scarcity communities are still emerging. of resources, and the lack of political commitment (Figure 1).

"In Lusophone Africa, the demand Characterization of M&E for evaluation is still largely driven in Lusophone Africa

and conditioned by development The creation of evaluation associations partners, who still favor the comforts and communities of practice throughout of their own “language” and practice". Africa was pioneered by Anglophone and later by Francophone countries (e.g. the creation of the African Evaluation Association in 1999). This drive triggered In Lusophone Africa, the demand for the dissemination of evaluation knowledge evaluation is still largely driven and and learning across the continent, but conditioned by development partners, mostly in English and French languages. who still favor the comforts of their own In contrast, in Lusophone Africa, despite “language” and practice. High-quality some commendable initiatives on M&E, evaluations do not abound and are efforts have been more scattered as commissioned and managed more o"en compared to those made by these other by donors than by governments (with two linguistic groups. some exceptions for academic institutions). Consequently, with governments shying In Portuguese-speaking Africa, like in away from a more direct participation, most countries of the continent, M&E evaluations are less likely to be used in policy. was first introduced as a requirement of donors and international financing Evaluation demand has also traditionally organizations, which o"en offer support been weak and inconsistent, largely due through technical assistance and capacity to a deeply rooted historical culture of building in line with the inner workings patronage and fluid policies. Moreover, few of their systems and frameworks, in their countries have adopted all the elements of own languages. In general, governments a Results-Based Management (rbm) cycle have been passively dealing with these and performance-based budgeting, to requirements, leaving to international align expenditure with strategic goals and organizations and other funding agencies priorities. Underlying issues constraining the design and implementation of such the advancement of M&E efforts, with systems, which are in general abandoned varying degrees in different countries, after project conclusion due to

Figure 1: Underlying bottlenecks in M&E in Lusophone countries

Reactive Lack of Weak institutional Lack of political capacity culture resources commitment

50 Speak the Language of Lusophone Africa eVALUation Matters Third Quarter 2019

inadequate buy-in and ownership, lack Africa, are still to obtain all the resources of capacity to maintain them, and in some needed to engender a more prolific buy-in countries, sheer lack of interest. and diffusion of evaluation culture, along with the development of a professional In Lusophone countries, the field of class able to promote evaluation-related evaluation is still fragmented and research and learning in Portuguese. somehow neglected, with a shortage of strong professional organizations In Lusophone Africa, M&E efforts cannot dedicated to evaluation, absence of be decoupled from other known existing specialized evaluation journals, and frail structural challenges: local capacity to conduct advocacy and influence policy-making. Despite initiatives ❚ Lack of data demand from policymakers in Brazil and Portugal, where sectorial and lack of support granted to National and national evaluation societies along Statistical Offices (NSOs) for the with Non-Governmental Organizations adequate and timely production of data (NGOs) have emerged, African Lusophone for policymaking; countries lag much more behind, with only small and informal networks providing ❚ Weak statistical systems and lack of short-term training, without a long-term political support for well-coordinated perspective. A few cases of dedicated statistical systems; academic training in Portuguese (e.g., Masters in Brazil and a post-graduate ❚ Limited supply of statisticians and course in Portugal) are well established, but relevant expertise within statistical existing proactive communities of practice, systems, with NSOs using traditional such as CLEAR Brazil and Lusophone tools for data collection and analysis;

Table 1: Brief characterization of evaluation relevance, supply and demand in African Lusophone countries

Country Evaluation Evaluation Supply Evaluation Demand Relevance in the national context Cabo Verde Growing Mostly done by donors and Sporadic initiatives supported by recognition international development partners. foreign partners. Minor government The National Directorate of Planning involvement but the situation of the Ministry of Finance and Units has changed, as the joint work of Planning from Ministries, within with the UNDP is advancing and the context of the implementation consolidating previous M&E of a national M&E system. systems, with relevant training provided. Active civil society demand, with several evaluation- related activities organized. Mozambique Growing Mostly donor-based. Sporadic initiatives supported recognition by foreign partners. Angola Marginal Mostly donor-based. Marginal. São Tomé Marginal Almost inexistent, donor- Minor government involvement, and Princípe and NGO-based. demand from some donors. Guinea- Marginal Almost inexistent at national level. Marginal at the national level Bissau Performed by donors and executed Increasing demand, as Guinea- by international consultants, namely Bissau is a priority country for from Portugal, Italy or France. most of the international donors. There are no national professionals However, all the demand is or knowledge to perform related to projects fnal evaluation. evaluations at an acceptable level. No demand related to policies or at government level. Equatorial Marginal Almost inexistent at the national Minor government involvement, Guinea1 level, donor- and NGO-based. demand from some donors.

Speak the Language of Lusophone Africa 51 eVALUation Matters Third Quarter 2019

52 Speak the Language of Lusophone Africa eVALUation Matters Third Quarter 2019

❚ Issues of validity and reliability In Cabo Verde, evaluation is not only valued of data (in some cases, data released within the context of policy-making, but by international institutions does not also by academia and civil society. The coincide with data released by Member University of Cabo Verde (UniCV) has States for the same indicators); conducted several initiatives to promote M&E. The joint work of the Government ❚ Reluctance to openly share data of Cabo Verde with the United Nations in some countries, especially for Development Programme (UNDP) and the evaluation work. World Bank in consolidating its national M&E system is still ongoing, with relevant Table 1 highlights some of the most training provided to officials. Moreover, important characteristics of evaluation there is an informed and active civil society activities in Portuguese-speaking African demanding more accountability, which led countries. Common challenges these to the materialization of some initiatives. countries face relate to: the fragmentation Relevant joint events recently organized of the evaluation field, the lack of strong include one by the Ministry of Finance´s professional organizations dedicated to National Planning Department of Cabo evaluation, and the overall lack of capacity Verde (DNP), The World Bank Group to conduct advocacy and influence (Cabo Verde´s Office) and undp, under policy-making. the gLOCAL Evaluation Week (June 2019), which engaged the government, academia Despite the challenges pointed above, it and civil society actors. This was the is critical to highlight the key features of only M&E event organized in Lusophone each one of these countries (see Table 2 Africa during the gLOCAL Evaluation below). Week, whereas on the same occasion,

Table 2: Comparative advantages of each country

Cabo Verde Mozambique ❚ Good governance and democratic setting ❚ Wide presence and interest of donors ❚ Institutional maturity ❚ Variety of academia and civil society organizations ❚ Presence of donors interested in development and evaluation ❚ Well-developed statistical system, i.e. a ❚ RBM national M&E system (Orçamento por National M&E system (Sistema de Seguimento resultados) linked to the Plano Quinquenal e Avaliação) at the Ministry of Finance (MF/ do Governo (PQG) 2015-2019, strategic DNP has a planning and M&E unit) matrix for the PQG developed ❚ Civil society pledge for government ❚ Ministry of Economy and Planning: accountability and growing awareness National planning department with an for evaluation (e.g., GERA/FDI-CV) M&E unit and a national M&E plan ❚ Academia and civil society organizations ❚ Easy hiring of national researchers to support interested in development and evaluation data collection and evaluation efforts ❚ National Statistical Council ❚ Easy hiring of national researchers to support data collection and evaluation efforts Angola São Tomé and Princípe ❚ Academia and civil society organizations ❚ Presence of donors interested in development and evaluation ❚ Civil society organizations interested (e.g., CEIC - Centro de Estudos e Investigação in development and evaluation Científca, Universidade Católica de Angola) ❚ Growing awareness of evaluation Guinea-Bissau ❚ Priority country for most international donors ❚ Several major international NGOs operating in the country, and civil society organizations interested in development and evaluation ❚ Strong presence of donors such as the European Union, UN agencies and the Portuguese Development Agency

Speak the Language of Lusophone Africa 53 eVALUation Matters Third Quarter 2019

the Anglophone and Francophone and M&E. To tackle emerging challenges evaluation communities in Africa hosted such as financial stability, climate 14 and 16 events, respectively. change, and technology, reliable data is needed. Data collection and analysis In Mozambique, different entities within requirements have given rise to new tools the public sector have been conducting and approaches, which in turn created public policy evaluations, especially in new opportunities for finding sounder education, health, rural development, evidence to support policies. However, and other areas related to international the lack of large-scale revenues and a development cooperation. Mozambique persistently constrained tax base are presents a historical window of hampering Mozambique’s efforts to opportunity to garner support for adequately finance the development integrated public M&E systems, coupled of advanced M&E systems, its labor with impressive political will and force’s education and capacity building, understanding of statistical development particularly in line ministries.

54 Speak the Language of Lusophone Africa eVALUation Matters Third Quarter 2019

As for Angola, its levels of evaluation is lost in translation. Consequently, in supply and demand have been among Lusophone countries, the validity of the lowest in the region. In 2009, a research findings and evaluation results M&E Readiness Assessment and a is very o"en conditioned by the subjective National M&E system at the municipal, interpretation of outsiders, who do not provincial and central levels was master the language, history, life goals and conceived within the scope of the aspirations of specific communities. “2008-2010 Implementation Plan of the Decentralization and Local Governance Project of Angola”, but this was not followed up by the Government. "Together with the poor knowledge In São Tomé and Príncipe, Guinea-Bissau of local context, language is and Equatorial Guinea, there is little internal capacity to perform evaluations currently the main barrier to at an acceptable level. During 2019, building, disseminating and the clear Initiative delivered an M&E using credible and contextualized training to the Government of Guinea- evaluation in Lusophone Africa". Bissau, as part of a partnership between clear Brazil and Lusophone Africa and clear Francophone Africa. Another key point to raise is that the field of evaluation can draw on research Language as the first barrier methodologies which can be invasive to credible evaluation in or ill adapted to native communities, Lusophone Africa war-ravaged regions or remote areas. Conducting evaluative research might Together with the poor knowledge of be problematic in certain se!ings when local context, language is currently the dealing with communities who have main barrier to building, disseminating suffered from a history of conflict, and using credible and contextualized abuse and/or intrusive studies which evaluation in Lusophone Africa. might have resulted in cultural bias. Understanding the nuances of language, Credible evaluations are expected to deconstructing historical and cultural integrate all these complexities, whilst issues, such as the traumas of conflict responding to the assumptions imposed is a prerequisite for conducting sound by the funders and to those from self- research and obtaining reliable and determining communities. accurate findings. Within such a setting, evaluation as Very o"en, the mastery of the Portuguese a practice ought to be grounded in language is considered secondary (for more endogenous epistemologies and instance, it is clear in most language frameworks, with a focus on learning requirements in Terms of Reference and being mindful of local languages, as or ToRs). In addition, most evaluation well as cultural values. These concerns teams limit themselves to temporarily do not undermine the imperative of hiring a local translator to assist them providing relevant and valid findings, during field work. These hired translators which ensure the production of suitable may also have a limited knowledge of recommendations to inform policy the Portuguese language and of the and programming design. All these ambitions of the applied methodologies. requirements collectively imply the As a result, key nuanced information mastering not only of the language

Speak the Language of Lusophone Africa 55 eVALUation Matters Third Quarter 2019

but also of the communication for the needs of local populations, but also skills, necessary to obtain the unbiased due to the fact that o"en research teams views of communities. Moreover, it is do not approach evaluation work with important to adapt evaluation research these unique lenses. to the local context, to build capacity among local researchers and evaluators, and to ensure that external evaluators Conclusion master the language and the culture of the community under evaluation. Such In all Lusophone African countries, key aspects are widely acknowledged there are exceptional opportunities to by the relevant academic literature (e.g., strengthen evaluation activities, with Smith, 2012; LaFrance, J. and Nichols, R., governments playing a more active role in 2010; Kovach, 2010; Hood, Hopson, and demanding and managing evaluations and Frierson, 2005; Tillman, 2002). using evidence. However, these countries must explore these opportunities within Furthermore, research methodologies their considerably different political which truly establish guidelines for economy se!ings. Donors and partners community-based research and evaluation, can provide opportunities for learning honouring and building on the culture through country-led support. and context of indigenous people, are a rare find in Lusophone countries. Most A basic pre-condition for “Made in Africa o"en than not, evaluation assignments Evaluations” in Lusophone Africa is that have a short amount of time in the field, M&E practitioners should be able to with interactions with more remote master the official language and build communities limited to single interviews their practice on these countries’ context or focus groups. While these are common and self-determination culture for greater challenges shared by many evaluators relevance and validity of findings. This practicing in Africa, it should be noted also means that donors need to set stricter that they generally occur due to the lack language requirements in their ToR, along of resources (time and budget) to develop with tighter guidance for true community- more targeted methodologies to account based research.

56 Speak the Language of Lusophone Africa eVALUation Matters Third Quarter 2019

Misreading Lusophone Africa’s needed: the individual level (such as linguistic, historical and social nuances capacity building needs), the creation of in evaluation work hinders the quality, institutional systems and organizations, rigor and usefulness of evaluative and the enabling environment. evidence. African Lusophone countries seldom have a say in the development Lusophone countries are to take the of new or context-adapted evaluation lead in this process and map their methodologies and approaches. The lack own needs, building on their specific of Lusophone evaluators is preventing contexts and internal resources, but these regions from contributing to the also on international initiatives, so development of evaluation theory and as to strengthen their own national practice through their own endogenous evaluation systems and capacities. Sound knowledge and skills. Solving the 21st M&E systems and evaluation capacity century´s big challenges requires that development are urgently required in all take part in devising solutions. most African countries, and much more Empowering Lusophone M&E experts is in those speaking Portuguese. critical to ensure diversity and promote more innovative and respectful global However, given existing constraints and evaluative thinking. the risks of these efforts´ crowding out, they must work collaboratively to create There are no ready-made fast-tracking the necessary partnerships to make this solutions for achieving M&E goals work. By drawing on current initiatives and thus supporting the global to monitor the 2030 Agenda and the development agenda. For Lusophone African Union´s (AU) 2063 Agenda, African countries, sound M&E systems and by sharing evaluation experiences and capacity development are urgently with other African countries, regional required. However, it is fundamental and linguistic blocks, Lusophone to conduct a participatory M&E countries can create and affirm their diagnostic beforehand and involve local own evaluation networks. They can stakeholders in this exercise. Such a also contribute to the development of needs assessment is a steppingstone for local evaluation capacities and sustain understanding which kind of capacity an endogenous demand for evaluation, and resources ought to be put in place, ensuring that policymakers are aware who are the champions, and what are the of the value of evaluation knowledge to next steps. There are three important improve policymaking, and ultimately tiers to uncover the type of support development results.

Speak the Language of Lusophone Africa 57 eVALUation Matters Third Quarter 2019

Endnote

1. Spanish-speaking, but a member of CPLP and in included in AfDB´s Lusophone Compact..

References

Hood, S., Hopson, R., & Frierson, H. (Eds.). (2005). The role of LaFrance, J. and Nichols, R. (2010). “Reframing evaluation: culture and cultural context. Greenwich, CT: Information Age. defining an indigenous evaluation framework”, The Canadian Journal of Program Evaluation, Vol. 23 No. 2 Pages 13–31. Kovach, M. (2010). Indigenous methodologies: Characteristics, conversations, and contexts. Toronto, ON, Canada: University of Smith, L. T. (2012). Decolonizing methodologies: Research and Toronto Press. Indigenous peoples (2nd ed.). New York, NY: Zed.

Tillman, L. C. (2002). Culturally sensitive research approaches: An African-American perspective. Educational Researcher, 31, 3–12.

Elsa de Morais Sarmento is an applied economist and evaluator with experience in institutional support; policy formulation, implementation and evaluation; international development; and capacity building. She is an associate researcher at NOVAFRICA, the Nova Business School of Management and Economics in Portugal. She has

le worked for a variety of international organizations and national f governments, namely for the African Development Bank, the European Commission, the World Bank (Independent Evaluation Group and International Finance Corporation), a number of UN agencies, the Millennium Challenge Corporation, NEPAD, the Organization of Eastern Caribbean States (OECS), and CARICOM and led several international assignments in over 140 countries. She has also lectured for over a decade in several universities in Portugal, Spain and the UK, and acted as a researcher at the CEP (LSE) and the European Parliament. She had several Director positions at the Research Office of the Portuguese Ministry of Economy and Authors’ pro Development. She also edited the “The Emerald Handbook of Public- Private Partnerships in Developing and Emerging Economies”, and the Evaluation Ma!ers´ edition on “Impact Evaluation: Insights from Practitioners” for AfDB´s Independent Development Evaluation (idev).

58 Speak the Language of Lusophone Africa eVALUation Matters Third Quarter 2019

Carla Felix Silva is a social expert and evaluator with thirteen years of experience in Africa, in monitoring and evaluation, analytical sector work and project management. Key sectors of experience include human development, civil society engagement through participatory action research, social change and international development. Since joining the African Development Bank (AfDB) in 2010, she worked with the Human Development Department on Fragile States, and from 2015 she worked with IDEV in a number of country strategy and program evaluations, including in Portuguese-speaking countries in Africa. These comprise the AfDB’s Comprehensive Evaluation of Development Results and the Independent Evaluation of the Implementation of the Development and Business Delivery Model of the AfDB. Prior to joining the AfDB, Carla worked with the Non-Governmental (NGO) sector, providing support to actions under European Commission- funded programs and evaluations in Sub-Saharan Africa, including Cabo Verde, Guinea-Bissau and Angola. Carla studied International Relations at the Technical University of Lisbon. le f Mariana Branco is an economist and evaluation expert at the World Bank Group’s Independent Evaluation Group (IEG). Within IEG´s Evaluation Capacity Development, she supports the expansion of the CLEAR Initiative and IPDET, and as part of IEG´s Methods Advisory team she promotes innovation in evaluation and internal evaluation capacity building. She is also a Board Member of the European Evaluation Society; a leader of the European Young and Emerging Evaluators movement; and a member of EvalYouth management group. Prior to joining the IEG, Mariana Authors’ pro implemented impact M&E systems, conducted Value for Money evaluations and delivered accredited training across public and corporate projects in Europe, Angola and Brazil. At the public sector level, she collaborated with the Portugal2020 agenda, the Angola Sovereign Wealth Fund, the Brazilian Social Industry Service and Guile Foundation. In the impact investing sphere, she assessed the corporate approaches to sustainability of Uniliever, Danone, HSBC, among other companies. Mariana was also a research fellow in Development Studies and Health Economics at the University of Porto in Portugal. Mariana holds a Bachelor’s degree in Economics, a Master’s in International Cooperation, and a post-graduate diploma in Evidence-based Policy Research Methods from the United Nations University.

Speak the Language of Lusophone Africa 59