Ensaio: Avaliação e Políticas Públicas em Educação ISSN: 0104-4036 [email protected] Fundação Cesgranrio Brasil
Dannemann, Angela Cristina; Penna Firme, Thereza; Letichevsky, Ana Carolina Setting Up the Brazilian Evaluation Network: a Challenging Work With no Boundaries Ensaio: Avaliação e Políticas Públicas em Educação, vol. 13, núm. 49, octubre-diciembre, 2005, pp. 523-533 Fundação Cesgranrio Rio de Janeiro, Brasil
Available in: http://www.redalyc.org/articulo.oa?id=399537941009
How to cite Complete issue Scientific Information System More information about this article Network of Scientific Journals from Latin America, the Caribbean, Spain and Portugal Journal's homepage in redalyc.org Non-profit academic project, developed under the open access initiative Capacitação de professores em Matemática Contextualizada: Projeto bem-sucedido no Brasil 523 Informes e Participações
Setting Up the Brazilian Evaluation Network: a Challenging Work With no Boundaries *
Angela Cristina Dannemann** Thereza Penna Firme** Ana Carolina Letichevsky**
Abstract Angela Cristina Dannemann debate. However, cultural and historical aspects create Evaluation may be a Master in Business limits for the discussions and relevant tool for the practice Administration by IBMEC-RJ Specialization in Program boundaries among its of democracy. In order for it Evaluation by CEATS/FEA/USP participants. This paper to occur, quality and proper Leader of the Avina Foundation intends to shed light into utilization of its results must Board Member of FIRJAN’s some of these limits and be assured (PATTON, 1997). Social Responsibility Council boundaries, reflecting over Therefore, the consolidation Thereza Penna Firme how they are being overcome. of democratic principles in a Ph.D. Education and society and the promotion of Psychology – Stanford effective social interactions University – California Resumo (FETTERMAN; Coordinator of the Evaluation Center – Cesgranrio Foundation Estabelecendo a WANDERSMAN, 2005) will Senior Evaluation Consultant benefit enormously from a Teacher (retired) of the Federal rede brasileira greater and better use of University of Rio de Janeiro de avaliação: um evaluation in all possible Ana Carolina Letichevsky formats. The Brazilian Doctoral (in process) in Methods trabalho Evaluation Network was of Support to Decision/ desafiador sem created approximately three Department of Electrical years ago in a country with a Engineering PUC—RJ limites. recent history of democratic Master in Production Uma avaliação pode ser Engineering PUC-RJ um importante instrumento practice. Within a theoretical Statistician at Cesgranrio Foundation para o exercício da demo- framework networks do not Professor of Statistics at PUC-RJ have boundaries or limits for cracia. Para que isso ocorra
* Paper presented at the 2005 Joint Conference of the Canadian Evaluation Society and American Evaluation Association “Crossing Borders, Crossing Boundaries”, Toronto, Canada, October 24-30, 2005 ** The authors want to thank all members of the Brazilian Evaluation Network for their indirect contribution in the development of this paper.
Ensaio: aval. pol. públ. Educ., Rio de Janeiro, v.13, n.49, p. 515-533,515-522, out./dez. 2005 524 Angela Cristina Dannemann, Thereza Penna Firme e Ana Carolina Letichevsky
é preciso assegurar a qualidade da avalia- Rationale ção, bem como a utilização apropriada dos Brazil, with its 500 years of history has seus resultados (PATTON,1997). Desta for- only recently reached a full democratic ma, a construção de uma sociedade de- condition, the dream of generations – free mocrática, que busca a melhoria permanen- presidential elections happened again only te dos seus processos sociais (FETTERMAN; 16 years ago, after a long military WANDERSMAN, 2005), depende do aprimo- dictatorship that abolished civil rights and ramento contínuo da sua prática de avalia- reduced social initiatives to a bare minimum. ção, em suas variadas manifestações. No This new situation has met with a population Brasil, há aproximadamente 3 anos, foi cri- not entirely prepared for its demands, ada a Rede Brasileira de Avaliação. Uma especially due to a historical lack of rede que, em sua concepção teórica, não importance to universal education with high possui limites ou fronteiras de discussão. quality. Some of the reasons for this are Contudo, existem aspectos culturais e histó- summarized below: ricos (envolvendo valores), que criam limi- - lack of vision of the influential elite and tes para as referidas discussões e estabele- nonexistent pressure groups; cem fronteiras entre as diferentes instânci- - vested interests of powerful political as. Este trabalho apresenta alguns destes groups, who benefited from keeping limites e fronteiras, refletindo sobre a forma entire populations uneducated in order como aqueles que encomendam e que con- to delay the natural process of demand duzem avaliações vêm contornando os li- for social and political rights resulting mites e ultrapassando fronteiras. from educational enlightenment; - education not a priority because it is a Introduction task demanding high investment, After 3 years of steady growth of the constant supervision, and aimed at Brazilian Evaluation Network (hereafter long term results; BEN) several members are arguing its - historical difficulties with mid and long challenges. Presently, its members place term activities, due to the short term of emphasis on transparency of planning and political mandates and excessive implementation, respect for stakeholders dependence on government decisions. values and taking into account all characteristics of the situation; application Thus, the need for a liberating evaluation of proper methods; and assurance that all practice had to face the lack of education stakeholders must understand the and democratic experience, added to a significance of evaluation, agree with its punitive vision of evaluation, due to it being application and, most important, be able introduced within the context of an to understand, debate and use its results. educational tradition. Radical groups of As a consequence, this paper’s proposal several denominations introduced and is to further promote this debate guided by sustained formal education in Brazil for the following question: centuries and are to be praised for this, but - How to evolve to a more transparent must also bear the responsibility for the model, with a well defined identity and well culture of blaming individuals and not organized internal evaluation processes? processes, methods or concepts for
Ensaio: aval. pol. públ. Educ., Rio de Janeiro, v.13, n.49, p. 515-533, out./dez. 2005 Setting Up the Brazilian Evaluation Network: a Challenging Work With no Boundaries 525
something gone wrong. Also, it must be promoting and sponsoring activities related to observed that the practice of evaluation as it. In other words, the network is promoting the a systematic process of data collection for development of the practice of evaluation in proper value judgment was established only Brazil, pushing boundaries ever further and 30 years ago. Not the least important is the overcoming moral, political and academic fact that the network dynamic is a very recent barriers. A debate over some critical aspects of organizational arrangement, especially in this network is extremely relevant towards Brazil – the first networks were started in the achieving better interactions within a traditionally 90’s, aimed at environmental education. horizontal, non-hierarchical system, such as this one, for members coming from an ingrained Added to that, Brazil’s huge social vertical, highly hierarchical culture. The inequality will benefit enormously from considerations developed herein may be an adequate measurement of the results and important contribution towards the improvement outcomes of its public policies and programs of evaluation everywhere specially in developing - 10% of Brazil’s population amass 50% of countries such as Brazil (GUARNIERI, 2004). the GDP, which in terms of monthly salaries means that all those earning US$ 200 or above Overview comprise these 10%. In spite of commendable The Brazilian Evaluation Network efforts from several social organizations, very (hereafter BEN) was originally born in low or non-existent income does not promote August, 2002, as part of an international sustainability and certainly promotes alternative cooperation for the promotion and ways of survival such as begging, child labor, development of evaluation stimulated by UN and other worse situations. organisms such as UNICEF (SEGONE, 2002). It was initially constituted as a network Belief in evaluation as a means to of individuals and individuals within promote important values for the institutions with the following mission: improvement of democracy (FETTERMAN, “develop and strengthen public interest 2001), such as transparency, reflection, evaluation culture and practice by means critical thinking, (SCRIVEN, 2003) of articulating individuals and participation and debate, has been the organizations, knowledge management, reason behind the creation of the Brazilian professional development and the advocacy Evaluation Network whose current mission of evaluation as a tool for political statement reads: “Develop and enhance the transformation.” Its founding members culture and practice of public interest believed in evaluation as a social learning evaluation within the Brazilian society.” process that contributes with society’s development, being also an important In fact the Brazilian Evaluation Network1 has organizational development tool that been bringing invaluable contributions to this promotes transparency and disseminates best cause, either by supporting this practice among practices, reliable information and results its members (persons from several different sectors (ENCONTRO ANUAL DE PLANEJAMENTO of society interested in evaluation), or by ESTRATÉGICO, 2003).
1 Disponível em:
Ensaio: aval. pol. públ. Educ., Rio de Janeiro, v.13, n.49, p. 515-533, out./dez. 2005 526 Angela Cristina Dannemann, Thereza Penna Firme e Ana Carolina Letichevsky
The first strategic planning meeting for PLANEJAMENTO ESTRATÉGICO, BEN was held in São Paulo on August 2003). 2003. It was attended by 13 persons, representing the 5 established hubs at the In addition, a fifth goal aimed at BEN’s time (see Table III below). BEN’s mission own organizational development was statement was consolidated (already agreed upon: to strengthen its structure mentioned at the beginning of this and expand its reach. Issues about BEN’s chapter), its goals were reviewed, a vision organization were extensively debated for its future and an action plan for the breaching the following themes: next 18 months were designed and constitution (by individuals and consensus on organizational formats was individuals within institutions), financing achieved. (only specifically per project and under the responsibility of an executive office The vision of the group about the future rotating voluntarily between hubs) and of evaluation and its practice: government (by a group of facilitators a learning process chosen at the rate of 2 per regional hub). promoter of empowerment and transparency Almost one year later, on July 2004, accessible, ethical and participative a second strategic planning meeting was part of the organizational culture held in Brasilia with 16 persons a management tool attending, representing 6 regional hubs. a reliable source of information Only 3 of the persons attending the first a tool for political transformation meeting were present to provide continuity – the majority was not aware The four main goals were clearly of the previous planning and was stated within the mission statement itself: updated during the first day. Having (i) articulate individuals and overcome this difficulty, with the help of organizations, directly or indirectly the “veterans”, activities proceeded with involved with evaluation debate and a report per regional hub, reflections and practice, (ii) create, disseminate and debate about the advances and setbacks manage knowledge on evaluation, (iii) on the application of the previous promote the training and development strategic planning, review of the mission of professional evaluators, and (iv) statement and the setting of thematic advocate the inclusion of evaluation goals and the action plan for the 2004/ practice in management and planning 5 year. Initially, reports of the 2003/2004 strategies of public and private year were presented by representatives of institutions (ENCONTRO ANUAL DE the hubs. See below.
Ensaio: aval. pol. públ. Educ., Rio de Janeiro, v.13, n.49, p. 515-533, out./dez. 2005 Setting Up the Brazilian Evaluation Network: a Challenging Work With no Boundaries 527
Table I – Highlights of 2003/4 Reports
Hub Main Gains Challenges Membership Bahia Civil society Majority of members want to Few members working in the learn evaluation concepts evaluation field and principles Frequent Better definition of what BEN meetings with high has to offer attendance Difficulties in understanding a Workshop with Michael people network and a need to Quinn Patton involve institutions
Brasília Government and Workshop of Public Policy Low activity Multilateral Evaluation with Elliot Stern Agencies
Rio de Janeiro Academy Workshop with Michael Low activity and Quinn Patton high turnover
São Paulo ND Frequent meetings with good Low participation due to attendance excessive professional Workshop of Participative duties of members Evaluation with Rita O’Sullivan
Pernambuco ND Representation at a local – General Policy Event
Minas Gerais ND Frequent meetings with good – attendance Focused on stabilizing the operation
ND – not declared
Improvement was identified overall, as well out at Table I above, BEN’s growth met with as per regional hub, with the sustainability of several difficulties: the existing hubs, the creation of a new hub communication failure between at Minas Gerais, participation in the creation hubs and within hubs of the international evaluation network long distance between hubs (IOCE) and in the Latin American and need of an office or decision level to Caribbean network (RELAC), and the centralize information celebration of the agreement with the Third need to aggregate and advertise local Sector Information Network (RITS) to host and national references for BEN BEN’s site and e-group. Notwithstanding need to invest in obtaining precise these improvements, and the gains pointed knowledge about evaluation practice
Ensaio: aval. pol. públ. Educ., Rio de Janeiro, v.13, n.49, p. 515-533, out./dez. 2005 528 Angela Cristina Dannemann, Thereza Penna Firme e Ana Carolina Letichevsky
and development overall debate about the propriety or necessity (government, private sector and NG) of formalizing BEN into a national lack of a clear definition of identity for association BEN need to have a better knowledge of individual members’ profiles and Discussion was then initiated in order to activities in order to stimulate reach consensus about a preliminary interchange and a sense of belonging agenda for the two and a half days of work between members, and generate ahead. The list of agenda issues included further knowledge through joint paper the following: publication revision of previous mission to ratify the concept of diversity – there emphasize evaluation as a public shall not be a consensus about asset and reassessment of the evaluation methodology, concepts, designation as goal of the previously themes or professional approach assigned fifth goal of organizational approach the themes of web site development development and the creation of a open a broad debate on the two video conferencing channel by essential themes of BEN’s identity cooperation between institutions (name and abbreviation publicly After ample discussion of several themes applied) and governance (decision the outcomes were very positive. The mission making and representation) statement and the four original goals were discussion on transparency with kept in their essence, with slight alterations the management of funds and in structure for simplification’s sake. See fund raising flowchart below.
Table II – Mission and Goals 2004/5
MISSION Develop and enhance the culture and practice of public STATEMENT interest evaluation within Brazilian society.
articulate create, produce stimulate the advocate the individuals and and disseminate development of practice of organizations, knowledge on evaluation evaluation in all directly or evaluation professionals initiatives of indirectly public and involved with GOALS private evaluation debate institutions and practice
Ensaio: aval. pol. públ. Educ., Rio de Janeiro, v.13, n.49, p. 515-533, out./dez. 2005 Setting Up the Brazilian Evaluation Network: a Challenging Work With no Boundaries 529
Discussions were organized into five broad for the rules and procedures Work Group themes: governance, network identity, to build a proposal on the issue of sustainability, communication and development. admission and dismissal of members; Five Work Groups were formed to deal with BEN’s name discussion must be reinitiated specific activities, and attendees volunteered as at the next meeting. members and suggested other possible group Sustainability: funds will continue to be members amongst BEN, the latter to be raised per project; recommendation for confirmed after a first contact. The groups were the rules and procedures Work Group organized into activities that are self-explanatory: to build a proposal on the issue of fund rules and procedures group, relationship with raising. RITS group, video conferencing group, web site Communication: urgent compliance development group and support for hub startup with all items of the agreement with group. The activities to be executed by these RITS; discuss and disseminate ways to committees were defined as the 2004/5 action make evaluation more accessible for plan and a date for the 2005/6 strategic planning diverse stakeholders as well as society; meeting was proposed by the representative of immediate development of the new web the Pernambuco Hub and accepted: the 2nd half site; strengthen its political action in of August, 2005 in Recife (ENCONTRO ANUAL order to influence public policy and DE PLANEJAMENTO ESTRATÉGICO, 2004). decision making as well as to further include evaluation in planning within Common ground was found for several the governmental, non-governmental items of the pre-defined agenda: and private sectors. Governance: no formalized national Development: employ the best efforts association; an Executive Committee to acquire information on training created to coordinate the action plan and development in evaluation, by articulating Work Groups, solely disseminate these events and facilitate comprised of attendees to this meeting; attendance for BEN members; not to the executive office to be experimented develop its own training program; as a shared model – seat in Rio de take advantage of the experience of Janeiro and shared decisions with other senior managers to promote the hubs; national rules and procedures development of new members; to be defined by a Work Group and stimulate the creation of study groups. hub’s sub-networks will have independent decision making. Discussion Network Identity: survey to be developed for achieving in-depth knowledge of and Conclusion membership; preferential public for new By December, 2004 BEN had 264 membership are evaluation professionals, subscribed members and regional hubs in 6 management professionals, academic state capitals within the Northeastern, Central community, students, educators and and Southeastern regions of Brazil. Most of social leaders; spokesperson or group of these hubs had their own independent people must be chosen for public regional sub-networks, with diverse levels of representation of BEN; recommendation activity. The profile was as follows:
Ensaio: aval. pol. públ. Educ., Rio de Janeiro, v.13, n.49, p. 515-533, out./dez. 2005 530 Angela Cristina Dannemann, Thereza Penna Firme e Ana Carolina Letichevsky
Table III - Regional Hubs (by order of creation) State City Year Startup District Capital (DF) Brasilia 2002 São Paulo (SP) São Paulo 2002 Rio de Janeiro (RJ) Rio de Janeiro 2002 Bahia (BA) Salvador 2003 Pernambuco (PE) Recife 2003 Minas Gerais (MG) Belo Horizonte 2004 A close look at email address extensions gave us the following characteristics for the 264 members:
Table IV - Members Affiliation (according to email extensions) Type Extension Qty Non-governmental .org 20 .org.br 32 Commercial and providers .com 25 .com.br 94 Governmental .gov 1 .gov.br 54 Educational .edu 4 .edu.br* 27 Other 7 * includes educational institutions in Brazil who apply the format institutionname.br
Members within the Brazilian Internet possibility is that there are many members comprise 78.4% (207 email addresses) of who work as consultants in monitoring and the Network, where the overwhelming evaluation, may not be associated to any majority may be considered isolated institution, and are typically connecting individuals or persons within businesses – themselves through personal emails within they are 45.4% of the above, almost double email providers (by this angle we may also the amount of those connected with add the .com addresses and raise the government 26.1% and three times or more previous 45.4 total to 51.3%). On the of those within NGO’s and educational other hand it shows us how much still has institutions, 15.5% and 13% respectively. to be done towards achieving BEN’s mission – the need for enhancement and This profile may give us a strong development is clearly demonstrated by the indication of the use of personal emails small number of members and the for the Network in order to avoid the concentration of hubs in parts of the clogging of professional inboxes. Another country with higher income.
Ensaio: aval. pol. públ. Educ., Rio de Janeiro, v.13, n.49, p. 515-533, out./dez. 2005 Setting Up the Brazilian Evaluation Network: a Challenging Work With no Boundaries 531
Proper measurement of a public policy arguments and promoted the creation of the Rules or program will assure that its results and and Procedures Work Group, is a good example. outcomes are clearly identified and success With a transversal aspect to it, this group had or failure is distinguished. If you can’t clearly the potential to influence decisions that could be identify success you may be rewarding failure; positive or negative for all 4 strategic goals. But if you can’t see success you cannot learn from this group, in order to be “useful”, must in fact it; and if the results of a program cannot be produce suggestions with high “accuracy” and clearly demonstrated, you cannot win public post them within the predefined dates, for ample support (PATTON, 1997). Thus was created discussion, after reflecting on its own “feasibility” and still is maintained in Brazil this condition and the “propriety” of its stance. Some risks of lack of trust in government and low level associated to our attributes may be predicted of trust with civil society’s public initiatives (a here – distance problems due to having members recent survey associates high credibility with from 2 or more different states, lack of time firemen and the post office, but not with the availability because of members professional three levels of government or with NGO’s). commitments, difficulties with task division and assignment of responsibility with group Enhancement and development of the coordination in order to concentrate and culture and practice of public interest evaluation organize the work at hand and to post the group within civil society, governmental and consensus. These issues should have been raised educational institutions are tantamount at this at the very beginning, at the risk of non- time in Brazil, and the sheer existence of a strong completion of the task at hand. In fact they weren’t institution advocating evaluation practice and raised and the task was not completed – some development is decisive for the adequate of us were part of this group and these issues transition into a fully democratic condition. were not properly addressed while the group was Proper knowledge and application of created. A consequence of this failure was the evaluation’s 4 attributes of utility, feasibility, number of excessive complaints of inadequate propriety and accuracy, around which all posting, such as an anonymous survey trying to standards are organized, are the bare minimum confirm active membership. for evolution in this direction in Brazil, and they must be disseminated in all levels of public All 4 other themes, network identity, interest, especially in educational institutions. communication, sustainability and development, can undergo this same exercise, Let’s try this exercise of applying the 4 which we will not endeavor to do in this paper, attributes of evaluation and take BEN’s and they will all give us similar outcomes. In planning for instance. A close look at its goals fact some of the risks identified above, must be and the associated activities defined at the considered as an overall risk for BEN’s action 2004/2005 meeting with this approach, plan. Two of them are: lack of time availability trying to ask, for each one of them, how useful, because of members’ professional commitments proper, feasible and accurate they are, may and difficulties with task division and assignment be a good starting point. of responsibility with group coordination. Several Work Groups have not completed their The governance theme, for instance, which task due to these reasons. Voluntary dedication raised a high level of philosophical and political to any activity takes time and compromises
Ensaio: aval. pol. públ. Educ., Rio de Janeiro, v.13, n.49, p. 515-533, out./dez. 2005 532 Angela Cristina Dannemann, Thereza Penna Firme e Ana Carolina Letichevsky
financial survival in a country with low income development and improvement of all levels and this may be a good reason to start a stakeholders (PENNA FIRME, 2004). discussion on the possibility of coordinating an annual fund raising for the sustainability of the A good start for BEN would be to find rotating executive office – this would satisfy the guidance on these attributes of evaluation for any need for centralized information and and all of its future planning activities. Things could dissemination of knowledge and practice. Either improve even further through the revision of current that or the creation of a national association activities on the national level, maybe disseminating with hired professional staff for management of this knowledge towards the regional independent national activities, hosting of the e-group and sub-networks. “Do unto you…” is good advice centralization of knowledge and dissemination. for any and all relationships in society and application of the standards should better begin The crux of the matter here is to what inside our own house. Our role of measuring should we do when developing evaluations and placing proper value unto social programs in order to reduce uncertainty, improve and public policies might be made easier if we effectiveness and make relevant decisions. consider advocating and applying something as The noblest intent here is to strengthen a simple as the 4 attributes, in every single act. movement that is willing and dedicated to promote social transformation, and to follow BEN’s expansion finds solid ground on its course in order to intervene, whenever its internal and external debates and it must needed. The 4 attributes call for what is needed still meet the challenge of breaking – evaluations that are sensible to situational international boundaries to link the Brazilian and conditional aspects, methodologically evaluation community to the greater flexible, dynamic in the political sense and worldwide evaluation community, well substantially creative in order to promote the represented here in this conference.
References ENCONTRO ANUAL DE PLANEJAMENTO ESTRATÉGICO, 1., 2003, São Paulo. Relatório ... São Paulo: Rede Brasileira de Avaliação, 2003. Disponível em:
ENCONTRO ANUAL DE PLANEJAMENTO ESTRATÉGICO, 2., 2004, Brasília. Relatório… Brasília, DF: Rede Brasileira de Avaliação, 2004. Disponível em:
FETTERMAN, D. Foundations of empowerment evaluation. Thousand Oaks, CA: Sage Publications, 2001.
FETTERMAN, D.; WANDERSMAN, A. (Ed.). Empowerment evaluation principles in practice. New York, NY: The Guilford Press, 2005.
Ensaio: aval. pol. públ. Educ., Rio de Janeiro, v.13, n.49, p. 515-533, out./dez. 2005 Setting Up the Brazilian Evaluation Network: a Challenging Work With no Boundaries 533
GUARNIERI, M. C. L. Governança de (em) redes sociais: uma contradição? Rio de Janeiro, 2004. Disponível em: < http://www.rits.org.br/redes_teste/ rd_tmes_ago2004.cfm>. Acesso em: ago. 2004.
MARTINHO, C. Redes: uma introdução às dinâmicas da conectividade e da auto- organização. Brasília, DF: WWF, 2003. Disponível em:
PATTON, M. Utilization-focused evaluation: the new century text. Thousand Oaks, CA: Sage Publications, 1997.
PENNA FIRME, T. Avaliação: tendências e tendenciosidades. Ensaio: avaliação de políticas públicas em educação: revista da Fundação Cesgranrio, Rio de Janeiro, v.1, n. 2, p. 5-12, jan./mar. 1994.
PENNA FIRME, T. et al. Avaliação em rede. Rio de Janeiro, set. 2003. Disponível em:
SCRIVEN, M. Evaluation in the new millenium: the transdisciplinary vision. In: DONALDSON, S. I.; SCRIVEN, M. Evaluating social programs and problems: visions for the new millenium. Manwah, New Jersey: Lawrence Earlbaum Associates Publishers, 2003.
SEGONE, M. A comunidade internacional de monitoramento e avaliação: um movimento global para fortalecer a cultura e função de M&A no mundo. Brasília, DF, 2002. Disponível em: < http://avaliabrasil.org.br>. Acesso em: abr. 2005.
Ensaio: aval. pol. públ. Educ., Rio de Janeiro, v.13, n.49, p. 515-533, out./dez. 2005