http://www.jmde.com/ The Theory, Method, and Practice of Metaevaluation

The Contribution of Metaevaluation to Program Evaluation: Proposition of a Model

Helga C. Hedler and Namara Gibram Faculdade Alvorada, Curso de Administração, Brasília/DF

Background: This theoretical article points to the Findings: Metaevaluation and meta-analysis are different fundamental difference between meta-analysis and research methods with a different approach. Meta metaevaluation. A model of metaevaluation for social evaluation is a qualitative method useful when evaluating programs is presented based on prior practical research. prior evaluations. Yet the quantitative approach of meta- analysis applies better for first evaluations. Meta Purpose: The purpose is to present a model for evaluation may include other methods to help strengthen metaevaluation as a tool that can be used in other studies. the evaluation results. Theory points to the need of a qualitative framework to go beyond the understanding of meta-analysis for Conclusions: Metaevaluation aligns theory and practice program evaluation. for program evaluation. The proposed model for metaevaluation may hold value for future theoretical and Setting: This theoretical article is based on an empirical empirical work. research conducted at a Brazilian Governmental audit agency. Keywords: metaevaluation; program evaluation; evaluation use Subjects: The Government agency where the practical ______research was conducted is responsible for the effectiveness and accountability of social programs through audits that occurred from 2003 to 2006.

Intervention: Meetings and interviews were held with auditors that participated in the evaluation process going from planning to final reports as the model proposes.

Research Design: The model for metaevaluation has a qualitative approach used to evaluate prior evaluations for social programs.

Data Collection and Analysis: Data collection included structured interview with the chief manager of the agency in charge of evaluating governmental programs. Documents and reports were analyzed using qualitative method for content analysis. Synthesis of categories was applied to compare different analysis and summarize findings.

Journal of MultiDisciplinary Evaluation, Volume 6, Number 12 210 ISSN 1556-8180 June 2009 Helga Hedler & Namara G. Gibram

his theoretical article underwent the production of knowledge” (Barreira, 2002, p. Tchallenge of increasing knowledge on 17). program evaluation: metaevaluation, a theme In the case of public policies that bring with few studies conducted in Brazil. The model forth plans and goals by program action, proposed herein was based on data obtained by evaluation is a tool that propitiates information an auditing study carried through by the of the results reached by these programs (Ala- Brazilian Federal Audit Court (TCU). The result Harja & Sigurdur, 2000). Rossi and Freeman of this metaevaluation research composes the (1993) understand that the evaluative research scope of another article; therefore the present must use the scientific method as a means to work focuses on the theoretical and explicative investigate social problems. traces of the premises which sustain the Oskamp (1984) characterizes evaluation of metaevaluation model and its applications. programs as an attempt to evaluate the operation, the impact, and the effectiveness of Evaluation of Programs and Their programs in public and private organizations. Program evaluation was developed by applying Concepts a scientific method to the knowledge of reality based on the stages and demands for such The term evaluation can take several lato sensu methods. Moreover, the collection and meanings; among them, evaluations which are systematization of data for the conduct of generally made in daily relation to things, people program evaluation requires the adoption of or situations (Cano, 2004). In such evaluations, valid and trustful procedures, in order to have value judgments are made. Therefore, in this considerable and useful results (Aguilar & sense, evaluating consists in issuing a value Ander-Egg, 1995). judgment or attributing value to something. Aguilar and Ander-Egg (1995), revised This generic definition may be applied to several several definitions for program evaluation and deliberations performed regularly and it refers proposed one that summarizes what other to evaluation in the informal sense. Formal and authors such as Stufflebeam and Shinkfield, systematic evaluation is used to evaluate services (1987), Fernández-Ballesteros et al., Vedung and or professional activities; it utilizes the same Seyfried (1998), Cano (2002), Posavac and methods and techniques present in social Carey (2003) have declared. The definition research (Aguilar & Ander-Egg, 1995). states that program evaluation is “a kind of Evaluating means to determine merit, cost social research applied in a systematic, planned and value (Fernández-Ballesteros, Vedung, & and directive way in order to identify, obtain Seyfried, 1998; Posavac & Carey, 2003; and provide valid and trustful data…to support Stufflebeam & Shinkfield, 1987). Evaluation is a judgment of merit and value of different necessary task that constitutes part of programs, components in a program…” This definition public policies, private projects, public expresses the sense of utility that program regulations, public and private interventions. evaluation bears as a practice connected to The evaluation of programs, referred in this reality and to the needs of users, stakeholders, article as evaluative research, goes beyond these and those involved with the program, aiming concepts and presents the discussion of for the enhancement of service rendering. evaluation as method, subject and establishment Regarding service rendering, according to of scientific patterns. “...The development of Gray, Jenkins, and Segsworth (1993), quoted by the evaluative research presents at its core not Fernández-Ballesteros et al. (1998), the control only the importance of the evaluation as a of public expenses and management of judgment tool for procedures and actions, but assistance programs or policies have been the also the concept that the evaluation represents

Journal of MultiDisciplinary Evaluation, Volume 6, Number 12 211 ISSN 1556-8180 June 2009 Helga Hedler & Namara Gibram main focus of program evaluations in the past management of strategy analysis, evaluation of three decades. Therefore, there would be two human resources, and attitude research perspectives within the evaluation of programs: regarding the program. In some cases formative the first directed to contribution, planning and evaluation involves the development of field improvement of the program and the second research on a small scale before a more considering the verification of its effectiveness comprehensive implementation. The and impact. Other than legal principles, informative evaluator works in a team along regulation and financial management of public with the formulators and program finances aiding actions, program evaluations are administrators, and they participate directly in instruments for controlling government actions the decision making to perform all the necessary within the public scope. changes. Ex-post evaluation deals with the evaluation Evaluation Ex-ante, Intermediate, and Ex- of a working program. This kind of evaluation is post conducted when the program has been implanted, in order to reach stated objectives A definition of evaluation ex-ante is provided (Ala-Harja & Sigurdur, 2000). For this reason, it by the Evaluation Research Society (ERS) is also called additive evaluation. Additive (1988) which defines it as analysis of start-end, evaluation influence programs, projects, and pre-installation, viability analysis or contextual plans. analysis. This definition includes evaluative activities that come before the implantation of a Program, Project, and Plan program. Ex-ante evaluation aims to ratify, to research or to emit a precise estimative of The program, project, and plan modalities are conception sufficiency, operational viability, social interventions which differ in scope and sources of financial resources, and availability of duration. Hence, the project is a “minimal unit organizational support. The results provide a for the destination of resources and by means useful direction to refine the program planning, of an integrated set of activities, a way to determining the appropriate implantation level, transform part of reality, provisioning for a and the decision regarding the installation or scarcity or altering a problematic situation” not of the program. (Cotta, 1998, p. 104). A set of projects aiming The intermediate evaluation is one of the ways for the same objective form a program. Finally, of obtaining knowledge about the program. It the plan aggregates similar programs, thus aims to subsidize the program management defining the directives for social interventions. procedure as feedback for its implantation and The plan conception demands a wider development. In this case, the evaluators and comprehension when dealing with social clients are generally internal, most likely intervention. For instance, in Brazilian public program managers. Evaluation issues assessed policies, plans are developed to establish are those related to event management, which directives for a policy. Multiyear plans created are connected to program impact (Ala-Harja & by the government have a wide scope: they Sigurdur, 2000). Its main contribution lies in the predict directives, costs, budgets for the areas in program formulation (Posavac & Carey, 2003). which the government will work on, and enable According to ERS (1998), an intermediate programs to be unfolded in several areas. kind of evaluation is formative evaluation, also known as the evaluation process in a continuous program, aiming for modifications and improvements. Its activities may include the

Journal of MultiDisciplinary Evaluation, Volume 6, Number 12 212 ISSN 1556-8180 June 2009 Helga Hedler & Namara Gibram

Similarities and Differences between Auditing of syntheses with general conclusions, regarding and Program Evaluation several studies investigating similar areas of research...” (Smith & Bond, 1999, p. 15). The The plan conception demands a wider meta-analysis calculates the effect size regardless comprehension when dealing with social of the complexion of the real standard used by a intervention. For instance, in Brazilian public certain researchers. The size effect of a study is policies, plans are developed to establish the the result of the difference between the scores directives for a policy. The multiyear plans obtained by the experimental subjects and the created by the government have a wide scope: control group, divided by the standard deviation they predict directives, costs, budgets for the of the scores of subjects in the control group. areas in which the government will work on, The size effect provides the average of different and enable programs to be unfolded in several studies determining whether or not the areas. experimental effect that was being investigated In the early fifties, the auditing searched for in a consistent way, could be found. If the the rationalization of management and sampling of studies is large enough, the distribution of resources for the Defense influence of variation in the experimental Programs and Missions of the Government. delineation, geographic localization, study data This effort increased in the Department of and size effect can be predicted. Defense, Planning, Programming and Budget In areas where there is a great quantity of Systems. However, such effort was peripheral studies about a certain object, it is possible to and related to the accounting-perspective have quantitative literature reviews and these verification, whose main goals were: (1) studies become known as meta-analysis (Hunter planning the cost-effectiveness of the program & Schmidt, 1996, 1999; Rossi & Freeman, and then evaluating such cost-effectiveness, 1993). and; (2) checking if the cost-effectiveness was Although in the exact sense, meta-analysis is the result of the planning procedures. Despite not a delineation of research, it is an alternative this restrict approach, analyses such as: to evaluation projects that can be useful in some technical-political analyses, cost-benefit and situations, for instance, when more time is cost-effectiveness, were also conducted by the necessary for collection of original data. The economic area as an attempt to comprehend findings of the meta-analysis are particularly program activities. However, the focus of the useful in the delineation stage of a program, auditing was on planning so that the techniques because they summarize the existing knowledge could outline the probable future results of the regarding similar programs which have been programs. They were not aimed at identifying implanted, therefore providing knowledge for the current effects of program implantation of the new program. existing policies (Chelimsky, 1985). Some authors confuse the meta-analysis procedure with the metaevaluation method. For instance, Ashworth, Cebulla, Greenberg, and Metaevaluation: Characterization, Walker (2004) conducted a metaevaluation Background and Differences utilizing the meta-analysis procedure. They Regarding Meta-analysis justify the metaevaluation in the meta-analysis procedure because they believe it favors the Meta-analysis explicative power of replication, rigorous accumulation of evidence, revision and

summarizing. Besides that, they disagree with Meta-analysis can be described as “…a authors such as Patton (2001) and Günther statistical technique utilized in the development

Journal of MultiDisciplinary Evaluation, Volume 6, Number 12 213 ISSN 1556-8180 June 2009 Helga Hedler & Namara Gibram

(2006), who define metaevaluation as a 3. They provide strong evidence regarding qualitative method. The previous researchers the program impact, subsiding the believe that the qualitative approach is decision making process regarding it. insufficient to support the metaevaluation and Hence, the results of the metaevaluation have not yet realized that this kind of assist and justify the increase of trust by perspective is now surpassed in scientific the interested parts and managers of literature. The research method must be chosen programs in the evaluation results. by taking into consideration, among other factors, the characteristics of the phenomenon Historically speaking, metaevaluation started to be studied and whenever possible, both in 1960 when evaluators such as Scriven, Stake, approaches should be applied for a wider and and Stufflebeam began discussing procedures deeper comprehension of data and the reality in and formal criteria (Worthen, Sanders, & discussion. Fitzpatrick, 2004). The term “evaluation of the The objective of the meta-analysis is to evaluation” was created by Orata in 1940 and provide a description of real correlation metaevaluation by Scriven in 1969 (Cook & distributions between independent and Gruder, 1978). In accordance to Patton (2001), dependant variables. Therefore, if all the studies a metaevaluation is a re-analysis of an evaluative have been correctly conducted, the distribution study, which has been already concluded; taking of correlations can be directly used to estimate into consideration several aspects of the the distribution of the real correlation and if previous study such as methodology, subject not, be submitted to corrections (Hunter & selection, adopted criteria, results and analysis. Schmidt, 1996). The meta-analysis approach will Guba and Lincoln (quoted by Schwandt, significantly contribute to the delineation of 1989), stated that the concept of metaevaluation programs that have considerable gain, taking was conceptually modeled as an inspection into account the existing social science research audit, introduced to establish the validity of and professional reports of established naturalistic research (qualitative). The definition programs. of Schwandt corroborates the definition of Patton (2001), because it comprehends the Metaevaluation metaevaluation as a method of checking the quality of an evaluation. For such, it requires the Metaevaluations bear three main characteristics examination of the evaluation method and its (Woodside & Sakay, 2001): procedures to reach results and conclusions. Yet, for Woodside and Sakay (2001), 1. They are syntheses of findings and metaevaluation includes the evaluation of utility inferences of evaluative research about and validation of two or more studies, which the program performance. They report comprise the same issue. the effectiveness of managing the goals Sometimes, metaevaluation is confused with achieved by the programs and provide meta-analysis mainly by those that do not use information about two characteristics: this method to evaluate programs. In order to Well managed programs and poorly clarify this matter, there is a need to compare managed programs. them pointing to their similarities and 2. They inform about the validity and differences. Table 1 shows the characteristics of utility of evaluation methods, offering the study object, application procedures, data guidance regarding useful evaluation analysis and distinction between meta-analysis methods. and metaevaluation.

Journal of MultiDisciplinary Evaluation, Volume 6, Number 12 214 ISSN 1556-8180 June 2009 Helga Hedler & Namara G. Gibram

Table 1 Comparing Meta-analysis and Metaevaluation

Characteristics Meta-analysis Metaevaluation Study object Any kind of study Concluded evaluation (s) Data source Secondary Secondary Different studies are organized, Aplication following a criterion or variable, Selection of the concluded evaluation (s) regarding the evaluative procedures utilizing a temporal or thematic study or different studies with the same thematic approach Quantitative (statistics). A Qualitative (content analysis, criteria analysis). A new evaluation synthesis of similar findings, is done. The procedures and methods are compared to prior Data analysis calculating the size effect among studies applying pre-established criteria. Improvements are the studies. suggested or a new model is presented. Generally academic, but can also Either academic or professional. Serves as a reference for Usage subsidize professional practices. programs of the specific field studied.

As highlighted in Table 1, there are Posavac and Carey (2003) wrote a chapter in similarities between meta-analysis and their work Evaluation to clarify the establishment metaevaluation. Similarities occur when of criteria and standards in the evaluation of secondary sources of data and their applications projects. Therefore, evaluating would demand are present. The specifications appear in relation the issuing of a judgment based in values and to the study object and the data analysis also the establishment of criteria and standards. procedure. In the meta-analysis, the object can According to Posavac and Carey, the be any type of study. In the metaevaluation, the established criteria and standards need to be study object is exclusively composed of clear and explicit so they can subsidize useful evaluations which have been already concluded. evaluations. Therefore, they must represent the In meta-analysis, the procedure for data objectives of the program, the institutional analysis is quantitative and statistical, with the efforts, measurable and trustful characteristics, calculation of the size effect providing the including those selected with the stakeholders. average between different studies, determining The evaluators of the period from 1960 to if there was or there was not the investigated 1970 created lists to check what would experimental effect. It also indicates the constitute a “good” or “poor” program evaluation. consistency among the findings. Yet, the Hence, at the end of the seventies, a project was metaevaluation uses qualitative analysis launched aiming to develop a set of directives procedures such as: Content analysis or the applied to the educational evaluations to be criteria checking by international organizations established as a general consensus of evaluation of evaluation as the Joint Committee or ERS quality. The formulation of these directives (1998). started in 1975, and was coordinated by Stufflebeam, with authorization granted by the Quality Standards for Joint Committee on Standards for Educational Evaluation, since then known as the Joint Metaevaluation Committee (Worthen et al., 2004). The directives of the Joint Committee In the area of government program evaluation, consist of 30 standards, including definitions, there is not a unique set of standards for the fundamental logic, directives, common errors, auditor’s procedures as a meta-evaluator illustrative cases, and descriptions of evaluation (Schwandt, 1989). The quality standards for practices. In accordance to Stufflebeam and metaevaluation originate together with it.

Journal of MultiDisciplinary Evaluation, Volume 6, Number 12 215 ISSN 1556-8180 June 2009 Helga Hedler & Namara Gibram

Shinkfield (1987), the norms for program The procedure for content analysis starts by evaluation are: utility, viability, propriety, and reading the common parts of the text. After precision. that, counting rules are established for the The utility norms are those directed to recurrence of words or sentences based on the people and groups that have the task of theme. The procedure continues with the theme evaluating in other words, those directly analysis to identify the nucleus sense for the text responsible for the evaluation process. Such sentences, considering the frequency in which norms should help identify the “good” and they appear and their relevance to the research “bad” functioning of the evaluated object, interest. providing clear information regarding the The previously selected themes are then virtues and defects in the evaluation, besides grouped in categories, considering how often providing suggestions for improvement. they appear, homogeneity among them, The viability norms refer to the use of pertinence and exclusivity (Bardin, 1977). It is evaluative procedures, which can be utilized, advisable that the categories be submitted to considering and applying the possible control judges for semantic analysis. The categories can over political forces, which may somehow be previously established or they can freely interfere in the evaluation. emerge from the analyzed text. The norms related to ethics in evaluation relate to the explicit commitments, which assure Synthesis of categories. The synthesis of categories the cooperation, protection, the rights of those was developed by Gibram (2004) as a way to involved in the evaluation and the accuracy of broaden the scope for content analysis. The results. procedure consists in regrouping the categories Finally, the precision norms are those that in thematic axes. These axes can be previously clearly describe the evaluated object in its constructed according to theoretical parameters evolution, context, revealing virtues, defects in being studied or defined from the analyzed evaluation planning, proceedings and contents. The axes are important in research conclusions. dealing with great quantity of information or complex themes. The procedure of creating Methodology for Metaevaluation thematic axis can also be useful when there are many categories, hindering the conclusion of The metaevaluation data can be worked by the results. different qualitative analysis techniques. The The thematic axes formed by grouping authors of this article suggest: Content analysis, categories provide a broader and more realistic summary of categories and conceptual model of view of the problem being studied. If all themes the program and checking of criteria by the Joint were analyzed, as Bardin (1977) suggests, Committee. grouped in simple categories, the study of more complex problematic issues would lose the Content analysis. The content analysis is a representation of relational and textual technique for text analysis aiming to obtain significance. Hence, the thematic axis, formed through systematic and objective procedures, by several related categories preserve the recurring themes grouped to compose an significance while organizing multiple themes empirically defined category. These categories for analysis and result reports. facilitate the interpretation of data related to the research object. Among the several types of Program conceptual models. The social programs are category analysis, the theme analysis is widely delineated for action under different utilized (Bardin, 1977). problematic situations, and as such, they bear a

Journal of MultiDisciplinary Evaluation, Volume 6, Number 12 216 ISSN 1556-8180 June 2009 Helga Hedler & Namara Gibram conceptual and technical structure to support were chosen based on the criteria referring to them in accordance to the area of program the methodology of evaluative research for execution. For example, the programs in the which consulting of experts in specific program area of social assistance are based on individual theme domain was not necessary. rights and social policies that follow the Worthen et al. (2004), believe that the Operational Norms (NOB) established by the verification of the Joint Committee checklist Unique System of Social Assistance (SUAS). should go beyond the indication of the use or These are guidelines for the implantation and not of the criteria (check yes or no). They management of program procedures. Therefore, suggest the adoption of a scale with scoring during the execution of the metaevaluation, it is points to measure the criteria. For example, a necessary to search in the specific literature scale from zero to three as indicated in Table 2. related to the program, for sources that In order to correctly score the questions in contextualize the program thematic as well as the Checklist presented in Table 2, the evaluator other program evaluations similar to the meta- must comprehend that: evaluated ones. In the specific literature about 9 In Question 15, the measures to metaevaluation and in the search of a proper guarantee the minimum quantity of model for execution, different procedures were errors refer to: Application of a pilot found for data analysis as means to subside test; control group; data collection results and conclusions. For instance, Woodside before and after; random sampling or and Sakai (2001) utilized as procedure for another procedure of control for metaevaluation analysis the theoretical model of internal and external validity in Kotler (mentioned in Woodside & Sakai, 2001), evaluative research. designed specifically for the area of Federal 9 In Question 18, in order to evaluate the Marketing and Tourism. In order to evaluate the adequate training of the team for the execution program planning, they compared the of the auditing, the scoring of all questions procedures applied in the previous evaluation must be considered because they refer with the premises of a SWOT analysis. To to the adequate use and how pertinent evaluate the program implementation they the methodological processes are, what traced comparisons of the previously adopted kind of analysis, results, conclusions and procedure with the concepts of Mintzberg recommendations are produced. Hence: (mentioned by Woodside & Sakai, 2001), about ƒ Without training: the team who gets planned and deliberated strategy. The results of scores greater or equal to 49% of the previous study were analyzed under the the total (less than 9 questions); perspective of the use of impact indicators for ƒ Partial training: scores from 50 to the Federal Marketing and Tourism Program. 69% (from 9 to 12 questions); Moreover, at the end of this article, Woodside ƒ Adequate training: the team who gets and Sakai proposed a model for future scores “yes” in more than 70% of evaluations of similar programs. the total (13 questions). 9 In Question 19, there was no participation Checking the Criteria According to the Joint Committee. refers to when there was no hiring of According to the orientation of the Joint specialists or any consultation asked of Committee, in order to execute metaevaluation, them. Partial was considered when a checklist must be constructed based on the specialists participated in only one stage criteria to be contemplated in an evaluation. In (planning or evaluation execution). There the example showed in Picture 2, questions

Journal of MultiDisciplinary Evaluation, Volume 6, Number 12 217 ISSN 1556-8180 June 2009 Helga Hedler & Namara Gibram

was participation when consultants were criteria. Such questions are related to the hired or consulted. political context, program characteristics, approach, methods, techniques and difficulties Table 2 shows a sample for verifying the for the execution of evaluation. questions referring to the Joint Committee

Table 2 Checklist of Questions Based on the Criteria of the Joint Committee

Questions Does not apply no partially yes Politic context 0 1 2 3 1. Were the program audience and participants of the audit identified? 2. Did the report clearly describe the program context? 3. Did the audit consider how the different groups of interest acted in the program? Characteristics of the program 4. Was the collected data broad enough to understand the functioning of the program? 5. Did the report clearly describe the program? 6. Did the report clearly describe the objectives of the program? Auditing approach 7. Were the in formations collected sufficient to reflect the objectives of the audit? 8. Did the report clearly describe the results of the audit? 9. Did the report clearly describe the conclusions of the audit? 10. Did the report clearly justify the recommendations made by the audit? Methods and techniques 11. Were the techniques of data analysis explicit? 12. Did the report clearly describe the methodological procedures of the audit? 13. Were the procedures of information collection clearly described? 14. Were the instruments for information collection valid? 15. Were all the necessary measures taken in order to assure the minimum amount of errors during the data collection? 16. Was the quantitative information adequately analyzed? 17. Was the qualitative information adequately analyzed? Auditing accomplishment difficulties 18. Did the auditing team have adequate training to undergo the audit? 19. Did external consultants for specific areas participate in the audit? 20. Were the audit’s resources (time, money and employees) adequate for

accomplishing the foreseen activities?

Proposition of a Model for According to Patton (2001), Schwandt (1989), and Woodside and Sakai (2001), Metaevaluation metaevaluation can be defined as a method of research where one or more stages of the Metaevaluation itself can be the object of evaluative studies concluded are re-analyzed; another metaevaluation; in this case several there is a comparison of the previous requirements must be considered. The evaluations with quality standards and validity metaevaluation demands a set of procedures, accepted in the scientific community and at the standards and criteria for judging the evaluation end there is a new evaluation issued regarding quality (Schwandt, 1989). the analyzed evaluative study.

Journal of MultiDisciplinary Evaluation, Volume 6, Number 12 218 ISSN 1556-8180 June 2009 Helga Hedler & Namara Gibram

In the revised literature (Ashworth et al., execution becomes independent of the 2004; Chelimsky, 1985; Cook & Gruder, 1978; government policy, being assured by the Laws Patton, 2001; Schwandt, 1989; Stufflebeam & and Policies of the State. Shinkfield, 1987; Woodside & Sakai, 2001;), The TCU bears a social function associated studies pointed to different procedures for with the control and supervision of the public conducting metaevaluations and also different affairs, as well as it’s patrimonial and economic conceptions regarding the process. In this sense, aspects regarding public administration (Mendes no direct response to the question of the et al., 1999). Therefore, the court is responsible necessary stages and techniques for the for conducting the audit of social programs. conduction of a meta-evaluative study were The evaluation modalities used are auditing of found. operational performance and program The procedure of meta-analysis as means to evaluation (Brasil, 2000). obtain a metaevaluation was not considered In Brazil, the discontinuity of social because it would only answer to the main programs is very common. This is more evident purpose of metaevaluation if, at the end, a new in large-scale programs that produce little evaluation regarding the analyzed evaluation documented and systematized results. procedure had been drawn. The meta-analysis Notwithstanding governmental planning, focus can be utilized in metaevaluation only if it is is generally on the development of plans, used together with other qualitative procedures programs and projects, neglecting the stages of validated by program evaluation associations, to inspection, procedure evaluation, results and finally generate a new evaluation. impacts (Silva, 2002). The authors of the present article disagree The social reality in which the social with Ashworth et al. (2004) that consider meta- programs are inserted present challenges for the analysis the best way or a self-sufficient management of programs, both in the procedure for conducting a metaevaluation. effectiveness of actions and in program evaluation. Hence, the social reality being fluid Conceptual Premise: The Programs in the Social and mutable supports the drawn programs that Reality need constant evaluation and monitoring. These evaluations also need to be meta-evaluated. The Brazilian social reality has structural problems, which produce hunger, poverty and The Conceptual Model of Metaevaluation social disaggregation. In this article, the social program is understood as a “systematic The graphic model presented in Figure 1 is intervention planned with the purpose of supported by the supposition that the achieving change in the social reality” (Cano, metaevaluations are applied to evaluative studies 2004, p. 9). The social programs are developed within the context of social reality, and that this by public policies and emerge to supply the context may influence its realization. needs detected in the environment of a certain The meta-evaluative studies contemplate population (Posavac & Carey, 2003). previous studies, in any program phase, whether The social programs are created to intervene they are ex-ante, intermediate and ex-post and in these situations; however, due to their they influence any study bearing an evaluative originating complexity, the possibility of action drawing (e.g., evaluation, policy, plan, project is limited and may cause both advances and program, auditing). regressions. An advance is considered when Besides that, the metaevaluation depends on these programs transcend governments and a set of quality criteria to make it valid. Such become continuous services. This way their value judgment criteria are shared by the

Journal of MultiDisciplinary Evaluation, Volume 6, Number 12 219 ISSN 1556-8180 June 2009 Helga Hedler & Namara Gibram international community of evaluators as a quality standards and the methodology used in necessary guide for the evaluation of another metaevaluation involving techniques of data evaluation. In the current model, the quality analysis including case studies, content analysis, standards and validity of metaevaluation syntheses of categories and conceptual models adopted were the ones recommended by the in the specific program area. Joint Committee to attest the global quality of The results of these qualitative data analyses the evaluation and issue a new evaluation in subsidize at the end of the process, the relation to the previously conducted one. In the judgment of the previous evaluative study example described in Table 2, the checklist of comparing it to the criteria established in the questions was based on the criteria of the Joint conducted analysis, therefore enabling the Committee. emission of a new evaluation. Hence, the new The metaevaluation may have a set of evaluation will present the strong aspects to be analysis procedures in order to reach a final valued and the weak aspects to be corrected. result – the emission of a new value judgment, a new evaluation. This article has presented some

Figure 1. Conceptual Model of Metaevaluaiton

Journal of MultiDisciplinary Evaluation, Volume 6, Number 12 220 ISSN 1556-8180 June 2009 Helga Hedler & Namara Gibram

Conclusion The metaevaluation can be stimulated by several interests such as academic research or Some considerations will be made in relation to demands of the agencies that coordinate and the applicability of the metaevaluation. The supervise the program. It should be made clear metaevaluation can take several shapes and vary that the evaluator does not have to accept the from professional critiques to evaluation reports original results obtained by previous studies. or be used in procedures of re-analysis of This method is a qualitative instrument which original data. It can be formative or summative provides the means for analyzing and in kind. The summative metaevaluation is a implanting improvements in the existing flashback activity developed by an independent evaluations. external agent over the process and the product This article has presented the procedures of the evaluation comparing it to a group of for data analyzes, such as synthesis of categories patterns for evaluation. In order to do this kind for content analysis which bears the potential to of metaevaluation, the evaluator must have an enhance the understanding of wide range extremely coherent political and prudent categories originated from the reading of attitude and behavior towards the correct extensive printed material about the meta- actions and procedures. evaluated auditing. Metaevaluation demands hard work from The applicability of this model can be those conducting it, as well as the collaboration observed in the propositions of improvements of other consultants or researchers to judge the presented in the audit conducted by the Federal criteria utilized in the analysis of previous Audit Court. The method of data analysis evaluations. Sometimes the meta-evaluator may provided the knowledge of the procedure for develop different hypotheses and/or collect realization of ANOP, enhancing its strengths new information about the program studied in and weaknesses. Above all, it was verified that the previous evaluation. In case of programs the auditing model adopted by the Federal generating wide public interest, the Audit Court was positively evaluated by the metaevaluations analyze the results of different Joint Committee in most of the established evaluations of these programs (including criteria. The suggestions for improvement evaluations of units or program components) in referred to the methodological aspects regarding order to verify their global impact. Thus, it is the sample that was used, the instruments for important to remember the need to preserve the data collection and the need to improve the ethics, precision, and fidelity to the new qualitative analyses performed. metaevaluation results. When divulging the The metaevaluation carried through by metaevaluation results, a great deal of caution Hedler (2007) can also contribute to the must be taken in regard to questioning the improvement of social programs since the same previous evaluation. An ethical and cautious suggestions for improvement presented in the positioning is required from those conducting it previous evaluation can be applied by the in order to avoid bias and the inadequate use of programs themselves. They can implant them in results. A negative evaluation can harm the their monitoring and internal evaluations. credibility and merit of a given institution or Therefore, this article proposes to enhance group of evaluators. It is important to have in the discussion about the utility of mind that the meta-evaluators do not analyze metaevaluation, as well as the discussion about the collected data, but the inferences other the model presented and its future applicability. evaluators previously made about them, so they emit an evaluation based on personal inferences over the results obtained by other people.

Journal of MultiDisciplinary Evaluation, Volume 6, Number 12 221 ISSN 1556-8180 June 2009 Helga Hedler & Namara Gibram

References Administração Pública. 49, nº.2. Brasília: ENAP. Aguilar, M. J., & Ander-Egg, E. (1995). Evaluation Research Society [ERS] (1998). Avaliação de serviços e programas sociais. Standards for program evaluation. San Francisco, Petrópolis: Vozes. CA: Jossey-Bass. Ala-Harja, M., & Sigurdur, H. (2000, Outubro- Fernández-Ballesteros, R., Vedung, E., & Dezembro). Em direção às melhores práticas de Seyfried, E. (1998). Psychology in program avaliação. Revista do Serviço Público. evaluation. European Psychologist, 3,143-154. Fundação Escola Nacional de Gibram, N. F. R. (2004). Trabalho e familia: um Administração Pública, v.1, nº 1. 51. estudo da congruência dinâmica de demandas Brasília: ENAP. multiplas (Work and family: a study on the Ashworth, K., Cebulla. A., & Greenberg, D. W, dynamic congruency of multiple demands). Thesis Robert (2004). Metaevaluation: discovering presented for Doctor’s degree in Psychology what works beast in welfare provision. by the University of Brasília. Evaluation. Vol. 10. [On-line] Obtido em 18 Günther, H. (2006). Pesquisa qualitativa versus de fevereiro de 2007, de pesquisa quantitativa: Esta é a questão? Série: http://evi.sagepub.com/cgi/content/abstra Textos de Psicologia Ambiental, Nº 07. Brasília, ct. DF: UnB, Laboratório de Psicologia Bardin, L. (1977). Análise de Conteúdo. Lisboa: Ambiental. Edições 70 Hedler, H. C. (2007). Meta-avaliação em de Barreira, M. C. R. N. (2002). Avaliação Auditorias de Natureza Operacional do Tribunal Participativa de Programas Sociais. São Paulo: de Contas da União: Um estudo sobre Editoria Veras CPIHTS. auditorias de Programas Sociais. Thesis Brasil. (2000). Tribunal de Contas da União. presented for Doctor’s degree in Psychology Manual de auditoria de natureza operacional. by the University of Brasilia. Brasília: TCU. Coordenadoria de Hunter, J. E., & Schmidt, F. L. (1996). Fiscalização e Controle. 114p. Measurement error in psychological Cano, I. (2004). Introdução à avaliação de programas research: lessons from 26 research sociais. (2ª ed.) Rio de Janeiro: Editora. FGV. scenarios. Psychological Methods, 1(2), 199-223 Chelimsky, E. (1985). Comparing and Hunter, J. E., & Schmidt, F. L. (1999). contrasting auditing and evaluation: some Comparison of three meta-analysis revisited: notes on their relationship. Evaluation Review. An analysis of Johnson, Lullen, and Salas vol. 9, nº 4, p. 483-503 [On-line], obtido em (1995). Journal of Applied Psychology, 84(1), 18 de fevereiro de 2007, de 144-148. http://erx.sagepub.com/cgi/content/abstra Mendes, A. M. B., Tamayo, A., Paz, M. G. T., ct/9/4/483. Neiva, E. R., Tamayo, N., Silva, P. T., Cook, T. D., & Gruder, C. L. (1978). Souza, A. C., Martins, A. J., & David, R. G. Metaevaluation research. Evaluation Review. (1999). Análise da cultura organizacional do [On-line], February 18th 2007, de Tribunal de Contas da União – TCU. Brasília: http://erx.sagepub.com/cgi/content/abstra Relatório Final. O&T ct/2/1/5. Consultoria/FINATEC/Unb. Cotta, T. C. (1998, Abril - Junho). Metodologias de Oskamp, S. (1984). Applied social psychology. New avaliação de programas e projetos sociais: análise Jersey: Prentice Hall. Patton, M. Q. (2001). Qualitative research and do resultado de impacto. Revista do Serviço rd Público. Fundação Escola Nacional de evaluation methods (3 ed.). Thousand Oaks, CA: Sage.

Journal of MultiDisciplinary Evaluation, Volume 6, Number 12 222 ISSN 1556-8180 June 2009 Helga Hedler & Namara Gibram

Posavac, E. J., & Carey, R. G. (2003). Program evaluation. Methods and case Studies (6th ed.). New Jersey: Prentice Hall. Rossi, P. H., & Freeman, H., E. (1993). Evaluation: A systematic approach (5th ed.). Thousand Oaks, CA: Sage. Schwandt, T. A. (1989). The politics of verifying trustworthiness in evaluation auditing. American Journal of Evaluation, 10, 33-40. Silva, P. L. B. (2002). A avaliação de programas públicos: Reflexões sobre a experiência brasileira. Relatório técnico. Brasília: IPEA. Smith, P. B., & Bond, M. H. (1999). Social psychology across cultures. Allyn e Bacon: EUA. Stufflebeam, D. L., & Shinkfield, A. J. (1987). Evaluación sistemática: guía teórica y práctica. Barcelona: Ediciones Paidós Ibérica. Woodside, A. G., & Sakai, M. Y. (2001). Metaevaluation of performance audits of government tourism-marketing programs. Journal of Travel Research; 39, 369. [On-line], Obtido em 18 de fevereiro de 2007, de http://jtr.sagepub.com/cgi/content/abstrac t/39/4/369 Worthen, B. R., Sanders, J. R., & Fitzpatrick, J. L. (2004). Avaliação de programas: concepções e práticas. São Paulo: Editora Gente.

Journal of MultiDisciplinary Evaluation, Volume 6, Number 12 223 ISSN 1556-8180 June 2009