http://www.jmde.com/ The Theory, Method, and Practice of Metaevaluation The Contribution of Metaevaluation to Program Evaluation: Proposition of a Model Helga C. Hedler and Namara Gibram Faculdade Alvorada, Curso de Administração, Brasília/DF Background: This theoretical article points to the Findings: Metaevaluation and meta-analysis are different fundamental difference between meta-analysis and research methods with a different approach. Meta metaevaluation. A model of metaevaluation for social evaluation is a qualitative method useful when evaluating programs is presented based on prior practical research. prior evaluations. Yet the quantitative approach of meta- analysis applies better for first evaluations. Meta Purpose: The purpose is to present a model for evaluation may include other methods to help strengthen metaevaluation as a tool that can be used in other studies. the evaluation results. Theory points to the need of a qualitative framework to go beyond the understanding of meta-analysis for Conclusions: Metaevaluation aligns theory and practice program evaluation. for program evaluation. The proposed model for metaevaluation may hold value for future theoretical and Setting: This theoretical article is based on an empirical empirical work. research conducted at a Brazilian Governmental audit agency. Keywords: metaevaluation; program evaluation; evaluation use Subjects: The Government agency where the practical _____________________________________ research was conducted is responsible for the effectiveness and accountability of social programs through audits that occurred from 2003 to 2006. Intervention: Meetings and interviews were held with auditors that participated in the evaluation process going from planning to final reports as the model proposes. Research Design: The model for metaevaluation has a qualitative approach used to evaluate prior evaluations for social programs. Data Collection and Analysis: Data collection included structured interview with the chief manager of the agency in charge of evaluating governmental programs. Documents and reports were analyzed using qualitative method for content analysis. Synthesis of categories was applied to compare different analysis and summarize findings. Journal of MultiDisciplinary Evaluation, Volume 6, Number 12 210 ISSN 1556-8180 June 2009 Helga Hedler & Namara G. Gibram his theoretical article underwent the production of knowledge” (Barreira, 2002, p. Tchallenge of increasing knowledge on 17). program evaluation: metaevaluation, a theme In the case of public policies that bring with few studies conducted in Brazil. The model forth plans and goals by program action, proposed herein was based on data obtained by evaluation is a tool that propitiates information an auditing study carried through by the of the results reached by these programs (Ala- Brazilian Federal Audit Court (TCU). The result Harja & Sigurdur, 2000). Rossi and Freeman of this metaevaluation research composes the (1993) understand that the evaluative research scope of another article; therefore the present must use the scientific method as a means to work focuses on the theoretical and explicative investigate social problems. traces of the premises which sustain the Oskamp (1984) characterizes evaluation of metaevaluation model and its applications. programs as an attempt to evaluate the operation, the impact, and the effectiveness of Evaluation of Programs and Their programs in public and private organizations. Program evaluation was developed by applying Concepts a scientific method to the knowledge of reality based on the stages and demands for such The term evaluation can take several lato sensu methods. Moreover, the collection and meanings; among them, evaluations which are systematization of data for the conduct of generally made in daily relation to things, people program evaluation requires the adoption of or situations (Cano, 2004). In such evaluations, valid and trustful procedures, in order to have value judgments are made. Therefore, in this considerable and useful results (Aguilar & sense, evaluating consists in issuing a value Ander-Egg, 1995). judgment or attributing value to something. Aguilar and Ander-Egg (1995), revised This generic definition may be applied to several several definitions for program evaluation and deliberations performed regularly and it refers proposed one that summarizes what other to evaluation in the informal sense. Formal and authors such as Stufflebeam and Shinkfield, systematic evaluation is used to evaluate services (1987), Fernández-Ballesteros et al., Vedung and or professional activities; it utilizes the same Seyfried (1998), Cano (2002), Posavac and methods and techniques present in social Carey (2003) have declared. The definition research (Aguilar & Ander-Egg, 1995). states that program evaluation is “a kind of Evaluating means to determine merit, cost social research applied in a systematic, planned and value (Fernández-Ballesteros, Vedung, & and directive way in order to identify, obtain Seyfried, 1998; Posavac & Carey, 2003; and provide valid and trustful data…to support Stufflebeam & Shinkfield, 1987). Evaluation is a judgment of merit and value of different necessary task that constitutes part of programs, components in a program…” This definition public policies, private projects, public expresses the sense of utility that program regulations, public and private interventions. evaluation bears as a practice connected to The evaluation of programs, referred in this reality and to the needs of users, stakeholders, article as evaluative research, goes beyond these and those involved with the program, aiming concepts and presents the discussion of for the enhancement of service rendering. evaluation as method, subject and establishment Regarding service rendering, according to of scientific patterns. “...The development of Gray, Jenkins, and Segsworth (1993), quoted by the evaluative research presents at its core not Fernández-Ballesteros et al. (1998), the control only the importance of the evaluation as a of public expenses and management of judgment tool for procedures and actions, but assistance programs or policies have been the also the concept that the evaluation represents Journal of MultiDisciplinary Evaluation, Volume 6, Number 12 211 ISSN 1556-8180 June 2009 Helga Hedler & Namara Gibram main focus of program evaluations in the past management of strategy analysis, evaluation of three decades. Therefore, there would be two human resources, and attitude research perspectives within the evaluation of programs: regarding the program. In some cases formative the first directed to contribution, planning and evaluation involves the development of field improvement of the program and the second research on a small scale before a more considering the verification of its effectiveness comprehensive implementation. The and impact. Other than legal principles, informative evaluator works in a team along regulation and financial management of public with the formulators and program finances aiding actions, program evaluations are administrators, and they participate directly in instruments for controlling government actions the decision making to perform all the necessary within the public scope. changes. Ex-post evaluation deals with the evaluation Evaluation Ex-ante, Intermediate, and Ex- of a working program. This kind of evaluation is post conducted when the program has been implanted, in order to reach stated objectives A definition of evaluation ex-ante is provided (Ala-Harja & Sigurdur, 2000). For this reason, it by the Evaluation Research Society (ERS) is also called additive evaluation. Additive (1988) which defines it as analysis of start-end, evaluation influence programs, projects, and pre-installation, viability analysis or contextual plans. analysis. This definition includes evaluative activities that come before the implantation of a Program, Project, and Plan program. Ex-ante evaluation aims to ratify, to research or to emit a precise estimative of The program, project, and plan modalities are conception sufficiency, operational viability, social interventions which differ in scope and sources of financial resources, and availability of duration. Hence, the project is a “minimal unit organizational support. The results provide a for the destination of resources and by means useful direction to refine the program planning, of an integrated set of activities, a way to determining the appropriate implantation level, transform part of reality, provisioning for a and the decision regarding the installation or scarcity or altering a problematic situation” not of the program. (Cotta, 1998, p. 104). A set of projects aiming The intermediate evaluation is one of the ways for the same objective form a program. Finally, of obtaining knowledge about the program. It the plan aggregates similar programs, thus aims to subsidize the program management defining the directives for social interventions. procedure as feedback for its implantation and The plan conception demands a wider development. In this case, the evaluators and comprehension when dealing with social clients are generally internal, most likely intervention. For instance, in Brazilian public program managers. Evaluation issues assessed policies, plans are developed to establish are those related to event management, which directives for a policy. Multiyear plans created are connected to program impact (Ala-Harja & by the government have a wide scope: they Sigurdur, 2000).